Devops

netstat Host 머신에서 netstat 명령으로 docker container의 네트워크 상태가 확인 안됨 물론 container 내부에서 실행하면 되지만… docker container는 bridge 네트워크 기반으로 운영이 되므로 Host Network 에서는 노출이 안됨 # docker 실행 $ docker run -d -p 8081:80 --name=mvcapp cdecl/mvcapp 4fafaf418f84bf6541a1301b4422f825c58fa20b11d1190e87a3e23eea7a6825 # Host 에서는 publsh port (listen) 정보만 노출 $ netstat -ntl | grep 8081 tcp6 0 0 :::8081 :::* LISTEN # ip 현황 $ ip a ... 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:15:5d:3a:4a:00 brd ff:ff:ff:ff:ff:ff inet 192.168.137.100/24 brd 192.168.137.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever 4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default <-- docker bridge link/ether 02:42:58:c6:1b:23 brd ff:ff:ff:ff:ff:ff inet 10.1.0.1/24 brd 10.1.0.255 scope global docker0 valid_lft forever preferred_lft forever ... # bridge network $ docker inspect -f '{{.NetworkSettings.Gateway}}' mvcapp 10.1.0.1 # 라우팅 정보 $ netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface default gateway 0.0.0.0 UG 0 0 0 eth0 10.1.0.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0 <-- docker bridge 10.42.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0 192.168.137.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 nsenter https://github.com/jpetazzo/nsenter{:target="_blank"} 격리되어 있는 namespace enter 진입하는 명령 It is a small tool allowing to enter into namespaces. Technically, it can enter existing namespaces, or spawn a process into a new set of namespaces. ...

Docker Private Registry 구성 Registry 실행 docker-compose.yml version: '3' services: registry: image: registry container_name: registry restart: always ports: - 5000:5000 environment: REGISTRY_HTTP_ADDR: 0.0.0.0:5000 volumes: - registry:/var/lib/registry - ./config.yml:/etc/docker/registry/config.yml volumes: registry: # 실행 $ docker-compose up -d $ curl -s http://localhost:5000/v2/_catalog {"repositories":[]} SSL 적용 version: '3' services: registry: image: registry container_name: private_registry restart: always ports: - 5000:5000 environment: REGISTRY_HTTP_ADDR: 0.0.0.0:5000 REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain-pem.pem REGISTRY_HTTP_TLS_KEY: /certs/domain-pem.key volumes: - registry:/var/lib/registry - ./config.yml:/etc/docker/registry/config.yml - ./certs:/certs volumes: registry: Image Push 테스트용 이미지 태깅 # get alpine images $ docker pull alpine $ docker pull alpine:3.13 # tagging $ docker tag alpine localhost:5000/cdecl/alpine:latest $ docker tag alpine:3.13 localhost:5000/cdecl/alpine:3.13 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alpine latest ae607a46d002 2 weeks ago 5.33MB localhost:5000/cdecl/alpine latest ae607a46d002 2 weeks ago 5.33MB alpine 3.13 81efc9693413 2 months ago 5.35MB localhost:5000/cdecl/alpine 3.13 81efc9693413 2 months ago 5.35MB Image Push # image push $ docker push localhost:5000/cdecl/alpine:3.13 The push refers to repository [localhost:5000/cdecl/alpine] c55d5dbdab40: Layer already exists 3.13: digest: sha256:55fc95a51d97f7b76b124f3b581a58ebf5555d12574f16087de3971a12724dd4 size: 528 $ docker push localhost:5000/cdecl/alpine:latest The push refers to repository [localhost:5000/cdecl/alpine] 5bfa694cc00a: Layer already exists latest: digest: sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e size: 528 # Catalog list $ curl -s http://localhost:5000/v2/_catalog {"repositories":["cdecl/alpine"]} # Tag 정보 $ curl -s http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":["latest","3.13"]} Image Pull $ docker pull localhost:5000/cdecl/alpine Using default tag: latest latest: Pulling from cdecl/alpine Digest: sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e Status: Downloaded newer image for localhost:5000/cdecl/alpine:latest localhost:5000/cdecl/alpine:latest Image Delete Tag 확인 -> Manifests 확인 -> Manifests 이미지 삭제 # Tag 정보 확인 $ curl -s http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":["latest","3.13"]} # Tag의 Manifest 정보확인 - headers 확인 # <registry-url>/v2/<image-path-name>/manifests/<tag> $ curl -s -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ http://localhost:5000/v2/cdecl/alpine/manifests/latest HTTP/1.1 200 OK Content-Length: 528 Content-Type: application/vnd.docker.distribution.manifest.v2+json Docker-Content-Digest: sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e Docker-Distribution-Api-Version: registry/2.0 Etag: "sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e" # Manifest 정보로 이미지 삭제 # Docker-Content-Digest: <sha256:로 시작하는 해쉬 문자열> $ curl -s -XDELETE http://localhost:5000/v2/cdecl/alpine/manifests/sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e # 확인 $ curl -s http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":["3.13"]} Delete marked manifests $ curl -sS http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":null} $ docker exec -it registry bin/registry garbage-collect /etc/docker/registry/config.yml cdecl/alpine 0 blobs marked, 3 blobs and 0 manifests eligible for deletion blob eligible for deletion: sha256:81efc9693413c6292235ea2c29ea07c149701140b98df6c1998bb91d41acf802 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/81/81efc9693413c6292235ea2c29ea07c149701140b98df6c1998bb91d41acf802 go.version=go1.11.2 instance.id=d8bbbd10-232a-457a-9ceb-312b7317da5f service=registry blob eligible for deletion: sha256:55fc95a51d97f7b76b124f3b581a58ebf5555d12574f16087de3971a12724dd4 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/55/55fc95a51d97f7b76b124f3b581a58ebf5555d12574f16087de3971a12724dd4 go.version=go1.11.2 instance.id=d8bbbd10-232a-457a-9ceb-312b7317da5f service=registry blob eligible for deletion: sha256:595b0fe564bb9444ebfe78288079a01ee6d7f666544028d5e96ba610f909ee43 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/59/595b0fe564bb9444ebfe78288079a01ee6d7f666544028d5e96ba610f909ee43 go.version=go1.11.2 instance.id=d8bbbd10-232a-457a-9ceb-312b7317da5f service=registry 위 작업으로는 물리적인 데이터가 삭제 안됨 # 물리적인 파일 삭제 $ docker exec -it registry rm -rf /var/lib/registry/docker/registry/v2/repositories/cdecl Script registry-image-delete.sh #!/bin/bash URI="$1" NAME="$2" TAG="$3" if [[ -z "$URI" ]]; then echo "Usage: $0 <URI> <NAME> [<TAG>]" exit -1 fi if [[ -z "$NAME" ]]; then echo "Usage: $0 $URI <NAME> [<TAG>]" echo curl -sS $URI/v2/_catalog | jq -r '.repositories[]' echo exit -1 fi if [[ -z "$TAG" ]]; then echo "Usage: $0 $URI $NAME [<TAG>]" echo curl -sS $URI/v2/$NAME/tags/list | jq -r '.tags[]' echo exit -1 fi MANIFESTS=$(curl -sS -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ $URI/v2/$NAME/manifests/$TAG | grep -i Docker-Content-Digest | awk '{print $2}') if [[ -z "$MANIFESTS" ]]; then echo "No manifest found for $NAME:$TAG" exit -1 fi echo $MANIFESTS echo echo curl -sS -XDELETE $URI/v2/$NAME/manifests/$MANIFESTS echo

oh-my-zsh 이 느린 경우 해결 방법 (특히 WSL) oh-my-zsh so slow Theme 가 느린 경우는 Pass git 관리 디렉토리 진입시 현저히 느려지는 경우 $ git config --global oh-my-zsh.hide-status 1 $ git config --global oh-my-zsh.hide-dirty 1 $ git config --global -l oh-my-zsh.hide-status=1 oh-my-zsh.hide-dirty=1

Mac 설치 어플리케이션 brew 패키지 관리자 https://brew.sh/index_ko{:target="_blank"} # 설치 $ brew search iterm2 $ brew install iterm2 karabiner-elements brew install brew search karabiner https://karabiner-elements.pqrs.org/{:target="_blank"} 오른쪽 command 버튼 한영키로 매핑 iTerm2 터미널 brew install iterm2 Zsh, Oh My Zsh https://github.com/ohmyzsh/ohmyzsh{:target="_blank"} # zsh install $ brew install zsh # oh my zsh VSCode https://code.visualstudio.com/{:target="_blank"} Docker https://www.docker.com/get-started{:target="_blank"} Dbeaver DB Tool brew install dbeaver-community IINA 동영상 플레이어 brew install iina https://iina.io/{:target="_blank"} Mounty NTFS Mount brew search mounty https://mounty.app/{:target="_blank"} Keka 압축프로그램 brew install keka https://www.keka.io/ko/{:target="_blank"}

vim 최소 설정 .vimrc set ts=4 set autoindent set cindent set hlsearch set showmatch if has("syntax") syntax on endif

Docker default bridge 네트워크 대역 변경 내부 사설 IP와의 출동 등의 이슈 daemon.json /etc/docker/daemon.json { "default-address-pools": [ { "base": "10.1.0.0/16", "size": 24 } ] } Docker restart $ sudo systemctl restart docker 반영 전 $ ip a ... 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:82:58:25:33 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever ... 반영 후 $ ip a ... 1519: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:c0:d9:e2:56 brd ff:ff:ff:ff:ff:ff inet 10.1.0.1/24 brd 10.1.0.255 scope global docker0 valid_lft forever preferred_lft forever ... Data root directory 경로 변경 기본 경로 : /var/lib/docker daemon.json /etc/docker/daemon.json { "data-root": "/docker", "default-address-pools": [ { "base": "10.1.0.0/16", "size": 24 } ] } $ sudo systemctl restart docker $ sudo ls -l /docker 합계 0 drwx--x--x 4 root root 120 8월 18 11:51 buildkit drwx------ 2 root root 6 8월 18 11:51 containers drwx------ 3 root root 22 8월 18 11:51 image drwxr-x--- 3 root root 19 8월 18 11:51 network drwx------ 4 root root 112 8월 18 11:51 overlay2 drwx------ 4 root root 32 8월 18 11:51 plugins drwx------ 2 root root 6 8월 18 11:51 runtimes drwx------ 2 root root 6 8월 18 11:51 swarm drwx------ 2 root root 6 8월 18 11:51 tmp drwx------ 2 root root 6 8월 18 11:51 trust drwx------ 2 root root 50 8월 18 11:51 volumes Docker log 용량 관리 https://docs.docker.com/config/containers/logging/json-file/{:target="_blank"} /containers/ daemon.json /etc/docker/daemon.json { "data-root": "/docker", "log-opts": { "max-size": "100m" }, "default-address-pools": [ { "base": "10.1.0.0/16", "size": 24 } ] }

Lightweight Kubernetes : The certified Kubernetes distribution built for IoT & Edge computing 특징 https://k3s.io/{:target="_blank"} Kubernetes의 경량화 버전으로 아래와 같은 특징 기본 설치만으로 바로 배포 테스트 가능 Overlay Netowrk(Flannel), Load balancer, Ingress(Traefik), CoreDNS 등이 기본 설치 됨 https://rancher.com/docs/k3s/latest/en/networking/{:target="_blank"} etcd 대신 sqlite 운영 High Availability with an External DB High Availability with Embedded DB (Experimental) Master node schedulable uncordon 제외 가능 Worker node 필요 없음 (필요시 추가 가능) 사용 목적 Edge Computing 개발 테스트 및 스테이징 서버 구성 기타 어플리케이션 테스트 용 Master 설치 설치 curl -sfL https://get.k3s.io | sh - 실행으로 끝 systemd 관리 kubectl 설치 및 심볼릭 링크 설정 해줌 이미 kubectl 가 설치 되어 있는 경우는 심볼릭 링크 실패 # alias 필요시 아래 참고 $ alias kubectl='sudo k3s kubectl' # Install $ curl -sfL https://get.k3s.io | sh - # master node $ kubectl get node NAME STATUS ROLES AGE VERSION centos1 Ready control-plane,master 37s v1.21.3+k3s1 $ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-5ff76fc89d-wh9cg 1/1 Running 0 2m35s kube-system coredns-7448499f4d-2d7pb 1/1 Running 0 2m35s kube-system metrics-server-86cbb8457f-x9l6n 1/1 Running 0 2m35s kube-system helm-install-traefik-crd-w27q7 0/1 Completed 0 2m35s kube-system helm-install-traefik-2zllj 0/1 Completed 1 2m35s kube-system svclb-traefik-55qfd 2/2 Running 0 113s kube-system traefik-97b44b794-smzl9 1/1 Running 0 114s K8S 서비스 테스트 서비스 타입 : NodePort https://kubernetes.github.io/ingress-nginx/deploy/baremetal/{:target="_blank"} apiVersion: apps/v1 kind: Deployment metadata: name: mvcapp spec: selector: matchLabels: app: mvcapp replicas: 2 # --replicas=2 옵션과 동일 template: metadata: labels: app: mvcapp spec: containers: - name: mvcapp image: cdecl/mvcapp:0.6 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: mvcapp spec: type: NodePort selector: app: mvcapp ports: - port: 80 targetPort: 80 $ kubectl apply -f mvcapp-deploy-service.yaml deployment.apps/mvcapp created service/mvcapp created $ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-5ff76fc89d-wh9cg 1/1 Running 0 9m29s kube-system coredns-7448499f4d-2d7pb 1/1 Running 0 9m29s kube-system metrics-server-86cbb8457f-x9l6n 1/1 Running 0 9m29s kube-system helm-install-traefik-crd-w27q7 0/1 Completed 0 9m29s kube-system helm-install-traefik-2zllj 0/1 Completed 1 9m29s kube-system svclb-traefik-55qfd 2/2 Running 0 8m47s kube-system traefik-97b44b794-smzl9 1/1 Running 0 8m48s default mvcapp-79874d888c-6htvq 1/1 Running 0 62s default mvcapp-79874d888c-clslc 1/1 Running 0 62s $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 10m mvcapp NodePort 10.43.36.139 <none> 80:32105/TCP 106s # Nodeport IP $ curl 10.43.36.139:80 * Project : Mvcapp * Version : 0.5 / net5.0 * Hostname : mvcapp-79874d888c-6htvq * RemoteAddr : 10.42.0.1 * X-Forwarded-For : * Request Count : 1 * User-Agent : curl/7.29.0 $ curl localhost:32105 * Project : Mvcapp * Version : 0.5 / net5.0 * Hostname : mvcapp-79874d888c-clslc * RemoteAddr : 10.42.0.1 * X-Forwarded-For : * Request Count : 1 * User-Agent : curl/7.29.0 Agent 추가 Master Node만으로도 테스트 가능하나 Scale 테스트시 Agent(Worker Node) 추가 가능 환경변수 세팅 : 필요시 참고 $ sudo cat /var/lib/rancher/k3s/server/node-token > ~/.node-token $ K3S_TOKEN=$(< ~/.node-token) $ HOST_IP=$(ip a | sed -rn 's/.*inet ([0-9\.]+).*eth0/\1/p') Agent 등록 : 원격실행 OR Agent 머신에서 실행 HostIP, Token 정보 필요 (위 환경변수 세팅 참고) # Agent 머신에서 실행 $ curl -sfL https://get.k3s.io | K3S_URL=https://$HOST_IP:6443 K3S_TOKEN=$K3S_TOKEN sh - # Agent 추가 다른방법 $ ansible node01 -m shell -a "curl -sfL https://get.k3s.io | sh -s - agent --server https://$HOST_IP:6443 --token $K3S_TOKEN" -v K3S 삭제 ls /usr/local/bin/k3s-* | xargs -n1 sh -

MinIO 101 Introduction https://docs.min.io/{:target="_blank"} Open Source, S3 Compatible, Enterprise Hardened and Really, Really Fast S3 Compatible : Client(mc), SDK (Java, Javascript, Python, Golang, .Net ..) high performance, distributed object storage system Private cloud object storage Getting Started MinID 는 Golang으로 제작되어 의존성 없는 단일 파일로 운영 가능 Docker의 경우 Alpine linux 로 배포 Downloads : https://min.io/download{:target="_blank"} Quickstart Server https://docs.min.io/docs/minio-quickstart-guide.html{:target="_blank"} # linux - server run $ wget https://dl.min.io/server/minio/release/linux-amd64/minio $ chmod +x minio $ export MINIO_ROOT_USER=minio $ export MINIO_ROOT_PASSWORD=miniopass # ./minio server --address 0.0.0.0:9000 /data $ ./minio server /data Status: 1 Online, 0 Offline. Endpoint: http://192.168.144.5:9000 http://127.0.0.1:9000 Browser Access: http://192.168.144.5:9000 http://127.0.0.1:9000 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide docker-compose 9000 : 데이터 I/F 포트 9001 : Web Console Port ./data:/data : 데이터 영역 version: '3' services: minio: image: minio/minio command: server /data --console-address ":9001" container_name: minio environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9000:9000" - "9001:9001" volumes: - ./data:/data $ docker-compose up -d Creating network "minio_default" with the default driver Pulling minio (minio/minio:)... latest: Pulling from minio/minio c2c17d84f25a: Pull complete 46cdcde062b2: Pull complete c88923a3df19: Pull complete 1afaaeffed49: Pull complete 6c066ed8931e: Pull complete b889e4f29831: Pull complete 51b722521628: Pull complete Digest: sha256:ff4892c4248ad0ef73981d9f2e7b8a721dae45c55bdd25d7a23e1670540f36e1 Status: Downloaded newer image for minio/minio:latest Creating minio ... done Quickstart Client https://docs.min.io/docs/minio-client-quickstart-guide.html{:target="_blank"} # linux # wget https://dl.min.io/client/mc/release/linux-amd64/mc $ curl -O https://dl.min.io/client/mc/release/linux-amd64/mc $ chmod +x mc $ mv ./mc /usr/bin/ $ mc --help # add server config $ mc alias set local http://localhost:9000 minio miniopass Added `local` successfully. # creates a new bucket $ minio mc mb local/backup Bucket created successfully `local/backup`. # copy $ mc cp docker-compose.yml local/backup docker-compose.yml: 322 B / 322 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 30.39 KiB/s 0s # list $ mc ls local/backup [2021-08-18 11:01:50 KST] 322B docker-compose.yml # remove $ mc rm --recursive --force local/backup/ Removing `local/backup/docker-compose.yml`. MinIO Erasure Code https://docs.min.io/docs/minio-erasure-code-quickstart-guide.html{:target="_blank"} Erasure Code 누락되거나 손상된 데이터를 재구성하는 수학적 알고리즘 Erasure code와 Checksums 사용하여 하드웨어 오류 및 자동 데이터 손상으로부터 데이터를 보호 중복 수준이 높으면 전체 드라이브의 최대 절반 (N/2)이 손실 되어도 데이터를 복구 가능 드라이브를 4, 6, 8, 10, 12, 14 또는 16 개의 erasure-coding sets 구성 (최소 4개) Run MinIO Server with Erasure Code 4 drives setup Drives를 물리적인 디스크로 구성하면 별도의 Raid 구성이 필요 없음 $ minio server /data1 /data2 /data3 /data4 docker-compose docker-compose.yml version: '3' services: minio1: image: minio/minio command: server /data1 /data2 /data3 /data4 --console-address ":9001" container_name: minio1 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9000:9000" - "9001:9001" volumes: - ./data1:/data1 - ./data2:/data2 - ./data3:/data3 - ./data4:/data4 Distributed MinIO https://docs.min.io/docs/distributed-minio-quickstart-guide.html{:target="_blank"} MinIO in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server. 서버를 분리하여 다중 Drives를 지원 Data protection High availability Consistency Guarantees Run distributed MinIO MINIO_ROOT_USER and MINIO_ROOT_PASSWORD 같은 키로 구성 Erasure Code와 동일한 Drivers 정책으로 분산 Distributed MinIO 와 서버별로 Erasure Code 같이 적용 가능 docker-compose : docker로 4대 서버 시뮬레이션 version: '3' services: minio1: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio1 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9101:9000" - "9001:9001" volumes: - ./minio1:/data minio2: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio2 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9102:9000" volumes: - ./minio2:/data minio3: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio3 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9103:9000" volumes: - ./minio3:/data minio4: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio4 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9104:9000" volumes: - ./minio4:/data MinIO Admin Guide mc 명령어를 통해 Admin 기능을 수행 https://docs.min.io/docs/minio-admin-complete-guide.html{:target="_blank"} service restart and stop all MinIO servers update update all MinIO servers info display MinIO server information user manage users group manage groups policy manage policies defined in the MinIO server config manage MinIO server configuration heal heal disks, buckets and objects on MinIO server profile generate profile data for debugging purposes top provide top like statistics for MinIO trace show http trace for MinIO server console show console logs for MinIO server prometheus manages prometheus config kms perform KMS management operations user - Manage users User 생성 및 삭제 ## create User $ mc admin user add myinfo cdecl cdeclpass ## remove User # mc admin user remove myinfo cdecl $ mc admin user info myinfo cdecl AccessKey: cdecl Status: enabled PolicyName: MemberOf: Policy $ mc admin policy set myinfo readonly user=cdecl Policy readonly is set on user `cdecl` $ mc admin policy set myinfo writeonly user=cdecl Policy writeonly is set on user `cdecl` $ mc admin policy set myinfo readwrite user=cdecl Policy readwrite is set on user `cdecl` heal - Heal disks, buckets and objects on MinIO server This command is only applicable for MinIO erasure coded setup (standalone and distributed). Erasure Code 상태에서 특정 디스크의 데이터가 문제가 있을 경우, 균등하게 데이터를 복구 해줌 $ mc admin heal -r myinfo - data1 2/2 objects; 403 MiB in 1s ┌────────┬───┬─────────────────────┐ │ Green │ 5 │ 100.0% ████████████ │ │ Yellow │ 0 │ 0.0% │ │ Red │ 0 │ 0.0% │ │ Grey │ 0 │ 0.0% │ └────────┴───┴─────────────────────┘

MySQL 8 Docker 실행 및 백업, 복원 MySQL Docker 실행 docker-compose data:/var/lib/mysql : Data 파일 conf.d:/etc/mysql/conf.d : my.cnf 등의 설정파일 root:/root : login-path 사용시 ports : “3380:3306” : mysql port “33800:33060” : mysql-shell port version: '3' services: db: image: mysql:8 container_name: mysql8 command: --default-authentication-plugin=mysql_native_password restart: always environment: MYSQL_ROOT_PASSWORD: passwd TZ: Asia/Seoul ports: - "3380:3306" - "33800:33060" volumes: - data:/var/lib/mysql - conf.d:/etc/mysql/conf.d - root:/root volumes: data: conf.d: root: $ docker-compose up -d Creating network "mysql_default" with the default driver Creating volume "mysql_data" with default driver Creating volume "mysql_conf.d" with default driver Creating volume "mysql_root" with default driver Pulling db (mysql:8)... 8: Pulling from library/mysql e1acddbe380c: Pull complete bed879327370: Pull complete 03285f80bafd: Pull complete ccc17412a00a: Pull complete 1f556ecc09d1: Pull complete adc5528e468d: Pull complete 1afc286d5d53: Pull complete 6c724a59adff: Pull complete 0f2345f8b0a3: Pull complete c8461a25b23b: Pull complete 3adb49279bed: Pull complete 77f22cd6c363: Pull complete Digest: sha256:d45561a65aba6edac77be36e0a53f0c1fba67b951cb728348522b671ad63f926 Status: Downloaded newer image for mysql:8 Creating mysql8 ... done 연결 테스트 docker -t 옵션 제외 : the input device is not a TTY # echo 'select host, user from mysql.user' | docker exec -i mysql8 mysql -uroot -ppasswd $ docker exec -i mysql8 mysql -uroot -ppasswd <<< 'select host, user from mysql.user' mysql: [Warning] Using a password on the command line interface can be insecure. host user % root localhost mysql.infoschema localhost mysql.session localhost mysql.sys localhost root mysql: [Warning] Using a password on the command line interface can be insecure. ...

ElasticSearch to MySQL ETL 준비 curl : Http 기반 ElasticSearch Query (SQL) jq : ElasticSearch 결과 JSON 변환 (NDJSON) mysql-shell : MySQL 데이터 Bulk Insert (Json 타입의 Table) ElasticSearch Query curl 쿼리 후, 해당 내용 JSON 저장 $ curl -s -XPOST -H 'content-type: application/json' \ -d '{"query" : "select timestamp, activesession, loadavg, processor from \"mysql-perf-*\" limit 100"}' \ 'http://elasticsearch-server:7200/_sql' > result.json $ cat result.json | jq . { "columns": [ { "name": "timestamp", "type": "datetime" }, { "name": "activesession", "type": "long" }, { "name": "loadavg", "type": "float" }, { "name": "processor", "type": "float" } ], "rows": [ [ "2021-05-02T03:40:47.000Z", 2, 0.019999999552965164, 0 ], [ "2021-06-01T00:04:24.000Z", 4, 2.0399999618530273, 14.579999923706055 ], ... NDJSON 형태로 변환 jq -c : compact instead of pretty-printed output $ cat result.json | \ jq -c ' { "col": [.columns[] | .name], "row" : .rows[] } | [.col, .row] | transpose | map({ (.[0]): .[1] }) | add ' > result_t.json $ cat result_t.json | jq . { "timestamp": "2021-05-02T03:40:47.000Z", "activesession": 2, "loadavg": 0.019999999552965164, "processor": 0 } { "timestamp": "2021-06-22T05:25:24.000Z", "activesession": 3, "loadavg": 1.809999942779541, "processor": 4.599999904632568 } { "timestamp": "2021-07-01T00:04:21.000Z", "activesession": 1, "loadavg": 0.2800000011920929, "processor": 0.41999998688697815 MySQL JSON Import mysql-shell 테이블내 doc json 필드 생성 및 JSON 데이터 Row 단위 적재 # truncate table $ mysqlsh --sql user@mysql-server:3360/dbname -p<PASSWD> \ --execute 'truncate table data_table ;' # import $ mysqlsh --mysqlx user@mysql-server:3360/dbname -p<PASSWD> \ --import result_t.json --collection=data_table ...