Docker

netstat Host 머신에서 netstat 명령으로 docker container의 네트워크 상태가 확인 안됨 물론 container 내부에서 실행하면 되지만… docker container는 bridge 네트워크 기반으로 운영이 되므로 Host Network 에서는 노출이 안됨 # docker 실행 $ docker run -d -p 8081:80 --name=mvcapp cdecl/mvcapp 4fafaf418f84bf6541a1301b4422f825c58fa20b11d1190e87a3e23eea7a6825 # Host 에서는 publsh port (listen) 정보만 노출 $ netstat -ntl | grep 8081 tcp6 0 0 :::8081 :::* LISTEN # ip 현황 $ ip a ... 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:15:5d:3a:4a:00 brd ff:ff:ff:ff:ff:ff inet 192.168.137.100/24 brd 192.168.137.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever 4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default <-- docker bridge link/ether 02:42:58:c6:1b:23 brd ff:ff:ff:ff:ff:ff inet 10.1.0.1/24 brd 10.1.0.255 scope global docker0 valid_lft forever preferred_lft forever ... # bridge network $ docker inspect -f '{{.NetworkSettings.Gateway}}' mvcapp 10.1.0.1 # 라우팅 정보 $ netstat -r Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface default gateway 0.0.0.0 UG 0 0 0 eth0 10.1.0.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0 <-- docker bridge 10.42.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0 192.168.137.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 nsenter https://github.com/jpetazzo/nsenter{:target="_blank"} 격리되어 있는 namespace enter 진입하는 명령 It is a small tool allowing to enter into namespaces. Technically, it can enter existing namespaces, or spawn a process into a new set of namespaces. ...

Docker Private Registry 구성 Registry 실행 docker-compose.yml version: '3' services: registry: image: registry container_name: registry restart: always ports: - 5000:5000 environment: REGISTRY_HTTP_ADDR: 0.0.0.0:5000 volumes: - registry:/var/lib/registry - ./config.yml:/etc/docker/registry/config.yml volumes: registry: # 실행 $ docker-compose up -d $ curl -s http://localhost:5000/v2/_catalog {"repositories":[]} SSL 적용 version: '3' services: registry: image: registry container_name: private_registry restart: always ports: - 5000:5000 environment: REGISTRY_HTTP_ADDR: 0.0.0.0:5000 REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain-pem.pem REGISTRY_HTTP_TLS_KEY: /certs/domain-pem.key volumes: - registry:/var/lib/registry - ./config.yml:/etc/docker/registry/config.yml - ./certs:/certs volumes: registry: Image Push 테스트용 이미지 태깅 # get alpine images $ docker pull alpine $ docker pull alpine:3.13 # tagging $ docker tag alpine localhost:5000/cdecl/alpine:latest $ docker tag alpine:3.13 localhost:5000/cdecl/alpine:3.13 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alpine latest ae607a46d002 2 weeks ago 5.33MB localhost:5000/cdecl/alpine latest ae607a46d002 2 weeks ago 5.33MB alpine 3.13 81efc9693413 2 months ago 5.35MB localhost:5000/cdecl/alpine 3.13 81efc9693413 2 months ago 5.35MB Image Push # image push $ docker push localhost:5000/cdecl/alpine:3.13 The push refers to repository [localhost:5000/cdecl/alpine] c55d5dbdab40: Layer already exists 3.13: digest: sha256:55fc95a51d97f7b76b124f3b581a58ebf5555d12574f16087de3971a12724dd4 size: 528 $ docker push localhost:5000/cdecl/alpine:latest The push refers to repository [localhost:5000/cdecl/alpine] 5bfa694cc00a: Layer already exists latest: digest: sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e size: 528 # Catalog list $ curl -s http://localhost:5000/v2/_catalog {"repositories":["cdecl/alpine"]} # Tag 정보 $ curl -s http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":["latest","3.13"]} Image Pull $ docker pull localhost:5000/cdecl/alpine Using default tag: latest latest: Pulling from cdecl/alpine Digest: sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e Status: Downloaded newer image for localhost:5000/cdecl/alpine:latest localhost:5000/cdecl/alpine:latest Image Delete Tag 확인 -> Manifests 확인 -> Manifests 이미지 삭제 # Tag 정보 확인 $ curl -s http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":["latest","3.13"]} # Tag의 Manifest 정보확인 - headers 확인 # <registry-url>/v2/<image-path-name>/manifests/<tag> $ curl -s -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ http://localhost:5000/v2/cdecl/alpine/manifests/latest HTTP/1.1 200 OK Content-Length: 528 Content-Type: application/vnd.docker.distribution.manifest.v2+json Docker-Content-Digest: sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e Docker-Distribution-Api-Version: registry/2.0 Etag: "sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e" # Manifest 정보로 이미지 삭제 # Docker-Content-Digest: <sha256:로 시작하는 해쉬 문자열> $ curl -s -XDELETE http://localhost:5000/v2/cdecl/alpine/manifests/sha256:bd9137c3bb45dbc40cde0f0e19a8b9064c2bc485466221f5e95eb72b0d0cf82e # 확인 $ curl -s http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":["3.13"]} Delete marked manifests $ curl -sS http://localhost:5000/v2/cdecl/alpine/tags/list {"name":"cdecl/alpine","tags":null} $ docker exec -it registry bin/registry garbage-collect /etc/docker/registry/config.yml cdecl/alpine 0 blobs marked, 3 blobs and 0 manifests eligible for deletion blob eligible for deletion: sha256:81efc9693413c6292235ea2c29ea07c149701140b98df6c1998bb91d41acf802 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/81/81efc9693413c6292235ea2c29ea07c149701140b98df6c1998bb91d41acf802 go.version=go1.11.2 instance.id=d8bbbd10-232a-457a-9ceb-312b7317da5f service=registry blob eligible for deletion: sha256:55fc95a51d97f7b76b124f3b581a58ebf5555d12574f16087de3971a12724dd4 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/55/55fc95a51d97f7b76b124f3b581a58ebf5555d12574f16087de3971a12724dd4 go.version=go1.11.2 instance.id=d8bbbd10-232a-457a-9ceb-312b7317da5f service=registry blob eligible for deletion: sha256:595b0fe564bb9444ebfe78288079a01ee6d7f666544028d5e96ba610f909ee43 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/59/595b0fe564bb9444ebfe78288079a01ee6d7f666544028d5e96ba610f909ee43 go.version=go1.11.2 instance.id=d8bbbd10-232a-457a-9ceb-312b7317da5f service=registry 위 작업으로는 물리적인 데이터가 삭제 안됨 # 물리적인 파일 삭제 $ docker exec -it registry rm -rf /var/lib/registry/docker/registry/v2/repositories/cdecl Script registry-image-delete.sh #!/bin/bash URI="$1" NAME="$2" TAG="$3" if [[ -z "$URI" ]]; then echo "Usage: $0 <URI> <NAME> [<TAG>]" exit -1 fi if [[ -z "$NAME" ]]; then echo "Usage: $0 $URI <NAME> [<TAG>]" echo curl -sS $URI/v2/_catalog | jq -r '.repositories[]' echo exit -1 fi if [[ -z "$TAG" ]]; then echo "Usage: $0 $URI $NAME [<TAG>]" echo curl -sS $URI/v2/$NAME/tags/list | jq -r '.tags[]' echo exit -1 fi MANIFESTS=$(curl -sS -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ $URI/v2/$NAME/manifests/$TAG | grep -i Docker-Content-Digest | awk '{print $2}') if [[ -z "$MANIFESTS" ]]; then echo "No manifest found for $NAME:$TAG" exit -1 fi echo $MANIFESTS echo echo curl -sS -XDELETE $URI/v2/$NAME/manifests/$MANIFESTS echo

Docker default bridge 네트워크 대역 변경 내부 사설 IP와의 출동 등의 이슈 daemon.json /etc/docker/daemon.json { "default-address-pools": [ { "base": "10.1.0.0/16", "size": 24 } ] } Docker restart $ sudo systemctl restart docker 반영 전 $ ip a ... 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:82:58:25:33 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever ... 반영 후 $ ip a ... 1519: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:c0:d9:e2:56 brd ff:ff:ff:ff:ff:ff inet 10.1.0.1/24 brd 10.1.0.255 scope global docker0 valid_lft forever preferred_lft forever ... Data root directory 경로 변경 기본 경로 : /var/lib/docker daemon.json /etc/docker/daemon.json { "data-root": "/docker", "default-address-pools": [ { "base": "10.1.0.0/16", "size": 24 } ] } $ sudo systemctl restart docker $ sudo ls -l /docker 합계 0 drwx--x--x 4 root root 120 8월 18 11:51 buildkit drwx------ 2 root root 6 8월 18 11:51 containers drwx------ 3 root root 22 8월 18 11:51 image drwxr-x--- 3 root root 19 8월 18 11:51 network drwx------ 4 root root 112 8월 18 11:51 overlay2 drwx------ 4 root root 32 8월 18 11:51 plugins drwx------ 2 root root 6 8월 18 11:51 runtimes drwx------ 2 root root 6 8월 18 11:51 swarm drwx------ 2 root root 6 8월 18 11:51 tmp drwx------ 2 root root 6 8월 18 11:51 trust drwx------ 2 root root 50 8월 18 11:51 volumes Docker log 용량 관리 https://docs.docker.com/config/containers/logging/json-file/{:target="_blank"} /containers/ daemon.json /etc/docker/daemon.json { "data-root": "/docker", "log-opts": { "max-size": "100m" }, "default-address-pools": [ { "base": "10.1.0.0/16", "size": 24 } ] }

MySQL 8 Docker 실행 및 백업, 복원 MySQL Docker 실행 docker-compose data:/var/lib/mysql : Data 파일 conf.d:/etc/mysql/conf.d : my.cnf 등의 설정파일 root:/root : login-path 사용시 ports : “3380:3306” : mysql port “33800:33060” : mysql-shell port version: '3' services: db: image: mysql:8 container_name: mysql8 command: --default-authentication-plugin=mysql_native_password restart: always environment: MYSQL_ROOT_PASSWORD: passwd TZ: Asia/Seoul ports: - "3380:3306" - "33800:33060" volumes: - data:/var/lib/mysql - conf.d:/etc/mysql/conf.d - root:/root volumes: data: conf.d: root: $ docker-compose up -d Creating network "mysql_default" with the default driver Creating volume "mysql_data" with default driver Creating volume "mysql_conf.d" with default driver Creating volume "mysql_root" with default driver Pulling db (mysql:8)... 8: Pulling from library/mysql e1acddbe380c: Pull complete bed879327370: Pull complete 03285f80bafd: Pull complete ccc17412a00a: Pull complete 1f556ecc09d1: Pull complete adc5528e468d: Pull complete 1afc286d5d53: Pull complete 6c724a59adff: Pull complete 0f2345f8b0a3: Pull complete c8461a25b23b: Pull complete 3adb49279bed: Pull complete 77f22cd6c363: Pull complete Digest: sha256:d45561a65aba6edac77be36e0a53f0c1fba67b951cb728348522b671ad63f926 Status: Downloaded newer image for mysql:8 Creating mysql8 ... done 연결 테스트 docker -t 옵션 제외 : the input device is not a TTY # echo 'select host, user from mysql.user' | docker exec -i mysql8 mysql -uroot -ppasswd $ docker exec -i mysql8 mysql -uroot -ppasswd <<< 'select host, user from mysql.user' mysql: [Warning] Using a password on the command line interface can be insecure. host user % root localhost mysql.infoschema localhost mysql.session localhost mysql.sys localhost root mysql: [Warning] Using a password on the command line interface can be insecure. ...

https://docs.docker.com/engine/swarm/ Docker Swarm Node Manager Node 초기화 https://docs.docker.com/engine/reference/commandline/swarm_init/ 초기화 Initialize a swarm docker swarm init $ docker swarm init Swarm initialized: current node (bvz81updecsj6wjz393c09vti) is now a manager. To add a worker to this swarm, run the following command: docker swarm join \ --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \ 172.17.0.2:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. Initialize a swarm with advertised address docker swarm init --advertise-addr <ip|interface>[:port] $ docker swarm init --advertise-addr 192.168.99.121 Swarm initialized: current node (bvz81updecsj6wjz393c09vti) is now a manager. To add a worker to this swarm, run the following command: docker swarm join \ --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \ 172.17.0.2:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 노드 추가 Join Node Join a swarm as a node and/or manager https://docs.docker.com/engine/reference/commandline/swarm_join/ ...

Kubernetes 설치 및 운영 101 사전 준비 Kubernetes 설치 전 서버 구성 변경 참고 : https://www.mirantis.com/blog/how-install-kubernetes-kubeadm/{:target="_blank"} Swap 영역을 비활성화 # 일시적인 설정 $ sudo swapoff -a # 영구적인 설정, 아래 swap 파일 시스템을 주석처리 $ sudo vi /etc/fstab ... # /dev/mapper/kube--master--vg-swap_1 none swap sw 0 0 SELinux Disable # 임시 $ sudo setenforce 0 # 영구 $ sudo vi /etc/sysconfig/selinux ... SELinux=disabled 방화벽 Disable $ sudo systemctl disable firewalld $ sudo systemctl stop firewalld 브릿지 네트워크 할성화 # Centos $ sudo vim /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 # Ubuntu $ sudo vim /etc/ufw/sysctl.conf net/bridge/bridge-nf-call-ip6tables = 1 net/bridge/bridge-nf-call-iptables = 1 net/bridge/bridge-nf-call-arptables = 1 Docker Install Centos Install : https://docs.docker.com/engine/install/centos/{:target="_blank"} Cgroup 드라이버 이슈 최신 Kubernetes는 docker cgroup driver를 cgroupfs → systemd 변경 필요 Master Init 및 Worker Join 시 WARNING 발생 https://kubernetes.io/ko/docs/setup/production-environment/container-runtimes/{:target="_blank"} kubeadm init --pod-network-cidr 10.244.0.0/16 ... [init] Using Kubernetes version: v1.19.3 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". ... ... 드라이버 변경 작업 /etc/docker/daemon.json 파일 작성 $ cat <<EOF | sudo tee /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ] } EOF # 도커 재시작 $ sudo systemctl restart docker # 확인 $ sudo docker info | grep -i cgroup Cgroup Driver: systemd Kubernetes (kubeadm, kubelet, kubectl) 설치 참고 : https://kubernetes.io/docs/setup/independent/install-kubeadm/{:target="_blank"} Kubernetes 설치 : Centos7 기준 Docker 설치 sudo yum install -y docker sudo systemctl enable docker && systemctl start docker sudo usermod -aG docker $USER kubeadm, kubelet, kubectl : Repo 추가 및 패키지 설치 $ cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF $ sudo yum install -y kubelet kubeadm kubectl $ sudo systemctl enable kubelet && systemctl start kubelet # 버전이 안맞을 경우 지정 # sudo yum install kubelet-[version] kubeadm-[version] kubectl-[version] kubectl 자동완성 # sh source <(kubectl completion sh) echo "source <(kubectl completion sh)" >> ~/.shrc # zsh source <(kubectl completion zsh) echo "if [ $commands[kubectl] ]; then source <(kubectl completion zsh); fi" >> ~/.zshrc Master Node Init 및 Worker Node Join Master Node 설정 Master 초기화 네트워크 클래스 대역을 설정 필요 : --pod-network-cidr 10.244.0.0/16 sudo kubeadm init --pod-network-cidr 10.244.0.0/16 Kubectl 사용 : To start using your cluster.. 아래 항목 3줄 실행 [init] Using Kubernetes version: v1.10.5 ... To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... You can now join any number of machines by running the following on each node as root: kubeadm join 192.168.28.15:6443 --token 1ovd36.ft4mefr909iotg0a --discovery-token-ca-cert-hash sha256:82953a3ed178aa8c511792d0e21d9d3283e7575f3d3350a00bea3e34c2b87d29 Pod 상태 확인 coredns STATUS → Pending (∵ Overlay network 미설치) $ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-66bff467f8-ktvsz 0/1 Pending 0 19s kube-system coredns-66bff467f8-nvvjz 0/1 Pending 0 19s kube-system etcd-node1 1/1 Running 0 29s kube-system kube-apiserver-node1 1/1 Running 0 29s kube-system kube-controller-manager-node1 1/1 Running 0 29s kube-system kube-proxy-s582x 1/1 Running 0 19s kube-system kube-scheduler-node1 1/1 Running 0 29s Overlay network : Calico 설치 Overlay network 종류 https://kubernetes.io/docs/concepts/cluster-administration/networking/{:target="_blank"} Install Calico for on-premises deployments https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises{:target="_blank"} # Install Calico for on-premises deployments $ kubectl apply -f https://docs.projectcalico.org/manifests/calico-typha.yaml coredns 서비스가 정상적으로 Running $ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-799fb94867-bcntz 0/1 CrashLoopBackOff 3 2m6s kube-system calico-node-jtcmt 0/1 Running 1 2m7s kube-system calico-typha-6bc9dd6468-x2hjj 0/1 Pending 0 2m6s kube-system coredns-66bff467f8-ktvsz 0/1 Running 0 3m23s kube-system coredns-66bff467f8-nvvjz 0/1 Running 0 3m23s kube-system etcd-node1 1/1 Running 0 3m33s kube-system kube-apiserver-node1 1/1 Running 0 3m33s kube-system kube-controller-manager-node1 1/1 Running 0 3m33s kube-system kube-proxy-s582x 1/1 Running 0 3m23s kube-system kube-scheduler-node1 1/1 Running 0 3m33s Worker Node 추가 (Join) Worker Node 실행 # Join 명령 가져오기 $ kubeadm token create --print-join-command kubeadm join 192.168.28.15:6443 --token 1ovd36.ft4mefr909iotg0a --discovery-token-ca-cert-hash sha256:82953a3ed178aa8c511792d0e21d9d3283e7575f3d3350a00bea3e34c2b87d29 # Worker node 에서 실행 $ kubeadm join 192.168.28.15:6443 --token 1ovd36.ft4mefr909iotg0a --discovery-token-ca-cert-hash sha256:82953a3ed178aa8c511792d0e21d9d3283e7575f3d3350a00bea3e34c2b87d29 노드 상태 확인 > kubectl get node NAME STATUS ROLES AGE VERSION node1 Ready master 8m50s v1.18.6 node2 Ready <none> 16s v1.18.6 node3 Ready <none> 16s v1.18.6 서비스 배포 : 명령어(CLI) 기반 ...