K3s

K3s traefik ingress 사용 서비스 테스트 traefik ingress traefik : https://cdecl.github.io/devops/traefik-proxy/{:target="_blank"} traefik 을 활용한 ingress 구현체 : K3s 에서 번들로 제공 서비스 테스트 traefik/whoami 서비스 테스트 type: NodePort # whoami-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: whoami spec: selector: matchLabels: app: whoami replicas: 2 template: metadata: labels: app: whoami spec: containers: - name: whoami image: traefik/whoami imagePullPolicy: Always ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: whoami spec: type: NodePort selector: app: whoami ports: - port: 80 targetPort: 80 nodePort: 30080 $ kubectl apply -f whoami-deploy.yaml $ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-7b7dc8d6f5-jnmt5 1/1 Running 0 11m kube-system coredns-b96499967-brb9f 1/1 Running 0 11m kube-system helm-install-traefik-crd-hrfr5 0/1 Completed 0 11m kube-system metrics-server-668d979685-xj5jj 1/1 Running 0 11m kube-system helm-install-traefik-nr5jq 0/1 Completed 1 11m kube-system svclb-traefik-632fd507-mdwnl 2/2 Running 0 11m kube-system traefik-7cd4fcff68-47ms4 1/1 Running 0 11m default whoami-6bbfdbb69c-phxts 1/1 Running 0 102s default whoami-6bbfdbb69c-9rpxv 1/1 Running 0 102s $ kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 26m kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 26m kube-system metrics-server ClusterIP 10.43.90.12 <none> 443/TCP 26m kube-system traefik LoadBalancer 10.43.147.171 192.168.136.5 80:32391/TCP,443:31718/TCP 25m default whoami NodePort 10.43.116.252 <none> 80:30080/TCP 16m $ curl localhost:30080 Hostname: whoami-6bbfdbb69c-9rpxv IP: 127.0.0.1 IP: ::1 IP: 10.42.0.10 IP: fe80::7c46:beff:fe57:f495 RemoteAddr: 10.42.0.1:43930 GET / HTTP/1.1 Host: localhost:30080 User-Agent: curl/7.81.0 Accept: */* Ingress 적용 # ingress-rule.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: main-ingress spec: rules: # - host: domain - http: paths: - path: / pathType: ImplementationSpecific backend: service: name: whoami port: number: 80 Ingress 가 Proxy 역할로 X-Forwarded-For 등이 추가됨 ...

Kubernetes 패키지 매니저 도구인 helm을 통해 chart 생성 및 Kubernetes 배포 K3S 환경에서 테스트 Helm https://helm.sh/{:target="_blank"} Kubernetes 배포를 위한 패키지 매니저 툴 (e.g yum, choco) chart 라는 yaml 파일 기반의 템플릿 파일을 통해 패키지화 및 Kubernetes 설치 관리 Deployment, Service, Ingress 등 Kubernetes 서비스의 manifest 생성 및 설치 Helm Repository 를 통해 패키지 등록 및 다른 패키지 설치 가능 Helm Install 바이너리 직접 설치 및 설치 Script 활용 Homebrew, Chocolatey 등의 패키지로도 설치 가능 바이너리 다운로드 https://github.com/helm/helm/releases $ curl -LO https://get.helm.sh/helm-v3.7.1-linux-amd64.tar.gz $ tar -zxvf helm-v3.7.1-linux-amd64.tar.gz $ tree linux-amd64 linux-amd64 ├── LICENSE ├── README.md └── helm $ sudo cp linux-amd64/helm /usr/local/bin/ 설치 Script 활용 $ chmod 700 get_helm.sh $ ./get_helm.sh Creating Your Own Charts kubernetes 설치를 위한 chart 생성 및 세팅 https://helm.sh/docs/helm/helm_create/{:target="_blank"} # chart 생성 $ helm create mvcapp Creating mvcapp Chart directory 구조 Chart.yaml : Chart 버전, 이미지버전, 설명등을 기술하는 파일 values.yaml : manifest template 파일 기반, 기준 값을 세팅하는 파일 templates/ : kubernetes manifest template 파일 charts/ : chart 의존성 파일 $ tree mvcapp mvcapp ├── Chart.yaml ├── charts ├── templates │ ├── NOTES.txt │ ├── _helpers.tpl │ ├── deployment.yaml │ ├── hpa.yaml │ ├── ingress.yaml │ ├── service.yaml │ ├── serviceaccount.yaml │ └── tests │ └── test-connection.yaml └── values.yaml Chart.yaml 수정 version : Chart 버전 appVersion : Deploy 되는 image 버전 apiVersion: v2 name: mvcapp description: .net core test mvc application # ... 생략 type: application # ... 생략 version: 0.1.0 # ... 생략 appVersion: "0.6" # appVersion: "1.16.0" values.yaml 수정 replicaCount : Pod 의 replica 개수, 2개로 수정 image.repository : docker image 이름, cdecl/mvcapp 로 수정 service.type : On-Premise에서 테스트 목적, NodePort로 수정 service.nodePort : nodePort를 적용하기 위해 신규 추가 # Default values for mvcapp. # This is a YAML-formatted file. # Declare variables to be passed into your templates. replicaCount: 2 image: repository: cdecl/mvcapp pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart appVersion. tag: "" imagePullSecrets: [] nameOverride: "" fullnameOverride: "" # ... 생략 service: type: NodePort # ClusterIP port: 80 nodePort: 30010 ingress: # ... 생략 resources: {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following # lines, adjust them as necessary, and remove the curly braces after 'resources:'. # limits: # cpu: 100m # memory: 128Mi # requests: # cpu: 100m # memory: 128Mi # ... 생략 templates/service.yaml 수정 nodePort 를 적용하기 위해 template 수정 spec.ports.nodePort: {{ .Values.service.nodePort }} 추가 apiVersion: v1 kind: Service metadata: name: {{ include "mvcapp.fullname" . }} labels: {{- include "mvcapp.labels" . | nindent 4 }} spec: type: {{ .Values.service.type }} ports: - port: {{ .Values.service.port }} targetPort: http pro name: http nodePort: {{ .Values.service.nodePort }} selector: {{- include "mvcapp.selectorLabels" . | nindent 4 }} helm lint : chart 파일 검사 https://helm.sh/docs/helm/helm_lint/{:target="_blank"} $ helm lint mvcapp ==> Linting mvcapp [INFO] Chart.yaml: icon is recommended 1 chart(s) linted, 0 chart(s) failed helm template : kubernetes manifest 생성 https://helm.sh/docs/helm/helm_template/{:target="_blank"} values.yaml 에 세팅한 기준으로 manifest 생성 $ helm template mvcapp --- # Source: mvcapp/templates/serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: RELEASE-NAME-mvcapp labels: helm.sh/chart: mvcapp-0.1.0 app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME app.kubernetes.io/version: "0.6" app.kubernetes.io/managed-by: Helm --- # Source: mvcapp/templates/service.yaml apiVersion: v1 kind: Service metadata: name: RELEASE-NAME-mvcapp labels: helm.sh/chart: mvcapp-0.1.0 app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME app.kubernetes.io/version: "0.6" app.kubernetes.io/managed-by: Helm spec: type: NodePort ports: - port: 80 targetPort: http pro name: http selector: app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME --- # Source: mvcapp/templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: RELEASE-NAME-mvcapp labels: helm.sh/chart: mvcapp-0.1.0 app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME app.kubernetes.io/version: "0.6" app.kubernetes.io/managed-by: Helm spec: replicas: 2 selector: matchLabels: app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME template: metadata: labels: app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME spec: serviceAccountName: RELEASE-NAME-mvcapp securityContext: {} containers: - name: mvcapp securityContext: {} image: "cdecl/mvcapp:0.6" imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 pro livenessProbe: httpGet: path: / port: http readinessProbe: httpGet: path: / port: http resources: {} --- # Source: mvcapp/templates/tests/test-connection.yaml apiVersion: v1 kind: Pod metadata: name: "RELEASE-NAME-mvcapp-test-connection" labels: helm.sh/chart: mvcapp-0.1.0 app.kubernetes.io/name: mvcapp app.kubernetes.io/instance: RELEASE-NAME app.kubernetes.io/version: "0.6" app.kubernetes.io/managed-by: Helm annotations: "helm.sh/hook": test spec: containers: - name: wget image: busybox command: ['wget'] args: ['RELEASE-NAME-mvcapp:80'] restartPolicy: Never helm install : chart 활용 kubernetes service install https://helm.sh/docs/helm/helm_install/{:target="_blank"} install : helm install [NAME] [CHART] [flags] # 설치하지는 않고 테스트 $ helm install mvcapp-svc mvcapp --dry-run # 로컬 Chart 를 통한 설치 $ helm install mvcapp-svc mvcapp NAME: mvcapp-svc LAST DEPLOYED: Thu Nov 4 13:29:38 2021 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services mvcapp-svc) export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 78d mvcapp-svc NodePort 10.43.202.254 <none> 80:31503/TCP 29s $ kubectl get pod NAME READY STATUS RESTARTS AGE mvcapp-svc-78ff4d97f9-hd9rf 1/1 Running 0 37s mvcapp-svc-78ff4d97f9-x4984 1/1 Running 0 37s KS3 export KUBECONFIG=/etc/rancher/k3s/k3s.yaml 세팅 ...

Lightweight Kubernetes : The certified Kubernetes distribution built for IoT & Edge computing 특징 https://k3s.io/{:target="_blank"} Kubernetes의 경량화 버전으로 아래와 같은 특징 기본 설치만으로 바로 배포 테스트 가능 Overlay Netowrk(Flannel), Load balancer, Ingress(Traefik), CoreDNS 등이 기본 설치 됨 https://rancher.com/docs/k3s/latest/en/networking/{:target="_blank"} etcd 대신 sqlite 운영 High Availability with an External DB High Availability with Embedded DB (Experimental) Master node schedulable uncordon 제외 가능 Worker node 필요 없음 (필요시 추가 가능) 사용 목적 Edge Computing 개발 테스트 및 스테이징 서버 구성 기타 어플리케이션 테스트 용 Master 설치 설치 curl -sfL https://get.k3s.io | sh - 실행으로 끝 systemd 관리 kubectl 설치 및 심볼릭 링크 설정 해줌 이미 kubectl 가 설치 되어 있는 경우는 심볼릭 링크 실패 # alias 필요시 아래 참고 $ alias kubectl='sudo k3s kubectl' # Install $ curl -sfL https://get.k3s.io | sh - # master node $ kubectl get node NAME STATUS ROLES AGE VERSION centos1 Ready control-plane,master 37s v1.21.3+k3s1 $ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-5ff76fc89d-wh9cg 1/1 Running 0 2m35s kube-system coredns-7448499f4d-2d7pb 1/1 Running 0 2m35s kube-system metrics-server-86cbb8457f-x9l6n 1/1 Running 0 2m35s kube-system helm-install-traefik-crd-w27q7 0/1 Completed 0 2m35s kube-system helm-install-traefik-2zllj 0/1 Completed 1 2m35s kube-system svclb-traefik-55qfd 2/2 Running 0 113s kube-system traefik-97b44b794-smzl9 1/1 Running 0 114s K8S 서비스 테스트 서비스 타입 : NodePort https://kubernetes.github.io/ingress-nginx/deploy/baremetal/{:target="_blank"} apiVersion: apps/v1 kind: Deployment metadata: name: mvcapp spec: selector: matchLabels: app: mvcapp replicas: 2 # --replicas=2 옵션과 동일 template: metadata: labels: app: mvcapp spec: containers: - name: mvcapp image: cdecl/mvcapp:0.6 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: mvcapp spec: type: NodePort selector: app: mvcapp ports: - port: 80 targetPort: 80 $ kubectl apply -f mvcapp-deploy-service.yaml deployment.apps/mvcapp created service/mvcapp created $ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-5ff76fc89d-wh9cg 1/1 Running 0 9m29s kube-system coredns-7448499f4d-2d7pb 1/1 Running 0 9m29s kube-system metrics-server-86cbb8457f-x9l6n 1/1 Running 0 9m29s kube-system helm-install-traefik-crd-w27q7 0/1 Completed 0 9m29s kube-system helm-install-traefik-2zllj 0/1 Completed 1 9m29s kube-system svclb-traefik-55qfd 2/2 Running 0 8m47s kube-system traefik-97b44b794-smzl9 1/1 Running 0 8m48s default mvcapp-79874d888c-6htvq 1/1 Running 0 62s default mvcapp-79874d888c-clslc 1/1 Running 0 62s $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 10m mvcapp NodePort 10.43.36.139 <none> 80:32105/TCP 106s # Nodeport IP $ curl 10.43.36.139:80 * Project : Mvcapp * Version : 0.5 / net5.0 * Hostname : mvcapp-79874d888c-6htvq * RemoteAddr : 10.42.0.1 * X-Forwarded-For : * Request Count : 1 * User-Agent : curl/7.29.0 $ curl localhost:32105 * Project : Mvcapp * Version : 0.5 / net5.0 * Hostname : mvcapp-79874d888c-clslc * RemoteAddr : 10.42.0.1 * X-Forwarded-For : * Request Count : 1 * User-Agent : curl/7.29.0 Agent 추가 Master Node만으로도 테스트 가능하나 Scale 테스트시 Agent(Worker Node) 추가 가능 환경변수 세팅 : 필요시 참고 $ sudo cat /var/lib/rancher/k3s/server/node-token > ~/.node-token $ K3S_TOKEN=$(< ~/.node-token) $ HOST_IP=$(ip a | sed -rn 's/.*inet ([0-9\.]+).*eth0/\1/p') Agent 등록 : 원격실행 OR Agent 머신에서 실행 HostIP, Token 정보 필요 (위 환경변수 세팅 참고) # Agent 머신에서 실행 $ curl -sfL https://get.k3s.io | K3S_URL=https://$HOST_IP:6443 K3S_TOKEN=$K3S_TOKEN sh - # Agent 추가 다른방법 $ ansible node01 -m shell -a "curl -sfL https://get.k3s.io | sh -s - agent --server https://$HOST_IP:6443 --token $K3S_TOKEN" -v K3S 삭제 ls /usr/local/bin/k3s-* | xargs -n1 sh -