Object Storage

MinIO 101 Introduction https://docs.min.io/{:target="_blank"} Open Source, S3 Compatible, Enterprise Hardened and Really, Really Fast S3 Compatible : Client(mc), SDK (Java, Javascript, Python, Golang, .Net ..) high performance, distributed object storage system Private cloud object storage Getting Started MinID 는 Golang으로 제작되어 의존성 없는 단일 파일로 운영 가능 Docker의 경우 Alpine linux 로 배포 Downloads : https://min.io/download{:target="_blank"} Quickstart Server https://docs.min.io/docs/minio-quickstart-guide.html{:target="_blank"} # linux - server run $ wget https://dl.min.io/server/minio/release/linux-amd64/minio $ chmod +x minio $ export MINIO_ROOT_USER=minio $ export MINIO_ROOT_PASSWORD=miniopass # ./minio server --address 0.0.0.0:9000 /data $ ./minio server /data Status: 1 Online, 0 Offline. Endpoint: http://192.168.144.5:9000 http://127.0.0.1:9000 Browser Access: http://192.168.144.5:9000 http://127.0.0.1:9000 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide docker-compose 9000 : 데이터 I/F 포트 9001 : Web Console Port ./data:/data : 데이터 영역 version: '3' services: minio: image: minio/minio command: server /data --console-address ":9001" container_name: minio environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9000:9000" - "9001:9001" volumes: - ./data:/data $ docker-compose up -d Creating network "minio_default" with the default driver Pulling minio (minio/minio:)... latest: Pulling from minio/minio c2c17d84f25a: Pull complete 46cdcde062b2: Pull complete c88923a3df19: Pull complete 1afaaeffed49: Pull complete 6c066ed8931e: Pull complete b889e4f29831: Pull complete 51b722521628: Pull complete Digest: sha256:ff4892c4248ad0ef73981d9f2e7b8a721dae45c55bdd25d7a23e1670540f36e1 Status: Downloaded newer image for minio/minio:latest Creating minio ... done Quickstart Client https://docs.min.io/docs/minio-client-quickstart-guide.html{:target="_blank"} # linux # wget https://dl.min.io/client/mc/release/linux-amd64/mc $ curl -O https://dl.min.io/client/mc/release/linux-amd64/mc $ chmod +x mc $ mv ./mc /usr/bin/ $ mc --help # add server config $ mc alias set local http://localhost:9000 minio miniopass Added `local` successfully. # creates a new bucket $ minio mc mb local/backup Bucket created successfully `local/backup`. # copy $ mc cp docker-compose.yml local/backup docker-compose.yml: 322 B / 322 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 30.39 KiB/s 0s # list $ mc ls local/backup [2021-08-18 11:01:50 KST] 322B docker-compose.yml # remove $ mc rm --recursive --force local/backup/ Removing `local/backup/docker-compose.yml`. MinIO Erasure Code https://docs.min.io/docs/minio-erasure-code-quickstart-guide.html{:target="_blank"} Erasure Code 누락되거나 손상된 데이터를 재구성하는 수학적 알고리즘 Erasure code와 Checksums 사용하여 하드웨어 오류 및 자동 데이터 손상으로부터 데이터를 보호 중복 수준이 높으면 전체 드라이브의 최대 절반 (N/2)이 손실 되어도 데이터를 복구 가능 드라이브를 4, 6, 8, 10, 12, 14 또는 16 개의 erasure-coding sets 구성 (최소 4개) Run MinIO Server with Erasure Code 4 drives setup Drives를 물리적인 디스크로 구성하면 별도의 Raid 구성이 필요 없음 $ minio server /data1 /data2 /data3 /data4 docker-compose docker-compose.yml version: '3' services: minio1: image: minio/minio command: server /data1 /data2 /data3 /data4 --console-address ":9001" container_name: minio1 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9000:9000" - "9001:9001" volumes: - ./data1:/data1 - ./data2:/data2 - ./data3:/data3 - ./data4:/data4 Distributed MinIO https://docs.min.io/docs/distributed-minio-quickstart-guide.html{:target="_blank"} MinIO in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server. 서버를 분리하여 다중 Drives를 지원 Data protection High availability Consistency Guarantees Run distributed MinIO MINIO_ROOT_USER and MINIO_ROOT_PASSWORD 같은 키로 구성 Erasure Code와 동일한 Drivers 정책으로 분산 Distributed MinIO 와 서버별로 Erasure Code 같이 적용 가능 docker-compose : docker로 4대 서버 시뮬레이션 version: '3' services: minio1: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio1 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9101:9000" - "9001:9001" volumes: - ./minio1:/data minio2: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio2 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9102:9000" volumes: - ./minio2:/data minio3: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio3 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9103:9000" volumes: - ./minio3:/data minio4: image: minio/minio command: server http://minio{1...4}:9000/data --console-address ":9001" container_name: minio4 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: miniopass restart: always ports: - "9104:9000" volumes: - ./minio4:/data MinIO Admin Guide mc 명령어를 통해 Admin 기능을 수행 https://docs.min.io/docs/minio-admin-complete-guide.html{:target="_blank"} service restart and stop all MinIO servers update update all MinIO servers info display MinIO server information user manage users group manage groups policy manage policies defined in the MinIO server config manage MinIO server configuration heal heal disks, buckets and objects on MinIO server profile generate profile data for debugging purposes top provide top like statistics for MinIO trace show http trace for MinIO server console show console logs for MinIO server prometheus manages prometheus config kms perform KMS management operations user - Manage users User 생성 및 삭제 ## create User $ mc admin user add myinfo cdecl cdeclpass ## remove User # mc admin user remove myinfo cdecl $ mc admin user info myinfo cdecl AccessKey: cdecl Status: enabled PolicyName: MemberOf: Policy $ mc admin policy set myinfo readonly user=cdecl Policy readonly is set on user `cdecl` $ mc admin policy set myinfo writeonly user=cdecl Policy writeonly is set on user `cdecl` $ mc admin policy set myinfo readwrite user=cdecl Policy readwrite is set on user `cdecl` heal - Heal disks, buckets and objects on MinIO server This command is only applicable for MinIO erasure coded setup (standalone and distributed). Erasure Code 상태에서 특정 디스크의 데이터가 문제가 있을 경우, 균등하게 데이터를 복구 해줌 $ mc admin heal -r myinfo - data1 2/2 objects; 403 MiB in 1s ┌────────┬───┬─────────────────────┐ │ Green │ 5 │ 100.0% ████████████ │ │ Yellow │ 0 │ 0.0% │ │ Red │ 0 │ 0.0% │ │ Grey │ 0 │ 0.0% │ └────────┴───┴─────────────────────┘