MinIO 101

업데이트: 2021.08.17

MinIO 101

Introduction

https://docs.min.io/
Open Source, S3 Compatible, Enterprise Hardened and Really, Really Fast
- S3 Compatible : Client(mc), SDK (Java, Javascript, Python, Golang, .Net ..)
high performance, distributed object storage system
Private cloud object storage

Getting Started

MinID 는 Golang으로 제작되어 의존성 없는 단일 파일로 운영 가능
Docker의 경우 Alpine linux 로 배포
Downloads : https://min.io/download

Quickstart Server

https://docs.min.io/docs/minio-quickstart-guide.html

# linux - server run
$ wget https://dl.min.io/server/minio/release/linux-amd64/minio
$ chmod +x minio

$ export MINIO_ROOT_USER=minio
$ export MINIO_ROOT_PASSWORD=miniopass

# ./minio server --address 0.0.0.0:9000 /data
$ ./minio server /data

Status:         1 Online, 0 Offline.
Endpoint:  http://192.168.144.5:9000  http://127.0.0.1:9000

Browser Access:
   http://192.168.144.5:9000  http://127.0.0.1:9000

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide

docker-compose
- 9000 : 데이터 I/F 포트
- 9001 : Web Console Port
- ./data:/data : 데이터 영역

version: '3'
services:
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    container_name: minio
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: miniopass
    restart: always
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - ./data:/data

$ docker-compose up -d 
Creating network "minio_default" with the default driver
Pulling minio (minio/minio:)...
latest: Pulling from minio/minio
c2c17d84f25a: Pull complete
46cdcde062b2: Pull complete
c88923a3df19: Pull complete
1afaaeffed49: Pull complete
6c066ed8931e: Pull complete
b889e4f29831: Pull complete
51b722521628: Pull complete
Digest: sha256:ff4892c4248ad0ef73981d9f2e7b8a721dae45c55bdd25d7a23e1670540f36e1
Status: Downloaded newer image for minio/minio:latest
Creating minio ... done

Quickstart Client

https://docs.min.io/docs/minio-client-quickstart-guide.html

# linux 
# wget https://dl.min.io/client/mc/release/linux-amd64/mc
$ curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
$ chmod +x mc
$ mv ./mc /usr/bin/ 
$ mc --help

# add server config 
$ mc alias set local http://localhost:9000 minio miniopass
Added `local` successfully.

# creates a new bucket
$ minio mc mb local/backup
Bucket created successfully `local/backup`.

# copy
$ mc cp docker-compose.yml local/backup
docker-compose.yml:   322 B / 322 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 30.39 KiB/s 0s

# list 
$ mc ls local/backup
[2021-08-18 11:01:50 KST]   322B docker-compose.yml

# remove 
$ mc rm --recursive --force local/backup/
Removing `local/backup/docker-compose.yml`.

MinIO Erasure Code

https://docs.min.io/docs/minio-erasure-code-quickstart-guide.html

Erasure Code

누락되거나 손상된 데이터를 재구성하는 수학적 알고리즘
Erasure code와 Checksums 사용하여 하드웨어 오류 및 자동 데이터 손상으로부터 데이터를 보호
중복 수준이 높으면 전체 드라이브의 최대 절반 (N/2)이 손실 되어도 데이터를 복구 가능
드라이브를 4, 6, 8, 10, 12, 14 또는 16 개의 erasure-coding sets 구성 (최소 4개)

Run MinIO Server with Erasure Code

4 drives setup
Drives를 물리적인 디스크로 구성하면 별도의 Raid 구성이 필요 없음

$ minio server /data1 /data2 /data3 /data4

docker-compose

docker-compose.yml

version: '3'
services:
  minio1:
    image: minio/minio
    command: server /data1 /data2 /data3 /data4 --console-address ":9001"
    container_name: minio1
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: miniopass
    restart: always
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - ./data1:/data1
      - ./data2:/data2
      - ./data3:/data3
      - ./data4:/data4

Distributed MinIO

https://docs.min.io/docs/distributed-minio-quickstart-guide.html
MinIO in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server.
서버를 분리하여 다중 Drives를 지원
- Data protection
- High availability
- Consistency Guarantees

Run distributed MinIO

MINIO_ROOT_USER and MINIO_ROOT_PASSWORD 같은 키로 구성
Erasure Code와 동일한 Drivers 정책으로 분산
Distributed MinIO 와 서버별로 Erasure Code 같이 적용 가능

docker-compose : docker로 4대 서버 시뮬레이션

version: '3'
services:
  minio1:
    image: minio/minio
    command: server http://minio{1...4}:9000/data --console-address ":9001"
    container_name: minio1
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: miniopass
    restart: always
    ports:
      - "9101:9000"
      - "9001:9001"
    volumes:
      - ./minio1:/data

  minio2:
    image: minio/minio
    command: server http://minio{1...4}:9000/data --console-address ":9001"
    container_name: minio2
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: miniopass
    restart: always
    ports:
      - "9102:9000"
    volumes:
      - ./minio2:/data

  minio3:
    image: minio/minio
    command: server http://minio{1...4}:9000/data --console-address ":9001"
    container_name: minio3
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: miniopass
    restart: always
    ports:
      - "9103:9000"
    volumes:
      - ./minio3:/data

  minio4:
    image: minio/minio
    command: server http://minio{1...4}:9000/data --console-address ":9001"
    container_name: minio4
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: miniopass
    restart: always
    ports:
      - "9104:9000"
    volumes:
      - ./minio4:/data

MinIO Admin Guide

mc 명령어를 통해 Admin 기능을 수행
https://docs.min.io/docs/minio-admin-complete-guide.html

service     restart and stop all MinIO servers
update      update all MinIO servers
info        display MinIO server information
user        manage users
group       manage groups
policy      manage policies defined in the MinIO server
config      manage MinIO server configuration
heal        heal disks, buckets and objects on MinIO server
profile     generate profile data for debugging purposes
top         provide top like statistics for MinIO
trace       show http trace for MinIO server
console     show console logs for MinIO server
prometheus  manages prometheus config
kms         perform KMS management operations

user - Manage users

User 생성 및 삭제

## create User
$ mc admin user add myinfo cdecl cdeclpass

## remove User
# mc admin user remove myinfo cdecl

$ mc admin user info myinfo cdecl             
AccessKey: cdecl                              
Status: enabled                               
PolicyName:                                   
MemberOf:                                     

Policy

$ mc admin policy set myinfo readonly user=cdecl
Policy readonly is set on user `cdecl`

$ mc admin policy set myinfo writeonly user=cdecl
Policy writeonly is set on user `cdecl`

$ mc admin policy set myinfo readwrite user=cdecl
Policy readwrite is set on user `cdecl`

heal - Heal disks, buckets and objects on MinIO server

This command is only applicable for MinIO erasure coded setup (standalone and distributed).
Erasure Code 상태에서 특정 디스크의 데이터가 문제가 있을 경우, 균등하게 데이터를 복구 해줌

$ mc admin heal -r myinfo
 -  data1
    2/2 objects; 403 MiB in 1s
    ┌────────┬───┬─────────────────────┐
    │ Green  │ 5 │ 100.0% ████████████ │
    │ Yellow │ 0 │   0.0%              │
    │ Red    │ 0 │   0.0%              │
    │ Grey   │ 0 │   0.0%              │
    └────────┴───┴─────────────────────┘

Byung Kyu KIM

MinIO 101

Introduction

Getting Started

Quickstart Server

Quickstart Client

MinIO Erasure Code

Erasure Code

Run MinIO Server with Erasure Code

docker-compose

Distributed MinIO

Run distributed MinIO

docker-compose : docker로 4대 서버 시뮬레이션

MinIO Admin Guide

user - Manage users

heal - Heal disks, buckets and objects on MinIO server

댓글남기기

참고

TLS와 SSH 프로토콜 기능

GNU C++로 정적 라이브러리 링크하는 방법

유니코드 NFD to NFC 변환

Ollama를 이용한 Mistral 로컬 실행 가이드