搭建 elasticsearch 叢集

發表於 更新於 閱讀次數： Disqus：

本文記錄搭建 elasticsearch cluster 的過程。

對於 ELK 的應用及操作都練習很多了，日常工作常用到，但如果是從零開始，這我反倒是沒有經驗，這篇其實就是我自己的摸索過程

因為情境是搭建在三台虛擬機上，每台一個elasticsearch節點，所以勢必不可能用一份 docker-compose.yml直接結案
我的錯誤嘗試很多，所以不打算全都列出來，所以我大概提一下重點

node1

.env

1
2
3
4
5
# .env
# Elasticsearch connection settings
ELASTIC_HOST=https://localhost:9200
ELASTIC_USERNAME=elastic
ELASTIC_PASSWORD=test123

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
services:
  setup:
    container_name: setup
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.1
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 172.27.98.106\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 172.27.106.246\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "      - 172.27.102.224\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"kibana123\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120
  es01:
    container_name: es01
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.1
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - ./esdata:/usr/share/elasticsearch/data
    environment:
      - node.name=es01
      - network.publish_host=172.27.98.106
      - cluster.name=es-cluster
      - cluster.initial_master_nodes=es01
      - discovery.seed_hosts=172.27.98.106,172.27.106.246,172.27.102.224
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
      - cluster.routing.allocation.disk.watermark.low=2gb
      - cluster.routing.allocation.disk.watermark.high=1gb
      - cluster.routing.allocation.disk.watermark.flood_stage=512mb
    mem_limit: 1073741824
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
  kibana:
    container_name: kibana
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:8.6.1
    volumes:
      - ./certs:/usr/share/kibana/config/certs
      - ./kibanadata:/usr/share/kibana/data
    ports:
      - 5601:5601
    user: "1000"
    environment:
      - SERVERNAME=kibana
      - theme:darkMode=true
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=kibana123
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - xpack.security.encryptionKey=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
      - xpack.encryptedSavedObjects.encryptionKey=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
      - xpack.reporting.encryptionKey=cccccccccccccccccccccccccccccccc

    mem_limit: 1073741824
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

上面這一段有一個很重要的是 setup 的部分，這是一個獨立的服務，用來建立憑證。

node2

.env

1
2
3
4
# .env
ELASTIC_HOST=https://localhost:9200
ELASTIC_USERNAME=elastic
ELASTIC_PASSWORD=test123

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
services:
  es02:
    container_name: es02
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.1
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - ./esdata:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - network.publish_host=172.27.106.246
      - cluster.name=es-cluster
      - discovery.seed_hosts=172.27.98.106,172.27.106.246,172.27.102.224
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
      - cluster.routing.allocation.disk.watermark.low=2gb
      - cluster.routing.allocation.disk.watermark.high=1gb
      - cluster.routing.allocation.disk.watermark.flood_stage=512mb
    mem_limit: 1073741824
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

node3

.env

1
2
3
4
# .env
ELASTIC_HOST=https://localhost:9200
ELASTIC_USERNAME=elastic
ELASTIC_PASSWORD=test123

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
services:
  es03:
    container_name: es03
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.1
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - ./esdata:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=es-cluster
      - network.publish_host=172.27.102.224
      - cluster.name=es-cluster
      - discovery.seed_hosts=172.27.98.106,172.27.106.246,172.27.102.224
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
      - cluster.routing.allocation.disk.watermark.low=2gb
      - cluster.routing.allocation.disk.watermark.high=1gb
      - cluster.routing.allocation.disk.watermark.flood_stage=512mb
    mem_limit: 1073741824
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

重點說明

因為這些筆記已經寫了有半年之久，今天才整理，所以我其實已經忘記了這是我第幾次嘗試後，寫的手動SOP
但印象是node1的 docker-compose.yml裡面有 setup的部分，其實已經把這些東西都自動化作掉了。
不過還是紀錄一下，如果需要手動作的話，大概會有哪些環節。

建立minimal security

  1. 加入xpack.security.enabled: true

查看節點

1
http://172.27.106.193:9200/_cat/nodes?v

第一次建立密碼

1
2
docker exec -it elasticsearch bash
./bin/elasticsearch-setup-passwords interactive

kibana 加入帳密設定

1
2
elasticsearch.username: kibana_system
elasticsearch.password: change-me

建立憑證(讓node 互相以 TLS 溝通)

  • 建立CA
  • 使用CA建立憑證並放到node中
  • 添加設定到yml，並重啟節點

node1 進入節點並建立CA、產生憑證

1
docker exec -it elasticsearch bash
1
2
3
4
# 建立CA
./bin/elasticsearch-certutil ca
# 接著用ca來建立憑證
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

在宿主機創建臨時目錄

mkdir ~/elastic-certs && cd ~/elastic-certs

使用 Elasticsearch 容器生成 CA 和節點證書（無需啟動容器）

docker run –rm -v $(pwd):/certs docker.elastic.co/elasticsearch/elasticsearch:8.17.4
bin/elasticsearch-certutil cert –silent –pem –in /usr/share/elasticsearch/config/certificates/instances.yml –out /certs/certs.zip

解壓證書文件

unzip certs.zip -d certs

將產生的憑證從容器複製出來，調整權限方便讓其他節點複製

1
2
3
docker cp elasticsearch:/usr/share/elasticsearch/elastic-certificates.p12 .
chmod 777 elastic-certificates.p12
# 接著把憑證放到 /home/art/es-auth/cert

node2 , node 3

先把憑證的目錄建好，放在 /home/art/es-auth/cert

1
sudo scp art@172.27.106.193:/home/art/es-auth/cert/elastic-certificates.p12 .

修改所有節點設定檔，加入下面設定 node1,2,3的 elasticsearch.yml

1
2
3
4
5
6
# 產生金鑰後加入下面設定
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

讓 docker-compose 掛載憑證

1
2
volumes:
  - /home/art/es-auth/cert/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12

參考資料

  1. # 為 Elasticsearch 設定最低安全性
  2. https://ithelp.ithome.com.tw/articles/10329516
  3. https://github.com/cr7258/elasticsearch-mcp-server/blob/main/docker-compose-elasticsearch.yml