I try to add a snapshot repository using the Azure Repository Plugin to an elasticsearch instance running on Azure (with K8s) deployed through helm using the chart from helm.elastic.com. The chart is patched to use a self-built image myprivaterepo:elasticsearch-azure:7.9.2 containing the Azure Repository Plugin and now I'm stuck adding the Azure credentials to the elasticsearch-keystore.

I added a secret containing the azure credentials and a second init-container to the elasticsearch statefulset to populate the keystore from the secret:

 - command: - sh - -c - | whoami echo $AZURE_ACCOUNT | bin/elasticsearch-keystore add --stdin --force azure.client.default.account echo $AZURE_SAS_TOKEN | bin/elasticsearch-keystore add --stdin --force azure.client.default.sas_token ls -l config bin/elasticsearch-keystore list env: - name: AZURE_ACCOUNT valueFrom: secretKeyRef: key: account name: snapshot-secret - name: AZURE_SAS_TOKEN valueFrom: secretKeyRef: key: sas name: snapshot-secret image: myprivateregistry/elasticsearch-azure-7.9.2 imagePullPolicy: IfNotPresent name: update-keystore resources: {} securityContext: runAsGroup: 1000 runAsNonRoot: true runAsUser: 1000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File

The command is slightly extended to check problems with permission. The Init Container runs sucessfully, producing the following output.

$ kubectl logs elasticsearch-master-2 -c update-keystore
elasticsearch
total 32
-rw-rw---- 1 elasticsearch elasticsearch 428 Mar 24 09:01 elasticsearch.keystore
-rw-rw---- 1 elasticsearch root 53 Sep 23 00:49 elasticsearch.yml
-rw-rw---- 1 elasticsearch root 2301 Sep 23 00:43 jvm.options
drwxrwxr-x 2 elasticsearch root 4096 Sep 23 00:47 jvm.options.d
-rw-rw---- 1 elasticsearch root 7734 Sep 23 00:49 log4j2.properties
-rw-rw---- 1 elasticsearch root 473 Sep 23 00:47 role_mapping.yml
-rw-rw---- 1 elasticsearch root 197 Sep 23 00:47 roles.yml
-rw-rw---- 1 elasticsearch root 0 Sep 23 00:47 users
-rw-rw---- 1 elasticsearch root 0 Sep 23 00:47 users_roles
azure.client.default.account
azure.client.default.sas_token
keystore.seed

The first line is the user which ran the command, the list shows /usr/share/elasticsearch/config followed by the output from bin/elasticsearch-keystore list which correctly shows the added keys.

My problem is that in the running elasticsearch container, the keystore is empty

$ kubectl exec -ti elasticsearch-master-2 -- bin/elasticsearch-keystore list
keystore.seed

same for -master-0 and -master-1.

I looked at several blogs and howtos, all doing it quite the same way, but I cannot find the mistake. Some use the elasticsearch controller which is currently no option and should not really be the showstopper here.

A Guide to Elasticsearch SnapshotsKeystore file created by initContainer is not accessible by the elasticsearch user #3332

I, too, deployed the chart into our private K8s cluster running elasticsearch 7.11.2 with the same result.

1 Answer

Solved it. I overlooked that the InitContainer and the real container need to mount a volume to share the generated secrets.

By adding a volume and properly mounting it to the container like this, the keystore has the expected values within the container.

spec: template: spec: volumes: - name: keystore emptyDir: {} containers: volumeMount: - name: keystore mountPath: /usr/share/elasticsearch/config/elasticsearch.keystore subPath: elasticsearch.keystore initContainer: - command: [... as above ...] cp -a /usr/share/elastichsearch/config/elasticsearch.keystore /tmp/keystore volumeMounts: - name: keystore mountPath: /tmp/keystore

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy