Table of Contents

Add elastic repo

helm repo add elastic https://helm.elastic.co

Install Elasticsearch

Create namespace

kubectl create ns elasticsearch

deploy Elasticsearch from helm

helm install --namespace elasticsearch --name elasticsearch elastic/elasticsearch --set replicas=1

Create Persistent Volume for Elasticsearch

sudo mkdir /kubernetes_volumes/elasticsearch-data
sudo chmod a+rw -R /kubernetes_volumes/elasticsearch-data/
apiVersion: v1
kind: PersistentVolume
metadata:
   name: elasticsearch-data
   namespace: elasticsearch
   labels:
     app: elasticsearch-master
spec:
  capacity:
    storage: 32Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: "/kubernetes_volumes/elasticsearch-data"
    type: Directory
  persistentVolumeReclaimPolicy: Retain

И дадим права на запись в индексы:

kubectl exec -it -n elasticsearch elasticsearch-master-0 -- curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Так как нода у меня одна, то мне нужно запретить создавать реплики. Иниче - в дальнейшем кластер просто перестанет работать:

kubectl exec -it -n elasticsearch elasticsearch-master-0 -- curl -H "Content-Type: application/json" -XPUT 'http://127.0.0.1:9200/_template/default' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}'

И почистим уже созданные индексы:

kubectl exec -it -n elasticsearch elasticsearch-master-0 -- curl -XDELETE 'http://127.0.0.1:9200/*'

Теперь в логах пода elasticsearch-master-0 появится строка:

Cluster health status changed from [YELLOW] to [GREEN]

И в этот elasticsearch можно будет писать логи.

Install Kibana

kubectl create ns kibana
helm install --namespace elasticsearch --name kibana elastic/kibana --set elasticsearchHosts=http://elasticsearch-master.elasticsearch.svc.cluster.local:9200
  helm upgrade kibana elastic/kibana --set elasticsearchHosts=http://openresty-oidc-http.elasticsearch.svc.cluster.local:9200 --set elasticsearch.requestHeadersWhitelist=[X-Auth-Username]  

Kibana Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
  name: kibana-ingress
  namespace: elasticsearch
spec:
  rules:
  - host: kibana.autosys.tk
    http:
      paths:
      - backend:
          serviceName: kibana-kibana
          servicePort: 5601
        path: /
  tls:
  - hosts:
    - kibana.autosys.tk
    secretName:  kibana-autosys-tk-tls

Kibana Saved Objects

В кибане есть сохраненные пользователем объекты. Например - Index Patterns.
Создать можно так:

kubectl exec -it -n elk elk-openresty-oidc-559f46d69d-shbwg -- curl -H 'Authorization: Basic dXN....o' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"attributes": {"title": "my-pattern-*"}}' http://elk-kb-http:5601/api/saved_objects/index-pattern/my-pattern

Посмотреть их можно так:

kubectl exec -it -n elk elk-openresty-oidc-559f46d69d-shbwg -- curl -H 'Authorization: Basic dXN....o' http://elk-kb-http:5601/api/saved_objects/_find?type=index-pattern | grep '"type":"index-pattern"'

Удалить какой-то можно так:

kubectl exec -it -n elk elk-openresty-oidc-559f46d69d-shbwg -- curl -X DELETE -H 'kbn-xsrf: true' -H 'Authorization: Basic dXN....o' http://elk-kb-http:5601/api/saved_objects/index-pattern/vrm

Logstash

https://habr.com/ru/post/421819/
https://helm.elastic.co/

helm install --namespace elasticsearch --name logstash elastic/logstash -f ./values.yaml

Настройка rsyslog и фильтра logstash

У меня rsyslog сыплет логи в таком виде:

Dec 25 14:06:25 kub kubelet[989]: W1225 14:06:25.516800     989 volume_linux.go:45] Setting volume ownership for /var/lib/kubelet/pods/d36eb244-fe63-4f5d-b133-c4e4232c68b3/volumes/kubernetes.io~configmap/sc-dashboard-provider and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699

Для того, чтобы logstash смог сформировать объект json-объект message нужно в конфигурацию прописать фильтр, который распарсит строку message , чтобы на выходе получился объект.
Встроенные паттерны тут: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
Проверить паттерны можно тут: http://grokdebug.herokuapp.com/
Для вышеприведенной стоки подходит такой фильтр:

%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{SYSLOGHOST:host}%{SPACE}%{SYSLOGPROG:process}:%{SPACE}%{GREEDYDATA:SYSLOGMESSAGE}

Для того, чтобы логи попадали в elasticsearch в конфиг rsyslog (/etc/rsyslog.conf) в конце надо дописать такое:

*.*  @@logstash.autosys.tk:1514

Тут *.* - это значит все записи, @@ - это TCP (@ - UDP), дальше хост logstash и порт.

logstash_values.yaml

---
replicas: 1

# Allows you to add any config files in /usr/share/logstash/config/
# such as logstash.yml and log4j2.properties
logstashConfig:
  logstash.yml: |
    http.host: "0.0.0.0"
    config.reload.automatic: "true"
    xpack.management.enabled: "false"
    
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value

# Allows you to add any pipeline files in /usr/share/logstash/pipeline/
logstashPipeline:
  input_main.conf: |
    input {
      udp {
        port => 1514
        type => syslog
      }
      tcp {
        port => 1514
        type => syslog
      }
      http {
        port => 8080
      }
      # kafka {
      #  ## ref: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
      #   bootstrap_servers => "kafka-input:9092"
      #   codec => json { charset => "UTF-8" }
      #   consumer_threads => 1
      #   topics => ["source"]
      #   type => "example"
      # }
    }
  filter_main.conf: |
    filter {
      if [type] == "syslog" {
       # Uses built-in Grok patterns to parse this standard format
        grok {
          match => {
            "message" => "%{SYSLOGTIMESTAMP:@timestamp}%{SPACE}%{SYSLOGHOST:host}%{SPACE}%{SYSLOGPROG:process}:%{SPACE}%{GREEDYDATA:syslogmessage}"
          }
        }
        # Sets the timestamp of the event to the timestamp of recorded in the log-data
        # By default, logstash sets the timestamp to the time it was ingested.
        #date {
        #  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        #}
        mutate {
          rename => ["syslogmessage", "message" ]
        }
      }
    }
  output_main.conf: |
    output {
     # stdout { codec => rubydebug }
      elasticsearch {
        hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
        manage_template => false
        index => "logs-%{+YYYY.MM.dd}"
     }
     # kafka {
     #   ## ref: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html
     #   bootstrap_servers => "kafka-output:9092"
     #   codec => json { charset => "UTF-8" }
     #   compression_type => "lz4"
     #   topic_id => "destination"
     # }
    }

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: 
  - name: "ELASTICSEARCH_HOST"
    value: "elasticsearch-master.elasticsearch.svc.cluster.local"
  - name: "ELASTICSEARCH_PORT"
    value: "9200"
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# A list of secrets and their paths to mount inside the pod
secretMounts: []

image: "docker.elastic.co/logstash/logstash"
imageTag: "7.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: []

podAnnotations: {}

# additionals labels
labels:
  app: logstash
  
logstashJavaOpts: "-Xmx1g -Xms1g"

resources:
  requests:
    cpu: "100m"
    memory: "1536Mi"
  limits:
    cpu: "1000m"
    memory: "1536Mi"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 1Gi

rbac:
  create: false
  serviceAccountName: ""

podSecurityPolicy:
  create: false
  name: ""
  spec:
    privileged: true
    fsGroup:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
      - secret
      - configMap
      - persistentVolumeClaim

persistence:
  enabled: false
  annotations: {}

extraVolumes: ""
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: ""
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true

extraContainers: ""
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

extraInitContainers: ""
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

httpPort: 9600

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000

securityContext:
  capabilities:
    drop:
    - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

# How long to wait for logstash to stop gracefully
terminationGracePeriod: 120

livenessProbe:
  httpGet:
    path: /
    port: http
  initialDelaySeconds: 300
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
  successThreshold: 1

readinessProbe:
  httpGet:
    path: /
    port: http
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
  successThreshold: 3

## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""

nodeSelector: {}
tolerations: []

nameOverride: ""
fullnameOverride: ""

lifecycle: {}
  # preStop:
  #   exec:
  #     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
  # postStart:
  #   exec:
  #     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]

service:
  annotations: {}
  type: LoadBalancer
  ports:
  - name: beats
    port: 5044
    protocol: TCP
    targetPort: 5044
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: syslog
    port: 1514
    targetPort: 1514

Create Persistent Volume for LogStash

sudo mkdir /kubernetes_volumes/logstash-data
sudo chmod a+rw -R /kubernetes_volumes/logstash-data
apiVersion: v1
kind: PersistentVolume
metadata:
   name: logstash-data
   namespace: elasticsearch
   labels:
     app: logstash
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: "/kubernetes_volumes/logstash-data"
    type: Directory
  persistentVolumeReclaimPolicy: Retain

Проверка работоспособности ELK

Теперь можно проверить, что всё вместе работает.
Можно отправить сообщение в logstash на input http:

curl -XPUT 'http://~~~logstash~IP~~~:8080/test/test1/1' -d 'hello'

В ответ должно быть ok.
А в kibana можно нажать Discovery, увидеть там новый индекс.

Filebeat

Для работы с файлами логов потребуется дополнительный сервис Filebeat, который будет читать логи из файла и отправлять в Logstash. Пример конфигурации:

filebeat.inputs:
- type: log
  enabled: true
  paths:
      - /var/log/nginx/access.log
  fields:
    type: nginx
  fields_under_root: true
  scan_frequency: 5s

output.logstash:
  hosts: ["logstash:5000"]

Filebeat Autodiscover

У Filebeat есть отличный функцмонал - Autodiscover. Он позволяет по заданным правилам автоматически обнаруживать файлы с логами. Прмиер конфига для K8S:

filebeat:
  autodiscover:
    providers:
    - node: ${HOSTNAME}
      templates:
      - config:
        - containers:
            ids:
            - ${data.kubernetes.container.id}
          paths:
          - /var/log/containers/*-${data.kubernetes.container.id}.log
          type: container
      type: kubernetes
output:
  elasticsearch:
    hosts:
    - https://elk-es-http.elk.svc:9200
    password: 7hhjEqA6oH3049D1oTT9s4O2
    ssl:
      certificate_authorities:
      - /mnt/elastic-internal/elasticsearch-certs/ca.crt
    username: elk-elk-beat-user
processors:
- drop_fields:
    fields:
    - log
    - container.id
    - container.runtime
    - container.image.name
    - input.type
    - tags
    - kubernetes.labels
    - kubernetes.pod.uid
    - kubernetes.replicaset.name
    - kubernetes.node
    - kubernetes.namespace_labels
    - kubernetes.namespace_uid
    - ecs.version
    - agent
    ignore_missing: false
setup:
  dashboards:
    enabled: true
  kibana:
    host: http://elk-kb-http.elk.svc:5601
    password: 6l2IPFK8mBR88w6ek49ePI71
    username: elk-elk-beat-kb-user

Этот конфиг сформирован оператором ECK 1.8.
Что делать, если Filebeat Kubernetes Autodiscover не работает без видимых причин?? В моем случае - ошибок не было, но и логи контейнеров Filebeat Autodiscover не обнаруживал. Лог выглядел так:

...
2021-10-24T11:11:24.565Z        INFO    [autodiscover.pod]      kubernetes/util.go:122  kubernetes: Using node MCS-K8S-201 provided in the config
2021-10-24T11:11:24.565Z        DEBUG   [autodiscover.pod]      kubernetes/pod.go:80    Initializing a new Kubernetes watcher using node: MCS-K8S-201
2021-10-24T11:11:24.587Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:90 Configured autodiscover provider: kubernetes
2021-10-24T11:11:24.587Z        INFO    [autodiscover]  autodiscover/autodiscover.go:113        Starting autodiscover manager
2021-10-24T11:11:24.687Z        DEBUG   [kubernetes]    kubernetes/watcher.go:184       cache sync done
2021-10-24T11:11:24.788Z        DEBUG   [kubernetes]    kubernetes/watcher.go:184       cache sync done
2021-10-24T11:11:24.889Z        DEBUG   [kubernetes]    kubernetes/watcher.go:184       cache sync done
...

Однако, при нормальной работе - должно быть что-то такое:

...
2021-10-24T11:49:30.633Z        INFO    [autodiscover.pod]      kubernetes/util.go:122  kubernetes: Using node mcs-k8s-203 provided in the config
2021-10-24T11:49:30.642Z        INFO    [autodiscover]  autodiscover/autodiscover.go:113        Starting autodiscover manager
2021-10-24T11:49:31.048Z        INFO    [input] log/input.go:164        Configured paths: [/var/log/containers/*-70600db8d4371ebdacc300edd825c030e6698a6b75467ea8b280c7aef1faa366.log]  {"input_id": "f60ce837-5828-4125-afe5-5a40d70fc613"}
2021-10-24T11:49:31.048Z        INFO    [input] log/input.go:164        Configured paths: [/var/log/containers/*-70600db8d4371ebdacc300edd825c030e6698a6b75467ea8b280c7aef1faa366.log]  {"input_id": "ba234dcb-e0bd-4395-baa4-ba47dd1f1f6d"}
...

Я потратил довольно много времени на выяснение причин такого поведения. дело оказалось в том, что HOSTNAME на ноде задан большими буквами (MCS-K8S-203), а в кластере нода имеет имя маленькими буквами (mcs-k8s-203). В итоге - всё вылечилось, когда я на хостах кластера выполнил:

  sudo hostnamectl set-hostname `echo "$HOSTNAME" | tr '[:upper:]' '[:lower:]'`

то есть переменовал хосты кластера в нижнем регистре.

Настройка безопасности

По-умолчанию стек ELK не обеспечивает безопасного доступа к данным и компоненты никак не аутентифицируются. То есть записать и читать данные может кто угодно.
Базовая бесплатная лицензия не позволяет аутентифицировать пользователей из каталогов LDAP, поэтому нужно либо платить, либо настраивать аутентификацют сторонними методами.

LDAP-аутентификация в Kibana и RBAC для elasticsearch помощью oidc и proxy

Общая идея такая:

Используемые компоненты:

Вот ссылочки, которые помогали мне в настройке oidc-proxy для LDAP-аутентификации :

Я изобретаю свои велосипеды, а вот есть некторые готовые компоненты:

Keycloak

Развернут с помощью helm - deploy_keycloak_using_helm.
REALM настроен так: https://habr.com/ru/post/441112/

Собираем образ openresty с модулем lua-resty-openidc

Dockerfile на базе https://github.com/Revomatico/docker-openresty-oidc/blob/master/Dockerfile :

FROM alpine:3.10
MAINTAINER Mikhail Usik <mike@autosys.tk>

ENV LUA_SUFFIX=jit-2.1.0-beta3 \
    LUAJIT_VERSION=2.1 \
    NGINX_PREFIX=/opt/openresty/nginx \
    OPENRESTY_PREFIX=/opt/openresty \
    OPENRESTY_SRC_SHA256=bf92af41d3ad22880047a8b283fc213d59c7c1b83f8dae82e50d14b64d73ac38 \
    OPENRESTY_VERSION=1.15.8.2 \
    LUAROCKS_VERSION=3.1.3 \
    LUAROCKS_SRC_SHA256=c573435f495aac159e34eaa0a3847172a2298eb6295fcdc35d565f9f9b990513 \
    LUA_RESTY_OPENIDC_VERSION=1.7.2-1 \
    VAR_PREFIX=/var/nginx

RUN set -ex \
  && apk --no-cache add \
    libgcc \
    libpcrecpp \
    libpcre16 \
    libpcre32 \
    libssl1.1 \
    libstdc++ \
    openssl \
    pcre \
    curl \
    unzip \
    git \
    dnsmasq \
    ca-certificates \
  && apk --no-cache add --virtual .build-dependencies \
    make \
    musl-dev \
    gcc \
    ncurses-dev \
    openssl-dev \
    pcre-dev \
    perl \
    readline-dev \
    zlib-dev \
    libc-dev \
  \
## OpenResty
  && curl -fsSL https://github.com/openresty/openresty/releases/download/v${OPENRESTY_VERSION}/openresty-${OPENRESTY_VERSION}.tar.gz -o /tmp/openresty.tar.gz \
  \
  && cd /tmp \
  && echo "${OPENRESTY_SRC_SHA256} *openresty.tar.gz" | sha256sum -c - \
  && tar -xzf openresty.tar.gz \
  \
  && cd openresty-* \
  && readonly NPROC=$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) \
  && ./configure \
    --prefix=${OPENRESTY_PREFIX} \
    --http-client-body-temp-path=${VAR_PREFIX}/client_body_temp \
    --http-proxy-temp-path=${VAR_PREFIX}/proxy_temp \
    --http-log-path=${VAR_PREFIX}/access.log \
    --error-log-path=${VAR_PREFIX}/error.log \
    --pid-path=${VAR_PREFIX}/nginx.pid \
    --lock-path=${VAR_PREFIX}/nginx.lock \
    --with-luajit \
    --with-pcre-jit \
    --with-ipv6 \
    --with-http_ssl_module \
    --without-http_ssi_module \
    --with-http_realip_module \
    --without-http_scgi_module \
    --without-http_uwsgi_module \
    --without-http_userid_module \
    -j${NPROC} \
  && make -j${NPROC} \
  && make install \
  \
  && rm -rf /tmp/openresty* \
  \
## LuaRocks
  && curl -fsSL http://luarocks.github.io/luarocks/releases/luarocks-${LUAROCKS_VERSION}.tar.gz -o /tmp/luarocks.tar.gz \
  \
  && cd /tmp \
  && echo "${LUAROCKS_SRC_SHA256} *luarocks.tar.gz" | sha256sum -c - \
  && tar -xzf luarocks.tar.gz \
  \
  && cd luarocks-* \
  && ./configure \
    --prefix=${OPENRESTY_PREFIX}/luajit \
    --lua-suffix=${LUA_SUFFIX} \
    --with-lua=${OPENRESTY_PREFIX}/luajit \
    --with-lua-lib=${OPENRESTY_PREFIX}/luajit/lib \
    --with-lua-include=${OPENRESTY_PREFIX}/luajit/include/luajit-${LUAJIT_VERSION} \
  && make build \
  && make install \
  \
  && rm -rf /tmp/luarocks* \
  && rm -rf ~/.cache/luarocks \
## Post install
  && ln -sf ${NGINX_PREFIX}/sbin/nginx /usr/local/bin/nginx \
  && ln -sf ${NGINX_PREFIX}/sbin/nginx /usr/local/bin/openresty \
  && ln -sf ${OPENRESTY_PREFIX}/bin/resty /usr/local/bin/resty \
  && ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luajit-* ${OPENRESTY_PREFIX}/luajit/bin/lua \
  && ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luajit-* /usr/local/bin/lua  \
  && ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luarocks /usr/local/bin/luarocks \
  && ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luarocks-admin /usr/local/bin/luarocks-admin \
  && echo user=root >> /etc/dnsmasq.conf \
## Install lua-resty-openidc
  && cd ~/ \
  # Fix for https://github.com/zmartzone/lua-resty-openidc/issues/213#issuecomment-432471572
#  && luarocks install lua-resty-hmac \
  && luarocks install lua-resty-openidc ${LUA_RESTY_OPENIDC_VERSION} \
## Install lua-resty-xacml-pep
#  && curl -fsSL https://raw.githubusercontent.com/zmartzone/lua-resty-xacml-pep/master/lib/resty/xacml_pep.lua -o /opt/openresty/lualib/resty/xacml_pep.lua \
## Cleanup
  && apk del .build-dependencies 2>/dev/null

WORKDIR $NGINX_PREFIX

CMD dnsmasq; openresty -g "daemon off; error_log /dev/stderr info;"


ConfigMap для openresty

Создано по мотивам https://developers.redhat.com/blog/2018/10/08/configuring-nginx-keycloak-oauth-oidc/ и https://daenney.github.io/2019/10/05/beyondcorp-at-home-authn-authz-openresty

Описаны два виртуальных сервра (оба - прокси).

kind: ConfigMap
apiVersion: v1
metadata:
  name: openresty-oidc-config
  namespace: elasticsearch
data:
  nginx.conf: |
    worker_processes  1;
    events {
        worker_connections  1024;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        sendfile        on;
        keepalive_timeout  65;
        gzip  on;
        ##
        # LUA options
        ##
        lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
        lua_package_path '~/lua/?.lua;;';
        resolver 192.168.77.1;
        # cache for discovery metadata documents
        lua_shared_dict discovery 1m;
        # cache for JWKs
        lua_shared_dict jwks 1m;
        # allow the server to close connection on non responding client, this will free up memory
        reset_timedout_connection on;
    
        server {
        listen 80 default_server;
        server_name kibana.autosys.tk;
        #access_log /dev/stdout;
        #error_log /dev/stdout;
        access_by_lua '
          local opts = {
            redirect_uri = "/redirect_uri",
            accept_none_alg = true,
            discovery = "https://sso.autosys.tk/auth/realms/Autosys/.well-known/openid-configuration",
            client_id = "kibana",
            client_secret = "af903747-905c-4342-a62c-83033d3289cc",
            redirect_uri_scheme = "https",
            logout_path = "/logout",
            redirect_after_logout_uri = "https://sso.autosys.tk/auth/realms/Autosys/protocol/openid-connect/logout?redirect_uri=https://kibana.autosys.tk/",
            redirect_after_logout_with_id_token_hint = false,
            session_contents = {id_token=true}
          }
          -- call introspect for OAuth 2.0 Bearer Access Token validation
          local res, err = require("resty.openidc").authenticate(opts)
          if err then
            ngx.status = 403
            ngx.say(err)
            ngx.exit(ngx.HTTP_FORBIDDEN)
          end
          -- set data from the ID token as HTTP Request headers
          ngx.req.set_header("X-Auth-Username", res.id_token.preferred_username)
          ngx.req.set_header("X-Auth-Groups", res.id_token.groups)
          -- ngx.req.set_header("X-Auth-Audience", res.id_token.aud)
          -- ngx.req.set_header("X-Auth-Email", res.id_token.email)
          -- ngx.req.set_header("X-Auth-ExpiresIn", res.id_token.exp)
          -- ngx.req.set_header("X-Auth-Roles", res.id_token.roles)
          -- ngx.req.set_header("X-Auth-Name", res.id_token.name)
          -- ngx.req.set_header("X-Auth-Subject", res.id_token.sub)
          -- ngx.req.set_header("X-Auth-Userid", res.id_token.preferred_username)
          -- ngx.req.set_header("X-Auth-Locale", res.id_token.locale)
          -- Output headers to nginx err log
          --ngx.log(ngx.ERR, "Got header X-Auth-Userid: "..res.id_token.preferred_username..";")
        ';
        expires           0;
        add_header        Cache-Control private;
        location / {
          proxy_connect_timeout 5s;
          proxy_pass http://kibana-kibana.elasticsearch.svc.cluster.local:5601;
        }
      }
      server {
        listen 9200;
        access_log /dev/stdout;
        error_log /dev/stdout;
        access_by_lua '
        local restrictions = {
          elasticsearch_ro = {
            ["^/$"]                             = { "GET" },
            ["^/?[^/]*/?[^/]*/_mget"]           = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/_doc"]            = { "GET" },
            ["^/?[^/]*/?[^/]*/_search"]         = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/_msearch"]        = { "GET" },
            ["^/?[^/]*/?[^/]*/_validate/query"] = { "GET" },
            ["/_aliases"]                       = { "GET" },
            ["/_cluster.*"]                     = { "GET" }
          },
          elasticsearch_rw = {
            ["^/$"]                             = { "GET" },
            ["^/?[^/]*/?[^/]*/_search"]         = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/_msearch"]        = { "GET", "POST" },
            ["^/?[^/]*/traffic*"]               = { "GET", "POST", "PUT", "DELETE" },
            ["^/?[^/]*/?[^/]*/_validate/query"] = { "GET", "POST" },
            ["/_aliases"]                       = { "GET" },
            ["/_cluster.*"]                     = { "GET" }
          },
          elasticsearch_full = {
            ["^/?[^/]*/?[^/]*/_bulk"]          = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/_refresh"]       = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/?[^/]*/_create"] = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/?[^/]*/_update"] = { "GET", "POST" },
            ["^/?[^/]*/?[^/]*/?.*"]            = { "GET", "POST", "PUT", "DELETE" },
            ["^/?[^/]*/?[^/]*$"]               = { "GET", "POST", "PUT", "DELETE" },
            ["/_aliases"]                      = { "GET", "POST" }
          }
        }
        local groups = ngx.req.get_headers()["x-auth-groups"]
        local authenticated_group = nil
        local ngx_re = require "ngx.re"
        if ( type(groups) == "string" and string.len(groups) >= 1 ) then
          groups = string.lower(groups)
          ngx.log(ngx.ERR, "Got header X-Auth-Groups: "..groups..";")
          local groups_table = ngx_re.split(groups, ", ")
          local groups_number = table.getn(groups_table)
          ngx.log(ngx.ERR, "Groups number : "..groups_number..";")
          for i=1,groups_number do
            local group = groups_table[i]:gsub("/","")
            group = group:gsub("%s","_")
            ngx.log(ngx.ERR, "Check group: "..group..";")
            if (restrictions[group] ~= nil) then
              ngx.log(ngx.ERR, "User belongs to Authenticated Group: "..group..";")
              authenticated_group = group
              break
            end
          end
          -- exit 403 when no matching role has been found
          if authenticated_group == nil then
            ngx.header.content_type = "text/plain"
            ngx.log(ngx.ERR, "Unauthenticated request... User - "..ngx.req.get_headers()["x-auth-username"]..";")
            ngx.status = 403
            ngx.say("403 Forbidden: You don\'t have access to this resource.")
            return ngx.exit(403)
          end
          
          -- get URL
          local uri = ngx.var.uri
          ngx.log(ngx.DEBUG, uri)
          -- get method
          local method = ngx.req.get_method()
          ngx.log(ngx.DEBUG, method)
          
          local allowed  = false
          
          for path, methods in pairs(restrictions[authenticated_group]) do
          
            -- path matched rules?
            local p = string.match(uri, path)
            
            local m = nil
            
            -- method matched rules?
            for _, _method in pairs(methods) do
              m = m and m or string.match(method, _method)
            end
            
            if p and m then
              allowed = true
              ngx.log(ngx.NOTICE, method.." "..uri.." matched: "..tostring(m).." "..tostring(path).." for "..authenticated_group)
              break
            end
          end

          if not allowed then
            ngx.header.content_type = "text/plain"
            ngx.log(ngx.WARN, "Group ["..authenticated_group.."] not allowed to access the resource ["..method.." "..uri.."]")
            ngx.status = 403
            ngx.say("403 Forbidden: You don\'t have access to this resource.")
            return ngx.exit(403)
          end
        end
        ';
        location / {
          proxy_connect_timeout 15s;
          proxy_pass http://elasticsearch-master.elasticsearch.svc.cluster.local:9200;
        }
      }
    }

В конфигурации первого прокси аутентификацию выполняет блок access_by_lua. В local opts прописано следующее:

В конфигурации второго прокси авторизацию также выполняет блок access_by_lua:

openresty-oidc_deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:    
  name: openresty-oidc
  namespace: elasticsearch
spec:
  replicas: 1
  selector:  
    matchLabels:
      app: openresty-oidc
  template:  
    metadata:
      labels:
        app: openresty-oidc
    spec:
      imagePullSecrets:
      - name: autosys-regcred    
      containers:    
        - name: openresty-oidc
          image: registry.autosys.tk/openresty-oidc
          volumeMounts:
            - name: openresty-oidc-config-volume
              mountPath: /opt/openresty/nginx/conf/nginx.conf
              subPath: nginx.conf
      volumes:
        - name: openresty-oidc-config-volume
          configMap:
            name: openresty-oidc-config

openresty-oidc-service

apiVersion: v1
kind: Service
metadata:
  name: openresty-oidc-http
  namespace: elasticsearch
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: elasticsearch
    port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    app: openresty-oidc
  sessionAffinity: None
  type: ClusterIP


kibana-values.yaml

В конфигурации kibana нужно внести следующие изменения с помощью helm upgrade:

elasticsearchHosts: "http://openresty-oidc-http.elasticsearch.svc.cluster.local:9200"
kibanaConfig:
  kibana.yml: |
    server.name: kibana
    server.host: "0"
    xpack.monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch.requestHeadersWhitelist:
    - authorization
    - x-auth-groups
    - x-auth-username

kibana-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
  name: kibana-ingress
  namespace: elasticsearch
spec:
  rules:
  - host: kibana.autosys.tk
    http:
      paths:
      - backend:
          serviceName: openresty-oidc-http
          servicePort: 80
        path: /
  tls:
  - hosts:
    - kibana.autosys.tk
    secretName: kibana-autosys-tk-tls


RBAC using Oidc proxy

  1. С одной стороны - можно продолжать наполнять таблицу привилегий OIDC-proxy. Недостатки - большая и сложная таблица, медленная работа.
  2. C другой стороны - в коде OIDC-proxy средствами ES API можно реализовать механизм проверки наличия пользователя в базе ES, создания ее в случае отсутствия, а также назначения ролей встроенных в ES, в соответствии с группами.

API - https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html
ROles - https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html

API RO RW FULL
/_cluster GET GET GET PUT POST DELETE
/_cat GET GET GET
/_nodes GET GET GET POST
/_remote GET GET GET
/_tasks GET GET GET
/_ccr GET GET GET
* GET GET GET PUT

Разные проблемы

Произвольная деаутентификация клиента openresty oidc proxy

Меня зачпокала ситуация, когда вроде всё работает, но в любой момент в логе nginx oidc-proxy может появиться ошибка

2020/04/07 18:46:22 [alert] 12#0: *12 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt) while SSL handshaking to upstream, client: 10.244.0.71, server: kibanadomain.local, request: "GET /built_assets/css/plugins/ml/application/index.light.css HTTP/1.1", upstream: "https://10.104.117.4:5601/built_assets/css/plugins/ml/application/index.light.css", host: "kibana.domain.local", referrer: "https://kibana.domain.local/app/kibana

и дальше cookie сессии сбрасываются, клиент деаутетифицируется и последующие запросы уже идут с кодом 302 и редиректом на redirect_uri.

Причина

Причина и решение описаны тут: https://github.com/bungle/lua-resty-session/issues/23
Судя по всему - причина кроется в механизмах шифрования-расшифровывания. А именно - в сессионном ключе. Если явно не задано значение переменной $session_secret, то каждый worker сгенерирует свой собственный секрет и они не смогут расшифровать данные зашифрованные разными ключами. Поэтому - глюк плавающий. Он повторяется рандомно.
Решение - либо добавить в секцию server значение $session_secret длинной 32 байта:

server {
    ...
    set $session_secret T62DGscdGyb4So4tLsXhNIRdlEpt4J2k;

либо использовать единственный worker.

Elasticsearch operator и OSS docker images

При попытке использовать имиджи Elasticsearch OSS 7.6.2, для разворачивания кластера средствами elasticsearch operator все поды всегда уходили в бесконечный InitError.

Диагностика

Смотрим describe любого пода и видим:

Events:
  Type     Reason     Age                From                       Message
  ----     ------     ----               ----                       -------
  Normal   Scheduled  44s                default-scheduler          Successfully assigned default/elasticsearch-es-master-2 to kub-dev-master01
  Normal   Pulled     22s (x3 over 41s)  kubelet, kub-dev-master01  Container image "registry.rdleas.ru:5000/elasticsearch/elasticsearch-oss:7.6.2" already present on machine
  Normal   Created    21s (x3 over 40s)  kubelet, kub-dev-master01  Created container elastic-internal-init-filesystem
  Normal   Started    21s (x3 over 40s)  kubelet, kub-dev-master01  Started container elastic-internal-init-filesystem
  Warning  BackOff    7s (x5 over 36s)   kubelet, kub-dev-master01  Back-off restarting failed container

Смотрим логи контейнера elastic-internal-init-filesystem и видим:

$ kubectl logs elasticsearch-es-master-0 -c elastic-internal-init-filesystem
unsupported_distribution

Легкое гугление приводит нас сюда: https://github.com/sebgl/cloud-on-k8s/commit/c1a88cee00bc583dc28217747d4a39160904f013 , где написано, что OSS имиджи не поддерживаются оператором:

The operator only works with the official ES distributions to enable the security
available with the basic (free), gold and platinum licenses in order to ensure that
all clusters launched are secured by default.

A check is done in the prepare-fs script by looking at the existence of the
Elastic License. If not present, the script exit with a custom exit code.

Then the ES reconcilation loop sends an event of type warning if it detects that
a prepare-fs init container terminated with this exit code.

Ошибки SSL

Кластер развернут с помощью elasticsearch operator 1.0.1.

$ kubectl exec -it elasticsearch-es-master-0 -- bin/elasticsearch-setup-passwords interactive
19:52:29.467 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [10.244.0.210]; the server provided a certificate with subject name [CN=elasticsearch-es-http.default.es.local,OU=elasticsearch] and fingerprint [74c38cee7c20e080613da16e86c9d5570d238717]; the certificate has subject alternative names [DNS:elasticsearch-es-http.default.es.local,DNS:elasticsearch-es-http,DNS:elasticsearch-es-http.default.svc,DNS:elasticsearch-es-http.default]; the certificate is issued by [CN=elasticsearch-http,OU=elasticsearch]; the certificate is signed by (subject [CN=elasticsearch-http,OU=elasticsearch] fingerprint [5d410bb81a7da9148cf71156ee07e36fa85ee5ef] {trusted issuer}) which is self-issued; the [CN=elasticsearch-http,OU=elasticsearch] certificate is trusted in this ssl context ([xpack.security.http.ssl])
java.security.cert.CertificateException: No subject alternative names matching IP address 10.244.0.210 found
....
....
....
SSL connection to https://10.244.0.210:9200/_security/_authenticate?pretty failed: No subject alternative names matching IP address 10.244.0.210 found
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

ERROR: Failed to establish SSL connection to elasticsearch at https://10.244.0.210:9200/_security/_authenticate?pretty. 
command terminated with exit code 78

В кластере единственный master. При попытке подключения утилита обращается по IP-адресу, а не по имени. Видно, что сертификат у сервера есть.
Как написано тут: https://www.elastic.co/guide/en/elasticsearch/reference/master/trb-security-setup.html достаточно привести конфиг к такому виду:

  xpack.security.enabled: true
  xpack.security.http.ssl.verification_mode: certificate

Ошибка при начальной смене паролей bin/elasticsearch-setup-passwords

При выполнении команды bin/elasticsearch-setup-passwords появляется ошибка :

$ kubectl exec -it elasticsearch-es-master-0 -- bin/elasticsearch-setup-passwords auto
Failed to authenticate user 'elastic' against https://10.244.0.219:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /usr/share/elasticsearch/config/elasticsearch.keystore

Это значит, что пароли встроенных учеток уже заданы elasticsearch operator в процессе установки elasticsearch, они хранятся в секретах elasticsearch-es-internal-users и elasticsearch-es-elastic-user:

kubectl get secrets elasticsearch-es-internal-users -o=jsonpath='{.data.elastic-internal}' | base64 --decode

и

kubectl get secret elasticsearch-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode

Если на перед создание экземпляра кластера elasticsearch создать секреты elasticsearch-es-internal-users и elasticsearch-es-elastic-user, то встроенным учетным записям будут назначены указанные там пароли. Это написано тут (там же написано как заскриптовать смену пароля): https://github.com/elastic/cloud-on-k8s/issues/967
А вообще - в ближайшее время elasticsearch operator позволит задавать пароли для встроенных учеток прямо в конфиге: https://github.com/elastic/cloud-on-k8s/pull/2682