helm repo add elastic https://helm.elastic.co
kubectl create ns elasticsearch
helm install --namespace elasticsearch --name elasticsearch elastic/elasticsearch --set replicas=1
sudo mkdir /kubernetes_volumes/elasticsearch-data sudo chmod a+rw -R /kubernetes_volumes/elasticsearch-data/
apiVersion: v1
kind: PersistentVolume
metadata:
name: elasticsearch-data
namespace: elasticsearch
labels:
app: elasticsearch-master
spec:
capacity:
storage: 32Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/kubernetes_volumes/elasticsearch-data"
type: Directory
persistentVolumeReclaimPolicy: Retain
И дадим права на запись в индексы:
kubectl exec -it -n elasticsearch elasticsearch-master-0 -- curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
Так как нода у меня одна, то мне нужно запретить создавать реплики. Иниче - в дальнейшем кластер просто перестанет работать:
kubectl exec -it -n elasticsearch elasticsearch-master-0 -- curl -H "Content-Type: application/json" -XPUT 'http://127.0.0.1:9200/_template/default' -d '{"index_patterns": ["*"],"order": -1,"settings": {"number_of_shards": "1","number_of_replicas": "0"}}'
И почистим уже созданные индексы:
kubectl exec -it -n elasticsearch elasticsearch-master-0 -- curl -XDELETE 'http://127.0.0.1:9200/*'
Теперь в логах пода elasticsearch-master-0 появится строка:
Cluster health status changed from [YELLOW] to [GREEN]
И в этот elasticsearch можно будет писать логи.
kubectl create ns kibana helm install --namespace elasticsearch --name kibana elastic/kibana --set elasticsearchHosts=http://elasticsearch-master.elasticsearch.svc.cluster.local:9200
helm upgrade kibana elastic/kibana --set elasticsearchHosts=http://openresty-oidc-http.elasticsearch.svc.cluster.local:9200 --set elasticsearch.requestHeadersWhitelist=[X-Auth-Username]
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt
name: kibana-ingress
namespace: elasticsearch
spec:
rules:
- host: kibana.autosys.tk
http:
paths:
- backend:
serviceName: kibana-kibana
servicePort: 5601
path: /
tls:
- hosts:
- kibana.autosys.tk
secretName: kibana-autosys-tk-tls
В кибане есть сохраненные пользователем объекты. Например - Index Patterns.
Создать можно так:
kubectl exec -it -n elk elk-openresty-oidc-559f46d69d-shbwg -- curl -H 'Authorization: Basic dXN....o' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{"attributes": {"title": "my-pattern-*"}}' http://elk-kb-http:5601/api/saved_objects/index-pattern/my-pattern
Посмотреть их можно так:
kubectl exec -it -n elk elk-openresty-oidc-559f46d69d-shbwg -- curl -H 'Authorization: Basic dXN....o' http://elk-kb-http:5601/api/saved_objects/_find?type=index-pattern | grep '"type":"index-pattern"'
Удалить какой-то можно так:
kubectl exec -it -n elk elk-openresty-oidc-559f46d69d-shbwg -- curl -X DELETE -H 'kbn-xsrf: true' -H 'Authorization: Basic dXN....o' http://elk-kb-http:5601/api/saved_objects/index-pattern/vrm
https://habr.com/ru/post/421819/
https://helm.elastic.co/
helm install --namespace elasticsearch --name logstash elastic/logstash -f ./values.yaml
У меня rsyslog сыплет логи в таком виде:
Dec 25 14:06:25 kub kubelet[989]: W1225 14:06:25.516800 989 volume_linux.go:45] Setting volume ownership for /var/lib/kubelet/pods/d36eb244-fe63-4f5d-b133-c4e4232c68b3/volumes/kubernetes.io~configmap/sc-dashboard-provider and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699
Для того, чтобы logstash смог сформировать объект json-объект message нужно в конфигурацию прописать фильтр, который распарсит строку message , чтобы на выходе получился объект.
Встроенные паттерны тут: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
Проверить паттерны можно тут: http://grokdebug.herokuapp.com/
Для вышеприведенной стоки подходит такой фильтр:
%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{SYSLOGHOST:host}%{SPACE}%{SYSLOGPROG:process}:%{SPACE}%{GREEDYDATA:SYSLOGMESSAGE}
Для того, чтобы логи попадали в elasticsearch в конфиг rsyslog (/etc/rsyslog.conf) в конце надо дописать такое:
*.* @@logstash.autosys.tk:1514
Тут *.* - это значит все записи, @@ - это TCP (@ - UDP), дальше хост logstash и порт.
---
replicas: 1
# Allows you to add any config files in /usr/share/logstash/config/
# such as logstash.yml and log4j2.properties
logstashConfig:
logstash.yml: |
http.host: "0.0.0.0"
config.reload.automatic: "true"
xpack.management.enabled: "false"
# key:
# nestedkey: value
# log4j2.properties: |
# key = value
# Allows you to add any pipeline files in /usr/share/logstash/pipeline/
logstashPipeline:
input_main.conf: |
input {
udp {
port => 1514
type => syslog
}
tcp {
port => 1514
type => syslog
}
http {
port => 8080
}
# kafka {
# ## ref: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
# bootstrap_servers => "kafka-input:9092"
# codec => json { charset => "UTF-8" }
# consumer_threads => 1
# topics => ["source"]
# type => "example"
# }
}
filter_main.conf: |
filter {
if [type] == "syslog" {
# Uses built-in Grok patterns to parse this standard format
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:@timestamp}%{SPACE}%{SYSLOGHOST:host}%{SPACE}%{SYSLOGPROG:process}:%{SPACE}%{GREEDYDATA:syslogmessage}"
}
}
# Sets the timestamp of the event to the timestamp of recorded in the log-data
# By default, logstash sets the timestamp to the time it was ingested.
#date {
# match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
#}
mutate {
rename => ["syslogmessage", "message" ]
}
}
}
output_main.conf: |
output {
# stdout { codec => rubydebug }
elasticsearch {
hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
manage_template => false
index => "logs-%{+YYYY.MM.dd}"
}
# kafka {
# ## ref: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html
# bootstrap_servers => "kafka-output:9092"
# codec => json { charset => "UTF-8" }
# compression_type => "lz4"
# topic_id => "destination"
# }
}
# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs:
- name: "ELASTICSEARCH_HOST"
value: "elasticsearch-master.elasticsearch.svc.cluster.local"
- name: "ELASTICSEARCH_PORT"
value: "9200"
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# A list of secrets and their paths to mount inside the pod
secretMounts: []
image: "docker.elastic.co/logstash/logstash"
imageTag: "7.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: []
podAnnotations: {}
# additionals labels
labels:
app: logstash
logstashJavaOpts: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "100m"
memory: "1536Mi"
limits:
cpu: "1000m"
memory: "1536Mi"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
rbac:
create: false
serviceAccountName: ""
podSecurityPolicy:
create: false
name: ""
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
persistence:
enabled: false
annotations: {}
extraVolumes: ""
# - name: extras
# emptyDir: {}
extraVolumeMounts: ""
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraContainers: ""
# - name: do-something
# image: busybox
# command: ['do', 'something']
extraInitContainers: ""
# - name: do-something
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"
# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
httpPort: 9600
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
# How long to wait for logstash to stop gracefully
terminationGracePeriod: 120
livenessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 300
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
readinessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 3
## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""
nodeSelector: {}
tolerations: []
nameOverride: ""
fullnameOverride: ""
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
service:
annotations: {}
type: LoadBalancer
ports:
- name: beats
port: 5044
protocol: TCP
targetPort: 5044
- name: http
port: 8080
protocol: TCP
targetPort: 8080
- name: syslog
port: 1514
targetPort: 1514
sudo mkdir /kubernetes_volumes/logstash-data sudo chmod a+rw -R /kubernetes_volumes/logstash-data
apiVersion: v1
kind: PersistentVolume
metadata:
name: logstash-data
namespace: elasticsearch
labels:
app: logstash
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/kubernetes_volumes/logstash-data"
type: Directory
persistentVolumeReclaimPolicy: Retain
Теперь можно проверить, что всё вместе работает.
Можно отправить сообщение в logstash на input http:
curl -XPUT 'http://~~~logstash~IP~~~:8080/test/test1/1' -d 'hello'
В ответ должно быть ok.
А в kibana можно нажать Discovery, увидеть там новый индекс.
Для работы с файлами логов потребуется дополнительный сервис Filebeat, который будет читать логи из файла и отправлять в Logstash. Пример конфигурации:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
type: nginx
fields_under_root: true
scan_frequency: 5s
output.logstash:
hosts: ["logstash:5000"]
У Filebeat есть отличный функцмонал - Autodiscover. Он позволяет по заданным правилам автоматически обнаруживать файлы с логами. Прмиер конфига для K8S:
filebeat:
autodiscover:
providers:
- node: ${HOSTNAME}
templates:
- config:
- containers:
ids:
- ${data.kubernetes.container.id}
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
type: container
type: kubernetes
output:
elasticsearch:
hosts:
- https://elk-es-http.elk.svc:9200
password: 7hhjEqA6oH3049D1oTT9s4O2
ssl:
certificate_authorities:
- /mnt/elastic-internal/elasticsearch-certs/ca.crt
username: elk-elk-beat-user
processors:
- drop_fields:
fields:
- log
- container.id
- container.runtime
- container.image.name
- input.type
- tags
- kubernetes.labels
- kubernetes.pod.uid
- kubernetes.replicaset.name
- kubernetes.node
- kubernetes.namespace_labels
- kubernetes.namespace_uid
- ecs.version
- agent
ignore_missing: false
setup:
dashboards:
enabled: true
kibana:
host: http://elk-kb-http.elk.svc:5601
password: 6l2IPFK8mBR88w6ek49ePI71
username: elk-elk-beat-kb-user
Этот конфиг сформирован оператором ECK 1.8.
Что делать, если Filebeat Kubernetes Autodiscover не работает без видимых причин?? В моем случае - ошибок не было, но и логи контейнеров Filebeat Autodiscover не обнаруживал. Лог выглядел так:
... 2021-10-24T11:11:24.565Z INFO [autodiscover.pod] kubernetes/util.go:122 kubernetes: Using node MCS-K8S-201 provided in the config 2021-10-24T11:11:24.565Z DEBUG [autodiscover.pod] kubernetes/pod.go:80 Initializing a new Kubernetes watcher using node: MCS-K8S-201 2021-10-24T11:11:24.587Z DEBUG [autodiscover] autodiscover/autodiscover.go:90 Configured autodiscover provider: kubernetes 2021-10-24T11:11:24.587Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager 2021-10-24T11:11:24.687Z DEBUG [kubernetes] kubernetes/watcher.go:184 cache sync done 2021-10-24T11:11:24.788Z DEBUG [kubernetes] kubernetes/watcher.go:184 cache sync done 2021-10-24T11:11:24.889Z DEBUG [kubernetes] kubernetes/watcher.go:184 cache sync done ...
Однако, при нормальной работе - должно быть что-то такое:
...
2021-10-24T11:49:30.633Z INFO [autodiscover.pod] kubernetes/util.go:122 kubernetes: Using node mcs-k8s-203 provided in the config
2021-10-24T11:49:30.642Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
2021-10-24T11:49:31.048Z INFO [input] log/input.go:164 Configured paths: [/var/log/containers/*-70600db8d4371ebdacc300edd825c030e6698a6b75467ea8b280c7aef1faa366.log] {"input_id": "f60ce837-5828-4125-afe5-5a40d70fc613"}
2021-10-24T11:49:31.048Z INFO [input] log/input.go:164 Configured paths: [/var/log/containers/*-70600db8d4371ebdacc300edd825c030e6698a6b75467ea8b280c7aef1faa366.log] {"input_id": "ba234dcb-e0bd-4395-baa4-ba47dd1f1f6d"}
...
Я потратил довольно много времени на выяснение причин такого поведения. дело оказалось в том, что HOSTNAME на ноде задан большими буквами (MCS-K8S-203), а в кластере нода имеет имя маленькими буквами (mcs-k8s-203). В итоге - всё вылечилось, когда я на хостах кластера выполнил:
sudo hostnamectl set-hostname `echo "$HOSTNAME" | tr '[:upper:]' '[:lower:]'`
то есть переменовал хосты кластера в нижнем регистре.
По-умолчанию стек ELK не обеспечивает безопасного доступа к данным и компоненты никак не аутентифицируются. То есть записать и читать данные может кто угодно.
Базовая бесплатная лицензия не позволяет аутентифицировать пользователей из каталогов LDAP, поэтому нужно либо платить, либо настраивать аутентификацют сторонними методами.
Общая идея такая:
Используемые компоненты:
Вот ссылочки, которые помогали мне в настройке oidc-proxy для LDAP-аутентификации :
Я изобретаю свои велосипеды, а вот есть некторые готовые компоненты:
Развернут с помощью helm - deploy_keycloak_using_helm.
REALM настроен так: https://habr.com/ru/post/441112/
Dockerfile на базе https://github.com/Revomatico/docker-openresty-oidc/blob/master/Dockerfile :
FROM alpine:3.10
MAINTAINER Mikhail Usik <mike@autosys.tk>
ENV LUA_SUFFIX=jit-2.1.0-beta3 \
LUAJIT_VERSION=2.1 \
NGINX_PREFIX=/opt/openresty/nginx \
OPENRESTY_PREFIX=/opt/openresty \
OPENRESTY_SRC_SHA256=bf92af41d3ad22880047a8b283fc213d59c7c1b83f8dae82e50d14b64d73ac38 \
OPENRESTY_VERSION=1.15.8.2 \
LUAROCKS_VERSION=3.1.3 \
LUAROCKS_SRC_SHA256=c573435f495aac159e34eaa0a3847172a2298eb6295fcdc35d565f9f9b990513 \
LUA_RESTY_OPENIDC_VERSION=1.7.2-1 \
VAR_PREFIX=/var/nginx
RUN set -ex \
&& apk --no-cache add \
libgcc \
libpcrecpp \
libpcre16 \
libpcre32 \
libssl1.1 \
libstdc++ \
openssl \
pcre \
curl \
unzip \
git \
dnsmasq \
ca-certificates \
&& apk --no-cache add --virtual .build-dependencies \
make \
musl-dev \
gcc \
ncurses-dev \
openssl-dev \
pcre-dev \
perl \
readline-dev \
zlib-dev \
libc-dev \
\
## OpenResty
&& curl -fsSL https://github.com/openresty/openresty/releases/download/v${OPENRESTY_VERSION}/openresty-${OPENRESTY_VERSION}.tar.gz -o /tmp/openresty.tar.gz \
\
&& cd /tmp \
&& echo "${OPENRESTY_SRC_SHA256} *openresty.tar.gz" | sha256sum -c - \
&& tar -xzf openresty.tar.gz \
\
&& cd openresty-* \
&& readonly NPROC=$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) \
&& ./configure \
--prefix=${OPENRESTY_PREFIX} \
--http-client-body-temp-path=${VAR_PREFIX}/client_body_temp \
--http-proxy-temp-path=${VAR_PREFIX}/proxy_temp \
--http-log-path=${VAR_PREFIX}/access.log \
--error-log-path=${VAR_PREFIX}/error.log \
--pid-path=${VAR_PREFIX}/nginx.pid \
--lock-path=${VAR_PREFIX}/nginx.lock \
--with-luajit \
--with-pcre-jit \
--with-ipv6 \
--with-http_ssl_module \
--without-http_ssi_module \
--with-http_realip_module \
--without-http_scgi_module \
--without-http_uwsgi_module \
--without-http_userid_module \
-j${NPROC} \
&& make -j${NPROC} \
&& make install \
\
&& rm -rf /tmp/openresty* \
\
## LuaRocks
&& curl -fsSL http://luarocks.github.io/luarocks/releases/luarocks-${LUAROCKS_VERSION}.tar.gz -o /tmp/luarocks.tar.gz \
\
&& cd /tmp \
&& echo "${LUAROCKS_SRC_SHA256} *luarocks.tar.gz" | sha256sum -c - \
&& tar -xzf luarocks.tar.gz \
\
&& cd luarocks-* \
&& ./configure \
--prefix=${OPENRESTY_PREFIX}/luajit \
--lua-suffix=${LUA_SUFFIX} \
--with-lua=${OPENRESTY_PREFIX}/luajit \
--with-lua-lib=${OPENRESTY_PREFIX}/luajit/lib \
--with-lua-include=${OPENRESTY_PREFIX}/luajit/include/luajit-${LUAJIT_VERSION} \
&& make build \
&& make install \
\
&& rm -rf /tmp/luarocks* \
&& rm -rf ~/.cache/luarocks \
## Post install
&& ln -sf ${NGINX_PREFIX}/sbin/nginx /usr/local/bin/nginx \
&& ln -sf ${NGINX_PREFIX}/sbin/nginx /usr/local/bin/openresty \
&& ln -sf ${OPENRESTY_PREFIX}/bin/resty /usr/local/bin/resty \
&& ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luajit-* ${OPENRESTY_PREFIX}/luajit/bin/lua \
&& ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luajit-* /usr/local/bin/lua \
&& ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luarocks /usr/local/bin/luarocks \
&& ln -sf ${OPENRESTY_PREFIX}/luajit/bin/luarocks-admin /usr/local/bin/luarocks-admin \
&& echo user=root >> /etc/dnsmasq.conf \
## Install lua-resty-openidc
&& cd ~/ \
# Fix for https://github.com/zmartzone/lua-resty-openidc/issues/213#issuecomment-432471572
# && luarocks install lua-resty-hmac \
&& luarocks install lua-resty-openidc ${LUA_RESTY_OPENIDC_VERSION} \
## Install lua-resty-xacml-pep
# && curl -fsSL https://raw.githubusercontent.com/zmartzone/lua-resty-xacml-pep/master/lib/resty/xacml_pep.lua -o /opt/openresty/lualib/resty/xacml_pep.lua \
## Cleanup
&& apk del .build-dependencies 2>/dev/null
WORKDIR $NGINX_PREFIX
CMD dnsmasq; openresty -g "daemon off; error_log /dev/stderr info;"
Создано по мотивам https://developers.redhat.com/blog/2018/10/08/configuring-nginx-keycloak-oauth-oidc/ и https://daenney.github.io/2019/10/05/beyondcorp-at-home-authn-authz-openresty
Описаны два виртуальных сервра (оба - прокси).
kind: ConfigMap
apiVersion: v1
metadata:
name: openresty-oidc-config
namespace: elasticsearch
data:
nginx.conf: |
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
##
# LUA options
##
lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
lua_package_path '~/lua/?.lua;;';
resolver 192.168.77.1;
# cache for discovery metadata documents
lua_shared_dict discovery 1m;
# cache for JWKs
lua_shared_dict jwks 1m;
# allow the server to close connection on non responding client, this will free up memory
reset_timedout_connection on;
server {
listen 80 default_server;
server_name kibana.autosys.tk;
#access_log /dev/stdout;
#error_log /dev/stdout;
access_by_lua '
local opts = {
redirect_uri = "/redirect_uri",
accept_none_alg = true,
discovery = "https://sso.autosys.tk/auth/realms/Autosys/.well-known/openid-configuration",
client_id = "kibana",
client_secret = "af903747-905c-4342-a62c-83033d3289cc",
redirect_uri_scheme = "https",
logout_path = "/logout",
redirect_after_logout_uri = "https://sso.autosys.tk/auth/realms/Autosys/protocol/openid-connect/logout?redirect_uri=https://kibana.autosys.tk/",
redirect_after_logout_with_id_token_hint = false,
session_contents = {id_token=true}
}
-- call introspect for OAuth 2.0 Bearer Access Token validation
local res, err = require("resty.openidc").authenticate(opts)
if err then
ngx.status = 403
ngx.say(err)
ngx.exit(ngx.HTTP_FORBIDDEN)
end
-- set data from the ID token as HTTP Request headers
ngx.req.set_header("X-Auth-Username", res.id_token.preferred_username)
ngx.req.set_header("X-Auth-Groups", res.id_token.groups)
-- ngx.req.set_header("X-Auth-Audience", res.id_token.aud)
-- ngx.req.set_header("X-Auth-Email", res.id_token.email)
-- ngx.req.set_header("X-Auth-ExpiresIn", res.id_token.exp)
-- ngx.req.set_header("X-Auth-Roles", res.id_token.roles)
-- ngx.req.set_header("X-Auth-Name", res.id_token.name)
-- ngx.req.set_header("X-Auth-Subject", res.id_token.sub)
-- ngx.req.set_header("X-Auth-Userid", res.id_token.preferred_username)
-- ngx.req.set_header("X-Auth-Locale", res.id_token.locale)
-- Output headers to nginx err log
--ngx.log(ngx.ERR, "Got header X-Auth-Userid: "..res.id_token.preferred_username..";")
';
expires 0;
add_header Cache-Control private;
location / {
proxy_connect_timeout 5s;
proxy_pass http://kibana-kibana.elasticsearch.svc.cluster.local:5601;
}
}
server {
listen 9200;
access_log /dev/stdout;
error_log /dev/stdout;
access_by_lua '
local restrictions = {
elasticsearch_ro = {
["^/$"] = { "GET" },
["^/?[^/]*/?[^/]*/_mget"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/_doc"] = { "GET" },
["^/?[^/]*/?[^/]*/_search"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/_msearch"] = { "GET" },
["^/?[^/]*/?[^/]*/_validate/query"] = { "GET" },
["/_aliases"] = { "GET" },
["/_cluster.*"] = { "GET" }
},
elasticsearch_rw = {
["^/$"] = { "GET" },
["^/?[^/]*/?[^/]*/_search"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/_msearch"] = { "GET", "POST" },
["^/?[^/]*/traffic*"] = { "GET", "POST", "PUT", "DELETE" },
["^/?[^/]*/?[^/]*/_validate/query"] = { "GET", "POST" },
["/_aliases"] = { "GET" },
["/_cluster.*"] = { "GET" }
},
elasticsearch_full = {
["^/?[^/]*/?[^/]*/_bulk"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/_refresh"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/?[^/]*/_create"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/?[^/]*/_update"] = { "GET", "POST" },
["^/?[^/]*/?[^/]*/?.*"] = { "GET", "POST", "PUT", "DELETE" },
["^/?[^/]*/?[^/]*$"] = { "GET", "POST", "PUT", "DELETE" },
["/_aliases"] = { "GET", "POST" }
}
}
local groups = ngx.req.get_headers()["x-auth-groups"]
local authenticated_group = nil
local ngx_re = require "ngx.re"
if ( type(groups) == "string" and string.len(groups) >= 1 ) then
groups = string.lower(groups)
ngx.log(ngx.ERR, "Got header X-Auth-Groups: "..groups..";")
local groups_table = ngx_re.split(groups, ", ")
local groups_number = table.getn(groups_table)
ngx.log(ngx.ERR, "Groups number : "..groups_number..";")
for i=1,groups_number do
local group = groups_table[i]:gsub("/","")
group = group:gsub("%s","_")
ngx.log(ngx.ERR, "Check group: "..group..";")
if (restrictions[group] ~= nil) then
ngx.log(ngx.ERR, "User belongs to Authenticated Group: "..group..";")
authenticated_group = group
break
end
end
-- exit 403 when no matching role has been found
if authenticated_group == nil then
ngx.header.content_type = "text/plain"
ngx.log(ngx.ERR, "Unauthenticated request... User - "..ngx.req.get_headers()["x-auth-username"]..";")
ngx.status = 403
ngx.say("403 Forbidden: You don\'t have access to this resource.")
return ngx.exit(403)
end
-- get URL
local uri = ngx.var.uri
ngx.log(ngx.DEBUG, uri)
-- get method
local method = ngx.req.get_method()
ngx.log(ngx.DEBUG, method)
local allowed = false
for path, methods in pairs(restrictions[authenticated_group]) do
-- path matched rules?
local p = string.match(uri, path)
local m = nil
-- method matched rules?
for _, _method in pairs(methods) do
m = m and m or string.match(method, _method)
end
if p and m then
allowed = true
ngx.log(ngx.NOTICE, method.." "..uri.." matched: "..tostring(m).." "..tostring(path).." for "..authenticated_group)
break
end
end
if not allowed then
ngx.header.content_type = "text/plain"
ngx.log(ngx.WARN, "Group ["..authenticated_group.."] not allowed to access the resource ["..method.." "..uri.."]")
ngx.status = 403
ngx.say("403 Forbidden: You don\'t have access to this resource.")
return ngx.exit(403)
end
end
';
location / {
proxy_connect_timeout 15s;
proxy_pass http://elasticsearch-master.elasticsearch.svc.cluster.local:9200;
}
}
}
В конфигурации первого прокси аутентификацию выполняет блок access_by_lua. В local opts прописано следующее:
В конфигурации второго прокси авторизацию также выполняет блок access_by_lua:
apiVersion: apps/v1
kind: Deployment
metadata:
name: openresty-oidc
namespace: elasticsearch
spec:
replicas: 1
selector:
matchLabels:
app: openresty-oidc
template:
metadata:
labels:
app: openresty-oidc
spec:
imagePullSecrets:
- name: autosys-regcred
containers:
- name: openresty-oidc
image: registry.autosys.tk/openresty-oidc
volumeMounts:
- name: openresty-oidc-config-volume
mountPath: /opt/openresty/nginx/conf/nginx.conf
subPath: nginx.conf
volumes:
- name: openresty-oidc-config-volume
configMap:
name: openresty-oidc-config
apiVersion: v1
kind: Service
metadata:
name: openresty-oidc-http
namespace: elasticsearch
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: elasticsearch
port: 9200
protocol: TCP
targetPort: 9200
selector:
app: openresty-oidc
sessionAffinity: None
type: ClusterIP
В конфигурации kibana нужно внести следующие изменения с помощью helm upgrade:
elasticsearchHosts: "http://openresty-oidc-http.elasticsearch.svc.cluster.local:9200"
kibanaConfig:
kibana.yml: |
server.name: kibana
server.host: "0"
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestHeadersWhitelist:
- authorization
- x-auth-groups
- x-auth-username
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt
kubernetes.io/ingress.class: nginx
name: kibana-ingress
namespace: elasticsearch
spec:
rules:
- host: kibana.autosys.tk
http:
paths:
- backend:
serviceName: openresty-oidc-http
servicePort: 80
path: /
tls:
- hosts:
- kibana.autosys.tk
secretName: kibana-autosys-tk-tls
API - https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html
ROles - https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-roles.html
| API | RO | RW | FULL | |
| /_cluster | GET | GET | GET PUT POST DELETE | |
| /_cat | GET | GET | GET | |
| /_nodes | GET | GET | GET POST | |
| /_remote | GET | GET | GET | |
| /_tasks | GET | GET | GET | |
| /_ccr | GET | GET | GET | |
| * | GET | GET | GET | PUT |
Меня зачпокала ситуация, когда вроде всё работает, но в любой момент в логе nginx oidc-proxy может появиться ошибка
2020/04/07 18:46:22 [alert] 12#0: *12 ignoring stale global SSL error (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt) while SSL handshaking to upstream, client: 10.244.0.71, server: kibanadomain.local, request: "GET /built_assets/css/plugins/ml/application/index.light.css HTTP/1.1", upstream: "https://10.104.117.4:5601/built_assets/css/plugins/ml/application/index.light.css", host: "kibana.domain.local", referrer: "https://kibana.domain.local/app/kibana
и дальше cookie сессии сбрасываются, клиент деаутетифицируется и последующие запросы уже идут с кодом 302 и редиректом на redirect_uri.
Причина и решение описаны тут: https://github.com/bungle/lua-resty-session/issues/23
Судя по всему - причина кроется в механизмах шифрования-расшифровывания. А именно - в сессионном ключе. Если явно не задано значение переменной $session_secret, то каждый worker сгенерирует свой собственный секрет и они не смогут расшифровать данные зашифрованные разными ключами. Поэтому - глюк плавающий. Он повторяется рандомно.
Решение - либо добавить в секцию server значение $session_secret длинной 32 байта:
server {
...
set $session_secret T62DGscdGyb4So4tLsXhNIRdlEpt4J2k;
либо использовать единственный worker.
При попытке использовать имиджи Elasticsearch OSS 7.6.2, для разворачивания кластера средствами elasticsearch operator все поды всегда уходили в бесконечный InitError.
Смотрим describe любого пода и видим:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 44s default-scheduler Successfully assigned default/elasticsearch-es-master-2 to kub-dev-master01 Normal Pulled 22s (x3 over 41s) kubelet, kub-dev-master01 Container image "registry.rdleas.ru:5000/elasticsearch/elasticsearch-oss:7.6.2" already present on machine Normal Created 21s (x3 over 40s) kubelet, kub-dev-master01 Created container elastic-internal-init-filesystem Normal Started 21s (x3 over 40s) kubelet, kub-dev-master01 Started container elastic-internal-init-filesystem Warning BackOff 7s (x5 over 36s) kubelet, kub-dev-master01 Back-off restarting failed container
Смотрим логи контейнера elastic-internal-init-filesystem и видим:
$ kubectl logs elasticsearch-es-master-0 -c elastic-internal-init-filesystem unsupported_distribution
Легкое гугление приводит нас сюда: https://github.com/sebgl/cloud-on-k8s/commit/c1a88cee00bc583dc28217747d4a39160904f013 , где написано, что OSS имиджи не поддерживаются оператором:
The operator only works with the official ES distributions to enable the security available with the basic (free), gold and platinum licenses in order to ensure that all clusters launched are secured by default. A check is done in the prepare-fs script by looking at the existence of the Elastic License. If not present, the script exit with a custom exit code. Then the ES reconcilation loop sends an event of type warning if it detects that a prepare-fs init container terminated with this exit code.
Кластер развернут с помощью elasticsearch operator 1.0.1.
$ kubectl exec -it elasticsearch-es-master-0 -- bin/elasticsearch-setup-passwords interactive
19:52:29.467 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [10.244.0.210]; the server provided a certificate with subject name [CN=elasticsearch-es-http.default.es.local,OU=elasticsearch] and fingerprint [74c38cee7c20e080613da16e86c9d5570d238717]; the certificate has subject alternative names [DNS:elasticsearch-es-http.default.es.local,DNS:elasticsearch-es-http,DNS:elasticsearch-es-http.default.svc,DNS:elasticsearch-es-http.default]; the certificate is issued by [CN=elasticsearch-http,OU=elasticsearch]; the certificate is signed by (subject [CN=elasticsearch-http,OU=elasticsearch] fingerprint [5d410bb81a7da9148cf71156ee07e36fa85ee5ef] {trusted issuer}) which is self-issued; the [CN=elasticsearch-http,OU=elasticsearch] certificate is trusted in this ssl context ([xpack.security.http.ssl])
java.security.cert.CertificateException: No subject alternative names matching IP address 10.244.0.210 found
....
....
....
SSL connection to https://10.244.0.210:9200/_security/_authenticate?pretty failed: No subject alternative names matching IP address 10.244.0.210 found
Please check the elasticsearch SSL settings under xpack.security.http.ssl.
ERROR: Failed to establish SSL connection to elasticsearch at https://10.244.0.210:9200/_security/_authenticate?pretty.
command terminated with exit code 78
В кластере единственный master. При попытке подключения утилита обращается по IP-адресу, а не по имени. Видно, что сертификат у сервера есть.
Как написано тут: https://www.elastic.co/guide/en/elasticsearch/reference/master/trb-security-setup.html достаточно привести конфиг к такому виду:
xpack.security.enabled: true xpack.security.http.ssl.verification_mode: certificate
При выполнении команды bin/elasticsearch-setup-passwords появляется ошибка :
$ kubectl exec -it elasticsearch-es-master-0 -- bin/elasticsearch-setup-passwords auto Failed to authenticate user 'elastic' against https://10.244.0.219:9200/_security/_authenticate?pretty Possible causes include: * The password for the 'elastic' user has already been changed on this cluster * Your elasticsearch node is running against a different keystore This tool used the keystore at /usr/share/elasticsearch/config/elasticsearch.keystore
Это значит, что пароли встроенных учеток уже заданы elasticsearch operator в процессе установки elasticsearch, они хранятся в секретах elasticsearch-es-internal-users и elasticsearch-es-elastic-user:
kubectl get secrets elasticsearch-es-internal-users -o=jsonpath='{.data.elastic-internal}' | base64 --decode
и
kubectl get secret elasticsearch-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode
Если на перед создание экземпляра кластера elasticsearch создать секреты elasticsearch-es-internal-users и elasticsearch-es-elastic-user, то встроенным учетным записям будут назначены указанные там пароли. Это написано тут (там же написано как заскриптовать смену пароля): https://github.com/elastic/cloud-on-k8s/issues/967
А вообще - в ближайшее время elasticsearch operator позволит задавать пароли для встроенных учеток прямо в конфиге: https://github.com/elastic/cloud-on-k8s/pull/2682