Elastic Stack on Docker, with preconfigured security, tools, self-monitoring, and Prometheus Metrics Exporters
Comes with tools like Curator, ElastAlert for Alerting.
Elastic Stack (AKA ELK) Docker Composition, preconfigured with Security, Monitoring, Tools such as ElastAlert for alerting and Curator.
- Configured as Production Single Node Cluster (With a multi-node option for experimenting).
- Use Docker-Compose and
.envto configure your stack.
- Security Enabled (under basic license).
- SSL Enabled for Transport Layer.
- Automated Script that initializes and persist Elasticsearch's Keystore and SSL Certifications.
- Curator Preconfigured for Automated Snapshotting (Need to setup S3 Repository).
- Self-Monitoring Metrics Enabled.
- Filebeat instance for shipping Stack logs to Elasticsearch itself.
- Prometheus Exporters for Stack Metrics.
- ElastAlert preconfigured for Alerting.
- Embedded Container Healthchecks for Stack Images.
- Go to repository directory
.envfile for your requirments, most importantly
ELASTIC_PASSWORDthat setup your superuser
elastic's password. and
ELK_VERSIONfor, yk, ELK Version.
- Initalize Elasticsearch Keystore and SSL Certificates
$ make setup
- Start Elastic Stack
$ make elk ---- OR ---- $ docker-compose up -d
- Visit Kibana at localhost:5601
ELASTIC_PASSWORD value in
To Start Monitoring and Promethus Exporters
$ make monitoring
To Start Tools (ElastAlert and Curator
$ make tools
To Start ELK, Tools and Monitoring
$ make all
To Start 2 Extra Elasticsearch nodes (for development only)
$ make nodes
Adding Two Extra Nodes to the cluster will make the cluster depending on them and won't start without them again.
Makefile is a wrapper around
make helpto know every command.
Elasticsearch will save its data to a volume named
Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the
./secretsdirectory by the setup command.
Linux Users must set the following configuration as
sysctl -w vm.max_map_count=262144
By default, Virtual Memory is not enough.
- Some Configuration are parameterized in the
elastic's password (default:
ELK_VERSIONElastic Stack Version (default:
ELASTICSEARCH_HEAP, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
LOGSTASH_HEAP, how much Logstash allocate from memory.
- Other configurations which their such as cluster name, and node name, etc.
- Elasticsearch Configuration in
- Logstash Configuration in
- Logstash Pipeline in
- Kibana Configuration in
- ElastAlert Configuration in
- ElastAlert Alert rules in
./tools/elastalert/rules, head to ElastAlert docs to lookup how to create alerts.
- Curator Actions at
Setting Up Keystore
You can extend the Keystore generation script by adding keys to
./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)
To Re-generate Keystore:
If you started Prometheus Exporters using
make monitoring command. Prometheus Exporters will expose metrics at the following ports.
||- To Monitor Each Container stats and metrics.|
Self-Monitoring is Enabled
Head to Stack Monitoring tab in Kibana to see cluster metrics for all stack components.
In Production, cluster metrics should be shipped to another dedicated monitoring cluster.
MIT License Copyright (c) 2019 Sherif Abdel-Naby
PR(s) are Open and Welcomed.