ELK Stack One-Click App Setup

Introduction

One of the many available One-Click Apps that we have is ELK. ELK is an acronym for a collection of three open-source products: Elasticsearch, Logstash, and Kibana. Elasticsearch is a NoSQL database that is based on the Lucene search engine. Logstash is a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. Kibana is a visualization layer that works on top of Elasticsearch.

Prerequisites

• Orbit created with ELK image

• Running Orbit

Creating Your Orbit

1. To create your application, you must create a Photon. You can do so by selecting the Photon tab after selecting ‘Create Orbit’

   

2. Once there, you will see a list of available photons that you can create. Select one and it will bring you to a detailed page

   

3. Scroll down to the details and you can start selecting options for your photon as well as setting up credentials.

   

4. Once everything is filled in, just press Create and you can move on to the next step

SSH into Orbit

1. First we need to check some details on our orbit by first navigating to the “My Orbits” tab.

   

2. Once there, click on the Orbit you want to connect to and check for three things:

   • IP Address

   • Running Status

   • Source Image(Ubuntu, Debian, Centos, Fedora

   

3. Once you have confirmed that your Orbit is running and you have the other information. Open up your console.

4. Now you can SSH into your Orbit by running.

   ssh image@address

   • IP Address: We got this from the detail page, in this case it is “216.200.116.60”

   • Image: We got this from the source image in the details page. It should be one of the following: ubuntu, debian, fedora, centos

   • Note: A full command for this example would be: “ssh ubuntu@216.200.116.60”

Starting ELK

1. Once you are SSH’d into the server you will see the following.

   

2. We first need to set a setting for Elasticsearch to function

   sudo sysctl -w vm.max_map_count=262144

   Finally we can deploy ELK to the docker

   docker stack deploy -c stack.yml elk

   

3. Before continuing, we need to wait a few minutes for the docker to set everything up. If you want to see if everything is ok to continue, you can run the following and it should look like the image below

   docker ps

   

4. Now that everything is up and running, you can go to the following pages in your browser to see if theyre loading

   • Kibana

     ipaddress:5601

     

   • Elasticsearch

     ipaddress:9200

     

Installing Filebeat on Ubuntu – Optional

1. Now that we have everything setup, we need another server that will obtain all our log files and Filebeat will do just that. First we need to install it by running the following

   curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-i386.deb
   sudo dpkg -i filebeat-5.6.1-i386.deb

   Note: You need to change i386 if you have a x64 architecture. You can change it to amd64

   

2. From your original Orbit that is running your ELK server we need to transfer 2 files: filebeat.yml and logstash-beats.crt
   First you must go to your home directory before running any of these commands

   cd nginx-filebeat
   scp logstash-beats..crt filebeat.yml ubuntu@192.168.16.58:~

   Here we are copying the two files we need from the nginx-filebeat directory to the home directory of our Ubuntu server whose IP is 192.168.16.58. Change the IP to reflect the system that you are transferring to

3. Once transferred we can go back to the system we transferred the files to and move them to the appropriate directories. First, the filebeat.yml file needs to be moved

   sudo mv filebeat.yml /etc/filebeat/filebeat.yml

   After the file gets transferred, we need to change ownership by running the following

   sudo chown root:root /etc/filebeat/filebeat.yml

   Now we need to create a directory for our certificate and then we can move the certificate over.

   sudo mkdir -p /etc/pki/tls/certs/
   sudo mv logstash-beats.crt /etc/pki/tls/certs/

4. Next we need to edit out host file to contain the IP of the ELK server. We do so by editing the hosts file and adding the following

   sudo vim /etc/hosts

   We add a new line with the IP address of our ELK server along with the name elk as shown in the image below

   

5. We can test that everything is working by pinging elk. We should see something like the image below

   ping elk

   

6. Now we need to make sure our connection between ELK and Filebeat are working as well. We can do that by running

   curl -XPUT 'http://elk:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json

   

7. We can finally start the service and check Kibana for our results. We can start the service by running the following

   sudo systemctl start filebeat

   You can also check the service’s status through this

   sudo systemctl status filebeat.service

   

8. Lastly, open up Kibana and you should see the logs like the image below