Grid Guides

Explore How System On Grid Can Work For You

ELK Stack One-Click App Setup


Introduction

One of the many available One-Click Apps that we have is ELK. ELK is an acronym for a collection of three open-source products: Elasticsearch, Logstash, and Kibana. Elasticsearch is a NoSQL database that is based on the Lucene search engine. Logstash is a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. Kibana is a visualization layer that works on top of Elasticsearch.

Prerequisites

  • Orbit created with ELK image

  • Running Orbit

Creating Your Orbit

  1. To create your application, you must create a Photon. You can do so by selecting the Photon tab after selecting 'Create Orbit'

    Photons
  2. Once there, you will see a list of available photons that you can create. Select one and it will bring you to a detailed page

    Photon List Photon Description
  3. Scroll down to the details and you can start selecting options for your photon as well as setting up credentials.

    Photon Details
  4. Once everything is filled in, just press Create and you can move on to the next step

SSH into Orbit

  1. First we need to check some details on our orbit by first navigating to the “My Orbits” tab.

    My Orbits
  2. Once there, click on the Orbit you want to connect to and check for three things:

    • IP Address

    • Running Status

    • Source Image(Ubuntu, Debian, Centos, Fedora)

    Orbit Details
  3. Once you have confirmed that your Orbit is running and you have the other information. Open up your console.

  4. Now you can SSH into your Orbit by running.

    ssh [email protected]
    • IP Address: We got this from the detail page, in this case it is “216.200.116.60”

    • Image: We got this from the source image in the details page. It should be one of the following: ubuntu, debian, fedora, centos

    • Note: A full command for this example would be: “ssh [email protected]

Starting ELK

  1. Once you are SSH'd into the server you will see the following.

    ELK CMD
  2. We first need to set a setting for Elasticsearch to function

    sudo sysctl -w vm.max_map_count=262144

    Finally we can deploy ELK to the docker

    docker stack deploy -c stack.yml elk
    ELK Stackguides-
  3. Before continuing, we need to wait a few minutes for the docker to set everything up. If you want to see if everything is ok to continue, you can run the following and it should look like the image below

    docker ps
    ELK Docker
  4. Now that everything is up and running, you can go to the following pages in your browser to see if theyre loading

    • Kibana

      ipaddress:5601
      ELK Kibana
    • Elasticsearch

      ipaddress:9200
      ELK Elastic

Installing Filebeat on Ubuntu - Optional

  1. Now that we have everything setup, we need another server that will obtain all our log files and Filebeat will do just that. First we need to install it by running the following

    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-i386.deb
    sudo dpkg -i filebeat-5.6.1-i386.deb

    Note: You need to change i386 if you have a x64 architecture. You can change it to amd64

    Filebeat Download
  2. From your original Orbit that is running your ELK server we need to transfer 2 files: filebeat.yml and logstash-beats.crt
    First you must go to your home directory before running any of these commands

    cd nginx-filebeat
    scp logstash-beats..crt filebeat.yml [email protected]:~

    Here we are copying the two files we need from the nginx-filebeat directory to the home directory of our Ubuntu server whose IP is 192.168.16.58. Change the IP to reflect the system that you are transferring to

  3. Once transferred we can go back to the system we transferred the files to and move them to the appropriate directories. First, the filebeat.yml file needs to be moved

    sudo mv filebeat.yml /etc/filebeat/filebeat.yml

    After the file gets transferred, we need to change ownership by running the following

    sudo chown root:root /etc/filebeat/filebeat.yml

    Now we need to create a directory for our certificate and then we can move the certificate over.

    sudo mkdir -p /etc/pki/tls/certs/
    sudo mv logstash-beats.crt /etc/pki/tls/certs/
  4. Next we need to edit out host file to contain the IP of the ELK server. We do so by editing the hosts file and adding the following

    sudo vim /etc/hosts

    We add a new line with the IP address of our ELK server along with the name elk as shown in the image below

    ELK Host
  5. We can test that everything is working by pinging elk. We should see something like the image below

    ping elk
    ELK Ping
  6. Now we need to make sure our connection between ELK and Filebeat are working as well. We can do that by running

    curl -XPUT 'http://elk:9200/_template/filebeat?pretty' [email protected]/etc/filebeat/filebeat.template.json
    ELK Curl
  7. We can finally start the service and check Kibana for our results. We can start the service by running the following

    sudo systemctl start filebeat

    You can also check the service's status through this

    sudo systemctl status filebeat.service
    Filebeat Service
  8. Lastly, open up Kibana and you should see the logs like the image below

    ELK Log