Install ELK Stack on Ubuntu/Debian

How to install ELK and configure it on your server (Ubuntu/Debian)?

You can follow the steps below to install and configure Elasticsearch-Logstash-Kibana (ELK) stack on Ubuntu/Debian server.

If you are familiar with ansible, you can also find below instructions via ansible installation for ELK stack here.

Logstash and elastic search require Java, preferably Java 1.6+. Check if you have Java installed.

java -version

Install Java Runtime:

sudo apt-get update
sudo apt-get install -y openjdk-7-jre`

Install Logstash

Add the key:
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

Add the logstash repo to /etc/apt/sources.list
deb http://packages.elasticsearch.org/logstash/1.4/debian stable main

Install Logstash:
sudo apt-get install logstash -y

With this you have installed Logstash as a service.

Install Elastic search:

Add elasticsearch key:
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

Add the following line to /etc/apt/sources.list:
deb http://packages.elasticsearch.org/elasticsearch/1.0/debian stable main

Update the package list:
sudo apt-get update

Install ElasticSearch:
sudo apt-get install elasticsearch -y

Install Kibana:

Get the archive and extract it:

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz
tar -xvf kibana-3.1.2.tar.gz

In the Kibana configuration file, find the line that specifies the elasticsearch, and set the port number to 80:
elasticsearch: "http://"+window.location.hostname+":80",

We will use Nginx to serve kibana web, so lets get set up for that:

sudo mkdir -p /var/www/kibana3

Copy the kibana directory contents to /var/www/kibana3:

sudo cp -r kibana-3.1.2/* /var/www/kibana3/

Install Nginx:

sudo apt-get update
sudo apt-get install nginx -y

Open the /etc/nginx/sites-enabled/default file and change the line below:

root /usr/share/nginx/www;
to
root /var/www/kibana3;

[UPDATE: Added instructions for serving Kibana via JBoss based on the commenter's request below.]

Install JBoss

If you would like to use JBoss instead of Nginx, then follow the steps below.

Download and extract JBoss:

cd /tmp
wget http://download.jboss.org/jbossas/7.1/jboss-as-7.1.1.Final/jboss-as-7.1.1.Final.tar.gz
tar -xvf jboss-as-7.1.1.Final.tar.gz

Move extracted JBoss to /usr/local/share

mv /tmp/jboss-as-7.1.1.Final /usr/local/share/jboss-7.1.1

Deplpoy Kibana content:

cd /usr/local/share/jboss-7.1.1/standalone/deployments
sudo mkdir -p kibana.war/WEB-INF
sudo cp -r kibana-3.1.2/* kibana.war
sudo touch kibana.war.dodeploy

Please do not forget to create .dodeploy file as mentioned in the last step above

Create web.xml with contents shown below: vi kibana.war/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1" ?>

<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
    version="2.4">
</web-app>

Now start JBoss:

cd /usr/local/share/jboss-7.1.1
sudo ./bin/standalone.sh -Djboss.bind.address=192.168.1.8 -Djboss.bind.address.management=192.168.1.8

Replace 192.168.1.8 with your server's IP address.

And kibana web will be available at: http://192.168.1.8:8080/kibana/

Configure Logstash:

Create logstash config file:
vi /etc/logstash/conf.d/simple.conf

And add the following content:

input {
  redis {
    host => "11.12.13.14"
    type => "redis"
    data_type => "list"
    key => "logstash"
  }
}
output {
stdout { }
  elasticsearch {
    cluster => "elasticsearch"
  }
}

Now start logstash:
logstash/bin/logstash -f /etc/logstash/conf.d/simple.conf

That is your ELK stack ready to serve you log content.

But before we can see any logs we need to ship the logs to the ELK stack.

Install Logstash shipper/forwarder on your client machines.

Same instructions as installing logstash on the ELK stack. Add the key:
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

Add the logstash repo to /etc/apt/sources.list
deb http://packages.elasticsearch.org/logstash/1.4/debian stable main

Install Logstash:
sudo apt-get install logstash -y

With this you have installed Logstash as a service.

Create logstash shipper config:
vi /etc/logstash/conf.d/logstash_shipper.conf

input {
  stdin { }
  file {
    path => "/opt/applications/logs/your_app/your_app.log"
    start_position => beginning
    codec =>  multiline {
      'negate' => true
      'pattern' => '^\d'
      'what' => 'previous'
    }
  }
}
output {
  stdout { codec => rubydebug }
  redis { host => "10.11.14.15" data_type => "list" key => "logstash" }
}

And then start logstash service on the client: sudo service logstash start

Hit Kibana by hitting the server IP in your favorite browser: