Logstash Shipper / Forwarder

Once you have the ELK stack installed, you can then ship/forward your logs to the redis database on the ELK stack. Once your logs are on the redis database, the logstash on the ELK server will read from it and forward it to elasticsearch. You can then use kibana to query the elasticsearch for the newly intercepted logs!

Here are the steps for setting up Logstash shipper/forwarder on Ubuntu/Debian family:

Add the key:
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

Add the logstash repo to /etc/apt/sources.list
deb http://packages.elasticsearch.org/logstash/1.4/debian stable main

Install Logstash:
sudo apt-get install logstash -y

With this you have installed Logstash as a service.

Create logstash shipper config:
vi /etc/logstash/conf.d/logstash_shipper.conf

input {
  file {
    path => "/opt/applications/logs/your_app/your_app.log"
    start_position => beginning
    codec =>  multiline {
      'negate' => true
      'pattern' => '^\d'
      'what' => 'previous'
    }
  }
}

filter {
  grok {
    match => {"message" => "%{DATESTAMP:logDateTime} \[%{LOGLEVEL:logLevel}\] %{GREEDYDATA:logMessage}" }
}

output {
  redis { host => "10.11.14.15" data_type => "list" key => "logstash" }
}

And then start logstash service on the client: sudo service logstash start. Logstash will start reading the file: /opt/application/logs/your_app/your_app.log from the beginning. Each line read will be concatenated into previous line unless the new line starts with a number. This way stacktraces will be read as one line instead of many. The input like then will be tokenized based on the grok pattern and these tokens will then be sent to the redis server: 10.11.14.15, as a list, under the key name: logstash.

The logstash in ELK stack will be reading from the redis server and seding it's output to Elasticsearch which then can be queried via Kibana.

Does the explaination above make sense? Would you like more detail? Did I miss something? Please let me know via comments below!

Thank you for your time!

P.S: Of course you would need to plug in the file name that you want logstash to read and the redis ip needs to be changed to your ELK server ip.