Email notifications via supervisor on abrupt termination of process

supervisor has some awesome plugins. One among them is superlance. It allows you to notify via email in case of certain event occurrence. For ex: you can send an email to the ops guys if a process terminates abruptly.

Here's a sample python application that will terminate abruptly every 30 seconds:

import time

i = 10  
while True:  
    print "I am still working"
    time.sleep(3)
    j=  1000 / i
    i -= 1

To have supervisor and superlance setup (for Ubuntu) you need to:

pip install supervisor  
pip install superlance  
sudo apt-get install sendmail  

Next is superlance config file. I am providing the program and email config part below. For full file you can look at my github project here:

[program:simple_app]
command=python simple_app.py  
process_name=%(program_name)s  
numprocs=1  
directory=/home/ravi/dev/projects/python/supervisor-play  
autostart=true  
startretries=3  
stdout_logfile=~/dev/projects/python/supervisor-play/s.out  
stderr_logfile=/home/ravi/dev/projects/python/supervisor-play/s.err 

[eventlistener:crashmail] 
command=/home/ravi/.virtualenvs/super-awesome/bin/crashmail -a -m james.bond@gmail.com  
events=PROCESS_STATE  

To start up your program simply do: supervisord -c supervisord.conf -n

And that's it! You have got mail!!! ;-)

Gotchas:

  1. I did not have sendmail installed and supervisor was failing silently.
  2. You need to fail abruptly and apparently sending process kill signal is not abrupt. I have to fail with some unexpected program error.

Written with StackEdit.