Somsip Blog

I know there are a load of solutions out there already, but nothing that I could see that were as simple as I needed, or as flexible. In particularly I have one project which uses the core code across three sets of servers that do very different things – a webserver, a number of emailservers that prepare and send emails for a short period each day so are brought into existence with an AWS auto-scaling group, and a number of imageservers which don’t serve static images and these are always running but scale up and down with load. Each one runs a different set of cronjobs. And some cronjobs run every minute.

As AWS auto-scaling groups rely on defining the AMI and changing these is a bit of messing about I would prefer to minimise, I wanted the cronjobs under source control, and for them to be updated when the code base is updated on boot for the new machines brought up when scaling. And I want to be able to push crontab changes to any servers that are running when I deploy new code. And I have some servers that run more than one codebase (it’s complicated), so I can’t overwrite any existing crontab when adding cronjobs.

My first thought was to have a few files in a crontabs folder under git, symlink to the correct one from /etc/cron.d and that would be enough. Well, any file linked to in /etc/cron.d must be owned by root and not readable by anyone else. Which led to some permissions changes which ultimately breaks my deployment process with fabric. And a change of thinking. In the end, I have gone with this:

From boot:

In crontab for login user:
@reboot /var/www/project/crontabs/reboot.sh suffix > /dev/null 2>&1

In reboot.sh:
#! /bin/bash
# Takes an argument that matches one of the crontabs.* files in the project webroot
# Strips all lines between the CRONTAB_START and CRONTAB_END and inserts the matching file

if [ -z "$1" ]
then
  exit
fi

WEBROOT=/var/www
PROJECT=myproject
CRONTAB_START='# MYPROJECT START'
CRONTAB_END='# MYPROJECT END'

(crontab -l | sed "/^$CRONTAB_START/,/^$CRONTAB_END/d"; cat $WEBROOT/$PROJECT/crontabs/crontabs.$1) | crontab -

So the crobjob just kicks off a bash script with a parameter of ‘suffix’. This will allow me to call different files for the different flavours of production servers, and dev server. The bash script has a few variables to define, but here they all use ‘myproject’ or some variant.

The clever bit outputs the existing crontab, strips anything between and including CRONTAB_START and CRONTAB_END, then appends the correct crontabs.suffix file for the login user. So on boot, the script deletes what was there already for this project without affecting anything else in the crontab file, adds in the current version of the crontabs for this server, based on the file suffix, and the job is done. The crontab.* files look something like:

# MYPROJECT START

# Build every minute throughout the day
* * * * * curl "http://myserver/endpoint" > /dev/null 2>&1

# MYPROJECT END

For running servers, I use a fabfile that does pretty much the same thing:

from fabric.api import *

env.user = 'user'
env.key_filename = '/home/mark/.pem/server.pem'

WEBROOT = '/var/www'
PROJECT = 'myproject'
CRONTAB_START = '# MYPROJECT START'
CRONTAB_END = '# MYPROJECT END'

# Other code stripped out for brevity
def production():
  env.hosts = ['www.myserver.com']

def crontab(file):
  run("(crontab -l | sed '/^%s/,/^%s/d'; cat %s/%s/crontabs/crontabs.%s) | crontab -" % (CRONTAB_START, CRONTAB_END, WEBROOT, PROJECT, file))

So to update the crontab on a running server, I can do something like this and the old cronjobs will be stripped from the crontab, and the contents of the file crontabs.emailserver (in this example) will be appended to it

fab production crontab:file=emailserver 

Time will show if it’s worthwhile doing this. The crontabs are reasonably static. But when they do change, this will avoid recreating AMIs for the sake of small alterations or additions.