Revision $Id: webjob-manage-cronjob.base,v 1.15 2010/12/10 05:33:34 klm Exp $ Purpose This recipe demonstrates how to deploy, remove, or update a specified cron job from one or more WebJob clients. Motivation WebJob clients typically check in and do work on an hourly or daily basis. Sometimes, they'll check in as often as every 15 minutes (e.g., heartbeat job). However, checking in more frequently than that may not be practical -- especially for large client populations. This creates a 15-60 minute hole in the admin's ability to schedule higher frequency jobs and get work done on the clients. Since cron has minute resolution, it's a natural choice for managing higher frequency work and filling that hole. However, this requires a modification to each client's crontab. That's not a problem for a few hosts, but if you have tens to hundreds of hosts, it could quickly turn into a real headache. That's where WebJob and this recipe come in. Requirements Cooking with this recipe requires an operational WebJob server. If you do not have one of those, refer to the instructions provided in the README.INSTALL file that comes with the source distribution. The latest source distribution is available here: http://sourceforge.net/project/showfiles.php?group_id=40788 The server must be running UNIX and have basic system utilities, Apache, and WebJob (1.9.0 or higher) installed. Each client must be running UNIX and have basic system utilities and WebJob (1.9.0 or higher) installed. This recipe assumes that you have read and implemented the hourly script from the following recipe: http://webjob.sourceforge.net/Files/Recipes/webjob-run-periodic.txt and the queue-worker.sh script from the following recipe: http://webjob.sourceforge.net/Files/Recipes/webjob-run-queued-jobs.txt Time to Implement Assuming that you have satisfied all the requirements/prerequisits, this recipe should take less than one hour to implement. Solution The following steps describe how to implement this solution. 1. Download cronjob-manager from the following location, and install it in the appropriate commands directory. If you want this script to be bound to a particular client, set WEBJOB_CLIENT as appropriate before running the following commands. Once the file is in place, set its ownership and permissions to 0:0 and mode 644, respectively. http://webjob.cvs.sourceforge.net/webjob/webjob/tools/cronjob-manager # WEBJOB_CLIENT=common # WEBJOB_COMMANDS=/var/webjob/profiles/${WEBJOB_CLIENT}/commands # cp cronjob-manager ${WEBJOB_COMMANDS}/ # chown 0:0 ${WEBJOB_COMMANDS}/cronjob-manager # chmod 644 ${WEBJOB_COMMANDS}/cronjob-manager This script has the following usage: Usage: cronjob-manager {-d|--deploy} [-n] [-f crontab] -c 'command-line' -t 'time-specification' cronjob-manager {-l|--lsjobs} cronjob-manager {-r|--remove} [-Fn] [-f crontab] -e 'crontab-entry' cronjob-manager {-u|--update} [-aFn] [-f crontab] -c 'command-line' -t 'time-specification' Note: If you are using DSV, you'll need to sign the script with webjob-dsvtool. # DSV_KEY=/var/webjob/config/dsv/webjob-key.pem # webjob-dsvtool -s -k ${DSV_KEY} ${WEBJOB_COMMANDS}/cronjob-manager 2. Decide what your job is going to be. For this recipe, we'll use the following example: * * * * * uptime >> /var/log/uptime.log 2> /dev/null This job runs uptime once a minute and appends any output to /var/log/uptime.log. 3. Test deployment and removal of your job on a small group of clients. To do this, create a new group called test. Assuming your clients are named client_host1, client_host2, and client_host3, you may create the group as follows: # webjob-jqd-create-group -g test client_host1 client_host2 client_host3k Next, create a job file and schedule it to run on the test group. --- cronjob-manager_deploy.job --- Command=cronjob-manager CommandAlias=cronjob-manager CommandLine=cronjob-manager --deploy -t '* * * * *' -c 'uptime >> /var/log/uptime.log 2> /dev/null' --- cronjob-manager_deploy.job --- # webjob-jqd-create-job -i %test -f cronjob-manager_deploy.job -t serial Once this job has completed, inspect the output to ensure that it was successful. The .out file should look similar to this: ACTION=deploy AUTO_CREATE=0 COMMAND_LINE=uptime >> /var/log/uptime.log 2> /dev/null CRONTAB_EXPR=uptime >> /var/log/uptime[.]log 2> /dev/null DRY_RUN=0 FORCE=0 TIME_SPECIFICATION=* * * * * USER=root --- crontab.bak --- existing jobs... --- crontab.bak --- --- crontab.new --- existing jobs... * * * * * uptime >> /var/log/uptime.log 2> /dev/null --- crontab.new --- UPDATE_STATUS=1 This output consists of three parts: runtime variables, a listing of the original crontab, and a listing of the new crontab. The job you deployed should show up at the bottom of the new crontab (i.e., crontab.new). To reverse the process (i.e., remove the cron job), create a second job file and schedule it to run on the test group. --- cronjob-manager_remove.job --- Command=cronjob-manager CommandAlias=cronjob-manager CommandLine=cronjob-manager --remove -e 'uptime >> /var/log/uptime.log 2> /dev/null' --- cronjob-manager_remove.job --- # webjob-jqd-create-job -i %test -f cronjob-manager_remove.job -t serial Once this job has completed, inspect the output to ensure that it was successful. The .out file should look similar to this: ACTION=remove AUTO_CREATE=0 COMMAND_LINE= CRONTAB_EXPR=uptime >> /var/log/uptime[.]log 2> /dev/null DRY_RUN=0 FORCE=0 TIME_SPECIFICATION= USER=root --- crontab.bak --- existing jobs... * * * * * uptime >> /var/log/uptime.log 2> /dev/null --- crontab.bak --- --- crontab.new --- existing jobs... --- crontab.new --- UPDATE_STATUS=1 4. Once you're satisfied that the deployment/removal process works as advertised, you can schedule a new job that'll run on all clients. After that, sit back, wait for the output to roll in, and check it to ensure that there weren't any complications. Closing Remarks Modifying crontabs can be risky -- especially if you rely on cron for mission critical operations. These risks can be reduced or eliminated through solid testing and a pinch of disaster recovery planning -- i.e., you'd better make backups of all your crontabs *before* you go messing with them. This can be done as follows: ${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg cronjob-manager --lsjobs --- cronjob-manager_lsjobs.job --- Command=cronjob-manager CommandAlias=cronjob-manager CommandLine=cronjob-manager --lsjobs --- cronjob-manager_lsjobs.job --- # webjob-jqd-create-job -i %test -f cronjob-manager_lsjobs.job -t serial As a defensive practice, you should get in the habit of enclosing the command-line ('-c') and time-specification ('-t') arguments in single quotes. This will help prevent unexpected variable expansion. Consider the following examples: # sh cronjob-manager --deploy -t '30 0 * * *' -c '${WEBJOB_HOME=/opt/local/webjob}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg daily > /dev/null 2>&1' or # sh cronjob-manager --deploy -t '30 0 * * *' -c "${WEBJOB_HOME=/opt/local/webjob}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg daily > /dev/null 2>&1" The first command does what you'd expect -- i.e., no variable expansion, and you get the following crontab entry: 30 0 * * * ${WEBJOB_HOME=/opt/local/webjob}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg daily > /dev/null 2>&1 The second command, however, encloses the command-line ('-c') argument with double quotes, so its variables are evaluated and replaced with their actual values. Assuming that WEBJOB_HOME was not already defined, the entry that ends up in the crontab would look like this: 30 0 * * * /opt/local/webjob/bin/webjob -e -f /opt/local/webjob/etc/upload.cfg daily > /dev/null 2>&1 Credits This recipe was brought to you by Klayton Monroe. References