Revision

  $Id: webjob-triggers-compress-upload.base,v 1.6 2007/01/23 17:51:51 klm Exp $

Purpose

  This recipe demonstrates how to use WebJob's server-side triggers to
  automatically compress (via bzip2, compress, or gzip) WebJob uploads
  for a specific command using configuration overrides.

Motivation

  This recipe grew out of the desire to compress WebJob uploads as
  they roll in rather than waiting to process them in bulk at some
  later point in time (e.g., via cron/find).

Requirements

  Cooking with this recipe requires an operational WebJob server.
  If you do not have one of those, refer to the instructions provided
  in the README.INSTALL file that comes with the source distribution.
  The latest source distribution is available here:

    http://sourceforge.net/project/showfiles.php?group_id=40788

  Each client must be running UNIX and have basic system utilities
  and WebJob (1.6.0 or higher) installed.

  The server must be running UNIX and have basic system utilities,
  Apache, and WebJob (1.6.0 or higher) installed.

  The commands presented throughout this recipe were designed to
  be executed within a Bourne shell (i.e., sh or bash).

  This recipe assumes that you have read and implemented the following
  recipe:

    http://webjob.sourceforge.net/Files/Recipes/webjob-run-periodic.txt

Time to Implement

  Assuming that you have satisfied all the requirements/prerequisites,
  this recipe should take less than one hour to implement.

Solution

  The solution is to configure your WebJob server to utilize triggers,
  install the compression script (Appendix 1), enable triggers for a
  particular command, and finally, test it all out.  To keep things
  simple, this recipe assumes that the output you wish to compress is
  produced by s_hlc_ps.  Further, it is assumed that you are already
  harvesting this data from your hourly and/or daily scripts.

  1. To configure your WebJob server to utilize triggers, you need to
     upgrade nph-webjob.cgi to revision 1.39 or higher.  If your
     server already has WebJob 1.5.0 or higher installed, you should
     be able to simply replace the existing CGI script with the newer
     revision.  Older installs will require more planning and care.

     Once the new script is in place, create a trigger logfile that
     lives in the same directory as nph-webjob.log.  This can be done
     as follows:

       # WEBJOB_BASE_DIR=/var/webjob
       # touch ${WEBJOB_BASE_DIR}/logfiles/nph-webjob-trigger.log
       # chown www:www ${WEBJOB_BASE_DIR}/logfiles/nph-webjob-trigger.log
       # chmod 644 ${WEBJOB_BASE_DIR}/logfiles/nph-webjob-trigger.log

  2. Download webjob-compress-upload from the following location, and
     install it in a suitable bin directory.

       http://webjob.cvs.sourceforge.net/webjob/webjob/tools/webjob-compress-upload

       # BIN_DIR=/usr/local/bin
       # cp webjob-compress-upload ${BIN_DIR}
       # chown root:wheel ${BIN_DIR}/webjob-compress-upload
       # chmod 755 ${BIN_DIR}/webjob-compress-upload

  3. Create a custom config file override, and populate it with the
     controls shown below.

       # mkdir -p ${WEBJOB_BASE_DIR}/config/nph-webjob/commands/s_hlc_ps

       --- nph-webjob.cfg ---
       PutTriggerEnable=Y
       PutTriggerCommandLine=webjob-compress-upload -m bzip2 -r %rdy
       --- nph-webjob.cfg ---

     Make sure that webjob-compress-upload and your chosen compression
     utility (bzip2 in this case) are in the Apache user's PATH.

     Now, sit back and wait for the data to roll in.

  4. Verify that each of the files in the upload group have been
     compressed.  There should be four files, and these files should
     have extensions that match the following:

       <file>.env.bz2
       <file>.err.bz2
       <file>.out.bz2
       <file>.rdy.bz2

Closing Remarks

  If you don't see the correct output, try looking through the WebJob
  trigger and Apache error logs for clues.  Also, check to see if the
  compression utility you specified is in the Apache user's PATH.

Credits

  This recipe was brought to you by Klayton Monroe.

References