NAME

nph-webjob.cgi - Process webjob GET/JOB/PUT requests.

SYNOPSIS

    --- http.conf ---
    ...
    Include /var/webjob/config/apache/webjob.conf
    ...
    --- http.conf ---

    --- /var/webjob/config/apache/webjob.conf ---
    SetEnv WEBJOB_PROPERTIES_FILE /var/webjob/config/nph-webjob/nph-webjob.cfg

    ScriptAlias /cgi-client/ "/usr/local/webjob/cgi/cgi-client/"

    <Directory "/usr/local/webjob/cgi/cgi-client/">
      AllowOverride AuthConfig
      Options None
      Order allow,deny
      Allow from all
    </Directory>
    --- /var/webjob/config/apache/webjob.conf ---

DESCRIPTION

This utility implements a Common Gateway Interface (CGI), and its purpose is to processes webjob GET/JOB/PUT requests.

The name of this utility must begin with the string ``nph-''. This naming convention, which stands for Non Parsed Headers, informs the web server that the utility is responsible for generating the entire HTTP response. Non Parsed Headers are used by this utility because they allow it to take full advantage of HTTP status code extensions. This simplifies client-side error checking without giving up the ability to generate application-specific responses.

The nph-webjob.cfg config file is used to control the CGI's runtime behavior. To make use of it, set WEBJOB_PROPERTIES_FILE as appropriate. Upon execution, the script looks for and processes the file specified by this environment variable.

The cgi-client directory is where nph-webjob.cgi lives by default. Access to this directory should be strictly controlled, and any utilities placed in this directory should operate in the same security context (or realm). This utility is designed to read/write data from/to the server's local disk on behalf of potentially hostile clients. Therefore, you'll want to maintain a tight rein to avoid any unwanted results.

CONFIGURATION CONTROLS

This section describes the various controls that this program recognizes. In general, controls either shape runtime behavior or provide information needed to perform a specific function. Controls and their values, one pair/line, are read from a file having the following format.

    <control> = <value>

All controls are case insensitive, but, in general, their values are not. Comments may occur anywhere on a given line, and must begin with a pound character (i.e., '#'). In any given line, all text to the right of the first comment will be ignored. White space surrounding controls and values is ignored.

BaseDirectory: <path>

BaseDirectory is the epicenter of activity. The default value is '/var/webjob'.

CapContentLength: [Y|N]

CapContentLength forces the script to abort when ContentLength exceeds MaxContentLength. The default value is 'N'.

ConfigSearchOrder: [clients|commands|clients:commands|commands:clients]

ConfigSearchOrder specifies the order in which custom config files are sought out and processed. Custom config files may be used to override a predefined subset of the site-specific properties. The following tree enumerates the locations where global and custom config files may exist.

    config
      |
      + nph-webjob
          |
          - nph-webjob.cfg # applies globally
          |
          + clients
          |   |
          |   - nph-webjob.cfg # applies globally
          |   |
          |   + <client-N>
          |       |
          |       - nph-webjob.cfg # applies to all commands for <client-N>
          |       |
          |       + <command-N>
          |           |
          |           - nph-webjob.cfg # applies only to <client-N>/<command-N>
          |
          + commands
              |
              - nph-webjob.cfg # applies globally
              |
              + <command-N>
                  |
                  - nph-webjob.cfg # applies to all clients for <command-N>
                  |
                  + <client-N>
                      |
                      - nph-webjob.cfg # applies only to <command-N>/<client-N>

As each config file is processed, its values trump those of any that came before -- including any values that came from global config files. Supported values for this variable are 'clients', 'commands', 'clients:commands', and 'commands:clients'. The default value is 'clients:commands'.

ConfigDirectory: <path>

ConfigDirectory is where various configuration files are maintained. The default value is '<BaseDirectory>/config'.

DsvMaxSignatureLength: <integer>

DsvMaxSignatureLength specifies the maximum signature length that the script is willing to allow. If the signature length exceeds this limit (in bytes), the script will abort. The default value is 256.

DsvNullSignature: <string>

DsvNullSignature specifies the signature for a null response. The script will abort if DsvRequireSignatures is enabled and this value is not defined.

DsvRequireSignatures: [Y|N]

When active, DsvRequireSignatures forces the script to abort if no signature file is found, or if the signature does not meet basic syntax checks. A signature file must have the same basename as the requested payload, and its suffix must match the value defined by DsvSignatureSuffix. The default value is 'N'.

DsvSignatureSuffix: <suffix>

DsvSignatureSuffix specifies the suffix assigned to and used by signature files. A signature file must have the same basename as the requested payload, and its suffix must match the value defined by this property. The default value is '.sig'.

DynamicDirectory: <path>

DynamicDirectory is where dynamic content is stored. The default value is '<BaseDirectory>/dynamic'.

EnableGetService: [Y|N]

When active, EnableGetService allows the script to accept and process valid GET requests. This control can be overridden, and its intended purpose is to administratively disable access for valid clients (e.g., to force a failover). The default value is 'Y'.

EnableHostAccessList: [Y|N]

When active, EnableHostAccessList causes the script to consult a host access list to determine whether or not the client will be granted access based on its IP address. Note: This mechanism will fail closed if it's enabled and the access list is missing or does not contain the necessary client ID to IP address mappings. The default value is 'N'.

EnableConfigOverrides: [Y|N]

When active, EnableConfigOverrides causes the script to seek out and process additional client- and/or command-specific config files (see ConfigSearchOrder). The default value is 'Y'.

EnableJobQueues: [Y|N]

When active, EnableJobQueues allows the script to pull jobs from job queues and answer JOB requests. The default value is 'Y'.

EnableLogging: [Y|N]

When active, EnableLogging forces the script to generate a log message for each request. If the designated LogFile can not be opened, the log message will be written to STDERR. The default value is 'Y'.

EnablePutService: [Y|N]

When active, EnablePutService allows the script to accept and process valid PUT requests. This control can be overridden, and its intended purpose is to administratively disable access for valid clients (e.g., to prevent unwanted uploads). The default value is 'Y'.

FolderList: <item[:item[...]]>

FolderList specifies locations where shared programs can be found. If a requested file does not exist in a given client's commands directory, the FolderList is searched according to the order given here. The list delimiter is a colon (i.e., ':'). An example list would be 'common:shared'. The default value is 'common'.

GetHookCommandLine: <command>

GetHookCommandLine is a command string consisting of zero or more conversion specifications optionally interspersed with zero or more plain text characters. The following conversion specifications are supported:

    %A   = Weekday as a full name
    %a   = Weekday as an abbreviated name
    %cid = Client ID as a string
    %cmd = Client-requested command as a string
    %d   = Day of the month as a decimal number (01-31)
    %dynamic_cmd = Alias for %dynamic_out
    %dynamic_dir = Full path of the dynamic directory
    %dynamic_out = Full path of the dynamic output file that is to be created
    %dynamic_sig = Full path of the dynamic signature file that is to be created
    %H   = Hour as a decimal number (00-23)
    %ip  = IP address as a dotted quad string
    %jid = Job ID as a string
    %M   = Minute as a decimal number (00-59)
    %m   = Month as a decimal number (01-12)
    %pid = Process ID of server-side CGI script
    %S   = Second as a decimal number (00-60)
    %s   = Number of seconds since the Epoch
    %sid = Server ID as a string
    %system_version = System version as given by the client
    %u   = Weekday as a decimal number (1-7)
    %w   = Weekday as a decimal number (0-6)
    %Y   = Year with century as a decimal number

For example, the following command string:

    { if [ "%H"X = "23"X ] ; then cp /<path>/checker %dynamic_out ; else { echo ; echo ":" ; } > %dynamic_out ; fi ; }

will copy the checker command to the filename represented by %dynamic_out if the current hour is 23. Otherwise, a script that simpy returns true is created on-the-fly -- effectively turning the job into a NOOP.

If the specified command is an empty string, then the hook mechanism is (effectively) disabled, and the condition is logged. However, if the hook is disabled (i.e., GetHookEnable=N), then this control is ignored.

GetHookLogDivertedOutput: [Y|N]

GetHookLogDivertedOutput causes the script to divert any output on stdout/stderr to nph-webjob-hook.{err,out}. Normally, output for these streams is discarded since hook commands should be written to keep them clean (e.g., by logging to a file). The default value is 'N'.

Note: If a given hook command is not working properly, you can use this option to capture any output that was written to stdout/stderr. That, in turn, may help you debug the problem.

Note: If logging is disabled (i.e., EnableLogging=N), this control is ignored.

GetHookEnable: [Y|N]

When active, GetHookEnable causes the script to execute the command line specified by GetHookCommandLine. The behavior of the hook mechanism is to launch a subprocess and wait for it to finish. The purpose of the subprocess is to create dynamic content that will be delivered to the client. The hook mechanism is highly configurable -- config file overrides are fully supported, multiple conversion tokens are available, and the user determines what, if any, commands are executed when the hook is activated. Currently, hooks are only activated if they are enabled and a hook command has been defined. The default value is 'N'.

GetHookStatus: <integer>

GetHookStatus determines the meaning of success for the hook. If the specified value does not match the exit status as returned by system(), the hook is considered a failure and the entire job is aborted. Otherwise, processing continues as normal. The default value is '0'.

GetHookStatusMap: <integer:integer>[,<integer:integer>[...]]

GetHookStatusMap allows the user to map hook exit codes into HTTP response codes. No mappings are defined by default.

GetTriggerCommandLine: <command>

GetTriggerCommandLine is a command string consisting of zero or more conversion specifications optionally interspersed with zero or more plain text characters. The following conversion specifications are supported:

    %A   = Weekday as a full name
    %a   = Weekday as an abbreviated name
    %cid = Client ID as a string
    %cmd = Client-requested command as a string
    %d   = Day of the month as a decimal number (01-31)
    %H   = Hour as a decimal number (00-23)
    %jid = Job ID as a string
    %ip  = IP address as a dotted quad string
    %M   = Minute as a decimal number (00-59)
    %m   = Month as a decimal number (01-12)
    %pid = Process ID of server-side CGI script
    %S   = Second as a decimal number (00-60)
    %s   = Number of seconds since the Epoch
    %sid = Server ID as a string
    %u   = Weekday as a decimal number (1-7)
    %w   = Weekday as a decimal number (0-6)
    %Y   = Year with century as a decimal number

For example, the following command string:

  echo "%Y-%m-%d %H:%M:%S GET %jid %cid" >> /var/log/%cid.jids

will append the current date, time, request method, job ID, and client ID to a client-specific file in /var/log.

If the specified command is an empty string, then the trigger mechanism is (effectively) disabled, and the condition is logged. However, if the trigger is disabled (i.e., GetTriggerEnable=N), then this control is ignored.

Note: Triggers are not currently supported on Windows platforms.

GetTriggerEnable: [Y|N]

When active, GetTriggerEnable causes the script to execute the command line specified by GetTriggerCommandLine. The behavior of the trigger mechanism is to launch a subprocess and continue with the main line of execution. In particular, the script will not block or wait for the subprocess to finish, nor will it attempt check the status or cleanup after the subprocess. The trigger mechanism is highly configurable -- config file overrides are fully supported, multiple conversion tokens are available, and the user determines what, if any, commands are executed when the trigger is pulled. Currently, triggers are only pulled if they are enabled, a trigger command has been defined, and the HTTP status code is 200. The default value is 'N'.

Note: Triggers are not currently supported on Windows platforms.

IncomingDirectory: <path>

IncomingDirectory is where various client uploads are stored. The default value is '<BaseDirectory>/incoming'.

JobQueueActive: [Y|N]

When active, JobQueueActive causes the script to pull jobs from the specified job queue. The default value is 'Y'.

JobQueueDirectory: <path>

JobQueueDirectory is where client job queues are maintained. The default value is '<SpoolDirectory>/jqd'.

JobQueuePqActiveLimit: <integer>

JobQueuePqActiveLimit specifies the maximum number of parallel jobs that may be in an active state (i.e., either 'sent' or 'open'). If JobQueueActive is disabled, this control is ignored. A value of zero means there is no limit. The default value is '0'.

JobQueuePqAnswerLimit: <integer>

JobQueuePqAnswerLimit specifies the maximum number of parallel jobs that will be returned to the client. If JobQueueActive is disabled, this control is ignored. A value of zero means there is no limit. The default value is '0'.

phProperties{'JobQueuePqAnswerLimit'} = 0;

JobQueueSqActiveLimit: <integer>

JobQueueSqActiveLimit specifies the maximum number of serial jobs that may be in an active state (i.e., either 'sent' or 'open'). If JobQueueActive is disabled, this control is ignored. A value of zero means there is no limit. The default value is '1'.

JobQueueSqAnswerLimit: <integer>

JobQueueSqAnswerLimit specifies the maximum number of serial jobs that will be returned to the client. If JobQueueActive is disabled, this control is ignored. A value of zero means there is no limit. The default value is '0'.

LogfilesDirectory: <path>

LogfilesDirectory is where log files are stored. The default value is '<BaseDirectory>/logfiles'.

MaxContentLength: <integer>

MaxContentLength specifies the largest upload in bytes the script will accept. If CapContentLength is disabled, this control has no effect. The default value is '100000000'.

OverwriteExistingFiles: [Y|N]

When active, OverwriteExistingFiles forces the script to unlink existing files prior to writing the uploaded data. The default PutNameFormat used by this script attempts to prevent filename collisions. However, that behavior is user-defined, and in some cases, it may be desirable to specify a PutNameFormat that is guaranteed to create collisions. In those situations, this control must be enabled to produce the desired outcome (i.e., allow existing files with the same name to be overwritten). The default value is 'N'.

ProfilesDirectory: <path>

ProfilesDirectory is where client profiles and shared commands are maintained. The default value is '<BaseDirectory>/profiles'.

PutHookCommandLine: <command>

PutHookCommandLine is a command string consisting of zero or more conversion specifications optionally interspersed with zero or more plain text characters. The following conversion specifications are supported:

    %A   = Weekday as a full name
    %a   = Weekday as an abbreviated name
    %cid = Client ID as a string
    %cmd = Client-requested command as a string
    %d   = Day of the month as a decimal number (01-31)
    %dynamic_dir = Full path of the dynamic directory
    %dynamic_out = Full path of the dynamic output file that is to be created
    %dynamic_sig = Full path of the dynamic signature file that is to be created
    %env = Full path to .env file as a string
    %err = Full path to .err file as a string
    %H   = Hour as a decimal number (00-23)
    %ip  = IP address as a dotted quad string
    %jid = Job ID as a string
    %job_time = Time between the creation of the job ID and the .rdy file is created
    %jqt = Job Queue Tag
    %lck = Full path to .lck file as a string
    %M   = Minute as a decimal number (00-59)
    %m   = Month as a decimal number (01-12)
    %out = Full path to .out file as a string
    %pid = Process ID of server-side CGI script
    %rdy = Full path to .rdy file as a string
    %S   = Second as a decimal number (00-60)
    %s   = Number of seconds since the Epoch
    %sid = Server ID as a string
    %system_version = System version as given by the client
    %u   = Weekday as a decimal number (1-7)
    %w   = Weekday as a decimal number (0-6)
    %Y   = Year with century as a decimal number

If the specified command is an empty string, then the hook mechanism is (effectively) disabled, and the condition is logged. However, if the hook is disabled (i.e., PutHookEnable=N), then this control is ignored.

PutHookLogDivertedOutput: [Y|N]

PutHookLogDivertedOutput causes the script to divert any output on stdout/stderr to nph-webjob-hook.{err,out}. Normally, output for these streams is discarded since hook commands should be written to keep them clean (e.g., by logging to a file). The default value is 'N'.

Note: If a given hook command is not working properly, you can use this option to capture any output that was written to stdout/stderr. That, in turn, may help you debug the problem.

Note: If logging is disabled (i.e., EnableLogging=N), this control is ignored.

PutHookEnable: [Y|N]

When active, PutHookEnable causes the script to execute the command line specified by PutHookCommandLine. The behavior of the hook mechanism is to launch a subprocess and wait for it to finish. The purpose of the subprocess is to create dynamic content that will be delivered to the client. The hook mechanism is highly configurable -- config file overrides are fully supported, multiple conversion tokens are available, and the user determines what, if any, commands are executed when the hook is activated. Currently, hooks are only activated if they are enabled and a hook command has been defined. The default value is 'N'.

PutHookStatus: <integer>

PutHookStatus determines the meaning of success for the hook. If the specified value does not match the exit status as returned by system(), the hook is considered a failure and the entire job is aborted. Otherwise, processing continues as normal. The default value is '0'.

PutHookStatusMap: <integer:integer>[,<integer:integer>[...]]

PutHookStatusMap allows the user to map hook exit codes into HTTP response codes. No mappings are defined by default.

PutNameFormat: <format>

PutNameFormat controls how files are named/saved in the incoming directory. In other words, it controls the directory's layout. Basically, PutNameFormat is a format string consisting of zero or more conversion specifications optionally interspersed with zero or more plain text characters. The following conversion specifications are supported:

    %A   = Weekday as a full name
    %a   = Weekday as an abbreviated name
    %cid = Client ID as a string
    %cmd = Client-requested command as a string
    %d   = Day of the month as a decimal number (01-31)
    %H   = Hour as a decimal number (00-23)
    %ip  = IP address as a dotted quad string
    %M   = Minute as a decimal number (00-59)
    %m   = Month as a decimal number (01-12)
    %pid = Process ID of server-side CGI script
    %S   = Second as a decimal number (00-60)
    %s   = Number of seconds since the Epoch
    %sid = Server ID as a string
    %u   = Weekday as a decimal number (1-7)
    %w   = Weekday as a decimal number (0-6)
    %Y   = Year with century as a decimal number

For example, the following format string:

  "%cmd/%ip_%Y-%m-%d_%H.%M.%S"

will cause uploaded files to be stored in sub-directories that correspond to the name of the command executed, and each output filename will consist of an IP address, date, and time.

The added flexibility provided by this scheme means that it is possible to create format strings that are problematic. Consider the following string:

  "%cid/%cmd"

While this is a legal format string, it is likely to cause name collisions (e.g., the same client runs the same command two or more times). Therefore, it is important to create format strings that contain enough job specific information to distinguish one set of uploaded files from another.

The default value is '%cid/%cmd/%Y-%m-%d/%H.%M.%S.%pid'.

PutTriggerCommandLine: <command>

PutTriggerCommandLine is a command string consisting of zero or more conversion specifications optionally interspersed with zero or more plain text characters. The following conversion specifications are supported:

    %A   = Weekday as a full name
    %a   = Weekday as an abbreviated name
    %cid = Client ID as a string
    %cmd = Client-requested command as a string
    %d   = Day of the month as a decimal number (01-31)
    %env = Full path to .env file as a string
    %err = Full path to .err file as a string
    %H   = Hour as a decimal number (00-23)
    %ip  = IP address as a dotted quad string
    %jid = Job ID as a string
    %job_time = Time between the creation of the job ID and the .rdy file is created
    %jqt = Job Queue Tag
    %lck = Full path to .lck file as a string
    %M   = Minute as a decimal number (00-59)
    %m   = Month as a decimal number (01-12)
    %out = Full path to .out file as a string
    %pid = Process ID of server-side CGI script
    %rdy = Full path to .rdy file as a string
    %S   = Second as a decimal number (00-60)
    %s   = Number of seconds since the Epoch
    %sid = Server ID as a string
    %u   = Weekday as a decimal number (1-7)
    %w   = Weekday as a decimal number (0-6)
    %Y   = Year with century as a decimal number

For example, the following command string:

  echo "%Y-%m-%d %H:%M:%S PUT %jid %cid" >> /var/log/%cid.jids

will append the current date, time, request method, job ID, and client ID to a client-specific file in /var/log.

If the specified command is an empty string, then the trigger mechanism is (effectively) disabled, and the condition is logged. However, if the trigger is disabled (i.e., PutTriggerEnable=N), then this control is ignored.

Note: Triggers are not currently supported on Windows platforms.

PutTriggerEnable: [Y|N]

When active, PutTriggerEnable causes the script to execute the command line specified by PutTriggerCommandLine. The behavior of the trigger mechanism is to launch a subprocess and continue with the main line of execution. In particular, the script will not block or wait for the subprocess to finish, nor will it attempt check the status or cleanup after the subprocess. The trigger mechanism is highly configurable -- config file overrides are fully supported, multiple conversion tokens are available, and the user determines what, if any, commands are executed when the trigger is pulled. Currently, triggers are only pulled if they are enabled, a trigger command has been defined, and the HTTP status code is 200. The default value is 'N'.

Note: Triggers are not currently supported on Windows platforms.

RequireMatch: [Y|N]

RequireMatch forces the script to abort unless ClientId matches RemoteUser. When this value is disabled, any authenticated user will be allowed to issue requests for a given client. Disabling RequireUser implicitly disables RequireMatch. The default value is 'Y'.

RequireUser: [Y|N]

RequireUser forces the script to abort unless RemoteUser has been set. The default value is 'Y'.

ServerId: <string>

ServerId specifies the identity assigned to the WebJob server. The default value is 'server_1'.

SpoolDirectory: <path>

SpoolDirectory is where spools and queues are maintained. The default value is '<BaseDirectory>/spool'.

SslRequireCn: [Y|N]

SslRequireCn forces the script to abort unless SslClientSDnCn has been set. If SslRequireSsl is disabled, this and all other SSL controls are ignored. The default value is 'N'.

SslRequireMatch: [Y|N]

SslRequireMatch forces the script to abort if ClientId does not match SslClientSDnCn. When this control is disabled, access will be governed by RequireMatch. Disabling SslRequireCn implicitly disables SslRequireMatch. Also, if SslRequireSsl is disabled, this and all other SSL controls are ignored. The SslRequireMatch check is performed prior to (not instead of) the RequireMatch check. The default value is 'N'.

SslRequireSsl: [Y|N]

SslRequireSsl forces the script to abort unless the client is speaking HTTPS. Disabling SslRequireSsl implicitly disables all SSL-related controls. The default value is 'Y'.

UseGmt: [Y|N]

When active, UseGmt forces the script to convert all time values to GMT. Otherwise, time values are converted to local time. The default value is 'N'.

FILES

BaseDirectory/logfiles/jqd.log

This utility writes JQD-related log messages to jqd.log. This file contains the following fields separated by whitespace in the given order: Date, Time, JobId, Program, Pid, Creator, Queue, QueueTag, OldQueueState, NewQueueState, Command, CommandSize, PoundName, JobGroup, Result, and Message. A single hyphen, '-', is used to denote the fact that a particular field had an empty or undefined value. The Message field is set off from the other fields with a double hyphen '--' because it uses a free form text format that can include whitespace.

BaseDirectory/logfiles/nph-webjob.log

This utility writes log messages to nph-webjob.log (see nph-webjob.cfg). This file contains the following fields separated by whitespace in the given order: Date, Time, JobId, RemoteUser, RemoteAddress, RequestMethod, ClientId, Filename, ContentLength, ServerContentLength, Duration, ReturnStatus, and ErrorMessage. A single hyphen, '-', is used to denote the fact that a particular field had an empty or undefined value. The ErrorMessage field is set off from the other fields with a double hyphen '--' because it uses a free form text format that can include whitespace.

BaseDirectory/logfiles/nph-webjob-hook.err

This utility captures hook-related output on stderr and writes it to nph-webjob-hook.err. This file is a bit bucket for dirty hook commands. It is also useful as a debugging aid when creating new hook commands.

BaseDirectory/logfiles/nph-webjob-hook.log

This utility writes hook-related log messages to nph-webjob-hook.log. This file contains the following fields separated by whitespace in the given order: Date, Time, JobId, RequestMethod, ClientId, Filename, PidLabel, Pid, State, and Message. A single hyphen, '-', is used to denote the fact that a particular field had an empty or undefined value. The Message field is set off from the other fields with a double hyphen '--' because it uses a free form text format that can include whitespace.

BaseDirectory/logfiles/nph-webjob-hook.out

This utility captures hook-related output on stdout and writes it to nph-webjob-hook.out. This file is a bit bucket for dirty hook commands. It is also useful as a debugging aid when creating new hook commands.

BaseDirectory/logfiles/nph-webjob-trigger.log

This utility writes trigger-related log messages to nph-webjob-trigger.log. This file contains the following fields separated by whitespace in the given order: Date, Time, JobId, RequestMethod, ClientId, Filename, PidLabel, Pid, State, and Message. A single hyphen, '-', is used to denote the fact that a particular field had an empty or undefined value. The Message field is set off from the other fields with a double hyphen '--' because it uses a free form text format that can include whitespace.

AUTHOR

Klayton Monroe

SEE ALSO

nph-config.cgi(1), webjob(1)

LICENSE

All documentation and code are distributed under same terms and conditions as WebJob.