WebJob - Download and execute a program over HTTP/HTTPS
webjob {-e|--execute} {-f|--file} config[,config[...]] program [options]
webjob {-g|--get-url} [{-f|--file} config[,config[...]]] url
webjob {-h|--hashsum} {-t|--type} digest file
webjob {-r|--run-embedded} {lua|perl} [...]
webjob {-v|--version}
Note: Command line arguments are position dependent, and argument
snuggling (e.g., '-ef', '-gf', or '-ht') is not supported. All
arguments and their values (if any) must be separated by whitespace.
However, any options associated with program, lua, or perl
are not bound by these restrictions. Instead, they are bound to the
argument syntax for program, lua, or perl, respectively.
WebJob's --execute mode downloads and executes a program in
one unified operation. WebJob's runtime behavior is controlled
through a config file identified by the --file argument. Among
other things, this file identifies a server willing to satisfy
WebJob GET requests. Once initialized, WebJob downloads the
target program, stores it on the local file system, and executes
it in a sub-process. The options, if any, are bound to the target
program at invocation. By default, any output generated by the
target program is directed to stdout/stderr. However, output may
also be directed to a WebJob server through a WebJob PUT
request.
WebJob's --get-url mode downloads a target url and writes the
output to stdout. This is useful in situations where a file
(accessible through the Web) is needed, but the target platform lacks
the necessary tools (e.g., curl, wget, etc.) to fetch it.
WebJob's --hashsum mode provides the ability to compute a
message digest of the specified file. This is useful in situations
where the downloaded executable needs to compute a hash, but the
target platform lacks the necessary tools (e.g., md5, md5sum, etc.) to
perform that task.
WebJob's --run-embedded mode provides the ability to execute Lua
or Perl scripts depending on how WebJob was compiled. This is
useful in situations where the target platform lacks the necessary
scripting tools (or the native tools are unable) to perform the task
at hand. It is also useful in that it provides a cross-platform
scripting environment, which allows you to standardize various jobs
across multiple platforms.
WebJob's GET/PUT requests may be bound to any available outbound
port. This increases the likelihood that distant WebJob servers
can be reached in spite of firewalls and filtering routers. WebJob
also provides limited proxy support.
When compiled with OpenSSL, WebJob supports HTTPS requests and
mutual certificate authentication.
WebJob also has many controls which dictate how it will execute.
The following table summarizes what controls apply to each mode of
operation. An 'X' indicates that the given control applies to the
selected mode.
===== MODES =====
r
u
n
-
e
e g h m v
x e a b e
e t s e r
c - h d s
u u s d i
t r u e o
e l m d n
====== CONTROL ======================
ClientId X . . . .
DsvCertificateTree X . . . .
DsvVerifySignature X . . . .
GetHookCommandLine X . . . .
GetHookEnable X . . . .
GetHookStatus X . . . .
GetHookSuffix X . . . .
GetTimeLimit X . . . .
HashType X . . . .
Import X . . . .
OverwriteExecutable X . . . .
Priority X . . . .
PutTimeLimit X . . . .
RunTimeLimit X . . . .
RunType X . . . .
SslBundledCAsFile X X . . .
SslExpectedPeerCN X X . . .
SslMaxChainLength X X . . .
SslPassPhrase X X . . .
SslPrivateKeyFile X X . . .
SslPublicCertFile X X . . .
SslUseCertificate X X . . .
SslVerifyPeerCert X X . . .
StrictControls X X . . .
TempDirectory X . . . .
TimeoutSignal X . . . .
UnlinkExecutable X . . . .
UnlinkOutput X . . . .
UploadOnGetFailure X . . . .
UrlAuthType X X . . .
UrlDownloadLimit X X . . .
UrlGetProxyAuthType X X . . .
UrlGetProxyEnable X X . . .
UrlGetProxyHost X X . . .
UrlGetProxyPassword X X . . .
UrlGetProxyPort X X . . .
UrlGetProxyUsername X X . . .
UrlGetUrl X . . . .
UrlPassword X X . . .
UrlPutProxyAuthType X . . . .
UrlPutProxyEnable X . . . .
UrlPutProxyHost X . . . .
UrlPutProxyPassword X . . . .
UrlPutProxyPort X . . . .
UrlPutProxyUsername X . . . .
UrlPutUrl X . . . .
UrlUsername X X . . .
The modes of operation described in this section are mutually
exclusive. In other words, only one mode may be specified per
invocation.
- {-e|--execute} {-f|--file} config[,config[...]] program [options]
-
Use config file settings to download and execute the specified
program. The config file (or comma delimited list of files), as
specified by the --file argument, may be the name of a regular file
or '-', which indicates that configuration information is to be read
from stdin. If a list of files is supplied, then the value for stdin
may only be specified once. Also, note that path information in the
target program argument is not allowed. The options, if any,
are bound to the target program at invocation. The various
config file controls and syntax rules are described in the
CONFIGURATION CONTROLS section of this document.
-
This mode of operation is roughly split into three stages: GET,
RUN, and PUT. Each stage provides an independent timer. In the event
that a given stage doesn't complete within the alloted time, WebJob
will attempt to abort gracefully. A timeout value of zero means there
is no time limit.
- {-g|--get-url} [{-f|--file} config[,config[...]]] url
-
Optionally use config file settings to download the specified
url. The config file (or comma delimited list of files), as
specified by the --file argument, may be the name of a regular file
or '-', which indicates that configuration information is to be read
from stdin. If a list of files is supplied, then the value for stdin
may only be specified once. The various config file controls and
syntax rules are described in the CONFIGURATION CONTROLS section of
this document.
-
URLs must use a scheme of http or https and satisfy the following
regular expression:
-
scheme://(user(:pass)?@)?host(:port)?(/path(\?query)?)?
- {-h|--hashsum} {-t|--type} digest file
-
Use digest algorithm, as specified by the --type argument, to
compute a hash of file. If file is specified as '-', then input
will be read from stdin. Currently, the following algorithms are
supported: MD5, SHA1, and SHA256. The digest value is not case
sensitive. In other words, 'md5' is equivalent to 'MD5'.
- {-r|--run-embedded} {lua|perl} [...]
-
Use the embedded Lua or Perl interpreter to execute or process the
lua or perl arguments, which can be combined in any way to form
valid command line expression. This mode of operation is only
available if embedded Lua or Perl support is compiled into the binary.
Note that only one embedded language can be compiled into a given
WebJob binary.
- {-v|--version}
-
Display version information and exit.
- --NoUpload
-
This option disables PUT requests (i.e., uploads) even if a PUT URL
has been specified via UrlPutUrl.
- --GetTimeLimit
-
This option is the command line counterpart to the GetTimeLimit
control, which is described in the CONFIGURATION CONTROLS Section. If
this value is specified in both the config file and on the command
line, the config file value will trump the the command line value.
Note that this behavior will likely be reversed in a future release.
- --PutTimeLimit
-
This option is the command line counterpart to the PutTimeLimit
control, which is described in the CONFIGURATION CONTROLS Section. If
this value is specified in both the config file and on the command
line, the config file value will trump the the command line value.
Note that this behavior will likely be reversed in a future release.
- --RunTimeLimit
-
This option is the command line counterpart to the RunTimeLimit
control, which is described in the CONFIGURATION CONTROLS Section. If
this value is specified in both the config file and on the command
line, the config file value will trump the the command line value.
Note that this behavior will likely be reversed in a future release.
- --TimeoutSignal
-
This option is the command line counterpart to the TimeoutSignal
control, which is described in the CONFIGURATION CONTROLS Section. If
this value is specified in both the config file and on the command
line, the config file value will trump the the command line value.
Note that this behavior will likely be reversed in a future release.
This section describes the various controls that WebJob recognizes.
In general, controls either shape runtime behavior or provide
information needed by the application to perform a specific function.
Controls and their values, one pair/line, are written to a file
having the following format.
<control> = <value>
All controls are case insensitive, but, in general, their values
are not. Comments may occur anywhere on a given line, and must
begin with a pound character (i.e., '#'). In any given line, all
text to the right of the first comment will be ignored. White
space surrounding controls and values is ignored.
This section describes each control that may be specified, defines
what values it may have, and states which modes of operation
recognize the control.
- ClientId: <client>
-
Applies to execute.
-
ClientId is required. It defines the identity of the client for
which the GET/PUT requests are being made. Typically, the ClientId
and UrlUsername are the same when basic authentication is enabled.
However, this is not a requirement.
- DsvCertificateTree: <directory>
-
Applies to execute.
-
DsvCertificateTree is optional. This control specifies a directory
that contains one or more PEM-encoded certificates whose public keys
may be used to verify the payload's signature -- invalid and
non-PEM-encoded files are silently ignored. Each certificate in the
certificate tree is used, in turn, to verify the payload's integrity.
The verification process does not stop until a match is found or all
certificates have been tried. Note that WebJob will not search for
certificates in sub-directories of the specified tree. This control
is ignored if DsvVerifySignature is disabled.
-
For more information on signing and verifying payloads refer to the
WebJob-DsvTool man page (i.e., webjob-dsvtool(1)). That document
also provides examples of how to create the necessary PEM-encoded keys
and certificates.
-
Note: If DsvCertificateTree does not exist or does not contain at
least one valid certificate, the program will abort.
-
Note: Digital Signature Verification (DSV) support is a compile-time
option, which is enabled by default. However, if DSV support is not
compiled into the binary, WebJob will abort with an error if you try
to use this (or any) DSV control in the config file. To determine if
DSV support is compiled into the binary, run the following command:
-
webjob --version
- DsvVerifySignature: [Y|N]
-
Applies to execute.
-
DsvVerifySignature is optional. When enabled ('Y' or 'y'), it
instructs the application to verify the signature of the requested
payload. Signatures are verified against one or more certificates
located in DsvCertificateTree. The default value is 'N'.
-
If this control is enabled and the payload's signature is missing or
invalid, WebJob will abort, and the downloaded payload will not be
executed. This helps to protect the client in the case where the
WebJob server can no longer be trusted (e.g., because it has been
compromised).
-
Note: Certificate chains and expiration dates are not checked in the
verification process. It is the responsibility of the WebJob
administrator to periodically manage (i.e., add, remove, update) any
certificates installed on the clients.
-
Note: Digital Signature Verification (DSV) support is a compile-time
option, which is enabled by default. However, if DSV support is not
compiled into the binary, WebJob will abort with an error if you try
to use this (or any) DSV control in the config file. To determine if
DSV support is compiled into the binary, run the following command:
-
webjob --version
- GetHookCommandLine: <command>
-
Applies to execute.
-
GetHookCommandLine is required when GetHookEnable is enabled.
The value of this control is a command or pipeline that will be
executed by the system's command interpreter once the requested
file has been downloaded. If GetHookSuffix is defined, a variant
of the target program is requested. This variant must be
transformed by the specified command to produce the target program.
If the command is successful, as defined by GetHookStatus, then
the target program will be executed as usual. Otherwise, the
target program will not be executed. The tokens '%tp', '%vp',
'%pid', and '%s' may be used as place holders in the specified
command line. When the hook is activated, these tokens are expanded
as the target program, variant program, current Process ID,
and suffix, respectively. The character '%' may be used to produce
a literal token. For example, '%%tp' would be expanded to '%tp'.
- GetHookEnable: [Y|N]
-
Applies to execute.
-
GetHookEnable is optional. When enabled ('Y' or 'y'), the caller
has the ability to interrupt or hook the normal course of events
by running a single system command. This mechanism provides users
with the ability veto or bless, as the case may be, subsequent
execution of the target program. The default value is 'N'.
- GetHookStatus: [0-255]
-
Applies to execute.
-
GetHookStatus is required when GetHookEnable is enabled. The
value of this control determines the meaning of success for the
hook. If the specified value matches the exit status as returned
by system(3), then the target program will be executed as usual.
Otherwise, the target program will not be executed.
- GetHookSuffix: <suffix>
-
Applies to execute.
-
GetHookSuffix is optional when GetHookEnable is enabled. The
hook suffix is appended to the target (program) name to create
a variant name. This variant name takes the place of the target
name in the GET request. To maintain order, the hook command must
transform the downloaded variant into the target program. If
this control is not defined, then the variant and target names will
be the same, and no transformation would be required.
- GetTimeLimit: [0-31536000]
-
Applies to execute.
-
GetTimeLimit is optional. The value of this control specifies
how long the program is willing wait before giving up on download
(i.e. GET) requests. If GetTimeLimit is non zero and the time
limit expires, the program will abort. If GetTimeLimit is not
set or is set to zero, no limit will be imposed. The default value
is zero.
- HashType: [md5|sha1]
-
Applies to execute.
-
HashType is optional. This control specifies the hash algorithm
that is to be used when calculating message digests. Currently, the
following algorithms are supported: MD5, SHA1, and SHA256. If not
specified, HashType defaults to MD5. This control is not case
sensitive.
- Import: <file>
-
Applies to execute and get-url.
-
Import is optional. When specified, the directives contained
in the file referenced by this control are included in the current
configuration. Multiple instances of this control are allowed per
file, and recursion is permitted up to three levels deep. Imports
may be specified using a relative path.
- OverwriteExecutable: [Y|N]
-
Applies to execute.
-
OverwriteExecutable is optional. When enabled ('Y' or 'y'),
existing executables in the specified TempDirectory will be
overwritten by subsequent GET requests. The default value is
'N'.
- Priority: [low|below_normal|normal|above_normal|high]
-
Applies to execute.
-
Priority is optional. Use it to set the priority of the target
process. The environment variable, WEBJOB_PRIORITY, trumps the value
of this control when set. No priority is set by default.
-
On UNIX platforms, the priority values listed above correspond to the
following priority levels:
-
low .............. 20
below_normal ..... 10
normal ........... 0
above_normal .....-10
high .............-20
-
On WINX platforms, the priority values listed above correspond to the
following priority classes:
-
low .............. IDLE_PRIORITY_CLASS
below_normal ..... BELOW_NORMAL_PRIORITY_CLASS
normal ........... NORMAL_PRIORITY_CLASS
above_normal ..... ABOVE_NORMAL_PRIORITY_CLASS
high ............. HIGH_PRIORITY_CLASS
-
Note: If the specified priority can't be set, the target process will
abort.
-
Note: If an environment variable is defined but exceeds the maximum
allowed length, it will be rejected.
- PutTimeLimit: [0-31536000]
-
Applies to execute.
-
PutTimeLimit is optional. The value of this control specifies
how long the program is willing wait before giving up on upload
(i.e. PUT) requests. If PutTimeLimit is non zero and the time
limit expires, the program will abort. If PutTimeLimit is not
set or is set to zero, no limit will be imposed. The default value
is zero.
- RunTimeLimit: [0-31536000]
-
Applies to execute.
-
RunTimeLimit is optional. The value of this control specifies
how long the program is willing wait for the target application to
finish. If RunTimeLimit is non zero and the time limit expires,
the program will kill the spawned process and abort. If RunTimeLimit
is not set or is set to zero, no limit will be imposed. The default
value is zero.
-
Note: Killing the spawned (i.e., child) process is done via
TerminateProcess() and kill() on Windows- and UNIX-based platforms,
respectively. This may result in orphaned grandkids in either case.
To help prevent this situation, proactively identify processes that
block or take a long time to run and make sure that they run as
kids. Using the shell's built-in exec or the family of exec() calls
can help in this regard.
- RunType: [linktest|snapshot]
-
Applies to execute.
-
RunType is optional. This control sets a corresponding flag
that classifies uploaded data as linktest or snapshot, The value of
this control does not affect the format or content of WebJob
output. When output is uploaded to a Web server, the value of
this control may be used to determine how the data is processed.
If not specified, RunType defaults to snapshot.
- SslBundledCAsFile: <file>
-
Applies to execute and get-url.
-
SslBundledCAsFile is required when SslVerifyPeerCert is
enabled. This control specifies the name of a PEM (Privacy Enhanced
Mail) encoded file that contains a bundled set of Certificate
Authority (CA) certificates. Any validated peer certificate that
is signed by one of these CAs will be accepted provided that the
SslMaxChainLength and SslExpectedPeerCN checks are also
satisfied. SslBundledCAsFile may be specified as a relative
path.
- SslExpectedPeerCN: <name>
-
Applies to execute and get-url.
-
SslExpectedPeerCN is required when SslVerifyPeerCert is
enabled. The value of this control represents the peer's expected
Common Name (CN). Conventionally, CNs are specified as fully
qualified domain names. This control eliminates the need to perform
a DNS lookup at the time of certificate validation. This, in turn,
may help to prevent attacks involving DNS spoofing.
- SslMaxChainLength: [1-10]
-
Applies to execute and get-url.
-
SslMaxChainLength is optional when SslVerifyPeerCert is
enabled. The value of this control determines how deep a certificate
chain may be before it is considered invalid. If not specified,
this control defaults to a value of 1.
- SslPassPhrase: <passphrase>
-
Applies to execute and get-url.
-
SslPassPhrase is optional when SslUseCertificate is enabled.
Its value, if specified, is used to decrypt the contents of the
client's private key file (see SslPrivateKeyFile). If a passphrase
is required to load the private key and this control has not been
set, the user will be prompted for one. Naturally, this will cause
problems for automated tasks, so keep that in mind.
- SslPrivateKeyFile: <file>
-
Applies to execute and get-url.
-
SslPrivateKeyFile is required when SslUseCertificate is
enabled. This control specifies the name of a PEM (Privacy Enhanced
Mail) encoded key file that can be used to sign SSL certificates.
SslPrivateKeyFile may be specified as a relative path.
- SslPublicCertFile: <file>
-
Applies to execute and get-url.
-
SslPublicCertFile is required when SslUseCertificate is
enabled. This control specifies the name of a PEM (Privacy Enhanced
Mail) encoded certificate to be used during SSL handshakes.
SslPublicCertFile may be specified as a relative path.
- SslUseCertificate: [Y|N]
-
Applies to execute and get-url.
-
SslUseCertificate is optional. When enabled ('Y' or 'y'), it
instructs the application to provide client side certificate
authentication, if necessary. The default value is 'N'.
- SslVerifyPeerCert: [Y|N]
-
Applies to execute and get-url.
-
SslVerifyPeerCert is optional. When enabled ('Y' or 'y'), it
instructs the application to verify the credentials of the peer
server. The default value is 'N'.
- StrictControls: [Y|N]
-
Applies to execute and get-url.
-
StrictControls is optional. When enabled ('Y' or 'y'), any unknown
or unsupported configuration controls will cause the program to abort.
When disabled ('N' or 'n'), any unknown or unsupported controls will
be silently ignored. The default value is 'N'.
-
Note: If you want to enforce strict controls within a single config
file (or chain of config files), StrictControls must be the first
control processed. You may also enable/disable this control multiple
times. This allows you to control the strictness of certain sections
or starting at a particular point in the entire configuration. Once
set, this control takes effect immediately.
-
Note: For versions 1.7.0 and below, this control was not supported,
and the default behavior was to abort on any unknown or unsupported
configuration control.
- TempDirectory: <directory>
-
Applies to execute.
-
TempDirectory is optional. This control specifies a directory
in which all temporary files are to be created. If TempDirectory
is not specified, the program attempts to obtain its value from the
environment. First, it looks for TMPDIR (TEMP for WIN32). Next,
it looks for TMP (all platforms). If a suitable environment variable
is not found, TempDirectory defaults to /tmp (\temp for WIN32).
TempDirectory may be specified as a relative path.
-
Note: If TempDirectory does not exist, the program will abort.
-
Note: If an environment variable is defined but exceeds the maximum
allowed length, it will be rejected.
- TimeoutSignal: <signal-number>
-
Applies to execute.
-
TimeoutSignal is optional. When the run timer expires (see
RunTimeLimit), the signal specified by this control is sent to
the spawned process. This control is rarely needed, and it should
be used with care. Sending the wrong signal to a child process may
not yield the desired result. However, there are cases when sending
the default signal (SIGKILL = 9) is not the preferred course of
action either. For example, you may want to run a tool that will
perform an orderly shutdown upon receipt of a particular signal
(e.g., SIGINT = 2).
-
Note: This control is UNIX-specific, and its value must be a valid
signal number (e.g., 1, 2, 9, etc.). Because the number of signals
can vary from OS to OS, you should check the man pages to determine
what values are valid for your systems. On many systems, the
following command will produce a mapping of signal names to their
assigned numbers:
-
$ kill -l
- UrlAuthType: [basic|none]
-
Applies to execute and get-url.
-
UrlAuthType is optional. It identifies what authentication
scheme to use when issuing HTTP/HTTPS requests. The value specified
by this control applies to any requests involving UrlGetUrl or
UrlPutUrl. When UrlAuthType is set to basic, user credentials
are base 64 encoded and incorporated into the request header. User
credentials specified in the URL take precedence over credentials
specified in the UrlUsername and UrlPassword controls. If
not specified, UrlAuthType defaults to none.
- UrlDownloadLimit: [0-100000000]
-
Applies to execute and get-url.
-
UrlDownloadLimit is optional. The value of this control specifies
how much data the client is willing to receive for any given
HTTP/HTTPS request. If UrlDownloadLimit is not set or is set
to zero, no limit will be imposed. The default value is zero.
-
Note: No limit, here, means that the Content-Length in the response
header can be any value up to 4 GB.
- UrlGetUrl: <url>
-
Applies to execute.
-
UrlGetUrl is required. It defines the scheme, user credentials,
host address, port, and CGI application to be used when making
requests. If a username/password pair is specified in the URL,
that pair takes precedence over the values specified by
UrlUsername/UrlPassword, if any. The tokens '%user' and
'%pass' may be used as place holders in the specified URL. Before
a GET/PUT request is issued, these tokens are replaced with the
actual values assigned to UrlUsername and UrlPassword,
respectively. URLs must use a scheme of http or https and satisfy
the following regular expression:
-
scheme://(user(:pass)?@)?host(:port)?/(path(\?query)?)?
-
Note: This has been relaxed to the following in version 1.7.0:
-
scheme://(user(:pass)?@)?host(:port)?(/path(\?query)?)?
- UrlPassword: <password>
-
Applies to execute and get-url.
-
UrlPassword is optional. It identifies the password to use when
accessing a remote Web server. The value specified by this control
is used in conjunction with UrlGetUrl and UrlPutUrl unless
those controls supply their own username/password pair.
- UrlPutUrl: <url>
-
Applies to execute.
-
UrlPutUrl is optional. It defines the scheme, user credentials,
host address, port, and CGI application to be used when making PUT
requests. If a username/password pair is specified in the URL, that
pair takes precedence over the values specified by
UrlUsername/UrlPassword, if any. In any event, user credentials
are only sent when basic authentication has been requested (See
UrlAuthType). UrlPutUrl uses the same syntax as UrlGetUrl.
- UrlUsername: <username>
-
Applies to execute and get-url.
-
UrlUsername is optional. It identifies the username to use when
accessing a remote Web server. The value specified by this control
is used in conjunction with UrlGetUrl and UrlPutUrl unless
those controls supply their own username/password pair.
- UnlinkExecutable: [Y|N]
-
Applies to execute.
-
UnlinkExecutable is optional. When enabled ('Y' or 'y'), the
downloaded executable is deleted before the program exits. The
default value is 'N'.
- UnlinkOutput: [Y|N]
-
Applies to execute.
-
UnlinkOutput is optional. When enabled ('Y' or 'y'), any output
files are unlinked before the program exits. This control has no
effect when output data is directed to stdout/stderr. The default
value is 'N'.
- UploadOnGetFailure: [Y|N]
-
Applies to execute.
-
UploadOnGetFailure is optional. When enabled ('Y' or 'y'), the
program will attempt to upload its results even if the GET fails.
Normally, attempting to upload results after a failed download is
futile (e.g., authentication failures). However, there may be times
when this behavior is desired. For example, suppose that UrlGetUrl
and UrlPutUrl point to different servers. In this scenario, a
GET failure does not imply that the PUT will fail too, so by
monitoring uploads, you could detect jobs that aren't functioning
properly. The default value for this control is 'N'.
-
Note: If multiple config files are specified on the command line, this
control only applies to the last one.
- UrlGetProxyAuthType: [basic|none]
-
Applies to execute and get-url.
-
UrlGetProxyAuthType is optional when UrlGetProxyEnable is
enabled. It identifies what authentication scheme to use when issuing
HTTP/HTTPS GET requests that must first pass through an outbound proxy
server. When UrlGetProxyAuthType is set to basic, user credentials
are base64 encoded and incorporated into the proxy request header. If
not specified, UrlGetProxyAuthType defaults to 'none'.
- UrlGetProxyEnable: [Y|N]
-
Applies to execute and get-url.
-
UrlGetProxyEnable is optional. When enabled ('Y' or 'y'), the
client submits its GET request to a proxy server that must be
specified using the UrlGetProxyHost control. All other controls in
the GET-Proxy group are optional. The default value is 'N'.
- UrlGetProxyHost: [<hostname>|<IP>]
-
Applies to execute and get-url.
-
UrlGetProxyHost is required when UrlGetProxyEnable is enabled.
The value of this control represents the proxy server's hostname or IP
address. Generally, hostnames should be specified as fully qualified
domain names.
- UrlGetProxyPassword: <password>
-
Applies to execute and get-url.
-
UrlGetProxyPassword is optional when UrlGetProxyEnable is
enabled. It identifies the password to use when issuing HTTP/HTTPS
GET requests that must first pass through an outbound proxy server.
- UrlGetProxyPort: <port>
-
Applies to execute and get-url.
-
UrlGetProxyHost is optional when UrlGetProxyEnable is enabled.
It identifies the port to use when issuing HTTP/HTTPS GET requests
that must first pass through an outbound proxy server. The default
value is '8080'.
- UrlGetProxyUsername: <username>
-
Applies to execute and get-url.
-
UrlGetProxyUsername is optional when UrlGetProxyEnable is
enabled. It identifies the username to use when issuing HTTP/HTTPS
GET requests that must first pass through an outbound proxy server.
- UrlPutProxyAuthType: [basic|none]
-
Applies to execute.
-
UrlPutProxyAuthType is optional when UrlPutProxyEnable is
enabled. It identifies what authentication scheme to use when issuing
HTTP/HTTPS PUT requests that must first pass through an outbound proxy
server. When UrlPutProxyAuthType is set to basic, user credentials
are base64 encoded and incorporated into the proxy request header. If
not specified, UrlPutProxyAuthType defaults to 'none'.
- UrlPutProxyEnable: [Y|N]
-
Applies to execute.
-
UrlPutProxyEnable is optional. When enabled ('Y' or 'y'), the
client submits its PUT request to a proxy server that must be
specified using the UrlPutProxyHost control. All other controls in
the PUT-Proxy group are optional. The default value is 'N'.
-
Note: All controls in the PUT-Proxy group will be ignored if
UrlPutUrl has not been defined.
- UrlPutProxyHost: [<hostname>|<IP>]
-
Applies to execute.
-
UrlPutProxyHost is required when UrlPutProxyEnable is enabled.
The value of this control represents the proxy server's hostname or IP
address. Generally, hostnames should be specified as fully qualified
domain names.
- UrlPutProxyPassword: <password>
-
Applies to execute.
-
UrlPutProxyPassword is optional when UrlPutProxyEnable is
enabled. It identifies the password to use when issuing HTTP/HTTPS
PUT requests that must first pass through an outbound proxy server.
- UrlPutProxyPort: <port>
-
Applies to execute.
-
UrlPutProxyHost is optional when UrlPutProxyEnable is enabled.
It identifies the port to use when issuing HTTP/HTTPS PUT requests
that must first pass through an outbound proxy server. The default
value is '8080'.
- UrlPutProxyUsername: <username>
-
Applies to execute.
-
UrlPutProxyUsername is optional when UrlPutProxyEnable is
enabled. It identifies the username to use when issuing HTTP/HTTPS
PUT requests that must first pass through an outbound proxy server.
Upon successful completion, a value of 0 (XER_OK) is returned.
Otherwise, one of the following error codes is returned:
-
1 = XER_Usage
-
2 = XER_Abort
-
3 = XER_BootStrap
-
4 = XER_ProcessArguments
-
5 = XER_ReadProperties
-
6 = XER_CheckDependencies
-
7 = XER_Configure
-
8 = XER_GetStage
-
9 = XER_RunStage
-
10 = XER_PutStage
-
11 = XER_MultiStage
-
12 = XER_GetUrl
-
13 = XER_CloneProperties
-
20 = XER_Cleanup
-
27 = XER_Cleanup_Configure (XER_Cleanup + XER_Configure)
-
28 = XER_Cleanup_GetStage (XER_Cleanup + XER_GetStage)
-
29 = XER_Cleanup_RunStage (XER_Cleanup + XER_RunStage)
-
30 = XER_Cleanup_PutStage (XER_Cleanup + XER_PutStage)
-
31 = XER_Cleanup_MultiStage (XER_Cleanup + XER_MultiStage)
If the command line does not conform to the required syntax, a value
of 1 (XER_Usage) is returned. A value of 2 (XER_Abort)
means the program encountered a critical or unexpected error and had
to abort immediately. The remaining exit codes roughly indicate which
subsystem encountered the fatal error. If the error code is
XER_MultiStage, then multiple stages (e.g., GET/RUN/PUT) failed.
- WebJob Configuration File
-
The config file specified on the command line contains a set of
controls that are used to configure WebJob's runtime behavior.
For more details on what controls may be specified, see the
CONFIGURATION CONTROLS section of this document.
- Target Program
-
The target program specified on the command line is retrieved from
the WebJob server and written to the local file system. The exact
location varies depending on the value of TempDirectory. When
UnlinkExecutable is enabled, the program is deleted before
WebJob exits. However, if WebJob aborts, this file may not be
deleted.
- WebJob Output Files (webjob_<time>_<nonce>[.d]/webjob.<suffix>)
-
The values for <time>, <nonce>, and <suffix> are the job epoch in
seconds, a random hex string used to help prevent name collisions and
guessing attacks, and one of {out,err,env}, respectively. The suffix
is used to associate a given file with its corresponding output stream
-- i.e., std{out,err,env}. When UrlPutUrl has been set, the
sub-process writes its data to the named out and err streams. Once
the sub-process has finished, WebJob writes information regarding
environment settings and exit status to the env stream. When
UnlinkOutput is enabled, these files are deleted after they have
been uploaded. However, if WebJob aborts or the upload fails,
these files may not be deleted. The ``.d'' extension is used to tag the
work directory as done. When WebJob starts, it creates an untagged
work directory. Before WebJob deletes any files and exits, it
attempts to tag (i.e., rename) the work directory as done. This is an
external indicator that the job has finished and it's OK to remove the
directory.
- Public Certificate and Private Key
-
When SslUseCertificate has been enabled, WebJob expects to
find certificate and key files in the locations specified by
SslPublicCert and SslPrivateKey controls, respectively. If
these files do not exist or have the proper format, WebJob will
abort.
- Bundled Certificate Authorities
-
When SslUseCertificate has been enabled, WebJob expects to
find a bundled certificate authorities file in the location
specified by SslBundledCAs control. If this file does not
exist or have the proper format, WebJob will abort.
The following examples are intended to demonstrate different ways
of configuring and using WebJob. Any text encapsulated between
'--- XXX ---' delimiters represents config file content.
This example demonstrates how to download and run ps(1) with output
being directed to stdout/stderr. It is assumed that a WebJob
server has already been configured to serve the appropriate ps(1)
binary for the system in question. Also, assume that the following
information has been provided.
ClientId = client_1
URL = http://webjob.server.net/cgi-client/nph-webjob.cgi
Username = client_1
Password = access
The simplest configuration file that will carry out this task is as
follows:
--- webjob.cfg ---
ClientId=client_1
UrlGetUrl=http://client_1:access@webjob.server.net/cgi-client/nph-webjob.cgi
UrlAuthType=basic
--- webjob.cfg ---
or, alternatively
--- webjob.cfg ---
ClientId=client_1
UrlGetUrl=http://webjob.server.net/cgi-client/nph-webjob.cgi
UrlUsername=client_1
UrlPassword=access
UrlAuthType=basic
--- webjob.cfg ---
Once the config file has been created, WebJob may be invoked as
follows:
webjob --execute --file webjob.cfg ps -aux
or in short form:
webjob -e -f webjob.cfg ps -aux
WebJob will also accept the config file via stdin as follows:
webjob -e -f - ps -aux < webjob.cfg
or
cat webjob.cfg | webjob -e -f - ps -aux
WebJob stores the target program in a job-specific directory
(webjob_<time>_<nonce>) under TempDirectory. In this case,
TempDirectory hasn't been specified, so data will be written to the
default TempDirectory, which is '/tmp' or '\temp' for UNIX or
Windows environments, respectively. To automatically remove the
target program and the sub-directory in which it resides after job
completion, enable UnlinkExecutable.
Suppose that you want to upload the output of ps(1) to a WebJob
server instead of stdout/stderr. Given that that the URL from the
previous example may be used for both GET and PUT requests, the new
config file becomes:
--- webjob.cfg ---
ClientId=client_1
UrlGetUrl=http://webjob.server.net/cgi-client/nph-webjob.cgi
UrlPutUrl=http://webjob.server.net/cgi-client/nph-webjob.cgi
UrlUsername=client_1
UrlPassword=access
UrlAuthType=basic
--- webjob.cfg ---
This job may be invoked as follows:
webjob -e -f webjob.cfg ps -aux
This time, WebJob will create three files, webjob.{out,err,env}, in
a job-specific directory (webjob_<time>_<nonce>) under
TempDirectory. These files contain the output data that will be
uploaded to the WebJob server in a PUT request. To automatically
remove these files after they have been uploaded, enable
UnlinkOutput. To automatically remove these files and the
sub-directory in which they reside, enable both UnlinkOutput and
UnlinkExecutable.
Suppose that you want to run netstat(1) over a secure, authenticated
channel (i.e., SSL with mutual certificate authentication). Suppose
you also want to delete the target program and its associated
output files upon job completion. Assuming that the WebJob server
is properly configured, then the information provided below is
sufficient to create the following config file.
GET/PUT URL = https://webjob.server.net:443/cgi-client/nph-webjob.cgi
SSL Public Cert File = /usr/local/webjob/ssl/mycrt.pem
SSL Private Key File = /usr/local/webjob/ssl/mykey.pem
SSL Bundled CAs File = /usr/local/webjob/ssl/CAs.pem
SSL PassPhrase = flimflam
Certificate Chain Length = 2
Web Resource Validates Client Certificates = Y
--- webjob.cfg ---
ClientId=client_1
UrlGetUrl=https://webjob.server.net:443/cgi-client/nph-webjob.cgi
UrlPutUrl=https://webjob.server.net:443/cgi-client/nph-webjob.cgi
UrlUsername=client_1
UrlPassword=access
UrlAuthType=basic
UnlinkOutput=Y
UnlinkExecutable=Y
SslUseCertificate=Y
SslPublicCertFile=/usr/local/webjob/ssl/mycrt.pem
SslPrivateKeyFile=/usr/local/webjob/ssl/mykey.pem
SslPassPhrase=flimflam
SslVerifyPeerCert=Y
SslBundledCAsFile=/usr/local/webjob/ssl/CAs.pem
SslExpectedPeerCN=webjob.server.net
SslMaxChainLength=2
--- webjob.cfg ---
This job may be invoked as follows:
webjob -e -f webjob.cfg netstat -an
To compute the MD5 hash of /bin/ps, run the following command:
webjob --hashsum --type md5 /bin/ps
or in short form:
webjob -h -t md5 /bin/ps
To compute the SHA1 hash of /bin/ps, run the following command:
webjob --hashsum --type sha1 /bin/ps
or in short form:
webjob -h -t sha1 /bin/ps
This example demonstrates how to download and run a gzipped copy of
ps(1). It is assumed that a WebJob server has already been
configured to serve unzipped and zipped copies of ps(1) as ps and
ps.gz, respectively.
The first step is to enable GetHook in the client's config file and
define the appropriate hook command, suffix, and return status. The
following config file snippet shows how this is done.
--- webjob.cfg ---
...
GetHookEnable=Y
GetHookCommandLine=gzip -d %vp
GetHookStatus=0
GetHookSuffix=.gz
...
--- webjob.cfg ---
In general, any valid command or pipeline may be specified for the
hook as long as it produces a valid ps executable. When the following
WebJob command is issued, the hook modifies the GET request such
that ps.gz is requested. After ps.gz has been downloaded, the hook
executes the specified command. In this case, '%vp' will be expanded
to 'ps.gz'. If the hook command returns a zero status, then the
operation is deemed successful and normal WebJob execution ensues.
If the return status is not zero, the target command will not be
executed.
webjob -e -f webjob.cfg ps -aux
This example demonstrates how to download and run a GPG signed copy of
ps(1). It is assumed that a WebJob server has already been
configured to serve unsigned or signed copies of ps(1) as ps or
ps.gpg, respectively.
The first step is to enable GetHook in the client's config file and
define the appropriate hook command, suffix, and return status. The
following config file snippet shows how this is done.
--- webjob.cfg ---
...
GetHookEnable=Y
GetHookCommandLine=gpg --batch --decrypt %vp > %tp
GetHookStatus=0
GetHookSuffix=.gpg
...
--- webjob.cfg ---
When the following WebJob command is issued, the hook modifies the
GET request such that ps.gpg is requested. After ps.gpg has been
downloaded, the hook executes the specified command. In this case,
'%tp' and '%vp' will be expanded to 'ps' and 'ps.gpg', respectively.
If the hook command returns a zero status, then the operation is
deemed successful and normal WebJob execution ensues. If the
return status is not zero, the target command will not be executed.
webjob -e -f webjob.cfg ps -aux
This example demonstrates how to log, using logger(1), the fact that
WebJob downloaded and attempted to run ps(1). It is assumed that a
WebJob server has already been configured to serve copies of ps(1)
as ps.
The first step is to enable GetHook in the client's config file and
define the appropriate hook command and return status. The following
config file snippet shows how this is done. Note how the suffix is
commented out -- it could be left blank also.
--- webjob.cfg ---
...
GetHookEnable=Y
GetHookCommandLine={ logger -p user.alert -t "webjob[%pid]" "%vp ---> %tp" ; true ; }
GetHookStatus=0
#GetHookSuffix=
...
--- webjob.cfg ---
When the following WebJob command is issued, the hook does not
modify the GET request because a suffix was not defined. After ps has
been downloaded, the hook executes the specified command. In this
case, '%tp', '%vp', and '%pid' will be expanded to 'ps', 'ps', and the
current Process ID (PID), respectively. The specified command always
returns a zero status due to true(1), so normal WebJob execution
ensues.
webjob -e -f webjob.cfg ps -aux
To download the file test.cfg located on www.somewhere.net and write
it to stdout, run the following command:
webjob --get-url http://www.somewhere.net/test.cfg
or in short form:
webjob -g http://www.somewhere.net/test.cfg
This example is similar to the previous one except that you must now
include a config file. The purpose of the config file is to enable
proxy support and to specify the server's address and port. Given
that the proxy server's address is 192.168.1.1, its port is 8888, and
it requires no authentication, the following config file would be
sufficient to perform the task:
--- webjob.cfg ---
UrlGetProxyEnable=Y
UrlGetProxyHost=192.168.1.1
UrlGetProxyPort=8888
--- webjob.cfg ---
To download the file test.cfg located on www.somewhere.net through the
specified proxy server and write it to a file called test.cfg on disk,
run the following command:
webjob --get-url --file webjob.cfg http://www.somewhere.net/test.cfg > test.cfg
or in short form:
webjob -g -f webjob.cfg http://www.somewhere.net/test.cfg > test.cfg
This example demonstrates how to use execute a simple Perl one-liner
script to URL-decode a string.
echo %74%65%73%74 | webjob -r perl -p -e 's/%([0-9A-Fa-f]{2})/pack("C", hex($1))/seg;'
This example demonstrates how to execute a simple Perl one-liner
script to generate a sequence of numbers between 0 and 255.
webjob -r perl -e 'foreach $i (0..255) { printf("%d\n", $i); }'
This example demonstrates how to execute a simple Perl one-liner
script to insert a comment above a line of selected text in a config
file called test.cfg.
webjob -r perl -p -i -e 's/^(DsvVerifySignature=.*)$/# No comment.\n$1/;' test.cfg
openssl(1), webjob-dsvtool(1)
Klayton Monroe
WebJob was initially written to assist Incident Response Handlers
in their efforts to investigate potentially compromised systems.
Often, these Handlers must work around the constraints imposed by
the surrounding environment. For example, lack of physical or shell
access, untrusted diagnostic programs, lack of encryption, many
machines in need of investigation, and so on. Therefore, I felt
that Handlers, or their eyes and ears in the field, needed an
efficient way to import and run known good diagnostic tools when
investigating live systems.
WebJob is lightweight, portable, and easy to use. This makes
it a good candidate for establishing a foothold on the target system.
Once there, all that is needed to begin diagnostics is a small
configuration file and access to a remote WebJob server. When
there are many machines to investigate, WebJob can be deployed
in parallel, and all harvested output can be directed to and
aggregated on a single server. This reduces the amount of manual
data processing involved in collecting, tagging, and storing evidence.
It also significantly reduces the amount time, effort, and resources
needed to arrive at a determination.
Message digest (i.e., --hashsum) support first appeared in WebJob
1.3.0.
GetHook and Jid (Job ID) support first appeared in WebJob 1.4.0.
Digital Signature Verification (DSV) support and server-side GET/PUT
triggers first appeared in WebJob 1.6.0.
Proxy support, get URL (i.e., --get-url) support, and dynamic
content via server-side GET hooks first appeared in WebJob 1.7.0.
Queueing support via Job Queue Directories (JQD) and embedded Perl
scripting (i.e., --run-embedded) first appeared in WebJob 1.8.0.
Embedded Lua scripting (i.e., --run-embedded) first appeared in
WebJob 1.9.0.
|
