Problem This recipe demonstrates how to harvest UNIX process information and generate reports when predetermined logical conditions are satisfied. Reports may be directed to stdout or a set of email addresses. Requirements Cooking with this recipe requires an operational WebJob server. If you do not have one of those, refer to the instructions provided in the README.INSTALL file that comes with the source distribution. The latest source distribution is available here: http://sourceforge.net/project/showfiles.php?group_id=40788 Each client system must be running UNIX and have at least the following utilities: crontab, ps, sh, and webjob. Likewise, the server must have at least the following utilities: awk, basename, cat, egrep, find, gzip, head, mail, mv, rm, sh, test, and wc. The commands presented throughout this recipe were designed to be executed within a Bourne shell (i.e. sh or bash). Solution The solution is to harvest process (ps) information using webjob and harvest_ps on the client. The uploaded output is subsequently processed on the server using process_ps. The following steps describe how to implement this solution. 1. Set WEBJOB_CLIENT and WEBJOB_COMMANDS as appropriate for your server. Next, extract harvest_ps from this recipe, and move it to the appropriate commands directory. If you want harvest_ps to be bound to a particular host, WEBJOB_CLIENT should be set to the appropriate client ID (e.g., client_0001) before running the following commands. Once harvest_ps is in place, set its ownership and permissions to 0:0 and mode 644, respectively. $ export WEBJOB_CLIENT=common $ export WEBJOB_COMMANDS=/integrity/profiles/${WEBJOB_CLIENT}/commands $ sed -e '1,/^--- harvest_ps ---$/d; /^--- harvest_ps ---$/,$d' webjob-monitor-ps.txt > harvest_ps $ mv harvest_ps ${WEBJOB_COMMANDS} $ chown 0:0 ${WEBJOB_COMMANDS}/harvest_ps $ chmod 644 ${WEBJOB_COMMANDS}/harvest_ps 2. Set WEBJOB_HOME as appropriate for your WebJob server. This variable should be set to the prefix directory that was used when WebJob was installed (e.g., /usr/local/integrity). $ export WEBJOB_HOME=/usr/local/integrity 3. Extract process_ps from this recipe, and move it to the server's ${WEBJOB_HOME}/bin directory. Set this file's ownership and permissions to 0:0 and mode 755, respectively. $ sed -e '1,/^--- process_ps ---$/d; /^--- process_ps ---$/,$d' webjob-monitor-ps.txt > process_ps $ mv -f process_ps ${WEBJOB_HOME}/bin $ chown 0:0 ${WEBJOB_HOME}/bin/process_ps $ chmod 755 ${WEBJOB_HOME}/bin/process_ps Next, extract process_ps.cfg from this recipe, and move it to the server's ${WEBJOB_HOME}/etc directory. Set this file's ownership and permissions to 0:0 and mode 644, respectively. $ sed -e '1,/^--- process_ps.cfg ---$/d; /^--- process_ps.cfg ---$/,$d' webjob-monitor-ps.txt > process_ps.cfg $ mv process_ps.cfg ${WEBJOB_HOME}/etc $ chmod 644 ${WEBJOB_HOME}/etc/process_ps.cfg $ chown 0:0 ${WEBJOB_HOME}/etc/process_ps.cfg 4. The config file, process_ps.cfg, contains various site- and report-specific variables. These variables control what reports are produced and how they are created. NOTE: You'll need to modify process_ps.cfg to meet your reporting needs. Two site-specific variables, process_ps_dropzone and process_ps_event_list, control the general operation of process_ps. These variables are described here: process_ps_dropzone This variable specifies the location on the server where harvest_ps output is initially stored by nph-webjob.cgi. Typically, this variable would be initialized as follows: process_ps_dropzone="/integrity/incoming" process_ps_event_list This variable is a space delimited list of events to be monitored. An event can be given any name, but using a single short word is recommended. For each event listed in process_ps_event_list, there are several report-specific variables that control how reports are created. These variables are described here: count_ This variable is an integer. It's used in conjunction with the corresponding logic_. email_ This variable is a space delimited list of email addresses that will be sent reports. hosts_ This variable is a space delimited list of hostnames that are to be monitored for the given event. logic_ This variable determines how ${ACTUAL_COUNT}, as calculated by process_ps, is compared to count_. It must be set to one of the following conditionals: Value Meaning ----- ------------------------ -eq equal to -ne not equal to -lt less than -le less than or equal to -gt greater than -ge greater than or equal to regex_ This variable is a regular expression that will be passed to egrep. title_ This variable is a string that summarizes the event. For each event defined in process_ps_event_list, process_ps counts the number of regex_s in each output file and produces reports based on the outcome of a predefined test. This sequence can be summarized, in pseudo code, as follows: ACTUAL_COUNT = number of regex_s in output file if [ ACTUAL_COUNT logic_ count_ ] then produce report fi For example, the following variables will cause process_ps to produce reports when the number of apache daemons (httpd) falls below 5 and sshd is not running as root with a PID of 87: process_ps_dropzone="/integrity/incoming/client_*/harvest_ps" process_ps_event_list="apache sshd" count_apache="5" email_apache="root@localhost" hosts_apache="client_1" logic_apache="-lt" regex_apache="httpd" title_apache="httpd: too few daemons" count_sshd="1" email_sshd="root@localhost" hosts_sshd="client_1" logic_sshd="-ne" regex_sshd="root +87 .*sshd" title_sshd="sshd: owner/pid out of spec" 5. Create a crontab entry on each client that periodically executes harvest_ps. Be sure to substitute the correct path for WEBJOB_HOME prior to committing the new crontab. For example, the following entry would execute harvest_ps hourly: 0 * * * * ${WEBJOB_HOME=/usr/local/integrity}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg harvest_ps -ef Here, the arguments supplied to harvest_ps were intended for use on a Solaris system. Depending on the platform and type of information you wish to collect, it may be necessary to supply different arguments to harvest_ps. For example, the following ps invocations are approximately equivalent: FreeBSD ps -ajx HP-UX ps -ef Linux ps -ef Solaris ps -ef 6. Create a crontab entry on the server that periodically executes process_ps. To ensure that each job is processed in a timely fashion and that no significant back-logging occurs, use a processing frequency that is slightly offset from and at least twice as fast as harvest_ps. Following the example above, we will execute process_ps every 30 minutes. This job is configured to write its output to mail and tag incoming files as done (i.e. add .d suffix) after they have been processed. 10,40 * * * * ${WEBJOB_HOME=/usr/local/integrity}/bin/process_ps tag mail Closing Remarks Rather than running process_ps from cron, it would be better to create a daemon that proactively manages the incoming directory. This daemon would be responsible for processing incoming files and invoking the proper follow-on analysis stages such as process_ps. Credits This recipe was brought to you by Andy Bair and Klayton Monroe, August 2003. Appendix 1 --- harvest_ps --- #!/bin/sh IFS=' ' PATH=/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin ps $* --- harvest_ps --- Appendix 2 --- process_ps.cfg --- process_ps_dropzone="/integrity/incoming/client_*/harvest_ps" process_ps_event_list="apache sshd sendmail" count_apache="5" email_apache="root@localhost" hosts_apache="client_1" logic_apache="-lt" regex_apache="httpd" title_apache="httpd: too few daemons" count_sshd="2" email_sshd=""root@localhost" hosts_sshd="client_1 client_2" logic_sshd="-ne" regex_sshd="root +268 .*sshd" title_sshd="sshd: owner/pid out of spec" count_sendmail="1" email_sendmail="root@localhost" hosts_sendmail="client_1 client_2" logic_sendmail="-ne" regex_sendmail="root +223 .*sendmail" title_sendmail="sendmail: owner/pid out of spec" --- process_ps.cfg --- --- process_ps --- #!/bin/sh IFS=' ' PATH=/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin PROGRAM=`basename $0` if [ $# -ne 2 ] ; then echo 1>&2 echo "Usage: ${PROGRAM} {remove|tag|ztag} {mail|stdout}" 1>&2 echo 1>&2 exit 1 else case "$1" in remove|REMOVE) ACTION=remove ;; tag|TAG) ACTION=tag DOGZIP=no ;; ztag|ZTAG) ACTION=tag DOGZIP=yes ;; *) echo "${PROGRAM}: Error='Argument one must be one of {remove|tag|ztag}.'" 1>&2 exit 1 ;; esac case "$2" in mail|MAIL) OUTPUT='mail -s "${EVENT}" ${EMAIL}' ;; stdout|STDOUT) OUTPUT='cat -' ;; *) echo "${PROGRAM}: Error='Argument tow must be one of {mail|stdout}.'" 1>&2 exit 1 ;; esac fi WEBJOB_HOME=${WEBJOB_HOME-/usr/local/integrity} if [ -r ${WEBJOB_HOME}/etc/process_ps.cfg ] ; then . ${WEBJOB_HOME}/etc/process_ps.cfg else echo "${PROGRAM}: Error='${WEBJOB_HOME}/etc/process_ps.cfg does not exist or is unreadable.'" 1>&2 exit 1 fi if [ -z "${process_ps_dropzone}" ] ; then echo "${PROGRAM}: Error='process_ps_dropzone is not defined'" 1>&2 exit 1 fi if [ -z "${process_ps_event_list}" ] ; then echo "${PROGRAM}: Error='process_ps_event_list is not defined'" 1>&2 exit 1 fi for RDY_FILE in `find ${process_ps_dropzone} -name "*.rdy" 2> /dev/null` ; do ENV_FILE=${RDY_FILE%rdy}env ERR_FILE=${RDY_FILE%rdy}err OUT_FILE=${RDY_FILE%rdy}out for FILE in "${ENV_FILE}" "${ERR_FILE}" "${OUT_FILE}" ; do if [ ! -r "${FILE}" ] ; then echo "${PROGRAM}: Error='${FILE} does not exist or is unreadable.'" 1>&2 exit 1 fi done HOSTNAME=`egrep "Hostname=" ${ENV_FILE} | awk -F= '{print $2}'` for ITEM in ${process_ps_event_list} ; do eval COUNT=\$count_${ITEM} eval EMAIL=\$email_${ITEM} eval HOSTS=\$hosts_${ITEM} eval LOGIC=\$logic_${ITEM} eval REGEX=\$regex_${ITEM} eval TITLE=\$title_${ITEM} if [ "${COUNT}" = "" -o "${EMAIL}" = "" -o "${HOSTS}" = "" -o "${LOGIC}" = "" -o "${REGEX}" = "" -o "${TITLE}" = "" ] ; then echo "${PROGRAM}: Error='Missing one or more variables.'" 1>&2 echo " COUNT: ${COUNT}" 1>&2 echo " EMAIL: ${EMAIL}" 1>&2 echo " HOSTS: ${HOSTS}" 1>&2 echo " LOGIC: ${LOGIC}" 1>&2 echo " REGEX: ${REGEX}" 1>&2 echo " TITLE: ${TITLE}" 1>&2 echo " Unable to process ${OUT_FILE}" 1>&2 exit 1 fi for HOST in ${HOSTS} ; do if [ "${HOST}" = "${HOSTNAME}" ] ; then ACTUAL_COUNT=`egrep "${REGEX}" ${OUT_FILE} | wc -l | awk '{print $1}'` if [ ${ACTUAL_COUNT} ${LOGIC} ${COUNT} ] ; then EVENT="${HOST}: ${TITLE}" { echo "=================================================================================" ; \ echo ; \ echo " EVENT: ${EVENT}" ; \ echo " GROUP: "`basename ${OUT_FILE} .out`".{env,err,out.rdy}" ; \ echo " REGEX: ${REGEX}" ; \ echo "TARGET_COUNT: ${COUNT}" ; \ echo "ACTUAL_COUNT: ${ACTUAL_COUNT}" ; \ echo " LOGIC: ${ACTUAL_COUNT} ${LOGIC} ${COUNT}" ; \ echo ; \ echo "--- EGREP OUTPUT ---" ; \ head -1 ${OUT_FILE} && egrep "${REGEX}" ${OUT_FILE} ; \ echo "--- EGREP OUTPUT ---" ; \ echo ; \ echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=" ; \ } | eval ${OUTPUT} fi fi done done if [ "${ACTION}" = "tag" -o ] ; then for FILE in "${RDY_FILE}" "${ENV_FILE}" "${ERR_FILE}" "${OUT_FILE}" ; do mv ${FILE} ${FILE}.d if [ "${DOGZIP}" = "yes" ] ; then gzip ${FILE}.d fi done else rm -f ${RDY_FILE} ${ENV_FILE} ${ERR_FILE} ${OUT_FILE} fi done --- process_ps ---