This recipe demonstrates how to configure WebJob to download and execute lsof -- a tool that lists open files for running processes. In particular, the focus will be to use lsof to collect information on all active TCP and UDP sockets. This yields the same, no, more information than an external port scan and is far more efficient. Cooking with this recipe requires that you already have an operational WebJob server. If this is not the case, refer to the instructions provided in the README.INSTALL file that comes with the source distribution. While this recipe was written for and tested on a FreeBSD system, the concepts should be portable across UNIX environments that are supported by both WebJob and lsof. This recipe assumes that for a given client, say client_0001, you wish to periodically run lsof and have the results posted to the WebJob server. 1. Obtain or compile an lsof binary (version 4.65 or later is recommended) and verify that it works on the target system. Make sure to conduct verification tests as as root -- i.e., the same level of privilege that WebJob will need. The URL for lsof is: ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/lsof_4.65.tar.gz If the target system is FreeBSD, you can build and test lsof from the ports tree as shown below. It's not necessary to install lsof on the target system since it will ultimately be stored on the WebJob server. cd /usr/ports/sysutils/lsof make cd work/lsof_4.65/lsof_4.65_src ./lsof -h When testing lsof, be sure to try the commands provided below. The first will print TCP and UDP socket information in an easy-to-read format. The second will produce output intended for other programs. lsof -Di -n -P -itcp -iudp lsof -Di -n -P -itcp -iudp -FLucRptPTn Next, copy lsof to client_0001's commands directory. In general, the permissions on all programs and scripts in this directory should be set to octal mode 644. This helps prevent accidental execution and serves to remind you that these files should not be run on the server. The server should now have an integrity tree similar to the one shown here: integrity | - incoming | - logfiles | - profiles | - clientid_0001 | - baseline | - commands | - lsof 2. At this point, the WebJob server has been configured. Now, it's time to focus on the target system. Create upload.cfg, as shown below. Set URLGetURL, URLPutURL, URLUsername, and URLPassword as appropriate. Install this file on the target system in /usr/local/integrity/etc. If necessary, create the specified TempDirectory with: mkdir -m 755 -p /usr/local/integrity/run --- upload.cfg --- URLGetURL=https://your.webjob.server.net/cgi-webjob/nph-webjob.cgi URLPutURL=https://your.webjob.server.net/cgi-webjob/nph-webjob.cgi URLUsername=client_0001 URLPassword=password URLAuthType=basic RunType=snapshot OverwriteExecutable=Y UnlinkOutput=Y UnlinkExecutable=Y GetTimeLimit=0 RunTimeLimit=0 PutTimeLimit=0 URLDownloadLimit=10000000 TempDirectory=/usr/local/integrity/run --- upload.cfg --- sed -e '1,/^--- upload.cfg ---$/d; /^--- upload.cfg ---$/,$d' webjob-run-lsof-socket.txt > upload.cfg install -m 600 -o root -g wheel upload.cfg /usr/local/integrity/etc 3. At this point, you are ready to test everything out. To start, run lsof so that its output is written to stdout. To do this, comment out URLPutURL in upload.cfg and invoke WebJob as follows: webjob -e -f upload.cfg lsof -Di -n -P -itcp -iudp If the command is successful, you should see output similar to that shown here: --- output --- COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslogd 77 root 4u IPv6 0xc85a3ec0 0t0 UDP *:514 syslogd 77 root 5u IPv4 0xc85a3e00 0t0 UDP *:514 sendmail 91 root 3u IPv4 0xc8601c60 0t0 TCP 127.0.0.1:25 (LISTEN) sshd 8710 root 5u IPv4 0xc8601600 0t0 TCP 10.0.1.1:22->10.0.1.2:4781 (ESTABLISHED) sshd 31005 root 3u IPv6 0xc8600d80 0t0 TCP *:22 sshd 31005 root 4u IPv4 0xc8601e80 0t0 TCP *:22 (LISTEN) --- output --- If your copy of lsof was compiled for a different version of the same OS, there's a good chance that it won't work properly. If you suspect this, look for a warning message such as the one shown here: --- output --- lsof: WARNING: compiled for FreeBSD release 4.6.1-RELEASE-p10; this is 4.3-RELEASE. --- output --- Now, try to run WebJob with the output being uploaded to the server. To do this, uncomment URLPutURL and rerun the previous command. If it is successful, you should find four files on the server in its incoming directory. The files listed here will give you an idea of what to look for: client_0001_20021119145123_lsof.env client_0001_20021119145123_lsof.err client_0001_20021119145123_lsof.out client_0001_20021119145123_lsof.rdy Inspect these files. Their content should be similar to that shown below. The .rdy file is simply a lock release mechanism. Therefore, it's content has not been listed. If lsof succeeds, the .err file is expected to be empty. --- client_0001_20021119145123_lsof.env --- CommandLine=lsof -Di -n -P -itcp -iudp JobPid=90251 KidPid=90252 KidStatus=0 KidSignal=0 KidReason=The kid exited cleanly. Hostname=your.target.system.net SystemOS=i386 FreeBSD 4.6.1-RELEASE-p10 JobEpoch=2002-11-19 15:53:56 GMT (1037721236.101192) GetEpoch=2002-11-19 15:53:56 GMT (1037721236.112715) RunEpoch=2002-11-19 15:53:56 GMT (1037721236.351608) PutEpoch=2002-11-19 15:53:56 GMT (1037721236.378183) --- client_0001_20021119145123_lsof.env --- --- client_0001_20021119145123_lsof.err --- --- client_0001_20021119145123_lsof.err --- --- client_0001_20021119145123_lsof.out --- COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslogd 77 root 4u IPv6 0xc85a3ec0 0t0 UDP *:514 syslogd 77 root 5u IPv4 0xc85a3e00 0t0 UDP *:514 sendmail 91 root 3u IPv4 0xc8601c60 0t0 TCP 127.0.0.1:25 (LISTEN) sshd 8710 root 5u IPv4 0xc8601600 0t0 TCP 10.0.1.1:22->10.0.1.2:4781 (ESTABLISHED) sshd 31005 root 3u IPv6 0xc8600d80 0t0 TCP *:22 sshd 31005 root 4u IPv4 0xc8601e80 0t0 TCP *:22 (LISTEN) --- client_0001_20021119145123_lsof.out --- 4. The next step is to modify lsof's command-line options so that its output can be processed more efficiently by other programs. The new command looks like this: webjob -e -f upload.cfg lsof -Di -n -P -itcp -iudp -FLucRptPTn The '-Di' argument instructs lsof to ignore the device cache file. The '-n' and '-P' arguments instruct lsof skip IP address and port name resolution. The '-itcp' and '-iudp' arguments instruct lsof to select data associated with TCP and UDP sockets. The flags tacked onto the '-F' argument carry the following meanings: L process login name u process user ID c process command name R parent process ID p process ID t file type P protocol name T TCP/TPI information n file name, comment, Internet address Running this command should produce output similar to that shown here: --- client_0001_20021119145438_lsof.env --- CommandLine=lsof -Di -n -P -itcp -iudp -FLucRptPTn JobPid=90256 KidPid=90257 KidStatus=0 KidSignal=0 KidReason=The kid exited cleanly. Hostname=your.target.system.net SystemOS=i386 FreeBSD 4.6.1-RELEASE-p10 JobEpoch=2002-11-19 15:57:10 GMT (1037721430.334340) GetEpoch=2002-11-19 15:57:10 GMT (1037721430.345317) RunEpoch=2002-11-19 15:57:10 GMT (1037721430.584356) PutEpoch=2002-11-19 15:57:10 GMT (1037721430.611326) --- client_0001_20021119145438_lsof.env --- --- client_0001_20021119145438_lsof.err --- --- client_0001_20021119145438_lsof.err --- --- client_0001_20021119145438_lsof.out --- p77 R1 csyslogd u0 Lroot tIPv6 PUDP n*:514 tIPv4 PUDP n*:514 p91 R1 csendmail u0 Lroot tIPv4 PTCP n127.0.0.1:25 TST=LISTEN TQR=0 TQS=0 p8710 R31005 csshd u0 Lroot tIPv4 PTCP n10.0.1.1:22->10.0.1.2:4781 TST=ESTABLISHED TQR=0 TQS=84 p31005 R1 csshd u0 Lroot tIPv6 PTCP n*:22 tIPv4 PTCP n*:22 TST=LISTEN TQR=0 TQS=0 --- client_0001_20021119145438_lsof.out --- As you can see, the lsof output has gone vertical, and every line is prefixed with a type identifier -- i.e. the first character on each line. The script lsof-socket-out2csv.pl, included below, provides an example of how to parse this data and convert it into a delimited format. Note, however, that this script only works for data collected precisely as follows: lsof -Di -n -P -itcp -iudp -FLucRptPTn Executing this script on client_0001_20021119145438_lsof.out should produce the output listed below. Your mileage with the script may vary, so be ready to dig into the code and tweak it as needed. lsof-socket-out2csv.pl -h -f client_0001_20021119145438_lsof.out > lsof.csv --- lsof.csv --- login|uid|command|ppid|pid|type|protocol|state|src_ip|src_port|dst_ip|dst_port root|0|syslogd|1|77|IPv6|UDP||*|514|| root|0|syslogd|1|77|IPv4|UDP||*|514|| root|0|sendmail|1|91|IPv4|TCP|LISTEN|127.0.0.1|25|| root|0|sshd|31005|8710|IPv4|TCP|ESTABLISHED|10.0.1.1|22|10.0.1.2|4781 root|0|sshd|1|31005|IPv6|TCP||*|22|| root|0|sshd|1|31005|IPv4|TCP|LISTEN|*|22|| --- lsof.csv --- If necessary, convince yourself that this output is consistent with that listed in client_0001_20021119145123_lsof.out. --- lsof-socket-out2csv.pl --- #!/usr/bin/perl -w ###################################################################### # # $RecipeId: lsof_socket_out2csv,v 1.4 2005/09/13 06:57:05 klm Exp $ # ###################################################################### # # Copyright 2002-2005 The WebJob Project, All Rights Reserved. # ###################################################################### # # Purpose: Convert lsof socket data to a delimited format. # ###################################################################### use strict; use File::Basename; use Getopt::Std; ###################################################################### # # Main Routine # ###################################################################### my $sProgram = basename(__FILE__); #################################################################### # # Field variables for 'lsof -Di -n -P -itcp -iudp -FLucRptPTn'. # #################################################################### my $sRequiredCodes = "LucRptPn"; my $sOptionalCodes = "T"; my $sFieldCodes = "L|u|c|R|p|t|P|TST|nsa|nsp|nda|ndp"; my %hFieldNames = ( 'L' => "login", 'P' => "protocol", 'R' => "ppid", 'T' => "tcp_tpi", 'TQR' => "rqueue", 'TQS' => "wqueue", 'TST' => "state", 'TWR' => "rwindow", 'TWW' => "wwindow", 'c' => "command", 'g' => "pgid", 'n' => "addresses", 'nda' => "dst_ip", 'ndp' => "dst_port", 'nsa' => "src_ip", 'nsp' => "src_port", 'p' => "pid", 't' => "type", 'u' => "uid" ); my $sDelimiterRegex = qq(^[ ,:|]\$); my $sNRegex = qq(^([\\d\\.\\*]+):(\\*|\\d+)(->([\\d\\.\\*]+):(\\*|\\d+))?\$); my $sNRegex6 = qq(^\\[([0-9A-Fa-f:]+)\\]:(\\*|\\d+)(->(\\[[0-9A-Fa-f:]+\\]):(\\*|\\d+))?\$); my $sTRegex = qq(^(ST|QR|QS|WR|WW)=(.+)\$); #################################################################### # # Get Options. # #################################################################### my %hOptions; if (!getopts('d:f:h', \%hOptions)) { Usage($sProgram); } #################################################################### # # A Delimiter, '-d', is optional. Default value is '|'. # #################################################################### my $sDelimiter = (exists $hOptions{'d'}) ? $hOptions{'d'} : "|"; if ($sDelimiter !~ /$sDelimiterRegex/) { print STDERR "$sProgram: Delimiter='$hOptions{'d'}' Regex='$sDelimiterRegex' Error='Unsupported delimiter.'\n"; exit(2); } #################################################################### # # A Filename, '-f', is required. It can be '-' or a regular file. # #################################################################### my ($sFileHandle); if (!exists $hOptions{'f'}) { Usage($sProgram); } else { my $sFilename = $hOptions{'f'}; if (-f $sFilename) { if (!open(IN, $sFilename)) { print STDERR "$sProgram: Filename='$sFilename' Error='$!'\n"; exit(2); } $sFileHandle = \*IN; } else { if ($sFilename ne '-') { print STDERR "$sProgram: Filename='$sFilename' Error='File not found.'\n"; exit(2); } $sFileHandle = \*STDIN; } } #################################################################### # # A PrintHeader flag, '-h', is optional. # #################################################################### my $sPrintHeader = (exists $hOptions{'h'}) ? 1 : 0; #################################################################### # # If any arguments remain in the array, it's an error. # #################################################################### if (scalar(@ARGV) > 0) { Usage($sProgram); } #################################################################### # # Print header line, if requested. # #################################################################### if ($sPrintHeader) { my @aValues; foreach my $sKey (split(/\|/, $sFieldCodes)) { push(@aValues, $hFieldNames{$sKey}); } print join($sDelimiter, @aValues),"\n"; } #################################################################### # # Loop over the input. # #################################################################### my ($sLine, $sLineNumber, $sPidCount, %hObservedCodes, %hRecordLoners, %hRecordValues, $sTypeCount); for ($sLineNumber = 1, $sPidCount = $sTypeCount = 0; $sLine = <$sFileHandle>; $sLineNumber++) { chomp($sLine); my ($sToken, $sValue) = $sLine =~ /^(.)(\S+)\s*$/; if ($sToken !~ /^[$sRequiredCodes]|[$sOptionalCodes]$/) { print STDERR "$sProgram: LineNumber='$sLineNumber' Line='$sLine' Error='Unknown field code.'\n"; exit(2); } $hObservedCodes{$sToken}++; if ($sToken eq "p") { if ($sPidCount++ >= 1) { my $sMissingCodes = CheckFieldCodes($sRequiredCodes, \%hObservedCodes); if ($sMissingCodes ne "") { print STDERR "$sProgram: MissingCodes='$sMissingCodes' Error='Missing one or more field codes.'\n"; exit(2); } PrintDelimited($sDelimiter, $sFieldCodes, \%hRecordValues); undef %hRecordValues; undef %hRecordLoners; $sTypeCount = 0; } $hRecordValues{$sToken} = $sValue; } elsif ($sToken eq "t") { if ($sTypeCount++ == 0) { %hRecordLoners = %hRecordValues; } else { my $sMissingCodes = CheckFieldCodes($sRequiredCodes, \%hObservedCodes); if ($sMissingCodes ne "") { print STDERR "$sProgram: MissingCodes='$sMissingCodes' Error='Missing one or more field codes.'\n"; exit(2); } PrintDelimited($sDelimiter, $sFieldCodes, \%hRecordValues); %hRecordValues = %hRecordLoners; } $hRecordValues{$sToken} = $sValue; } elsif ($sToken eq "T") { if ($sValue !~ /$sTRegex/) { print STDERR "$sProgram: LineNumber='$sLineNumber' Line='$sLine' Error='Invalid TCP/TPI data.'\n"; exit(2); } $sToken = "T" . $1; $sValue = $2; $hRecordValues{$sToken} = $sValue; } elsif ($sToken eq "n") { if ($sValue =~ /$sNRegex/) { $hRecordValues{'nsa'} = $1; $hRecordValues{'nsp'} = $2; $hRecordValues{'nda'} = $4; $hRecordValues{'ndp'} = $5; } elsif ($sValue =~ /$sNRegex6/) { $hRecordValues{'nsa'} = $1; $hRecordValues{'nsp'} = $2; $hRecordValues{'nda'} = $4; $hRecordValues{'ndp'} = $5; } else { print STDERR "$sProgram: LineNumber='$sLineNumber' Line='$sLine' Error='Invalid address data.'\n"; exit(2); } } else { $hRecordValues{$sToken} = $sValue; } } PrintDelimited($sDelimiter, $sFieldCodes, \%hRecordValues); 1; ###################################################################### # # CheckFieldCodes # ###################################################################### sub CheckFieldCodes { my ($sRequiredCodes, $phObservedCodes) = @_; my $sMissingCodes = ""; foreach my $sKey (split(//, $sRequiredCodes)) { if (!exists $$phObservedCodes{$sKey}) { $sMissingCodes .= $sKey; } } return $sMissingCodes; } ###################################################################### # # PrintDelimited # ###################################################################### sub PrintDelimited { my ($sDelimiter, $sFieldCodes, $phRecordValues) = @_; my @aValues; foreach my $sKey (split(/\|/, $sFieldCodes)) { push(@aValues, (defined $$phRecordValues{$sKey}) ? $$phRecordValues{$sKey} : ""); } print join($sDelimiter, @aValues),"\n"; } ###################################################################### # # Usage # ###################################################################### sub Usage { my ($sProgram) = @_; print STDERR "\n"; print STDERR "Usage: $sProgram [-d delimiter] [-h] -f {file|-}\n"; print STDERR "\n"; exit(1); } --- lsof-socket-out2csv.pl --- sed -e '1,/^--- lsof-socket-out2csv.pl ---$/d; /^--- lsof-socket-out2csv.pl ---$/,$d' webjob-run-lsof-socket.txt > lsof-socket-out2csv.pl 5. Once you are satisfied that all is well, add the following cron job to the target system's crontab -- this job runs on the hour. --- crontab.hourly --- 0 * * * * /usr/local/integrity/bin/webjob -e -f /usr/local/integrity/etc/upload.cfg lsof -Di -n -P -itcp -iudp -FLucRptPTn > /dev/null 2>&1 --- crontab.hourly ---