Revision

  $Id: webjob-process-uptime.base,v 1.8 2010/07/02 15:30:12 klm Exp $

Purpose

  This recipe demonstrates how to preprocess harvest_uptime data, load
  it into a Round Robin Database (RRD), and create a browsable set of
  HTML reports and graphs that depict average system loads for 1, 5,
  and 15 minute intervals.

Motivation

  Graphing harvest_uptime data makes it possible for you to visualize
  the health of your systems.  This can reveal trends or relationships
  that might otherwise be hard to identify.  It can also serve as an
  early warning indicator and as an aide for planning distributed
  jobs.

Requirements

  This recipe assumes that you have read and implemented the following
  recipe:

    http://webjob.sourceforge.net/Files/Recipes/webjob-harvest-uptime.txt

  Your server must be running UNIX and have Apache, Perl, and RRDTool
  (1.2.11 or higher) installed.

Time to Implement

  Assuming that you have satisfied all the requirements/prerequisites,
  this recipe should take less than one hour to implement.

Solution

  The following steps describe how to implement this recipe.

  1. Extract the script at the bottom of this recipe, and install it
     in /usr/local/bin.  Once the file is in place, set its ownership
     and permissions to 0:0 and 755, respectively.

       # sed -e '1,/^--- process_uptime ---$/d; /^--- process_uptime ---$/,$d' webjob-process-uptime.txt > process_uptime
       # cp process_uptime /usr/local/bin
       # chmod 755 /usr/local/bin/process_uptime
       # chown 0:0 /usr/local/bin/process_uptime

  2. Read the documentation provided in the script.  This can be done
     using perldoc as follows:

       $ perldoc process_uptime

  3. Locate your base input directory and make sure you have some
     data to process.  If you implemented the recipe cited above as
     written, then this directory should be located here:

       /var/rsync.all/uptime

     and it should have subdirectories and files that match the
     following format:

       /var/rsync.all/uptime/<hostname>/<YYYY-MM-DD>.out

     Next, decide where your base output directory will reside.  This
     recipe assumes the following:

       /usr/local/www/data/uptime

  4. Preprocess your data, load it into one or more RRDs, and create
     the HTML reports.  This can be done as follows:

       # process_uptime -c V -i /var/rsync.all/uptime -o /usr/local/www/data/uptime

     This should create the following directories under your base
     output directory:

     <base-out-dir>
       htm
       htm/png
       rrd

     View the HTML reports and graphs with a web browser. Then,
     experiment with the various arguments.  Once you've found a
     layout that you like, add a cron job to automatically regenerate
     the reports on a periodic basis.

     One option that you'll definitely want to use is '-t'.  The
     purpose of this option is to tag processed files as done.  If
     this is not done, all input files will be processed each time you
     run the script.

     Note: If you have more than 30 days of data, you should use the
     '-e' option the first time you run the script.  This will ensure
     that your RRDs have the proper epoch.

Closing Remarks

  This recipe is a work in progress.

  If you've been collecting data for a while, you should back it up
  prior to experimenting with this recipe.

Credits

  This recipe was brought to you by Andy Bair and Klayton Monroe.

References

Appendix 1

--- process_uptime ---
#!/usr/bin/perl -w
######################################################################
#
# $Id: process_uptime,v 1.130 2010/07/02 04:48:26 klm Exp $
#
######################################################################
#
# Copyright 2004-2010 The WebJob Project, All Rights Reserved.
#
######################################################################

BEGIN
{
  ####################################################################
  #
  # Use this section if Perl complains about not being able to find
  # RRDs.pm.
  #
  # RRD usually installs in /usr/local/rrdtool-X.X.X. To make life a
  # bit easier, we assume that you've created a symbolic link called
  # /usr/local/rrdtool that points to the installed version of RRD.
  #
  # With RRD 1.2.X, things changed a bit, so you'll need to push the
  # appropriate a Perl version/platform path onto @INC.
  #
  ####################################################################

  #push(@INC, "/usr/local/rrdtool/lib/perl"); # 1.0.X
  #push(@INC, "/usr/local/rrdtool/lib/perl/<version>/<platform>"); # 1.2.X
}

use strict;
use Fcntl qw(:flock);
use File::Basename;
use File::Copy;
use Getopt::Std;
use RRDs;
use Time::Local;

######################################################################
#
# Main Routine
#
######################################################################

  my ($sError, $sUpdateTime);

  $sUpdateTime = localtime();

  ####################################################################
  #
  # Punch in and go to work.
  #
  ####################################################################

  my ($sProgram);

  $sProgram = basename(__FILE__);

  ####################################################################
  #
  # Get Options.
  #
  ####################################################################

  my (%hOptions);

  if (!getopts('C:c:e:g:i:lNno:qR:r:S:s:tU:u:Z:', \%hOptions))
  {
    Usage($sProgram);
  }

  ####################################################################
  #
  # CpusFile, '-C', is optional.
  #
  ####################################################################

  my ($sCpusFile);

  $sCpusFile = (exists($hOptions{'C'})) ? $hOptions{'C'} : undef;

  ####################################################################
  #
  # ZoneColorOrientation, '-c', is optional.
  #
  ####################################################################

  my ($sZoneColorOrientation);

  $sZoneColorOrientation = (exists($hOptions{'c'})) ? uc($hOptions{'c'}) : "V";

  if ($sZoneColorOrientation !~ /^[HV]$/)
  {
    print STDERR "$sProgram: Invalid color orientation ($sZoneColorOrientation). Value must be 'H' or 'V'.\n";
    exit(2);
  }

  ####################################################################
  #
  # RrdEpoch, '-e', is optional.
  #
  ####################################################################

  my ($sRrdEpoch);

  $sRrdEpoch = (exists($hOptions{'e'})) ? $hOptions{'e'} : "-30d";

  ####################################################################
  #
  # GraphTimestamps, '-g', is optional.
  #
  ####################################################################

  my (@aGraphTimestamps, $sGraphTimestamps);

  $sGraphTimestamps = (exists($hOptions{'g'})) ? $hOptions{'g'} : "1m,1w,1d,2h";

  @aGraphTimestamps = split(/,/, $sGraphTimestamps);

  foreach my $sTimestamp (@aGraphTimestamps)
  {
    if (!ValidateGraphTimestamp($sTimestamp, \$sError))
    {
      print STDERR "$sProgram: $sError\n";
      exit(2);
    }
  }

  ####################################################################
  #
  # BaseInputDir, '-i', is required.
  #
  ####################################################################

  my ($sBaseInputDir);

  if (!exists($hOptions{'i'}) || !defined($hOptions{'i'}) || length($hOptions{'i'}) < 1)
  {
    Usage($sProgram);
  }
  $sBaseInputDir = $hOptions{'i'};

  $sBaseInputDir =~ s/\/+$// unless ($sBaseInputDir eq "/");

  ####################################################################
  #
  # LinkMode, '-l', is optional.
  #
  ####################################################################

  my ($sLinkMode);

  $sLinkMode = (exists($hOptions{'l'})) ? 1 : 0;

  ####################################################################
  #
  # NumericRanges, '-N', is optional.
  #
  ####################################################################

  my ($sNumericRanges);

  $sNumericRanges = (exists($hOptions{'N'})) ? 1 : 0;

  ####################################################################
  #
  # Normalize, '-n', is optional.
  #
  ####################################################################

  my ($sNormalize);

  $sNormalize = (exists($hOptions{'n'})) ? 1 : 0;

  ####################################################################
  #
  # BaseOutputDir, '-o', is required.
  #
  ####################################################################

  my ($sBaseOutputDir);

  if (!exists($hOptions{'o'}) || !defined($hOptions{'o'}) || length($hOptions{'o'}) < 1)
  {
    Usage($sProgram);
  }
  $sBaseOutputDir = $hOptions{'o'};

  my $sHtmDir = $sBaseOutputDir . "/" . "htm";
  my $sPngDir = $sHtmDir . "/png";

  ####################################################################
  #
  # BeQuiet, '-q', is optional.
  #
  ####################################################################

  my ($sBeQuiet);

  $sBeQuiet = (exists($hOptions{'q'})) ? 1 : 0;

  ####################################################################
  #
  # MetaRefresh, '-R', is optional.
  #
  ####################################################################

  my ($sMetaRefresh);

  $sMetaRefresh = (exists($hOptions{'R'})) ? $hOptions{'R'} : 3600;

  if ($sMetaRefresh !~ /^\d+$/ || $sMetaRefresh < 60)
  {
    print STDERR "$sProgram: The MetaRefresh value must be a number greater than 60 seconds.\n";
    exit(2);
  }

  ####################################################################
  #
  # BaseRrdDir, '-r', is optional.
  #
  ####################################################################

  my ($sBaseRrdDir, $sRrdDir);

  $sBaseRrdDir = (exists($hOptions{'r'})) ? $hOptions{'r'} : $sBaseOutputDir;

  $sRrdDir = $sBaseRrdDir . "/" . "rrd";

  ####################################################################
  #
  # LargeSize, '-S', is optional.
  #
  ####################################################################

  my ($sLargeHeight, $sLargeSize, $sLargeWidth);

  $sLargeSize = (exists($hOptions{'S'})) ? $hOptions{'S'} : "600x100";

  if ($sLargeSize !~ /^(\d+)x(\d+)$/)
  {
    print STDERR "$sProgram: Invalid size ($sLargeSize).\n";
    exit(2);
  }
  $sLargeWidth = $1;
  $sLargeHeight = $2;

  ####################################################################
  #
  # SmallSize, '-s', is optional.
  #
  ####################################################################

  my ($sSmallHeight, $sSmallSize, $sSmallWidth);

  $sSmallSize = (exists($hOptions{'s'})) ? $hOptions{'s'} : "150x25";

  if ($sSmallSize !~ /^(\d+)x(\d+)$/)
  {
    print STDERR "$sProgram: Invalid size ($sSmallSize).\n";
    exit(2);
  }
  $sSmallWidth = $1;
  $sSmallHeight = $2;

  ####################################################################
  #
  # TagAsDone, '-t', is optional.
  #
  ####################################################################

  my ($sTagAsDone);

  $sTagAsDone = (exists($hOptions{'t'})) ? 1 : 0;

  ####################################################################
  #
  # LargeUpperLimit, '-U', is optional.
  #
  ####################################################################

  my ($sLargeUpperLimit);

  $sLargeUpperLimit = (exists($hOptions{'U'})) ? $hOptions{'U'} : undef;

  if (defined($sLargeUpperLimit) && ($sLargeUpperLimit !~ /^\d+$/ || $sLargeUpperLimit < 1 || $sLargeUpperLimit > 100))
  {
    print STDERR "$sProgram: Large upper limit must be a number between 1 and 100.\n";
    exit(2);
  }

  ####################################################################
  #
  # SmallUpperLimit, '-u', is optional.
  #
  ####################################################################

  my ($sSmallUpperLimit);

  $sSmallUpperLimit = (exists($hOptions{'u'})) ? $hOptions{'u'} : undef;

  if (defined($sSmallUpperLimit) && ($sSmallUpperLimit !~ /^\d+$/ || $sSmallUpperLimit < 1 || $sSmallUpperLimit > 100))
  {
    print STDERR "$sProgram: Small upper limit must be a number between 1 and 100.\n";
    exit(2);
  }

  ####################################################################
  #
  # ZoneLimits, '-Z', is optional.  This option specifies the four
  # limits that create the five zones.  This table below shows the
  # default limits.
  #
  #            default
  #    limit    value
  #    -----   -------
  #      1      0.90
  #      2      0.80
  #      3      0.70
  #      4      0.60
  #
  # Tested with correct input:
  #
  #    Result   Parameters
  #    ------   -----------------------
  #    passed   -c h -l .6,.7,.8,.9
  #    passed   -c v -l .6,.7,.8,.9
  #    passed   -c h -l 0.6,0.7,0.8,0.9
  #    passed   -c v -l 0.6,0.7,0.8,0.9
  #    passed   -c h -l .3,.5,.7,.9
  #    passed   -c v -l .3,.5,.7,.9
  #    passed   -c h -l .7,.8,.9,1.0
  #    passed   -c v -l .7,.8,.9,1.0
  #    passed   -c h -l .7,.8,.9,1
  #    passed   -c v -l .7,.8,.9,1
  #    passed   -c h -l .7,.9,1.1,1.3
  #    passed   -c v -l .7,.9,1.1,1.3
  #
  # Tested with incorrect input:
  #
  #    Result   Parameters
  #    ------   ----------
  #    passed   -l
  #    passed   -l foo
  #    passed   -l a,b,c,d
  #    passed   -l a,2,3,4
  #    passed   -l 2,a,3,4
  #    passed   -l 2,3,a,4
  #    passed   -l 2,3,4,a
  #    passed   -l 4,1,2,3
  #    passed   -l 1,4,2,3
  #    passed   -l 1,2,4,3
  #    passed   -l 0,1,2,3
  #    passed   -l 1,0,2,3
  #    passed   -l 1,2,0,3
  #    passed   -l 1,2,3,0
  #    passed   -l -1,1,2,3
  #    passed   -l 1,-1,2,3
  #    passed   -l 1,2,-1,3
  #    passed   -l 1,2,3,-1
  #    passed   -l ,1,2,3,4
  #    passed   -l 1,2,3,4,
  #    passed   -l 1,2,3
  #    passed   -l 1,2
  #    passed   -l 1
  #    passed   -l 1,,,
  #    passed   -l ,1,,
  #    passed   -l ,,1,
  #    passed   -l ,,,1
  #    passed   -l ,,,1,
  #    passed   -l a.1,0.2,0.3,0.4
  #
  ####################################################################

  my ($sZoneLimits, $sZoneLimit1, $sZoneLimit2, $sZoneLimit3, $sZoneLimit4);

  $sZoneLimits = (exists($hOptions{'Z'})) ? $hOptions{'Z'} : undef;

  if (defined($sZoneLimits))
  {
    my @aZoneLimits = split(/,/, $sZoneLimits, -1);
    my $sLimitCount = scalar(@aZoneLimits);

    ##################################################################
    #
    # Verify that we only have four limits.
    #
    ##################################################################

    if ($sLimitCount != 4)
    {
      print STDERR "$sProgram: Invalid limit count. Found $sLimitCount limit fields, but 4 is the required number.\n";
      exit(2);
    }

    ##################################################################
    #
    # Verify that each limit is a number greater than zero.
    #
    ##################################################################

    foreach my $sLimit (@aZoneLimits)
    {
      if ($sLimit !~ /^[\d\.]+$/ || $sLimit <= 0)
      {
        print STDERR "$sProgram: Invalid limit ($sLimit). Value must be a number greater than zero.\n";
        exit(2);
      }
    }

    ##################################################################
    #
    # Verify values descend numerically and none are equal.
    #
    ##################################################################

    for (my $sLimitIndexLoHi = 0; $sLimitIndexLoHi <= $#aZoneLimits; $sLimitIndexLoHi++)
    {
      if ($sLimitIndexLoHi > 0)
      {
        for (my $sLimitIndexHiLo = ($sLimitIndexLoHi-1); $sLimitIndexHiLo >= 0; $sLimitIndexHiLo--)
        {
          if ($aZoneLimits[$sLimitIndexHiLo] >= $aZoneLimits[$sLimitIndexLoHi])
          {
            print STDERR "$sProgram: Invalid limit order, ", $aZoneLimits[$sLimitIndexHiLo], " >= ", $aZoneLimits[$sLimitIndexLoHi], ".\n";
            exit(2);
          }
        }
      }
    }

    ##################################################################
    #
    # Set the limits.
    #
    ##################################################################

    $sZoneLimit1 = $aZoneLimits[0];
    $sZoneLimit2 = $aZoneLimits[1];
    $sZoneLimit3 = $aZoneLimits[2];
    $sZoneLimit4 = $aZoneLimits[3];
  }
  else
  {
    $sZoneLimit1 = 0.60;
    $sZoneLimit2 = 0.70;
    $sZoneLimit3 = 0.80;
    $sZoneLimit4 = 0.90;
  }

  ####################################################################
  #
  # If any arguments remain in the array, it's an error.
  #
  ####################################################################

  if (scalar(@ARGV) > 0)
  {
    Usage($sProgram);
  }

  ####################################################################
  #
  # Open the base directory and create a list of clients.
  #
  ####################################################################

  my (@aClients, %hClients);

  if (!opendir(DIR, $sBaseInputDir))
  {
    die "$sProgram: Unable to open base directory ($sBaseInputDir): $!\n";
  }
  @aClients = grep(!/^[.]+$/, sort(readdir(DIR)));
  closedir(DIR);

  foreach my $sClient (@aClients)
  {
    my $sDirectory = $sBaseInputDir . "/" . $sClient;
    lstat($sDirectory);
    if ($sLinkMode)
    {
      unless (-l _)
      {
        print "Skipping $sDirectory, not a symlink\n" unless ($sBeQuiet);
        next;
      }
    }
    else
    {
      unless (-d _)
      {
        print "Skipping $sDirectory, not a directory\n" unless ($sBeQuiet);
        next;
      }
    }
    $hClients{$sClient} = $sDirectory;
  }

  ####################################################################
  #
  # Open each client directory and create a list of input files.
  #
  ####################################################################

  my (%hClientOutFiles);

  foreach my $sClient (sort(keys(%hClients)))
  {
    if (!opendir(DIR, $hClients{$sClient}))
    {
      die "$sProgram: Unable to open directory ($hClients{$sClient}): $!\n";
    }
    $hClientOutFiles{$sClient} = [ sort(grep(/\.out$/, map("$hClients{$sClient}/$_", readdir(DIR)))) ];
    closedir(DIR);
  }

  ####################################################################
  #
  # Create output directory structure.
  #
  ####################################################################

  foreach my $sDir ($sBaseOutputDir, $sRrdDir, $sHtmDir, $sPngDir)
  {
    if (!-d $sDir)
    {
      if (!mkdir($sDir, 0755))
      {
        die "$sProgram: Unable to create output directory ($sDir): $!\n";
      }
    }
  }

  ####################################################################
  #
  # Create a lock file. If the file is already locked, abort.
  #
  ####################################################################

  my ($sLockFile);

  $sLockFile = $sRrdDir . "/" . ".lock";

  if (-f $sLockFile)
  {
    if (!open(LH, "+< $sLockFile"))
    {
      print STDERR "$sProgram: Unable to open the lock file ($!).'\n";
      exit(2);
    }
  }
  else
  {
    if (!open(LH, "> $sLockFile"))
    {
      print STDERR "$sProgram: Unable to make the lock file ($!).'\n";
      exit(2);
    }
  }
  if (!flock(LH, LOCK_EX | LOCK_NB))
  {
    print STDERR "$sProgram: Unable to lock the lock file ($!).'\n";
    exit(2);
  }

  ####################################################################
  #
  # Create RRD database if it doesn't exist.
  #
  ####################################################################

  foreach my $sClient (sort(keys(%hClients)))
  {
    my $sRrdFile = SetRrdFile($sRrdDir, $sClient);

    if (!-e $sRrdFile)
    {
      if (!$sBeQuiet)
      {
        print "Creating: RRD database for $sClient\n";
      }

      RRDs::create(
        "$sRrdFile",
        "--start", $sRrdEpoch,
        "--step", "60",
        "DS:load_01min:GAUGE:90:0:U",
        "DS:load_05min:GAUGE:90:0:U",
        "DS:load_15min:GAUGE:90:0:U",
        "RRA:AVERAGE:0.5:1:525600",
        "RRA:MIN:0.5:1:525600",
        "RRA:MAX:0.5:1:525600"
        );

      if (($sError = RRDs::error))
      {
        die "$sProgram: Error creating RRD database ($sRrdFile): $sError\n";
      }
    }
  }

  ####################################################################
  #
  # Update RRD database as necessary.
  #
  ####################################################################

  foreach my $sClient (sort(keys(%hClients)))
  {
    my (@aTagList, $sLastOutFile, $sRrdFile);

    $sRrdFile = SetRrdFile($sRrdDir, $sClient);

    foreach my $sOutFile (@{$hClientOutFiles{$sClient}})
    {
      ################################################################
      #
      # Check for a tag file (i.e., <file>.d). If it exists, assume
      # that the corresponding out file has already been processed.
      #
      ################################################################

      if (-f ($sOutFile . ".d"))
      {
        next;
      }

      ################################################################
      #
      # Get the last RRD update time for this client.
      #
      ################################################################

      my ($sLastUpdate);

      $sLastUpdate = RRDs::last($sRrdFile); # Get last update time.
      if (($sError = RRDs::error))
      {
        print STDERR "$sProgram: Error obtaining last update time from RRD database ($sRrdFile): $sError\n";
      }

      ################################################################
      #
      # Open the input file, and read it in all at once.
      #
      ################################################################

      my (@aLines);

      if (!open(FH, "< $sOutFile"))
      {
         die "$sProgram: Unable to open input file ($sOutFile): $!\n";
      }
      @aLines = <FH>;
      close(FH);

      ################################################################
      #
      # If the last line is older than the last update, skip the
      # entire file. Otherwise, calculate the time delta, seek
      # backwards in the array by that amount and check the record
      # at that location. If that record is older than the last
      # update, truncate the array up to that point and continue.
      # Fall through and process the entire file in all other cases.
      #
      ################################################################

      my (@aFields, $sDelta, $sLineIndex);

      $sLineIndex = $#aLines;

      if ($sLineIndex >= 0 && ParseLine($aLines[$sLineIndex], \@aFields, \$sError))
      {
        if (defined($aFields[0]))
        {
          if ($aFields[0] <= $sLastUpdate)
          {
            if (!$sBeQuiet)
            {
              print "Ignoring: $sClient " . basename($sOutFile) . "\n";
            }
            if (defined($sLastOutFile))
            {
              push(@aTagList, $sLastOutFile);
            }
            $sLastOutFile = $sOutFile;
            next;
          }
          else
          {
            $sDelta = int(($aFields[0] - $sLastUpdate) / 60) + 1;
            $sLineIndex -= ($sDelta > $sLineIndex) ? $sLineIndex : $sDelta;
            if (ParseLine($aLines[$sLineIndex], \@aFields, \$sError))
            {
              if (defined($aFields[0]))
              {
                if ($aFields[0] <= $sLastUpdate)
                {
                  splice(@aLines, 0, $sLineIndex);
                }
              }
            }
          }
        }
      }

      ################################################################
      #
      # Loop through each remaining line and update the RRD as we go.
      #
      ################################################################

      foreach my $sLine (@aLines)
      {
        if (!ParseLine($sLine, \@aFields, \$sError))
        {
          print STDERR "$sProgram: Erroneous record in $sOutFile: '$sLine'\n";
          last;
        }
        next unless ($aFields[0]); # The parser returns undef for header lines.
        if ($aFields[0] > $sLastUpdate)
        {
          if (!$sBeQuiet)
          {
            print "Updating: $sClient " . join(":", @aFields) . "\n";
          }
          RRDs::update($sRrdFile, join(":", @aFields));
          if (($sError = RRDs::error))
          {
            print STDERR "$sProgram: Error updating RRD database ($sRrdFile): $sError\n";
          }
        }
        else
        {
          if (!$sBeQuiet)
          {
            print "Skipping: $sClient " . join(":", @aFields) . " ($aFields[0] <= $sLastUpdate)\n";
          }
        }
      }
      $sLastOutFile = $sOutFile;
    }

    ##################################################################
    #
    # Conditionally tag files as done.
    #
    ##################################################################

    if ($sTagAsDone)
    {
      foreach my $sOldFile (@aTagList)
      {
        my $sTagFile = $sOldFile . ".d";
        if (!$sBeQuiet)
        {
          print "Tagging: $sClient $sOldFile --> $sTagFile\n";
        }
        if (!symlink($sOldFile, $sTagFile))
        {
          print STDERR "$sProgram: Error creating tag file ($sTagFile): $!\n";
        }
      }
    }
  }

  ####################################################################
  #
  # Pull in CPU counts if necessary.
  #
  ####################################################################

  my (%hCpuCounts);

  if (defined($sCpusFile))
  {
    if (!open(FH, "< $sCpusFile"))
    {
      die "$sProgram: Unable to open CPUs file ($sCpusFile): $!\n";
    }

    while (my $sLine = <FH>)
    {
      $sLine =~ s/#.*$//; # Remove trailing comments.
      next unless ($sLine =~ /^\s*([\w-]+)\s*=\s*(\d+)\s*$/);
      if ($2 > 0) # Don't allow zero or negative numbers.
      {
        $hCpuCounts{$1} = $2;
      }
    }
    close(FH);
  }

  ####################################################################
  #
  # Set load ranges.
  #
  ####################################################################

  my ($sLoadLight, $sLoadActive, $sLoadBusy, $sLoadStressed, $sLoadOverloaded);

  if ($sNumericRanges)
  {
    $sLoadLight = "0-$sZoneLimit1";
    $sLoadActive = "$sZoneLimit1-$sZoneLimit2";
    $sLoadBusy = "$sZoneLimit2-$sZoneLimit3";
    $sLoadStressed = "$sZoneLimit3-$sZoneLimit4";
    $sLoadOverloaded = "$sZoneLimit4-";
  }
  else
  {
    $sLoadLight = "light";
    $sLoadActive = "active";
    $sLoadBusy = "busy";
    $sLoadStressed = "stressed";
    $sLoadOverloaded = "overloaded"
  }

  ####################################################################
  #
  # Create graphs. One reason for graphing all clients every time is
  # to allow the graphs to "slip" forward with time which can better
  # reveal a client that is tardy.
  #
  # Note: The following colors were taken from here:
  #
  #   http://www.childoflight.org/mcc/colorcodeF.html
  #
  ####################################################################

  my $sColorBlack    = "#000000";
  my $sColorBlueDark = "#0D3981";
  my $sColorBlueLite = "#99CCFF";
  my $sColorGreen    = "#66FF66";
  my $sColorOrange   = "#FF9900"; # Old value was #FF9966.
  my $sColorPurple   = "#940B63";
  my $sColorRed      = "#FF0000"; # Old value was #FF6699.
  my $sColorYellow   = "#FCE503";
  my $sInitialLimit1 = $sZoneLimit1;
  my $sInitialLimit2 = $sZoneLimit2;
  my $sInitialLimit3 = $sZoneLimit3;
  my $sInitialLimit4 = $sZoneLimit4;

  foreach my $sClient (sort(keys(%hClients)))
  {
    my $sRrdFile = SetRrdFile($sRrdDir, $sClient);

    my $sCpuCount = (exists($hCpuCounts{$sClient})) ? $hCpuCounts{$sClient} : 1;
    my $sDivisor = 1;

    if ($sNormalize)
    {
      $sDivisor = $sCpuCount;
    }
    else
    {
      $sZoneLimit1 = $sInitialLimit1 * $sCpuCount;
      $sZoneLimit2 = $sInitialLimit2 * $sCpuCount;
      $sZoneLimit3 = $sInitialLimit3 * $sCpuCount;
      $sZoneLimit4 = $sInitialLimit4 * $sCpuCount;
    }

    if (!$sBeQuiet)
    {
      print "Graphing: $sClient\n";
    }

    foreach my $sTimestamp (@aGraphTimestamps)
    {
      my $sPngFileLarge = SetPngFile($sHtmDir, $sClient, $sTimestamp, "lg");
      my $sPngFileSmall = SetPngFile($sHtmDir, $sClient, $sTimestamp, "sm");
      my $sGraphStartTimestamp = SetGraphTimestampRrd($sTimestamp);
      my $sGraphStartPhrase = SetGraphTimePhrase($sTimestamp);
      my @aOptsHeaderLarge = ();
      my @aOptsHeaderSmall = ();
      my @aOptsOrientation = ();
      my @aOptsTrailerLarge = ();
      my @aOptsTrailerSmall = ();

      @aOptsHeaderLarge =
      (
        "$sPngFileLarge",
        "--height", $sLargeHeight,
        "--imgformat", "PNG",
        "--lower-limit", "0",
        "--no-minor",
        "--start", $sGraphStartTimestamp,
        "--title", "Load Averages for $sClient ($sGraphStartPhrase)",
        "--width", $sLargeWidth,
        "DEF:load_01min_raw=$sRrdFile:load_01min:AVERAGE",
        "DEF:load_05min_raw=$sRrdFile:load_05min:AVERAGE",
        "DEF:load_15min_raw=$sRrdFile:load_15min:AVERAGE",
        "CDEF:load_01min=load_01min_raw,$sDivisor,/",
        "CDEF:load_05min=load_05min_raw,$sDivisor,/",
        "CDEF:load_15min=load_15min_raw,$sDivisor,/"
      );
      push(@aOptsHeaderLarge, "--upper-limit", $sLargeUpperLimit) if (defined($sLargeUpperLimit));

      @aOptsHeaderSmall =
      (
        "$sPngFileSmall",
        "--height", $sSmallHeight,
        "--imgformat", "PNG",
        "--lower-limit", "0",
        "--only-graph",
        "--start", $sGraphStartTimestamp,
        "--width", $sSmallWidth,
        "DEF:load_01min_raw=$sRrdFile:load_01min:AVERAGE",
        "DEF:load_05min_raw=$sRrdFile:load_05min:AVERAGE",
        "DEF:load_15min_raw=$sRrdFile:load_15min:AVERAGE",
        "CDEF:load_01min=load_01min_raw,$sDivisor,/",
        "CDEF:load_05min=load_05min_raw,$sDivisor,/",
        "CDEF:load_15min=load_15min_raw,$sDivisor,/"
      );
      push(@aOptsHeaderSmall, "--upper-limit", $sSmallUpperLimit) if (defined($sSmallUpperLimit));

      if ($sZoneColorOrientation =~ /^V$/)
      {
        @aOptsOrientation = (
        "CDEF:zone_1=load_15min,$sZoneLimit1,LT,load_15min,0,IF",
        "CDEF:zone_2=load_15min,$sZoneLimit1,GE,load_15min,$sZoneLimit2,LT,load_15min,0,IF,0,IF",
        "CDEF:zone_3=load_15min,$sZoneLimit2,GE,load_15min,$sZoneLimit3,LT,load_15min,0,IF,0,IF",
        "CDEF:zone_4=load_15min,$sZoneLimit3,GE,load_15min,$sZoneLimit4,LT,load_15min,0,IF,0,IF",
        "CDEF:zone_5=load_15min,$sZoneLimit4,GE,load_15min,0,IF"
        );
      }
      elsif ($sZoneColorOrientation =~ /^H$/)
      {
        @aOptsOrientation = (
        "CDEF:zone_1=load_15min,0.00,GT,load_15min,$sZoneLimit1,LT,load_15min,$sZoneLimit1,IF,0,IF",
        "CDEF:zone_2=load_15min,$sZoneLimit1,GE,load_15min,$sZoneLimit2,LT,load_15min,$sZoneLimit1,-,$sZoneLimit2,$sZoneLimit1,-,IF,0,IF",
        "CDEF:zone_3=load_15min,$sZoneLimit2,GE,load_15min,$sZoneLimit3,LT,load_15min,$sZoneLimit2,-,$sZoneLimit3,$sZoneLimit2,-,IF,0,IF",
        "CDEF:zone_4=load_15min,$sZoneLimit3,GE,load_15min,$sZoneLimit4,LT,load_15min,$sZoneLimit3,-,$sZoneLimit4,$sZoneLimit3,-,IF,0,IF",
        "CDEF:zone_5=load_15min,$sZoneLimit4,GE,load_15min,$sZoneLimit4,-,0,IF"
        );
      }

      my $sCodes   = "       Codes: "; # Note the alignment of all three comments -- it matters.
      my $sLoads   = "       Loads: ";
      my $sCreated = "Date Created:   $sUpdateTime";

      $sCodes =~ s/:/\\:/g;
      $sLoads =~ s/:/\\:/g;
      $sCreated =~ s/:/\\:/g;

      @aOptsTrailerLarge =
      (
#        "COMMENT:       Codes: ", # This works for 1.0.X.
        "COMMENT:$sCodes",
        "AREA:zone_1$sColorBlueLite:$sLoadLight",
        "STACK:zone_2$sColorGreen:$sLoadActive",
        "STACK:zone_3$sColorYellow:$sLoadBusy",
        "STACK:zone_4$sColorOrange:$sLoadStressed",
        "STACK:zone_5$sColorRed:$sLoadOverloaded (CPUs=$sCpuCount)\\n",
#        "COMMENT:       Loads: ", # This works for 1.0.X.
        "COMMENT:$sLoads",
        "LINE1:load_01min$sColorPurple:1 min",
        "LINE1:load_05min$sColorBlueDark: 5 min",
        "LINE1:load_15min$sColorBlack:15 min averages\\n",
#        "COMMENT:Date Created:   $sUpdateTime", # This works for 1.0.X.
        "COMMENT:$sCreated",
        "AREA:zone_1$sColorBlueLite",
        "STACK:zone_2$sColorGreen",
        "STACK:zone_3$sColorYellow",
        "STACK:zone_4$sColorOrange",
        "STACK:zone_5$sColorRed",
        "LINE1:load_15min$sColorBlack"
      );

      @aOptsTrailerSmall =
      (
        "AREA:zone_1$sColorBlueLite:$sLoadLight",
        "STACK:zone_2$sColorGreen:$sLoadActive",
        "STACK:zone_3$sColorYellow:$sLoadBusy",
        "STACK:zone_4$sColorOrange:$sLoadStressed",
        "STACK:zone_5$sColorRed:$sLoadOverloaded (CPUs=$sCpuCount)\\n",
        "LINE1:load_01min$sColorPurple:1   ",
        "LINE1:load_05min$sColorBlueDark:5     ",
        "LINE1:load_15min$sColorBlack:15 minute\\n",
        "AREA:zone_1$sColorBlueLite",
        "STACK:zone_2$sColorGreen",
        "STACK:zone_3$sColorYellow",
        "STACK:zone_4$sColorOrange",
        "STACK:zone_5$sColorRed",
        "LINE1:load_15min$sColorBlack"
      );

      RRDs::graph(@aOptsHeaderLarge, @aOptsOrientation, @aOptsTrailerLarge);
      if (($sError = RRDs::error))
      {
        print STDERR "$sProgram: Error creating PNG file ($sPngFileLarge): $sError\n";
      }

      RRDs::graph(@aOptsHeaderSmall, @aOptsOrientation, @aOptsTrailerSmall);
      if (($sError = RRDs::error))
      {
        print STDERR "$sProgram: Error creating PNG file ($sPngFileSmall): $sError\n";
      }
    }
  }

  ####################################################################
  #
  # Create header and/or footer notes (conditionally displayed).
  #
  ####################################################################

  my ($sBaseTitle, $sHeaderNotes, $sFooterNotes);

  $sHeaderNotes = "[ <a href=\"/\">Home</a> ] [ <a href=\"..\">Up</a> ]";
  $sFooterNotes = "The Load Average in UNIX refers to the average number of processes which want CPU time. A Load equal to the number of CPUs (1.0 on a 1-processor machine, 2.0 on a 2-processor machine) means that the system is 100% busy, but that it is not overloaded. A Load higher than the number of CPUs means that the system has more work than it can do -- some processes will have to wait for CPU time. For a general-purpose server or workstation, this is OK; it just means that some processes take a little more time to get their work done while sharing the CPU(s). But for essentially \"real-time\" workloads, like an IDS sensor processing a continuous stream of packets, there is no way to ever catch up. Therefore, a higher Load in these circumstances will negatively impact performance -- i.e., packets will be lost. The graphs above are calibrated so that a load approaching the number of CPUs in a given machine is red (i.e., overloaded).";
  $sBaseTitle = "Load Averages";

  ####################################################################
  #
  # Create client web pages.
  #
  ####################################################################

  foreach my $sClient (sort(keys(%hClients)))
  {
    my $sHtmFile = SetHtmFile($sHtmDir, $sClient);
    my $sTitle = "$sBaseTitle for $sClient";

    if (!$sBeQuiet)
    {
      print "Creating: client web page for $sClient\n";
    }

    if (!open(FH, "> $sHtmFile"))
    {
      die "$sProgram: Unable to open HTML file ($sHtmFile): $!\n";
    }

    print FH "<html>\n";
    print FH "<head>\n";
    print FH "<meta http-equiv=\"refresh\" content=\"$sMetaRefresh\">\n";
    print FH "<title>$sTitle</title>\n";
    print FH "</head>\n";
    print FH "<body>\n";
    print FH "<hr><p>$sHeaderNotes</p><hr>\n" if ($sHeaderNotes);
    print FH "<center>\n";
    print FH "<table border=\"1\">\n";
    foreach my $sTimestamp (@aGraphTimestamps)
    {
      my $sPngLink = SetPngLink($sClient, $sTimestamp, "lg");
      print FH "<tr><td align=\"center\"><img src=\"$sPngLink\"></img></td></tr>\n";
    }
    print FH "</table>\n";
    print FH "</center>\n";
    print FH "<hr><p>$sFooterNotes</p><hr>\n" if ($sFooterNotes);
    print FH "<center><p><tt><b>Page last updated: $sUpdateTime</b></tt></p></center>\n";
    print FH "</body>\n";
    print FH "</html>\n";
    close(FH);
  }

  ####################################################################
  #
  # Create time period web pages.
  #
  ####################################################################

  foreach my $sListTimestamp (@aGraphTimestamps)
  {
    my $sGraphStartPhrase = SetGraphTimePhrase($sListTimestamp);
    my $sGraphStartTimestampPadded = SetGraphTimestampPadded($sListTimestamp);
    my $sGraphLink = $sGraphStartTimestampPadded . ".html";
    my $sHtmFile = $sHtmDir . "/" . $sGraphLink;
    my $sTitle = "$sBaseTitle for $sGraphStartPhrase";

    if (!$sBeQuiet)
    {
      print "Creating: time period web page for $sGraphStartPhrase ($sListTimestamp)\n";
    }

    if (!open(FH, "> $sHtmFile"))
    {
      die "$sProgram: Unable to open HTML file ($sHtmFile): $!\n";
    }

    print FH "<html>\n";
    print FH "<head>\n";
    print FH "<meta http-equiv=\"refresh\" content=\"$sMetaRefresh\">\n";
    print FH "<title>$sTitle</title>\n";
    print FH "</head>\n";
    print FH "<body>\n";
    print FH "<hr><p>$sHeaderNotes</p><hr>\n" if ($sHeaderNotes);
    print FH "<center>\n";
    print FH "<table border=\"1\">\n";

    foreach my $sClient (sort(keys(%hClients)))
    {
      my $sPngLink = SetPngLink($sClient, $sListTimestamp, "lg");
      print FH "  <tr>\n";
      print FH "    <td align=\"center\"><a name=\"$sClient\"><img src=\"$sPngLink\"></img></a></td>\n";
      print FH "  </tr>\n";
    }

    print FH "</table>\n";
    print FH "</center>\n";
    print FH "<hr><p>$sFooterNotes</p><hr>\n" if ($sFooterNotes);
    print FH "<center><p><tt><b>Page last updated: $sUpdateTime</b></tt></p></center>\n";
    print FH "</body>\n";
    print FH "</html>\n";
    close(FH);
  }

  ####################################################################
  #
  # Create master web page
  #
  ####################################################################

  my ($sHtmFile, $sLineCount);

  $sHtmFile = $sHtmDir . "/index.html";

  if (!open(FH, "> $sHtmFile"))
  {
    die "$sProgram: Unable to open HTML file ($sHtmFile): $!\n";
  }

  print FH "<html>\n";
  print FH "<head>\n";
  print FH "<meta http-equiv=\"refresh\" content=\"$sMetaRefresh\">\n";
  print FH "<title>$sBaseTitle</title>\n";
  print FH "</head>\n";
  print FH "<body>\n";
  print FH "<hr><p>$sHeaderNotes</p><hr>\n" if ($sHeaderNotes);
  print FH "<center>\n";
  print FH "<table border=\"1\">\n";
  print FH "  <tr>\n";
  print FH "    <th align=\"center\" bgcolor=\"#cccccc\">&nbsp;</th>\n";
  print FH "    <th align=\"center\" bgcolor=\"#cccccc\">\n";
  print FH "      <table border=\"0\">\n";
  print FH "        <tr>\n";
  print FH "          <th bgcolor=\"#cccccc\" colspan=\"2\"><tt>Client</tt></th>\n";
  print FH "        </tr>\n";
  print FH "      </table>\n";
  print FH "    </th>\n";

  foreach my $sListTimestamp (@aGraphTimestamps)
  {
    my $sGraphStartPhrase = SetGraphTimePhrase($sListTimestamp);
    my $sGraphStartTimestampPadded = SetGraphTimestampPadded($sListTimestamp);
    my $sGraphLink = $sGraphStartTimestampPadded . ".html";
    print FH "    <th align=\"center\" bgcolor=\"#cccccc\">\n";
    print FH "      <table border=\"0\">\n";
    print FH "        <tr>\n";
    print FH "          <th bgcolor=\"#cccccc\" colspan=\"2\"><a href=\"$sGraphLink\"><tt>$sGraphStartPhrase</tt></a></th>\n";
    print FH "        </tr>\n";
    print FH "      </table>\n";
    print FH "    </th>\n";
  }
  print FH "  </tr>\n";

  foreach my $sClient (sort(keys(%hClients)))
  {
    my $sClientLink = $sClient . ".html";

    $sLineCount++;

    print FH "  <tr>\n";
    print FH "    <td bgcolor=\"#cccccc\" align=\"center\"><tt>$sLineCount</tt></td>\n";
    print FH "    <td bgcolor=\"#cccccc\" align=\"left\"><a href=\"$sClientLink\"><tt>$sClient</tt></a></td>\n";
    foreach my $sListTimestamp (@aGraphTimestamps)
    {
      my $sPngLinkLarge = SetPngLink($sClient, $sListTimestamp, "lg");
      my $sPngLinkSmall = SetPngLink($sClient, $sListTimestamp, "sm");
      print FH "    <td><a href=\"$sPngLinkLarge\"><img border=\"0\" src=\"$sPngLinkSmall\"></img></a></td>\n";
    }
    print FH "  </tr>\n";
  }

  print FH "</table>\n";
  print FH "</center>\n";
  print FH "<hr><p>$sFooterNotes</p><hr>\n" if ($sFooterNotes);
  print FH "<center><p><tt><b>Page last updated: $sUpdateTime</b></tt></p></center>\n";
  print FH "</body>\n";
  print FH "</html>\n";
  close(FH);

  ####################################################################
  #
  # Copy one of the master web pages over to the index web page.
  #
  ####################################################################

  my $sGraphStartTimestampPadded = SetGraphTimestampPadded($aGraphTimestamps[0]);
  my $sFirstSummaryClient = $sHtmDir . "/summary_client_" . $sGraphStartTimestampPadded . ".html";
  my $sIndex = $sHtmDir . "/index.html";
  copy($sFirstSummaryClient, $sIndex);

  ####################################################################
  #
  # Clean up and go home.
  #
  ####################################################################

  unlink($sLockFile);

  1;


######################################################################
#
# GetSeconds
#
######################################################################

sub GetSeconds
{
  my ($sDate, $sTime, $sPreference) = @_;

  my ($sYear, $sMonth, $sMonthDay) = split(/[-]/, $sDate);
  my ($sHours, $sMinutes, $sSeconds) = split(/[:]/, $sTime);

  if ($sPreference =~ /^local$/i)
  {
    return timelocal(
      $sSeconds,
      $sMinutes,
      $sHours,
      $sMonthDay,
      $sMonth - 1,
      $sYear - 1900
      );
  }
  else
  {
    return timegm(
      $sSeconds,
      $sMinutes,
      $sHours,
      $sMonthDay,
      $sMonth - 1,
      $sYear - 1900
      );
  }
}


######################################################################
#
# ParseLine
#
######################################################################

sub ParseLine
{
  my ($sLine, $paFields, $psError) = @_;

  my ($sLineRegex1, $sLineRegex2, $sLineRegex3, $sLineRegex4);

  $sLineRegex1 = qq(^(\\d{10})[|:]([\\d.]{1,})[|:]([\\d.]{1,})[|:]([\\d.]{1,})\$);
  $sLineRegex2 = qq(^(\\d{4}-\\d{2}-\\d{2})[|:](\\d{2}:\\d{2}:\\d{2})[|:]([\\d.]{1,})[|:]([\\d.]{1,})[|:]([\\d.]{1,})\$);
  $sLineRegex3 = qq(^(?:time|seconds)[|:]load_0?1min[|:]load_0?5min[|:]load_15min\$);
  $sLineRegex4 = qq(^date[|:]time[|:]load_0?1min[|:]load_0?5min[|:]load_15min\$);

  if ($sLine =~ /$sLineRegex1/)
  {
    my ($sSeconds, $s01MinAvg, $s05MinAvg, $s15MinAvg) = ($1, $2, $3, $4);
    @$paFields = ($sSeconds, $s01MinAvg, $s05MinAvg, $s15MinAvg);
  }
  elsif ($sLine =~ /$sLineRegex2/)
  {
    my ($sDate, $sTime, $s01MinAvg, $s05MinAvg, $s15MinAvg) = ($1, $2, $3, $4, $5);
    my $sSeconds = GetSeconds($sDate, $sTime, "local");
    @$paFields = ($sSeconds, $s01MinAvg, $s05MinAvg, $s15MinAvg);
  }
  elsif ($sLine =~ /$sLineRegex3/ || $sLine =~ /$sLineRegex4/)
  {
    @$paFields = (undef, undef, undef, undef); # It's a header line, so ignore it.
  }
  else
  {
    return undef;
  }

  1;
}


######################################################################
#
# SetGraphTimePhrase
#
######################################################################

sub SetGraphTimePhrase
{
  my ($sTimestamp) = @_;

  my $sTimePhrase;
  my $sTimeUnit;
  my $sTimeValue;

  $sTimeUnit  = SetGraphTimeUnit($sTimestamp);
  $sTimeValue = SetGraphTimeValue($sTimestamp);

  if ($sTimeUnit =~ /^d$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Day" : "Days";
  }
  elsif ($sTimeUnit =~ /^h$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Hour" : "Hours";
  }
  elsif ($sTimeUnit =~ /^m$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Month" : "Months";
  }
  elsif ($sTimeUnit =~ /^w$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Week" : "Weeks";
  }
  elsif ($sTimeUnit =~ /^y$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Year" : "Years";
  }
  else
  {
    return "unknown";
  }

  return (sprintf("%s %s", $sTimeValue, $sTimePhrase));
}


######################################################################
#
# SetGraphTimeUnit
#
######################################################################

sub SetGraphTimeUnit
{
  my ($sTimestamp) = @_;

  my $sTimeUnit;
  $sTimeUnit = $sTimestamp;
  $sTimeUnit =~ s/^\d+([dhmwy])$/$1/;
  return ($sTimeUnit);
}


######################################################################
#
# SetGraphTimeValue
#
######################################################################

sub SetGraphTimeValue
{
  my ($sTimestamp) = @_;

  my $sTimeValue;
  $sTimeValue = $sTimestamp;
  $sTimeValue =~ s/[dhmwy]$//;
  return ($sTimeValue);
}


######################################################################
#
# SetGraphTimestampPadded
#
######################################################################

sub SetGraphTimestampPadded
{
  my ($sTimestamp) = @_;

  return (sprintf("%06s", $sTimestamp));
}


######################################################################
#
# SetGraphTimestampRrd
#
######################################################################

sub SetGraphTimestampRrd
{
  my ($sTimestamp) = @_;

  return (sprintf("-%s", $sTimestamp));
}


######################################################################
#
# SetHtmFile
#
######################################################################

sub SetHtmFile
{
  my ($sHtmDir, $sClient) = @_;

  return (sprintf("%s/%s.html", $sHtmDir, $sClient));
}


######################################################################
#
# SetPngFile
#
######################################################################

sub SetPngFile
{
  my ($sHtmDir, $sClient, $sTimestamp, $sSize) = @_;

  return (sprintf("%s/%s", $sHtmDir, SetPngLink($sClient, $sTimestamp, $sSize)));
}


######################################################################
#
# SetPngLink
#
######################################################################

sub SetPngLink
{
  my ($sClient, $sTimestamp, $sSize) = @_;

  if ($sSize =~ /^lg$/i)
  {
    return (sprintf("png/%s_%s_lg.png", $sClient, SetGraphTimestampPadded($sTimestamp)));
  }
  elsif ($sSize =~ /^sm$/i)
  {
    return (sprintf("png/%s_%s_sm.png", $sClient, SetGraphTimestampPadded($sTimestamp)));
  }
  else
  {
    return (sprintf("png/%s_%s_xx.png", $sClient, SetGraphTimestampPadded($sTimestamp)));
  }
}


######################################################################
#
# SetRrdFile
#
######################################################################

sub SetRrdFile
{
  my ($sRrdDir, $sClient) = @_;

  return (sprintf("%s/%s.rrd", $sRrdDir, $sClient));
}


######################################################################
#
# Usage
#
######################################################################

sub Usage
{
  my ($sProgram) = @_;
  print STDERR "\n";
  print STDERR "Usage: $sProgram [-lNnqt] [-C cpus-file] [-c {H|V}] [-e rrd-epoch] [-g graph-times] [-R seconds] [-r base-rrd-dir] [-S WxH] [-s WxH] [-U limit] [-u limit] [-Z limits] -i base-in-dir -o base-out-dir\n";
  print STDERR "\n";
  exit(1);
}


######################################################################
#
# ValidateGraphTimestamp
#
######################################################################

sub ValidateGraphTimestamp
{
  my ($sTimestamp, $psError) = @_;

  my $sTimeValue;

  if ($sTimestamp !~ /^\d+[dhmwy]$/)
  {
    $$psError = "Invalid graph timestamp ($sTimestamp).";
    return undef;
  }

  $sTimeValue = SetGraphTimeValue($sTimestamp);

  if ($sTimeValue !~ /^\d+$/ || $sTimeValue <= 0)
  {
    $$psError = "Invalid graph time value ($sTimeValue).";
    return undef;
  }

  1;
}


=pod

=head1 NAME

process_uptime - Process harvest_uptime data

=head1 SYNOPSIS

B<process_uptime> [B<-lNnqt>] [B<-C cpus-file>] [B<-c {H|V}>] [B<-e rrd-epoch>] [B<-g graph-times>] [B<-R seconds>] [B<-r base-rrd-dir>] [B<-S WxH>] [B<-s WxH>] [B<-U limit>] [B<-u limit>] [B<-Z limits>] B<-i base-in-dir> B<-o base-out-dir>

=head1 DESCRIPTION

This utility processes harvest_uptime data and produces system load
graphs for 1 month, 1 week, 1 day, and 2 hours.

=head1 OPTIONS

=over 4

=item B<-C cpus-file>

Specifies the name of a file that contains key/value pairs of hostnames
and CPU counts.  The expected format of this file is as follows:

  <hostname>=<cpu-count>

The CPU count is used as a limit (B<-l>) multiplier unless the
normalizer option (B<-n>) is also specified.  If this option is not
specified, then all hosts will be assigned a CPU count of 1.

=item B<-c orientation>

Specifies the coloring orientation for the five color zones when
generating graphs.  The table below shows the colors for each zone.

 zone   color
 ----   ------
  1     blue
  2     green
  3     yellow
  4     orange
  5     red

This option allows these five colors to be drawn horizontally (H) or
vertically (V).  When coloring horizontally, zones are stacked on one
another something like this:

 red
 orange
 yellow
 green
 blue

When coloring vertically, zones are drawn side-by-side something like
this:

 r o y g b
 e r e r l
 d a l e u
   n l e e
   g o n
   e w

This option is optional with the default value of horizontal, 'V'.

=item B<-e rrd-epoch>

Specifies the start epoch for each RRD.  This value may be any legal
RRD start time (see rrdcreate(1)).  If you want all your systems
to be on the same page, it's probably best to specify an absolute
time.  The default value is "-30d".

=item B<-g time1,time2,time3...>

Specifies the order and time ranges graphs will be created.  Times are
specified in the following format:

  <number><unit>

Where <number> is a positive integer greater than zero and <unit> is
one of the letters represented in the table below.

  unit   meaning
  ----   --------
   d     Day(s)
   h     Hour(s)
   m     Month(s)
   w     Week(s)
   y     Year(s)

<number> and <unit> combined represent the amount of past time
(starting from now) to include in the graph that is to be created.
For example, '1d' produces a graph for the last day, '3m' produces
a graph for the last three months, '4w' produces a graph for the
last four weeks, and so on.

The default value for this option is "1m,1w,1d,2h".

=item B<-i base-in-dir>

Specifies the base input directory.  This option is mandatory.

=item B<-l>

Instructs the program to process symbolic links to client directories
rather than the actual directories.  This is useful for mapping
cryptic/encoded names (e.g., CLLI - Common Language Location
Identification ) into human-readable names.

=item B<-N>

Instructs the program to use numeric ranges in the legend.  If not
specified, the following codes are used instead: light, active,
busy, stressed, and overloaded.  It's up to the user to make sure
that the specified limits (B<-l>) make sense with these codes.

=item B<-n>

Instructs the program to normalize the load averages (rather than
scale the limit values) based on the number of CPUs.  This means
that the average loads will be divided by the number of CPUs and
the limit values will remain fixed.  If this option is not specified,
then the CPU count (if specified) is used as a limit multiplier.

=item B<-o base-out-dir>

Specifies the base output directory.  This is where web pages and
graphs are created.  This option is mandatory.

=item B<-q>

Instructs the program to run quietly.

=item B<-R seconds>

Specifies the META refresh rate that is to be included in each web
page.  Values must be a decimal number greater than 60 (seconds).  The
default value is 3600 (seconds).

=item B<-r base-rrd-dir>

Specifies the base RRD directory.  This is where RRDs are stored.  If
not specified, the value for this option defaults to B<base-out-dir>.

=item B<-S WxH>

Specifies the width and height (in pixels) for large graphs.  The
default value is 600x100.

=item B<-s WxH>

Specifies the width and height (in pixels) for small graphs.  The
default value is 150x25.

=item B<-t>

Instructs the program to tag (i.e., rename) old input files as done
(".d").  Note: There must be more than one input file for this
action to take place.  In other words, the current file is never
tagged as done.  This is because it may still be in flux.

=item B<-U limit>

Specifies the upper limit to use on large graphs.  Values may be 1-100.
There is no default value.

=item B<-u limit>

Specifies the upper limit to use on small graphs.  Values may be 1-100.
There is no default value.

=item B<-Z limit1,limit2,limit3,limit4>

Specifies the four limits that create the five coloring zones.  The
limits must be separated by commas, listed from highest to lowest, can
not be equal to one another, and must be greater than 0.  The table
below shows the definition for each zone using the limit values.

 zone   condition
 ----   ------------------------------------
  1     uptime < limit1
  2     uptime >= limit1 and uptime < limit2
  3     uptime >= limit2 and uptime < limit3
  4     uptime >= limit3 and uptime < limit4
  5     uptime >= limit4

This option is optional and the table below shows the default limits.

         default
 limit    value
 -----   -------
   1      0.60
   2      0.70
   3      0.80
   4      0.90

=back

=head1 AUTHORS

Andy Bair and Klayton Monroe

=cut
--- process_uptime ---