Revision

  $Id: webjob-process-df.base,v 1.6 2010/07/02 15:30:12 klm Exp $

Purpose

  This recipe demonstrates how to preprocess harvest_df data, load
  it into a Round Robin Database (RRD), and create a browsable set of
  HTML reports and graphs that depict percent utilization for 5-minute
  intervals.

Motivation

  Graphing harvest_df data makes it possible for you to visualize
  the health of your systems.  This can reveal trends or relationships
  that might otherwise be hard to identify.  It can also serve as an
  early warning indicator and as an aide for planning distributed
  jobs.

Requirements

  This recipe assumes that you have read and implemented the following
  recipe:

    http://webjob.sourceforge.net/Files/Recipes/webjob-harvest-df.txt

  Your server must be running UNIX and have Apache, Perl, and RRDTool
  (1.2.11 or higher) installed.

Time to Implement

  Assuming that you have satisfied all the requirements/prerequisites,
  this recipe should take less than one hour to implement.

Solution

  The following steps describe how to implement this recipe.

  1. Extract the script at the bottom of this recipe, and install it
     in /usr/local/bin.  Once the file is in place, set its ownership
     and permissions to 0:0 and 755, respectively.

       # sed -e '1,/^--- process_df ---$/d; /^--- process_df ---$/,$d' webjob-process-df.txt > process_df
       # cp process_df /usr/local/bin
       # chmod 755 /usr/local/bin/process_df
       # chown 0:0 /usr/local/bin/process_df

  2. Read the documentation provided in the script.  This can be done
     using perldoc as follows:

       $ perldoc process_df

  3. Locate your base input directory and make sure you have some
     data to process.  If you implemented the recipe cited above as
     written, then this directory should be located here:

       /var/rsync.all/df

     and it should have subdirectories and files that match the
     following format:

       /var/rsync.all/df/<hostname>/<YYYY-MM-DD>.out

     Next, decide where your base output directory will reside.  This
     recipe assumes the following:

       /usr/local/www/data/df

  4. Preprocess your data, load it into one or more RRDs, and create
     the HTML reports.  This can be done as follows:

       # process_df -i /var/rsync.all/df -o /usr/local/www/data/df

     This should create the following directories under your base
     output directory:

     <base-out-dir>
       htm
       htm/png
       rrd

     View the HTML reports and graphs with a web browser. Then,
     experiment with the various arguments.  Once you've found a
     layout that you like, add a cron job to automatically regenerate
     the reports on a periodic basis.

     One option that you'll definitely want to use is '-t'.  The
     purpose of this option is to tag processed files as done.  If
     this is not done, all input files will be processed each time you
     run the script.

     Note: If you have more than 30 days of data, you should use the
     '-e' option the first time you run the script.  This will ensure
     that your RRDs have the proper epoch.

Closing Remarks

  This recipe is a work in progress.

  If you've been collecting data for a while, you should back it up
  prior to experimenting with this recipe.

Credits

  This recipe was brought to you by Andy Bair and Klayton Monroe.

References

Appendix 1

--- process_df ---
#!/usr/bin/perl -w
######################################################################
#
# $Id: process_df,v 1.36 2008/10/16 04:57:30 klm Exp $
#
######################################################################
#
# Copyright 2006-2007 The WebJob Project, All Rights Reserved.
#
######################################################################

BEGIN
{
  ####################################################################
  #
  # Use this section if Perl complains about not being able to find
  # RRDs.pm.
  #
  # RRD usually installs in /usr/local/rrdtool-X.X.X. To make life a
  # bit easier, we assume that you've created a symbolic link called
  # /usr/local/rrdtool that points to the installed version of RRD.
  #
  # With RRD 1.2.X, things changed a bit, so you'll need to push the
  # appropriate a Perl version/platform path onto @INC.
  #
  ####################################################################

  #push(@INC, "/usr/local/rrdtool/lib/perl"); # 1.0.X
  #push(@INC, "/usr/local/rrdtool/lib/perl/<version>/<platform>"); # 1.2.X
}

use strict;
use Digest::MD5 qw(md5_hex);
use Fcntl qw(:flock);
use File::Basename;
use File::Copy;
use Getopt::Std;
use RRDs;
use Time::Local;

######################################################################
#
# Main Routine
#
######################################################################

  my ($sError, $sUpdateTime);

  $sUpdateTime = localtime();

  ####################################################################
  #
  # Punch in and go to work.
  #
  ####################################################################

  my ($sProgram);

  $sProgram = basename(__FILE__);

  ####################################################################
  #
  # Get Options.
  #
  ####################################################################

  my (%hOptions);

  if (!getopts('e:g:i:lo:qr:S:s:tU:u:Z:', \%hOptions))
  {
    Usage($sProgram);
  }

  ####################################################################
  #
  # RrdEpoch, '-e', is optional.
  #
  ####################################################################

  my ($sRrdEpoch);

  $sRrdEpoch = (exists($hOptions{'e'})) ? $hOptions{'e'} : "-30d";

  ####################################################################
  #
  # GraphTimestamps, '-g', is optional.
  #
  ####################################################################

  my (@aGraphTimestamps, $sGraphTimestamps);

  $sGraphTimestamps = (exists($hOptions{'g'})) ? $hOptions{'g'} : "30d,7d,1d";

  @aGraphTimestamps = split(/,/, $sGraphTimestamps);

  foreach my $sTimestamp (@aGraphTimestamps)
  {
    if (!ValidateGraphTimestamp($sTimestamp, \$sError))
    {
      print STDERR "$sProgram: $sError\n";
      exit(2);
    }
  }

  ####################################################################
  #
  # BaseInputDir, '-i', is required.
  #
  ####################################################################

  my ($sBaseInputDir);

  if (!exists($hOptions{'i'}) || !defined($hOptions{'i'}) || length($hOptions{'i'}) < 1)
  {
    Usage($sProgram);
  }
  $sBaseInputDir = $hOptions{'i'};

  $sBaseInputDir =~ s/\/+$// unless ($sBaseInputDir eq "/");

  ####################################################################
  #
  # LinkMode, '-l', is optional.
  #
  ####################################################################

  my ($sLinkMode);

  $sLinkMode = (exists($hOptions{'l'})) ? 1 : 0;

  ####################################################################
  #
  # BaseOutputDir, '-o', is required.
  #
  ####################################################################

  my ($sBaseOutputDir);

  if (!exists($hOptions{'o'}) || !defined($hOptions{'o'}) || length($hOptions{'o'}) < 1)
  {
    Usage($sProgram);
  }
  $sBaseOutputDir = $hOptions{'o'};

  my $sHtmDir = $sBaseOutputDir . "/" . "htm";
  my $sPngDir = $sHtmDir . "/png";

  ####################################################################
  #
  # BeQuiet, '-q', is optional.
  #
  ####################################################################

  my ($sBeQuiet);

  $sBeQuiet = (exists($hOptions{'q'})) ? 1 : 0;

  ####################################################################
  #
  # MetaRefresh, '-R', is optional.
  #
  ####################################################################

  my ($sMetaRefresh);

  $sMetaRefresh = (exists($hOptions{'R'})) ? $hOptions{'R'} : 3600;

  if ($sMetaRefresh !~ /^\d+$/ || $sMetaRefresh < 60)
  {
    print STDERR "$sProgram: The MetaRefresh value must be a number greater than 60 seconds.\n";
    exit(2);
  }

  ####################################################################
  #
  # BaseRrdDir, '-r', is optional.
  #
  ####################################################################

  my ($sBaseRrdDir, $sRrdDir);

  $sBaseRrdDir = (exists($hOptions{'r'})) ? $hOptions{'r'} : $sBaseOutputDir;

  $sRrdDir = $sBaseRrdDir . "/" . "rrd";

  ####################################################################
  #
  # LargeSize, '-S', is optional.
  #
  ####################################################################

  my ($sLargeHeight, $sLargeSize, $sLargeWidth);

  $sLargeSize = (exists($hOptions{'S'})) ? $hOptions{'S'} : "600x100";

  if ($sLargeSize !~ /^(\d+)x(\d+)$/)
  {
    print STDERR "$sProgram: Invalid size ($sLargeSize).\n";
    exit(2);
  }
  $sLargeWidth = $1;
  $sLargeHeight = $2;

  ####################################################################
  #
  # SmallSize, '-s', is optional.
  #
  ####################################################################

  my ($sSmallHeight, $sSmallSize, $sSmallWidth);

  $sSmallSize = (exists($hOptions{'s'})) ? $hOptions{'s'} : "150x25";

  if ($sSmallSize !~ /^(\d+)x(\d+)$/)
  {
    print STDERR "$sProgram: Invalid size ($sSmallSize).\n";
    exit(2);
  }
  $sSmallWidth = $1;
  $sSmallHeight = $2;

  ####################################################################
  #
  # TagAsDone, '-t', is optional.
  #
  ####################################################################

  my ($sTagAsDone);

  $sTagAsDone = (exists($hOptions{'t'})) ? 1 : 0;

  ####################################################################
  #
  # LargeUpperLimit, '-U', is optional.
  #
  ####################################################################

  my ($sLargeUpperLimit);

  $sLargeUpperLimit = (exists($hOptions{'U'})) ? $hOptions{'U'} : 100;

  if (defined($sLargeUpperLimit) && ($sLargeUpperLimit !~ /^\d+$/ || $sLargeUpperLimit < 1 || $sLargeUpperLimit > 100))
  {
    print STDERR "$sProgram: Large upper limit must be a number between 1 and 100.\n";
    exit(2);
  }

  ####################################################################
  #
  # SmallUpperLimit, '-u', is optional.
  #
  ####################################################################

  my ($sSmallUpperLimit);

  $sSmallUpperLimit = (exists($hOptions{'u'})) ? $hOptions{'u'} : 100;

  if (defined($sSmallUpperLimit) && ($sSmallUpperLimit !~ /^\d+$/ || $sSmallUpperLimit < 1 || $sSmallUpperLimit > 100))
  {
    print STDERR "$sProgram: Small upper limit must be a number between 1 and 100.\n";
    exit(2);
  }

  ####################################################################
  #
  # ZoneLimits, '-Z', is optional.
  #
  ####################################################################

  my ($sZoneLimits, $sZoneLimit1, $sZoneLimit2, $sZoneLimit3, $sZoneLimit4);

  $sZoneLimits = (exists($hOptions{'Z'})) ? $hOptions{'Z'} : undef;

  if (defined($sZoneLimits))
  {
    my @aZoneLimits = split(/,/, $sZoneLimits, -1);
    my $sLimitCount = scalar(@aZoneLimits);

    ##################################################################
    #
    # Verify that we only have four limits.
    #
    ##################################################################

    if ($sLimitCount != 4)
    {
      print STDERR "$sProgram: Invalid limit count. Found $sLimitCount limit fields, but 4 is the required number.\n";
      exit(2);
    }

    ##################################################################
    #
    # Verify that each limit is an integer in the range [1-100].
    #
    ##################################################################

    foreach my $sLimit (@aZoneLimits)
    {
      if ($sLimit !~ /^\d+$/ || $sLimit <= 0 || $sLimit > 100)
      {
        print STDERR "$sProgram: Invalid limit ($sLimit). Value must be an integer between 1 and 100.\n";
        exit(2);
      }
    }

    ##################################################################
    #
    # Verify values descend numerically and none are equal.
    #
    ##################################################################

    for (my $sLimitIndexLoHi = 0; $sLimitIndexLoHi <= $#aZoneLimits; $sLimitIndexLoHi++)
    {
      if ($sLimitIndexLoHi > 0)
      {
        for (my $sLimitIndexHiLo = ($sLimitIndexLoHi-1); $sLimitIndexHiLo >= 0; $sLimitIndexHiLo--)
        {
          if ($aZoneLimits[$sLimitIndexHiLo] >= $aZoneLimits[$sLimitIndexLoHi])
          {
            print STDERR "$sProgram: Invalid limit order, ", $aZoneLimits[$sLimitIndexHiLo], " >= ", $aZoneLimits[$sLimitIndexLoHi], ".\n";
            exit(2);
          }
        }
      }
    }

    ##################################################################
    #
    # Set the limits.
    #
    ##################################################################

    $sZoneLimit1 = $aZoneLimits[0];
    $sZoneLimit2 = $aZoneLimits[1];
    $sZoneLimit3 = $aZoneLimits[2];
    $sZoneLimit4 = $aZoneLimits[3];
  }
  else
  {
    $sZoneLimit1 = 10;
    $sZoneLimit2 = 70;
    $sZoneLimit3 = 85;
    $sZoneLimit4 = 95;
  }

  ####################################################################
  #
  # If any arguments remain in the array, it's an error.
  #
  ####################################################################

  if (scalar(@ARGV) > 0)
  {
    Usage($sProgram);
  }

  ####################################################################
  #
  # Open the base directory and create a list of clients.
  #
  ####################################################################

  my (@aClients, %hClients);

  if (!opendir(DIR, $sBaseInputDir))
  {
    die "$sProgram: Unable to open base directory ($sBaseInputDir): $!\n";
  }
  @aClients = grep(!/^[.]+$/, sort(readdir(DIR)));
  closedir(DIR);

  foreach my $sClient (@aClients)
  {
    my $sDirectory = $sBaseInputDir . "/" . $sClient;
    lstat($sDirectory);
    if ($sLinkMode)
    {
      unless (-l _)
      {
        print "Skipping $sDirectory, not a symlink\n" unless ($sBeQuiet);
        next;
      }
    }
    else
    {
      unless (-d _)
      {
        print "Skipping $sDirectory, not a directory\n" unless ($sBeQuiet);
        next;
      }
    }
    $hClients{$sClient} = $sDirectory;
  }

  ####################################################################
  #
  # Open each client directory and create a list of input files.
  #
  ####################################################################

  my (%hClientOutFiles);

  foreach my $sClient (sort(keys(%hClients)))
  {
    if (!opendir(DIR, $hClients{$sClient}))
    {
      die "$sProgram: Unable to open directory ($hClients{$sClient}): $!\n";
    }
    $hClientOutFiles{$sClient} = [ sort(grep(/\.out$/, map("$hClients{$sClient}/$_", readdir(DIR)))) ];
    closedir(DIR);
  }

  ####################################################################
  #
  # Create output directory structure.
  #
  ####################################################################

  foreach my $sDir ($sBaseOutputDir, $sRrdDir, $sHtmDir, $sPngDir)
  {
    if (!-d $sDir)
    {
      if (!mkdir($sDir, 0755))
      {
        die "$sProgram: Unable to create output directory ($sDir): $!\n";
      }
    }
  }

  ####################################################################
  #
  # Create a lock file. If the file is already locked, abort.
  #
  ####################################################################

  my ($sLockFile);

  $sLockFile = $sRrdDir . "/" . ".lock";

  if (-f $sLockFile)
  {
    if (!open(LH, "+< $sLockFile"))
    {
      print STDERR "$sProgram: Unable to open the lock file ($!).'\n";
      exit(2);
    }
  }
  else
  {
    if (!open(LH, "> $sLockFile"))
    {
      print STDERR "$sProgram: Unable to make the lock file ($!).'\n";
      exit(2);
    }
  }
  if (!flock(LH, LOCK_EX | LOCK_NB))
  {
    print STDERR "$sProgram: Unable to lock the lock file ($!).'\n";
    exit(2);
  }

  ####################################################################
  #
  # Create/Update RRD databases as necessary.
  #
  ####################################################################

  foreach my $sClient (sort(keys(%hClients)))
  {
    my (@aTagList, $sLastOutFile);

    foreach my $sOutFile (@{$hClientOutFiles{$sClient}})
    {
      ################################################################
      #
      # Track whether there are any updates for this file. If there
      # aren't any, the file can be tagged as done.
      #
      ################################################################

      my ($sAbortClient, $sHaveUpdates);

      $sAbortClient = 0;
      $sHaveUpdates = 0;

      ################################################################
      #
      # Check for a tag file/link (i.e., <file>.out.d). If it exists,
      # assume that the .out file has already been processed.
      #
      ################################################################

      lstat($sOutFile . ".d");
      if (-f _ || -l _)
      {
        next;
      }

      ################################################################
      #
      # Open and parse the input file.
      #
      ################################################################

      my (@aLines, %hRecords);

      if (!open(FH, "< $sOutFile"))
      {
        print STDERR "$sProgram: Unable to open input file ($sOutFile): $!\n";
        $sAbortClient = 1;
        last;
      }

      while (my $sLine = <FH>)
      {
        my (@aFields);
        chomp($sLine);
        if (!ParseLine($sLine, \@aFields, \$sError))
        {
          print STDERR "$sProgram: Erroneous record in $sOutFile: '$sLine'\n";
          next;
        }
        $hRecords{$aFields[0]}{$aFields[1]} = $aFields[2]; # Mount point, time, and percent.
      }
      close(FH);

      ################################################################
      #
      # Create RRD databases as needed -- one DB per mount point. To
      # ensure unqiue and valid DS names compute and use an MD5 hash
      # of the mount point's raw path.
      #
      ################################################################

      foreach my $sMountPoint (sort(keys(%hRecords)))
      {
        my $sRrdFile = SetRrdFile($sRrdDir, $sClient, $sMountPoint);

        if (!-e $sRrdFile)
        {
          if (!$sBeQuiet)
          {
            print "Creating: RRD database for $sClient:$sMountPoint\n";
          }

          my $sDsName = substr(md5_hex($sMountPoint), 0, 16);

          RRDs::create(
            "$sRrdFile",
            "--start", $sRrdEpoch,
            "--step", "300",
            "DS:$sDsName:GAUGE:450:0:100",
            "RRA:AVERAGE:0.5:1:105120",
            "RRA:MIN:0.5:1:105120",
            "RRA:MAX:0.5:1:105120"
            );

          if (($sError = RRDs::error))
          {
            print STDERR "$sProgram: Error creating RRD database ($sRrdFile): $sError\n";
            $sAbortClient = 1;
            last;
          }
        }
      }
      last if ($sAbortClient);

      ################################################################
      #
      # Get the last DB update time for each mount point.
      #
      ################################################################

      my (%hLastUpdate);

      foreach my $sMountPoint (sort(keys(%hRecords)))
      {
        my $sRrdFile = SetRrdFile($sRrdDir, $sClient, $sMountPoint);
        $hLastUpdate{$sMountPoint} = RRDs::last($sRrdFile); # Get last update time.
        if (($sError = RRDs::error))
        {
          print STDERR "$sProgram: Error obtaining last update time from RRD database ($sRrdFile): $sError\n";
          $sAbortClient = 1;
          last;
        }
      }
      last if ($sAbortClient);

      ################################################################
      #
      # Loop through each record updating the DB as we go.
      #
      ################################################################

      my (%hLastTime);

      foreach my $sMountPoint (sort(keys(%hRecords)))
      {
        my $sRrdFile = SetRrdFile($sRrdDir, $sClient, $sMountPoint);
        my @aRecords = ();
        foreach my $sTime (sort({ $a <=> $b } keys(%{$hRecords{$sMountPoint}})))
        {
          if ($sTime > $hLastUpdate{$sMountPoint})
          {
            $sHaveUpdates = 1;
            if (!$sBeQuiet)
            {
              print "Updating: $sClient:$sMountPoint ", join(":", $sTime, $hRecords{$sMountPoint}{$sTime}), "\n";
            }
            push(@aRecords, join(":", $sTime, $hRecords{$sMountPoint}{$sTime}));
          }
          else
          {
            if (!$sBeQuiet)
            {
              print "Skipping: $sClient:$sMountPoint ", join(":", $sTime, $hRecords{$sMountPoint}{$sTime}), " ($sTime <= $hLastUpdate{$sMountPoint})\n";
            }
          }
          $hLastTime{$sMountPoint} = $sTime;
        }
        if (scalar(@aRecords))
        {
          RRDs::update($sRrdFile, @aRecords);
          if (($sError = RRDs::error))
          {
            print STDERR "$sProgram: Error updating RRD database ($sRrdFile): $sError\n";
            $sAbortClient = 1;
            last;
          }
        }
      }
      last if ($sAbortClient);

      ################################################################
      #
      # If there were no updates and the last record for each mount
      # point has a time that is less than the last update, add this
      # file to the tag list. Always check to see if there was a
      # file that preceded this one. If yes, add it to the tag list.
      # The assumption is that it's too late to add data points from
      # it anyway -- RRD does not let you backfill missing data.
      #
      ################################################################

      if (!$sHaveUpdates)
      {
        my $sTally = 0;
        foreach my $sMountPoint (sort(keys(%hRecords)))
        {
          if ($hLastTime{$sMountPoint} >= $hLastUpdate{$sMountPoint})
          {
            $sTally++;
          }
        }
        if ($sTally == 0)
        {
          push(@aTagList, $sOutFile);
        }
      }
      if (defined($sLastOutFile) && length($sLastOutFile))
      {
        push(@aTagList, $sLastOutFile);
      }
      $sLastOutFile = $sOutFile;
    }

    ##################################################################
    #
    # Conditionally tag files as done.
    #
    ##################################################################

    if ($sTagAsDone)
    {
      foreach my $sOldFile (@aTagList)
      {
        my $sTagFile = $sOldFile . ".d";
        if (!$sBeQuiet)
        {
          print "Tagging: $sClient $sOldFile --> $sTagFile\n";
        }
        if (!symlink($sOldFile, $sTagFile))
        {
          print STDERR "$sProgram: Error creating tag file ($sTagFile): $!\n";
        }
      }
    }
  }

  ####################################################################
  #
  # Set categories.
  #
  ####################################################################

  my $sCategory1 = "empty";
  my $sCategory2 = "growing";
  my $sCategory3 = "filling";
  my $sCategory4 = "crowded";
  my $sCategory5 = "full";

  ####################################################################
  #
  # Set colors. Many of the colors below were found here:
  #
  #   http://www.childoflight.org/mcc/colorcodeF.html
  #
  ####################################################################

  my $sColorBlack    = "#000000";
  my $sColorBlueDark = "#0D3981";
  my $sColorBlueLite = "#99CCFF";
  my $sColorGreen    = "#66FF66";
  my $sColorOrange   = "#FF9900";
  my $sColorPurple   = "#940B63";
  my $sColorRed      = "#FF0000";
  my $sColorYellow   = "#FCE503";
  my @aColors =
  (
    "#C91F16", "#E68601", "#CED500", "#69B011", "#088343", "#1F9AD7", "#113279", "#5B0B5A",
    "#CC2A13", "#F1A60A", "#BED20F", "#5AAA1D", "#00844A", "#1D93D1", "#152C74", "#6B015A",
    "#CC3615", "#EEAD00", "#B0C50F", "#47A41E", "#0E8C62", "#0D84C4", "#182369", "#820060",
    "#D34810", "#F5B600", "#9FC40D", "#349B26", "#008C7C", "#0D73B3", "#181C67", "#940B63",
    "#D9580E", "#FCCF03", "#8FBB0C", "#2D9C1F", "#05949D", "#0363A3", "#1D1259", "#AE0964",
    "#DA5F0E", "#FCE503", "#7FB513", "#209426", "#0994A6", "#085293", "#230E59", "#BD0062",
    "#DD680B", "#EDE400", "#77B312", "#189425", "#0894B6", "#094A91", "#310C5A", "#C50059",
    "#E37509", "#DFDC09", "#6FB20F", "#008C33", "#009DCF", "#06438A", "#4B0A5B", "#C70542",
  );

  ####################################################################
  #
  # Create a current list of RRD files for this client. To do this,
  # we clear out the old clients hash. Then, we populate it for each
  # client that has one or more RRD files. This approach weeds out
  # clients that don't have any RRD files (e.g., someone could have
  # deleted files from an existing client directory since the script
  # began).
  #
  ####################################################################

  %hClients = (); # Recycle the clients hash.

  foreach my $sClient (sort(@aClients)) # Note the use of @aClients here.
  {
    if (!opendir(DIR, $sRrdDir))
    {
      die "$sProgram: Unable to open directory ($sRrdDir): $!\n";
    }
    my @aRrdFiles = sort(grep(/(?:^|\/)$sClient[_]%2f.*\.rrd$/, map("$sRrdDir/$_", readdir(DIR))));
    if (scalar(@aRrdFiles))
    {
      $hClients{$sClient} = [ @aRrdFiles ];
    }
    closedir(DIR);
  }

  ####################################################################
  #
  # Create graphs. One reason for graphing all clients every time is
  # to allow the graphs to "slip" forward with time which can better
  # reveal a client that is tardy.
  #
  ####################################################################

  foreach my $sClient (sort(keys(%hClients)))
  {
    if (!$sBeQuiet)
    {
      print "Graphing: $sClient\n";
    }

    ##################################################################
    #
    # Filter out excluded mount points. Note: this step overwrites
    # the original list of mount points.
    #
    ##################################################################

    my ($sExcludeFile, $sLocalError);

    if (!defined(FilterMountPoints($sRrdDir, $sClient, \@{$hClients{$sClient}}, \$sLocalError)))
    {
      print STDERR "$sProgram: $sLocalError\n";
    }

    ##################################################################
    #
    # Dynamically generate DEFS, CDEFS, and LINES.
    #
    ##################################################################

    my @aDynamicRrdCdefs = ();
    my @aDynamicRrdDefNames = ();
    my @aDynamicRrdDefs = ();
    my @aRrdLines = ();
    my $sColorIndex = 1;
    my $sDynamicRrdCdefs = undef;

    foreach my $sRrdFile (@{$hClients{$sClient}})
    {
      my $sBasename = basename($sRrdFile, ".rrd");
      my $sMountPoint = substr($sBasename, length($sClient) + 1); # Plus 1 for the "_" delimiter.
      $sMountPoint =~ s/%([0-9a-fA-F]{2})/pack('C', hex($1))/seg;
      my $sDsName = substr(md5_hex($sMountPoint), 0, 16);
      my $sGraphName = "mount_point" . $sColorIndex;
      my $sFixedGraphName = $sGraphName . "_fixed";
      push(@aDynamicRrdCdefs, "CDEF:$sFixedGraphName=$sGraphName,UN,0,$sGraphName,IF");
      push(@aDynamicRrdDefs, "DEF:$sGraphName=$sRrdFile:$sDsName:AVERAGE");
      push(@aRrdLines, "LINE3:$sGraphName$aColors[$sColorIndex % scalar(@aColors)]:$sMountPoint");
      push(@aDynamicRrdDefNames, $sGraphName);
      $sColorIndex++;
    }

    if (scalar(@aDynamicRrdDefNames))
    {
      my @aRpnArgs = ();
      my $sMountIndex = 0;
      foreach my $sDefName (@aDynamicRrdDefNames)
      {
        push(@aRpnArgs, $sDefName . "_fixed");
        push(@aRpnArgs, "MAX") if (++$sMountIndex >= 2);
      }
      $sDynamicRrdCdefs = "CDEF:max_mount_point=" . join(",", @aRpnArgs);
    }
    else
    {
      $sDynamicRrdCdefs = "CDEF:max_mount_point=mount_point1_fixed";
    }

    ##################################################################
    #
    # Finalize the large/small options, and create the graphs -- one
    # for each defined timestamp ('-g' option).
    #
    ##################################################################

    foreach my $sTimestamp (@aGraphTimestamps)
    {
      my $sPngFileLarge = SetPngFile($sHtmDir, $sClient, $sTimestamp, "lg");
      my $sPngFileSmall = SetPngFile($sHtmDir, $sClient, $sTimestamp, "sm");
      my $sGraphStartTimestamp = SetGraphTimestampRrd($sTimestamp);
      my $sGraphStartPhrase = SetGraphTimePhrase($sTimestamp);
      my @aOptionsLarge =
      (
        "$sPngFileLarge",
        "--height", $sLargeHeight,
        "--imgformat", "PNG",
        "--lower-limit", "0",
        "--no-minor",
        "--start", $sGraphStartTimestamp,
        "--title", "Disk Utilization for $sClient ($sGraphStartPhrase)",
        "--width", $sLargeWidth,
        @aDynamicRrdDefs,
        @aRrdLines,
      );
      my @aOptionsSmall =
      (
        "$sPngFileSmall",
        "--height", $sSmallHeight,
        "--imgformat", "PNG",
        "--lower-limit", "0",
        "--only-graph",
        "--start", $sGraphStartTimestamp,
        "--width", $sSmallWidth,
        @aDynamicRrdDefs,
        @aDynamicRrdCdefs,
        $sDynamicRrdCdefs,
        "CDEF:zone_1=max_mount_point,$sZoneLimit1,LT,max_mount_point,0,IF",
        "CDEF:zone_2=max_mount_point,$sZoneLimit1,GE,max_mount_point,$sZoneLimit2,LT,max_mount_point,0,IF,0,IF",
        "CDEF:zone_3=max_mount_point,$sZoneLimit2,GE,max_mount_point,$sZoneLimit3,LT,max_mount_point,0,IF,0,IF",
        "CDEF:zone_4=max_mount_point,$sZoneLimit3,GE,max_mount_point,$sZoneLimit4,LT,max_mount_point,0,IF,0,IF",
        "CDEF:zone_5=max_mount_point,$sZoneLimit4,GE,max_mount_point,0,IF",
        "AREA:zone_1$sColorBlueLite:$sCategory1",
        "STACK:zone_2$sColorGreen:$sCategory2",
        "STACK:zone_3$sColorYellow:$sCategory3",
        "STACK:zone_4$sColorOrange:$sCategory4",
        "STACK:zone_5$sColorRed:$sCategory5\\n",
        "LINE1:max_mount_point$sColorBlack:",
        "AREA:zone_1$sColorBlueLite",
        "STACK:zone_2$sColorGreen",
        "STACK:zone_3$sColorYellow",
        "STACK:zone_4$sColorOrange",
        "STACK:zone_5$sColorRed",
      );

      push(@aOptionsLarge, "--upper-limit", $sLargeUpperLimit) if (defined($sLargeUpperLimit));

      push(@aOptionsSmall, "--upper-limit", $sSmallUpperLimit) if (defined($sSmallUpperLimit));

      RRDs::graph(@aOptionsLarge);
      if (($sError = RRDs::error))
      {
        print STDERR "$sProgram: Error creating PNG file ($sPngFileLarge): $sError\n";
      }

      RRDs::graph(@aOptionsSmall);
      if (($sError = RRDs::error))
      {
        print STDERR "$sProgram: Error creating PNG file ($sPngFileSmall): $sError\n";
      }
    }
  }

  ####################################################################
  #
  # Create header and/or footer notes (conditionally displayed).
  #
  ####################################################################

  my ($sBaseTitle, $sHeaderNotes, $sFooterNotes);

  $sHeaderNotes = "[ <a href=\"/\">Home</a> ] [ <a href=\"..\">Up</a> ]";
  $sFooterNotes = undef;
  $sBaseTitle = "Disk Utilization";

  ####################################################################
  #
  # Create client web pages.
  #
  ####################################################################

  foreach my $sClient (sort(keys(%hClients)))
  {
    my $sHtmFile = SetHtmFile($sHtmDir, $sClient);
    my $sTitle = "$sBaseTitle for $sClient";

    if (!$sBeQuiet)
    {
      print "Creating: client web page for $sClient\n";
    }

    if (!open(FH, "> $sHtmFile"))
    {
      die "$sProgram: Unable to open HTML file ($sHtmFile): $!\n";
    }

    print FH "<html>\n";
    print FH "<head>\n";
    print FH "<meta http-equiv=\"refresh\" content=\"$sMetaRefresh\">\n";
    print FH "<title>$sTitle</title>\n";
    print FH "</head>\n";
    print FH "<body>\n";
    print FH "<hr><p>$sHeaderNotes</p><hr>\n" if ($sHeaderNotes);
    print FH "<center>\n";
    print FH "<table border=\"1\">\n";
    foreach my $sTimestamp (@aGraphTimestamps)
    {
      my $sPngLink = SetPngLink($sClient, $sTimestamp, "lg");
      print FH "<tr><td align=\"center\"><img src=\"$sPngLink\"></img></td></tr>\n";
    }
    print FH "</table>\n";
    print FH "</center>\n";
    print FH "<hr><p>$sFooterNotes</p><hr>\n" if ($sFooterNotes);
    print FH "<center><p><tt><b>Page last updated: $sUpdateTime</b></tt></p></center>\n";
    print FH "</body>\n";
    print FH "</html>\n";
    close(FH);
  }

  ####################################################################
  #
  # Create time period web pages.
  #
  ####################################################################

  foreach my $sListTimestamp (@aGraphTimestamps)
  {
    my $sGraphStartPhrase = SetGraphTimePhrase($sListTimestamp);
    my $sGraphStartTimestampPadded = SetGraphTimestampPadded($sListTimestamp);
    my $sGraphLink = $sGraphStartTimestampPadded . ".html";
    my $sHtmFile = $sHtmDir . "/" . $sGraphLink;
    my $sTitle = "$sBaseTitle for $sGraphStartPhrase";

    if (!$sBeQuiet)
    {
      print "Creating: time period web page for $sGraphStartPhrase ($sListTimestamp)\n";
    }

    if (!open(FH, "> $sHtmFile"))
    {
      die "$sProgram: Unable to open HTML file ($sHtmFile): $!\n";
    }

    print FH "<html>\n";
    print FH "<head>\n";
    print FH "<meta http-equiv=\"refresh\" content=\"$sMetaRefresh\">\n";
    print FH "<title>$sTitle</title>\n";
    print FH "</head>\n";
    print FH "<body>\n";
    print FH "<hr><p>$sHeaderNotes</p><hr>\n" if ($sHeaderNotes);
    print FH "<center>\n";
    print FH "<table border=\"1\">\n";

    foreach my $sClient (sort(keys(%hClients)))
    {
      my $sPngLink = SetPngLink($sClient, $sListTimestamp, "lg");
      print FH "  <tr>\n";
      print FH "    <td align=\"center\"><a name=\"$sClient\"><img src=\"$sPngLink\"></img></a></td>\n";
      print FH "  </tr>\n";
    }

    print FH "</table>\n";
    print FH "</center>\n";
    print FH "<hr><p>$sFooterNotes</p><hr>\n" if ($sFooterNotes);
    print FH "<center><p><tt><b>Page last updated: $sUpdateTime</b></tt></p></center>\n";
    print FH "</body>\n";
    print FH "</html>\n";
    close(FH);
  }

  ####################################################################
  #
  # Create master web page
  #
  ####################################################################

  my ($sHtmFile, $sLineCount);

  $sHtmFile = $sHtmDir . "/index.html";

  if (!open(FH, "> $sHtmFile"))
  {
    die "$sProgram: Unable to open HTML file ($sHtmFile): $!\n";
  }

  print FH "<html>\n";
  print FH "<head>\n";
  print FH "<meta http-equiv=\"refresh\" content=\"$sMetaRefresh\">\n";
  print FH "<title>$sBaseTitle</title>\n";
  print FH "</head>\n";
  print FH "<body>\n";
  print FH "<hr><p>$sHeaderNotes</p><hr>\n" if ($sHeaderNotes);
  print FH "<center>\n";
  print FH "<table border=\"1\">\n";
  print FH "  <tr>\n";
  print FH "    <th align=\"center\" bgcolor=\"#cccccc\">&nbsp;</th>\n";
  print FH "    <th align=\"center\" bgcolor=\"#cccccc\">\n";
  print FH "      <table border=\"0\">\n";
  print FH "        <tr>\n";
  print FH "          <th bgcolor=\"#cccccc\" colspan=\"2\"><tt>Client</tt></th>\n";
  print FH "        </tr>\n";
  print FH "      </table>\n";
  print FH "    </th>\n";

  foreach my $sListTimestamp (@aGraphTimestamps)
  {
    my $sGraphStartPhrase = SetGraphTimePhrase($sListTimestamp);
    my $sGraphStartTimestampPadded = SetGraphTimestampPadded($sListTimestamp);
    my $sGraphLink = $sGraphStartTimestampPadded . ".html";
    print FH "    <th align=\"center\" bgcolor=\"#cccccc\">\n";
    print FH "      <table border=\"0\">\n";
    print FH "        <tr>\n";
    print FH "          <th bgcolor=\"#cccccc\" colspan=\"2\"><a href=\"$sGraphLink\"><tt>$sGraphStartPhrase</tt></a></th>\n";
    print FH "        </tr>\n";
    print FH "      </table>\n";
    print FH "    </th>\n";
  }
  print FH "  </tr>\n";

  foreach my $sClient (sort(keys(%hClients)))
  {
    my $sClientLink = $sClient . ".html";

    $sLineCount++;

    print FH "  <tr>\n";
    print FH "    <td bgcolor=\"#cccccc\" align=\"center\"><tt>$sLineCount</tt></td>\n";
    print FH "    <td bgcolor=\"#cccccc\" align=\"left\"><a href=\"$sClientLink\"><tt>$sClient</tt></a></td>\n";
    foreach my $sListTimestamp (@aGraphTimestamps)
    {
      my $sPngLinkLarge = SetPngLink($sClient, $sListTimestamp, "lg");
      my $sPngLinkSmall = SetPngLink($sClient, $sListTimestamp, "sm");
      print FH "    <td><a href=\"$sPngLinkLarge\"><img border=\"0\" src=\"$sPngLinkSmall\"></img></a></td>\n";
    }
    print FH "  </tr>\n";
  }

  print FH "</table>\n";
  print FH "</center>\n";
  print FH "<hr><p>$sFooterNotes</p><hr>\n" if ($sFooterNotes);
  print FH "<center><p><tt><b>Page last updated: $sUpdateTime</b></tt></p></center>\n";
  print FH "</body>\n";
  print FH "</html>\n";
  close(FH);

  ####################################################################
  #
  # Copy one of the master web pages over to the index web page.
  #
  ####################################################################

  my $sGraphStartTimestampPadded = SetGraphTimestampPadded($aGraphTimestamps[0]);
  my $sFirstSummaryClient = $sHtmDir . "/summary_client_" . $sGraphStartTimestampPadded . ".html";
  my $sIndex = $sHtmDir . "/index.html";
  copy($sFirstSummaryClient, $sIndex);

  ####################################################################
  #
  # Clean up and go home.
  #
  ####################################################################

  unlink($sLockFile);

  1;


######################################################################
#
# FilterMountPoints
#
######################################################################

sub FilterMountPoints
{
  my ($sRrdDir, $sClient, $paRrdList, $psError) = @_;

  ####################################################################
  #
  # Read global/local exclude files, and build an array of filters.
  #
  ####################################################################

  my (@aFilters, $sGlobalExcludeFile, $sLocalExcludeFile);

  $sGlobalExcludeFile = $sRrdDir . "/" . "df.exclude";
  $sLocalExcludeFile = $sRrdDir . "/" . $sClient . ".exclude";
  foreach my $sExcludeFile ($sGlobalExcludeFile, $sLocalExcludeFile)
  {
    if (defined($sExcludeFile) && -f $sExcludeFile)
    {
      if (!open(HFilters, "< $sExcludeFile"))
      {
        $$psError = "Unable to open $sExcludeFile ($!)";
        return undef;
      }
      else
      {
        while (my $sLine = <HFilters>)
        {
          chomp($sLine);
          push(@aFilters, $sLine);
        }
        close(HFilters);
      }
    }
  }

  ####################################################################
  #
  # Filter out unwanted mount points. It's important to splice() the
  # array in reverse order because its indexes change each time it's
  # modified.
  #
  ####################################################################

  my (@aFilteredIndexes);

  for (my $sIndex = 0; $sIndex < scalar(@$paRrdList); $sIndex++)
  {
    my $sRrdFile = $$paRrdList[$sIndex];
    my $sBasename = basename($sRrdFile, ".rrd");
    my $sMountPoint = substr($sBasename, length($sClient) + 1); # Plus 1 for the "_" delimiter.
    $sMountPoint =~ s/%([0-9a-fA-F]{2})/pack('C', hex($1))/seg;
    foreach my $sFilter (@aFilters)
    {
      if ($sMountPoint =~ /$sFilter/)
      {
        push(@aFilteredIndexes, $sIndex);
        last;
      }
    }
  }
  foreach my $sIndex (reverse(sort({ $a <=> $b } @aFilteredIndexes)))
  {
    splice(@$paRrdList, $sIndex, 1);
  }

  1;
}


######################################################################
#
# GetSeconds
#
######################################################################

sub GetSeconds
{
  my ($sDate, $sTime, $sPreference) = @_;

  my ($sYear, $sMonth, $sMonthDay) = split(/[-]/, $sDate);
  my ($sHours, $sMinutes, $sSeconds) = split(/[:]/, $sTime);

  if ($sPreference =~ /^local$/i)
  {
    return timelocal(
      $sSeconds,
      $sMinutes,
      $sHours,
      $sMonthDay,
      $sMonth - 1,
      $sYear - 1900
      );
  }
  else
  {
    return timegm(
      $sSeconds,
      $sMinutes,
      $sHours,
      $sMonthDay,
      $sMonth - 1,
      $sYear - 1900
      );
  }
}


######################################################################
#
# ParseLine
#
######################################################################

sub ParseLine
{
  my ($sLine, $paFields, $psError) = @_;

  my $sLineRegex1 = '(\d{4}-\d{2}-\d{2})\|([0-9:]{8})\|[^|]+\|\d+\|\d+\|\d+\|(\d+)%\|([^|]+)';

  if ($sLine =~ /^$sLineRegex1$/)
  {
    my ($sDate, $sTime, $sPercentUsed, $sMountPoint) = ($1, $2, $3, $4);
    my $sSeconds = GetSeconds($sDate, $sTime, "local");
    @$paFields = ($sMountPoint, $sSeconds, $sPercentUsed);
  }
  else
  {
    return undef;
  }

  1;
}


######################################################################
#
# SetGraphTimePhrase
#
######################################################################

sub SetGraphTimePhrase
{
  my ($sTimestamp) = @_;

  my $sTimePhrase;
  my $sTimeUnit;
  my $sTimeValue;

  $sTimeUnit  = SetGraphTimeUnit($sTimestamp);
  $sTimeValue = SetGraphTimeValue($sTimestamp);

  if ($sTimeUnit =~ /^d$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Day" : "Days";
  }
  elsif ($sTimeUnit =~ /^h$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Hour" : "Hours";
  }
  elsif ($sTimeUnit =~ /^m$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Month" : "Months";
  }
  elsif ($sTimeUnit =~ /^w$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Week" : "Weeks";
  }
  elsif ($sTimeUnit =~ /^y$/)
  {
    $sTimePhrase = ($sTimeValue == 1) ? "Year" : "Years";
  }
  else
  {
    return "unknown";
  }

  return (sprintf("%s %s", $sTimeValue, $sTimePhrase));
}


######################################################################
#
# SetGraphTimeUnit
#
######################################################################

sub SetGraphTimeUnit
{
  my ($sTimestamp) = @_;

  my $sTimeUnit;
  $sTimeUnit = $sTimestamp;
  $sTimeUnit =~ s/^\d+([dhmwy])$/$1/;
  return ($sTimeUnit);
}


######################################################################
#
# SetGraphTimeValue
#
######################################################################

sub SetGraphTimeValue
{
  my ($sTimestamp) = @_;

  my $sTimeValue;
  $sTimeValue = $sTimestamp;
  $sTimeValue =~ s/[dhmwy]$//;
  return ($sTimeValue);
}


######################################################################
#
# SetGraphTimestampPadded
#
######################################################################

sub SetGraphTimestampPadded
{
  my ($sTimestamp) = @_;

  return (sprintf("%06s", $sTimestamp));
}


######################################################################
#
# SetGraphTimestampRrd
#
######################################################################

sub SetGraphTimestampRrd
{
  my ($sTimestamp) = @_;

  return (sprintf("-%s", $sTimestamp));
}


######################################################################
#
# SetHtmFile
#
######################################################################

sub SetHtmFile
{
  my ($sHtmDir, $sClient) = @_;

  return (sprintf("%s/%s.html", $sHtmDir, $sClient));
}


######################################################################
#
# SetPngFile
#
######################################################################

sub SetPngFile
{
  my ($sHtmDir, $sClient, $sTimestamp, $sSize) = @_;

  return (sprintf("%s/%s", $sHtmDir, SetPngLink($sClient, $sTimestamp, $sSize)));
}


######################################################################
#
# SetPngLink
#
######################################################################

sub SetPngLink
{
  my ($sClient, $sTimestamp, $sSize) = @_;

  if ($sSize =~ /^lg$/i)
  {
    return (sprintf("png/%s_%s_lg.png", $sClient, SetGraphTimestampPadded($sTimestamp)));
  }
  elsif ($sSize =~ /^sm$/i)
  {
    return (sprintf("png/%s_%s_sm.png", $sClient, SetGraphTimestampPadded($sTimestamp)));
  }
  else
  {
    return (sprintf("png/%s_%s_xx.png", $sClient, SetGraphTimestampPadded($sTimestamp)));
  }
}


######################################################################
#
# SetRrdFile
#
######################################################################

sub SetRrdFile
{
  my ($sRrdDir, $sClient, $sMountPoint) = @_;

  $sMountPoint =~ s/([%\/\\])/sprintf("%%%02x",unpack('C',$1))/seg;

  return (sprintf("%s/%s_%s.rrd", $sRrdDir, $sClient, $sMountPoint));
}


######################################################################
#
# Usage
#
######################################################################

sub Usage
{
  my ($sProgram) = @_;
  print STDERR "\n";
  print STDERR "Usage: $sProgram [-lqt] [-e rrd-epoch] [-g graph-times] [-r base-rrd-dir] [-S WxH] [-s WxH] [-U limit] [-u limit] [-Z limits] -i base-in-dir -o base-out-dir\n";
  print STDERR "\n";
  exit(1);
}


######################################################################
#
# ValidateGraphTimestamp
#
######################################################################

sub ValidateGraphTimestamp
{
  my ($sTimestamp, $psError) = @_;

  my $sTimeValue;

  if ($sTimestamp !~ /^\d+[dhmwy]$/)
  {
    $$psError = "Invalid graph timestamp ($sTimestamp).";
    return undef;
  }

  $sTimeValue = SetGraphTimeValue($sTimestamp);

  if ($sTimeValue !~ /^\d+$/ || $sTimeValue <= 0)
  {
    $$psError = "Invalid graph time value ($sTimeValue).";
    return undef;
  }

  1;
}


=pod

=head1 NAME

process_df - Process harvest_df data

=head1 SYNOPSIS

B<process_df> [B<-lqt>] [B<-e rrd-epoch>] [B<-g graph-times>] [B<-r base-rrd-dir>] [B<-S WxH>] [B<-s WxH>] [B<-U limit>] [B<-u limit>] [B<-Z limits>] B<-i base-in-dir> B<-o base-out-dir>

=head1 DESCRIPTION

This utility processes harvest_df data and produces disk utilization
graphs for 30, 7, and 1 days.

=head1 OPTIONS

=over 4

=item B<-e rrd-epoch>

Specifies the start epoch for each RRD.  This value may be any legal
RRD start time (see rrdcreate(1)).  If you want all your systems
to be on the same page, it's probably best to specify an absolute
time.  The default value is "-30d".

=item B<-g time1,time2,time3...>

Specifies the order and time ranges graphs will be created.  Times are
specified in the following format:

  <number><unit>

Where <number> is a positive integer greater than zero and <unit> is
one of the letters represented in the table below.

  unit   meaning
  ----   --------
   d     Day(s)
   h     Hour(s)
   m     Month(s)
   w     Week(s)
   y     Year(s)

<number> and <unit> combined represent the amount of past time
(starting from now) to include in the graph that is to be created.
For example, '1d' produces a graph for the last day, '3m' produces
a graph for the last three months, '4w' produces a graph for the
last four weeks, and so on.

The default value for this option is "30d,7d,1d".

=item B<-i base-in-dir>

Specifies the base input directory.  This option is mandatory.

=item B<-l>

Instructs the program to process symbolic links to client directories
rather than the actual directories.  This is useful for mapping
cryptic/encoded names (e.g., CLLI - Common Language Location
Identification ) into human-readable names.

=item B<-o base-out-dir>

Specifies the base output directory.  This is where web pages and
graphs are created.  This option is mandatory.

=item B<-q>

Instructs the program to run quietly.

=item B<-R seconds>

Specifies the META refresh rate that is to be included in each web
page.  Values must be a decimal number greater than 60 (seconds).  The
default value is 3600 (seconds).

=item B<-r base-rrd-dir>

Specifies the base RRD directory.  This is where RRDs are stored.  If
not specified, the value for this option defaults to B<base-out-dir>.

=item B<-S WxH>

Specifies the width and height (in pixels) for large graphs.  The
default value is 600x100.

=item B<-s WxH>

Specifies the width and height (in pixels) for small graphs.  The
default value is 150x25.

=item B<-t>

Instructs the program to tag (using a symbloc link) old input files
as done (".d").

Note: There must be more than one input file for this action to take
place.  In other words, the current file is never tagged as done.
This is because it may still be in flux.

=item B<-U limit>

Specifies the upper limit to use on large graphs.  Values may be 1-100.
There is no default value.

=item B<-u limit>

Specifies the upper limit to use on small graphs.  Values may be 1-100.
There is no default value.

=item B<-Z limit1,limit2,limit3,limit4>

Specifies the four limits that create the five coloring zones.  The
limits must be separated by commas, listed from highest to lowest, can
not be equal to one another, and must be greater than 0.  The table
below shows the definition for each zone using the limit values.

 zone   condition
 ----   ------------------------------------
  1     value < limit1
  2     value >= limit1 and value < limit2
  3     value >= limit2 and value < limit3
  4     value >= limit3 and value < limit4
  5     value >= limit4

This option is optional and the table below shows the default limits.

         default
 limit    value
 -----   -------
   1      10
   2      70
   3      85
   4      95

Note: This option applies only to the small images.

=back

=head1 AUTHOR

Klayton Monroe

=head1 CREDITS

This work is based on previous works created (in part or whole) by the
following people:

    Bair, Andy
    Leininger, Hank
    Monroe, Klayton

=cut
--- process_df ---