Revision $Id: webjob-run-periodic-bundles.base,v 1.2 2005/12/22 13:37:14 pab Exp $ Purpose This recipe demonstrates a solution for centrally managing WebJob tasks, which is called the periodic-bundle approach. Motivation The motivation for this recipe is to create a simple scheme that can be used to centrally manage and automate common administrative tasks. In large, complex production environments, ad hoc administrative practices tend to increase the amount of variance between similarly configured systems. As variance increases, the amount of time spent doing ad hoc administration also tends to increase. To combat this degenerative cycle, administrators should replace ad hoc techniques in favor of programmatic or mechanized techniques. This practice will also help to reduce the amount of oversights, errors, and duplicated effort. Requirements Cooking with this recipe requires an operational WebJob server. If you do not have one of those, refer to the instructions provided in the README.INSTALL file that comes with the source distribution. The latest source distribution is available here: http://sourceforge.net/project/showfiles.php?group_id=40788 The server must be running UNIX and have basic system utilities, Apache, WebJob (1.5.0 or higher), and PaD installed. Note: PaD tools are included in the WebJob distribution. Each client must be running UNIX and have basic system utilities (including tar and uuencode) and WebJob (1.5.0 or higher) installed. The commands presented throughout this recipe were designed to be executed within a Bourne shell (i.e., sh or bash). Note that this recipe was built and tested on FreeBSD 5.4 so mileage may vary if implementing on other operating systems. Time to Implement Assuming that you have satisfied all the requirements/prerequisites, this recipe should take less than one hour to implement. Plan on 1-3 hours for testing. Solution The solution is to create one or more shell scripts for one or more clients, then execute these shell scripts periodically with WebJob. The key to this solution is the framework shown below which has three distinct tiers: top, group, and client. The top tier consists of files in the directory, the group tier consists of files in the directory, and the client tier consists of files in the directory. <--- top tier | - header-00 - | + <--- group tier | - | + <--- client tier | - Each tier can contain zero or more files, where each file contains a single WebJob command. These files are called 'command files' and are generically depicted in the framework as , , and respectively. This framework enables you to perform two major functions: overriding a job assigned at a higher level, and running jobs in a certain time window. The tiered system is key to overriding jobs assigned at a higher level. Generally speaking, the script for each client is built using the contents of these command files: header-00, , , and respectively. The command files at the top tier can be overridden in the group or client tiers by using the same command file name. Similarly, command files at the group tier can be overridden in the client tiers by using the same command file name. Note that jobs are ordered alphabetically based on the file name. For example, assume there is a file named lsof in the top tier with the contents shown below. Since the command file (lsof) is in the top tier, all clients will include that command in their script. Assume that you do no want to run that command on a single client named client-1. To disable this job for client-1, just touch a file named lsof in the client-1 directory. When the script is built for client-1, the lowest level command file will be used. Since that file is blank, you will effectively turn off the lsof command for client-1. webjob -e -f upload.cfg lsof The second major function provided by this recipe is the ability to run jobs in a certain time window which is provided by the RunNow script function. This script is contained in the command file header-00, which is a special header file included in each . The RunNow function tests to see if the current time is equal to or contained in the range to . RunNow returns true if the conditions are satisfied, otherwise it returns false. Using RunNow and the logical AND (&&) you can create periodic jobs and one-shot jobs. For example, assume your WebJob clients run every 15 minutes. The first example below runs the hostname command once an hour in the first quarter hour. The second example below runs the hostname command on September 1, 2005 during the 1:00 AM hour. For more detailed information on the RunNow function, please read the function header in the header-00 file. RunNow "00-*-*-*-*-*" "15-*-*-*-*-*" && \ webjob -e -f upload.cfg hostname RunNow "00-01-01-09-2005-*" "15-01-01-09-2005-*" && \ webjob -e -f upload.cfg hostname The following steps describe how to implement this solution. 1. Set WEBJOB_BASE_DIR as appropriate for your server. Then, extract webjob-run-periodic-bundles.tgz.uu from this recipe (Appendix 1), uudecode it, and unpack its contents in a suitable location. # WEBJOB_BASE_DIR=/var/webjob # sed -e '1,/^--- webjob-run-periodic-bundles.tgz.uu ---$/d; /^--- webjob-run-periodic-bundles.tgz.uu ---$/,$d' webjob-run-periodic-bundles.txt > webjob-run-periodic-bundles.tgz.uu # uudecode webjob-run-periodic-bundles.tgz.uu # tar -C ${WEBJOB_BASE_DIR} -zxf webjob-run-periodic-bundles.tgz If you don't have uudecode on your system, try the following: # perl -ne 'next if (/^begin/); last if (/^end/); chomp; print unpack("u",$_);' < webjob-run-periodic-bundles.tgz.uu > webjob-run-periodic-bundles.tgz At this point, you should have a tree that looks like this: webjob-run-periodic-bundles/Mk/bundles.base webjob-run-periodic-bundles/Mk/bundles.inc webjob-run-periodic-bundles/Mk/clients.base webjob-run-periodic-bundles/Mk/clients.inc webjob-run-periodic-bundles/Mk/common.inc webjob-run-periodic-bundles/Mk/groups.base webjob-run-periodic-bundles/Mk/groups.inc webjob-run-periodic-bundles/Mk/header-00.base webjob-run-periodic-bundles/Mk/top.inc webjob-run-periodic-bundles/Makefile webjob-run-periodic-bundles/bundles/Makefile 2. Set your the Makefile variables as appropriate. First change to the webjob-run-periodic-bundles directory. # cd webjob-run-periodic-bundles Next, edit Mk/common.inc and set the PROJECT_DIR and PROFILE_DIR macros as appropriate for your environment. PROJECT_DIR Specifies the location to the webjob-run-periodic-bundles environment extracted in step 1. The default value is /var/webjob/webjob-run-periodic-bundles. PROFILE_DIR Specifies the location to the WebJob profiles directory for your WebJob server. The default value is /var/webjob/profiles. 3. Create the bundle and group structure as appropriate for your needs. First, choose one of two general approaches to implement discussed below. a. The first approach is to create a single bundle for each client and have the clients run the bundle through WebJob at the highest time frequency desired. The advantage of this technique is that it is easier to manage a single bundle on the server and simpler to setup execution of a single bundle on each client. The disadvantage of this technique is that it requires more extensive use of the RunNow function to implement jobs that occur less frequently than the execution time frequency. If implementing this technique, execute make as shown in the example below. The example commands create a single bundle named bundle. Note that B is the variable for bundle names, G is the variable for group names, and C is the variable for client names. The bundle names and group names can be anything you choose, but the client names must correspond to profiles setup in the WebJob profiles directory. See the internal documentation in the Makefile for explanations of other make targets. # make add B="bundle" G="group1" C="client1 client2" # make add B="bundle" G="group2" C="client2 client3" b. The second approach is to create several bundles for each execution period (e.g., hourly, daily, weekly, ...) and have each client run the appropriate bundle through WebJob at the appropriate time frequency. The advantage of this technique is that it may be more intuitive to segment the bundles based on their execution frequency and this technique will probably require less use of the RunNow function which will reduce the performance impact on the client. The disadvantages of this technique is that it requires more setup for each client. If implementing this technique, execute make as shown in the example below. The example commands create three bundles named hourly, daily, and weekly. Note that B is the variable for bundle names, G is the variable for group names, and C is the variable for client names. The bundle names and group names can be anything you choose, but the client names must correspond to profiles setup in the WebJob profiles directory. See the internal documentation in the Makefile for explanations of other make targets. # make add B="hourly" G="group1" C="client1 client2" # make add B="hourly" G="group2" C="client2 client3" # make add B="daily" G="group1" C="client1 client2" # make add B="daily" G="group2" C="client2 client3" # make add B="weekly" G="group1" C="client1 client2" # make add B="weekly" G="group2" C="client2 client3" Using either approach you can either run 'make show' or 'make tree' to view the webjob-run-periodic-bundles structure. The show target uses find to display results, and the tree target uses the tree program to display results. # make show # make tree Lastly, you can undo the add commands by using the 'del' target. The example command below will delete client1 and client2 from group1 in bundle weekly. # make del B="weekly" G="group1" C="client1 client2" 4. Now that you have a basic structure setup, you can manage jobs. Generally speaking, the script for each client is built using the contents of these command files: header-00, , , and respectively. The command files at the top tier can be overridden in the group or client tiers by using the same command file name. Similarly, command files at the group tier can be overridden in the client tiers by using the same command file name. Note that jobs are ordered alphabetically based on the file name. Although you are free to choose any naming convention for command files, I suggest placing one WebJob command in each command file and using the naming convention shown below for command file names. This will provide greater manageability of a large number of jobs. If using the recommended naming convention, is the program executed by WebJob, and is used to optionally define the command. For example, you might use when you are running the same command with different options. [-...] The examples below demonstrate different command file names. lsof ftimes-dig ftimes-map-all ftimes-map-slash ftimes-map-var The steps below demonstrate how to create periodic, one-shot, and overriding jobs. a. Create periodic jobs by creating one or more command files at the appropriate tier (top, group, or client) containing the WebJob to execute. For example, the ftimes-map-all command file shown below will execute ftimes in map mode through WebJob, mapping the whole drive. Note that if you place this command file at the top tier, every client configured for the bundle will execute this command. Similarly, if you place this command file at the group tier, every client configured for the respective group will execute this command. Lastly, if you place this command file at the client tier, only the respective clients will execute this command. --- ftimes-map-all --- webjob -e -f upload.cfg ftimes --mapauto all-magic --- ftimes-map-all --- Note that if this command file is contained in a bundle that executes every hour but you want to execute this command once a day, you should use the RunNow function as shown below. This example runs the WebJob command when the current time is during the 1:00 AM hour. For more detailed information on the RunNow function, please read the function header in the header-00 file. --- ftimes-map-all --- RunNow "00-01-*-*-*-*" "59-01-*-*-*-*" && \ webjob -e -f upload.cfg ftimes --mapauto all-magic --- ftimes-map-all --- b. Create one-shot jobs by creating one or more command files at the appropriate tier (top, group, or client) using the RunNow function AND'ed with the the WebJob to execute. For example, the ftimes-map-all command file shown below will execute ftimes in map mode through WebJob, mapping the whole drive September 1, 2005 during the 1:00 AM hour. For more detailed information on the RunNow function, please read the function header in the header-00 file. --- ftimes-map-all --- RunNow "00-01-01-09-2005-*" "15-01-01-09-2005-*" && \ webjob -e -f upload.cfg ftimes --mapauto all-magic --- ftimes-map-all --- c. Create overriding jobs as appropriate. To override a bundle command file create a command file of the same name in the group or client tiers as appropriate. Note that if the command file is empty, you will disable that command for the respective group or client. d. Make the command bundles by executing the command shown below. This will bundle concatenate the contents all command files into a single bundle file for each client. The bundle file is placed in the staging directory. # make build You can clean all files from the staging directory by executing the command below. # make clean e. Install the command bundles by executing the command shown below. This will copy the bundle files from the staging directory to the WebJob profiles directories. Note that the install target will overwrite any existing filenames and it is your responsibility to make backup copies of these files if you wish to rollback the changes. Currently there is no easy way to roll back any changes made by the install target. # make install You can uninstall all bundle files from the respective WebJob client profiles by executing the command below. # make uninstall 5. Periodically schedule each execution bundle through cron using WebJob. For example, if you created a single bundle named periodic-bundle for each client and wish to execute that bundle every hour, you would use a cron entry similar to the one shown below. See the crontab(5) man page for more information on cron. 0 * * * * webjob -e -f upload.cfg periodic-bundle On the other hand, you may have decided to create three bundles, one for each execution period: hourly, daily, and weekly respectively. In this case, you would use a cron entry similar to the one shown below. See the crontab(5) man page for more information on cron. 0 * * * * webjob -e -f upload.cfg hourly 0 0 * * * webjob -e -f upload.cfg daily 0 0 * * 0 webjob -e -f upload.cfg weekly Note that you should spread out client check-in times to the WebJob server to help reduce server load and allow for a maintenance window on the WebJob server. The maintenance window is typically used to process uploaded WebJob data. The example table below shows client hourly jobs staggered from minutes 0 through 29, creating a maintenance window from minutes 30 through 59 on the WebJob server. client crontab entry -------- ----------------------------------------- client0 0 * * * * webjob -e -f upload.cfg hourly client1 1 * * * * webjob -e -f upload.cfg hourly client2 2 * * * * webjob -e -f upload.cfg hourly ... client27 27 * * * * webjob -e -f upload.cfg hourly client28 28 * * * * webjob -e -f upload.cfg hourly client29 29 * * * * webjob -e -f upload.cfg hourly client30 0 * * * * webjob -e -f upload.cfg hourly client31 1 * * * * webjob -e -f upload.cfg hourly client32 2 * * * * webjob -e -f upload.cfg hourly ... 6. Test and implement your bundles. Closing Remarks This recipe uses make as the primary utility to consolidate job bundles because it relies heavily on file dependencies. This is a major capability of make and is accomplished using recursive calls to make. Unfortunately, the technique may fail when there are a large number of file dependencies, although this has not been tested. Another issue is that the make file is a bit clunky when determining what job files to use for overriding jobs. Given these issues, a more elegant approach may be a combination of make and Perl. Credits This recipe was brought to you by Andy Bair. References Appendix 1 --- webjob-run-periodic-bundles.tgz.uu --- begin 644 webjob-run-periodic-bundles.tgz M'XL(`(^`JT,"`^T=_5?;.))?R5^AYG*E[3;$=A)XI1NVE%+(78&^D!Z/5[J+ MB9W@:V)G':>4H_SOIR_+DBT[=N(`V[7VKDED:32:+\V,9'%M7O[7N:RZ4[LZ M-EW+,:Q>]7)J&T-S4EO)J2A*0]EL-N&GHJB-AO!)RXJJU)O:IJ)HZL:*HC:5 M9F,%-%?NH4PGGNX"L/)U.)K1SG0G*S]=N4[@_^'7VD/QOZ'5"_X_`O[3K^N7 M^L2EFW[-YP:IB@O+Y>@__C M>`Z?E$LK1?FI2TK]A[*PLBS]KVOUS;#^-SNS.$8?J&!]??V\#'Z`7?BS%_P4GO+/ M[LHA8(8YS`_8Y=0:&N'*WM#4[7"E94/)&P[#U5,[YL'DRKD.UWFN:98%&GVW M/*"62A*SWG-&(\-!"]RKU&.=Q&>`A> MDUF]!FRJT=;-VEU=7*[>'.O_?NL$9"/&!5WR*?N!OIM`\[577Z8SMB_&U`75"O_X*JN_/:V#M=NQ:M@8,D MHT+PA<0B..RVR/BE$J3A3SLW;!?P[.2]2R5L)!);4,.0V(:9C\162"AQ@SEF M#'_*)ETJ(6.4-]#"07M8_Z\WM$S;6WK\MQ&)_^I*X?\]1/P7N`N4\T446.C_ MPEMH-K&5P!FO97<+WK:/W M5>TU\+VXJ!-76]`_S&.$PIKG;_^9(*\LS?XK6C3_WU"UPO[?1_G8.?[7WF[W MCW?MSFH+U+[I;HV(1"U!,DJPU_OVASU)K['K]"W4Y.VGHW>L1>66&^?.SRJ7 M3KH[^S%-(%<&ECTH=/IA]7_@.M/Q@N%?BOBO'M;_#:5>Z/\#Q7\!VY'_`@H5 M+/1_J>N_UMS0(NM_LUC_B_@ODE/7\D\6..Q!>VP#9B]JJ$#:V- M)>]4R5<6(3(6-K/RV*S!4:0JBR+WL^U)S-K=2=RZ^DO/1)JM2)6J"#4"M,3E M*\+-X[,5V5,5ZHQ$PH)YBH7`_Z76_RM3-TRWJBB+A`#)Z[^F;JIT_P=Z_4U% MQ?L_6G'^YW[6_R>U2\NN3:YR78V.=@[W\-D(4CI3^\BY!E70-2<>LI&]J>N: M4'\\:P27V`E<98&''KFZ/3`QA).SH^./)^V3*)1?42?(M-%8W0:?@U_:]A?< M\]W>R6ZG_;';/C[B.G>OX##0F?4LVP2NZ4U=>P(7YJF)T($V.H*2^>=4'W(` M/$<U;?,B=X1-8,#71I8BJ:QCJ:+_'0N;;L`?SFCG1O M2\"$XG-XN%W]]>``_F,8\)_1"/YS`PO\N-Z.=("C3TSP31].(9ZZBYR./B;4 MI0F'6H\9`'ZTP*VB5)NO?KRX@UZ)94\],]H4HL&::G7<],J9NM&&$%76L*[B MAH9^`QS"<>B'>%?13G!JK).J$4102[_;C:E+AL*T0+TT!6'UZA690J-J6`/H M[,D[7:.14">UNHF;7YOF5XSBLT/';JDO07=JMK27R(E]CHG6V>M^ZAR!_^Q\ M^+1W\KC%.X34M34<`GTX<>#H$XB7#JYUUT92!S'D$;DRH;P@F3&_CRT7C@Y1 MF=K^#S;"A$C1SJ?NP7&'Z[]C&S?@K6ZY.;JRQ/(\>UZZA5[-X=D?W?;AWDEW MY_"CVJJHH2JM5=%(U?M/1]"M(GWAN@]`+LA`,.C_8.^[Y^H]#^BV@13-,G0/ MJK+E3CQ.[VGC7(B`YW38/E);%R2V@>X-3PKH14)7J3?U0-4XKX)J7[T@?0XZ M25W$/AKM<_ANYRRAE]"G[O%_C/JR\_SE\\KY1]2=X&-E<#7X4??!Q\%%?`J#Q7S`D^+!:A1L'0\Q`RQA903P<80."@&J MX@$X4L0"/I;0X2Q*!PW##?[Q0`0LT^A$Q+(1`*_':)V,HP&&F0V-XZ.YL(C# MX#@K%\ZR<@'Y%''#GV7EPFEV+B!G)8$'I^EX@$?G%M1RF:'&:W0+E%^4!<0B M'>E,.&-$-`L$[1TE90+.R)4PDPA9-[M!I+>[E2T,<1:LSA MY\H@GT6ID=W3E0$^E5!C#E^7@L;00_ZN)G,T,GN\&N?Q:B%'(^1J8,,FP05) M5$94)+ZO%OB^<^)!!#`K46*\8(WW@N=%Z/AH3GSB<#F>ET=GV7DD\8RUP#-. M@T<$B]-Y."3WD#7>0TZ-C+AP:]1/#GG*FM2#D'0..P*:Z"MS/D3*SO[P40(* M=G`AY'RO>2[L"`*QZ%&;MQA^O@,V'X($A3@,#Q9G[L$"S#U(9&X[!^:V%V%N M.XFYIWDP]W0AYIY&F+L$3WOWRNQ]Y2)Q?P/FF'E78C@=NG"GHZ39PIVACP)H`=*J#03"F+DKKDP?\ MML-DZ'CKR_#T4_GYO#7G`B!0-?\$A,6L]BXD))0F"C7)'/]C@0Z\>*"KT17C M=*=SU#[:WP*?HCL==+7@P*^!9Y(QMR,C/B=+2VDUM*;0*23AL3D9I3ON.4U#'D\?DVXN`&SB3GMC$/*?!&A'4_S],QN].)!P,"NFA. M,%SS\G>+]WT8S_/[71O'^?W'^>\;Y[[>M\_*E"CY?:NAXS)?S MLG]+T^+4 MAXC3'!^./3@L?_BW?_X&BAE[F7'2,9/O?V%3]^Q^:==00V?^F5KS_ M4]C_POZGL?_^5*Q^/USGFM],UY.B5>6'0I\HXY.KF]NF2Q%9@P!4LX'I39;R MAM$]7&^XZYHHLB"V$:Q=JFLO_1.R*-N%7Y0$:X-0-;D_!ZSUU+7UA%L257(5 MH3K[AD.NDS;[1L+40T@FJ,DGJ,DGHE&+9A#L>P.,.\'^NL^W!% M)X.AY1Y/]#4P'/PL3N7]GTM&#>.VWSG^])&BMG\'`MQF\8M@R*P#`A/EW*KA MV";]SGWE&8K7&V;9\<(3HAE/,M_"&)CU4J*)+%\=?34L-ZDU>6E[E6VFQ+6C M_*02N-H;@VH?A/Y62GQ_EE4IA_N+[UHG0&`-*0A$4$*]P'@\`/T>'5U"<@W% M.I#J%/.O<>(L4&(6*?B.OE#-D"J^BZ^4A)"A=T9!0!?N;JXT@",$QGHHB`^G MW87\/`[YN2]AP'/=_=#>.^J2R>YRD\TV6_2-0(I,//W,!2!,B5)KD="=SC"6 MB`$5^>O.,PX3HF@`W%>SD+[EF*+*^>(>MDL23`Y,A6E>'-D*MO<2F`J:@.J M;>#/+\0:D?#)C0GEED$7PQS2['*J7"QLCLYQQ>5AXY*DTB$HK`!\,J@D&"B? M1(T-R0C2%*`DQQ1*,P9P^1R59/9UR>SK491I:HRDI\@%*J137)(TP$>"2!HB MB=01.@M42,4\CB2/,H30(%;WF4-#P_V]70>,L^*X9W1Z#J]E.DQ%)EQ$A8(ZK?+`SB`NQ( MB#8+55G*)6O&):!I,)4,&9&$9$%L;!T-1=-/5!XASQT@AP)6.4ER]./Q@9D< M':W@F&%"A/.`EAKCYQ^MX^RT&%7%MPX:YQQ0X=-`.3(B.-")_T3$A_9)EU&= M_<$'253]8OU2_^J3#6D,ZDZ-*8,D&J^^_^Q.L%2LDDDN_I8[T=`1JKP)5_4/ ML*8A'B46K(IOR-2F#YBD26'/(N`#-(,N9L;>K@N1W;QIXV) M)4W'A)P-3&AM\:TZH3N0Y^NY/]^2D*TE2\DN";[IWW4!AN6:/<]Q;R!@:)G0 MZW/0`9EX6R`$UG^[19:)1=D*RYZ:X90_-'*KO&^[ M>\[X!NP>?SS[XZ2#73CI%`0'"K=^!V<=SQR!H*$]DZ5EELGYT1R5A@`L=$:N M,W\;52$N6A]D$W>"'WNMF^ZE?9L+2E9%"SG8'2S'II'`PB2%#?BX--5EY\ES MU%[Q%9M"@0L%SJI&(>4E'N:"NIBK!N8>XK*_F9=SJ.O#W0(]=.U'M><,A];$ M@IFZ<.@_Y,;]`/ANX.HW4/Z]4BFGVB)/LI=YP&<)6-T#NSM=/Z&:(OT*&Y]T M=D]:T:4KF/_2=T3#<5R..TLBY)B=[""VV^MTCN%:$>_E$>_% M>C3)<142QHOA)'8>M1?^V@,T_E;R'^"Z!ZI#BGB`L\K^;B>UOZ@WSK8-/"BK MHMT-].'"_#YV45M6=0=^`6I@:XEE`P`_WP+[^!@LM+_]H=7S<.901W]MSS2V M_&N?:<=T/3099_S M/&EU9,Y95"1&2B&?,X$_^5^SI,^ZPJCAK9$L4 M@53D-KSEG);2\]P.A]#0'A)U5N]*I>"[7X_\TCL^IX^BA-]$T0CDZDU(9LB; M;94W\']HR8>U>-N"CV;1W?+HYB8[ MNR6V]'?5T%2"1N4PP!\_R-WPHB3A8&BE*#_5_8_^Y\+W/RH-9;/9C+O_%Q5R M_^,F^FOP^/[?IJJL@&9Q_^.CX/^B]X#.N/]94>JJ>/^SIJC-XO[/>RGA"X/I MA=_EPI@7I2A%*4I1BE*4HA2E*$4I2E&*4I2B%*4H12E*48I2E*+\- &HK@`R``` ` end --- webjob-run-periodic-bundles.tgz.uu ---