From nobody@FreeBSD.ORG  Wed Aug  9 01:09:10 2000
Return-Path: <nobody@FreeBSD.ORG>
Received: by hub.freebsd.org (Postfix, from userid 32767)
	id 487C337B85F; Wed,  9 Aug 2000 01:09:10 -0700 (PDT)
Message-Id: <20000809080910.487C337B85F@hub.freebsd.org>
Date: Wed,  9 Aug 2000 01:09:10 -0700 (PDT)
From: brett@lariat.org
Sender: nobody@FreeBSD.ORG
To: freebsd-gnats-submit@FreeBSD.org
Subject: All FreeBSD systems trigger massive late-night activity at the same times
X-Send-Pr-Version: www-1.0

>Number:         20498
>Category:       conf
>Synopsis:       All FreeBSD systems trigger massive late-night activity at the same times
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    brian
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Aug 09 01:10:00 PDT 2000
>Closed-Date:    Wed May 23 08:19:52 PDT 2001
>Last-Modified:  Wed May 23 08:22:02 PDT 2001
>Originator:     Brett Glass
>Release:        Various
>Organization:
>Environment:
>Description:
All FreeBSD systems, for a very long time, have come with a fixed /etc/crontab
which triggers system security and statistics-gathering scripts at the same
time on each system. A room full of FreeBSD machines is a sight and sound to
behold at this "witching hour." The sudden churning of the hard drives, and the 
slowdown in the entire network (especially if the machines are mail, news, or 
Web servers) can be quite dramatic. So can the power surge; we see the load
indicators on our UPSes peak at that time.
>How-To-Repeat:

>Fix:
It'd be a good idea to randomize the times at which these scripts are 
triggered at install time. The run times should be distributed within 
a 3-hour period -- say, from 2 to 5 AM.

>Release-Note:
>Audit-Trail:

From: Neil Blakey-Milner <nbm@mithrandr.moria.org>
To: brett@lariat.org
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: conf/20498: All FreeBSD systems trigger massive late-night activity at the same times
Date: Wed, 9 Aug 2000 13:56:23 +0200

 On Wed 2000-08-09 (01:09), brett@lariat.org wrote:
 > >Number:         20498
 > >Category:       conf
 > >Synopsis:       All FreeBSD systems trigger massive late-night activity at the same times
 > >Description:
 > All FreeBSD systems, for a very long time, have come with a fixed /etc/crontab
 > which triggers system security and statistics-gathering scripts at the same
 > time on each system. A room full of FreeBSD machines is a sight and sound to
 > behold at this "witching hour." The sudden churning of the hard drives, and the 
 > slowdown in the entire network (especially if the machines are mail, news, or 
 > Web servers) can be quite dramatic. So can the power surge; we see the load
 > indicators on our UPSes peak at that time.
 > >How-To-Repeat:
 > 
 > >Fix:
 > It'd be a good idea to randomize the times at which these scripts are 
 > triggered at install time. The run times should be distributed within 
 > a 3-hour period -- say, from 2 to 5 AM.
 
 This is a system's administrator's job, surely?  A gratuitous change
 like this by default will confuse and probably irritate many.  And I
 personally wouldn't like to have to document it.
 
 One possible solution is to add a 000.time-wait.sh script to your
 periodic/daily directory, which has "perl -e 'sleep int(rand(180))' in
 it (overkill, I know).
 
 You can change '180' to get a number from a variable in periodic.conf,
 for example.  It would default to 0, or something.  If you don't want
 _each_ invocation randmomized, then make the number you get from a
 variable in periodic.conf be the exact offset.
 
 Neil
 -- 
 Neil Blakey-Milner
 Sunesi Clinical Systems
 nbm@mithrandr.moria.org
 

From: Brett Glass <brett@lariat.org>
To: Neil Blakey-Milner <nbm@mithrandr.moria.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: conf/20498: All FreeBSD systems trigger massive late-night
  activity at the same times
Date: Wed, 09 Aug 2000 13:45:56 -0600

 At 05:56 AM 8/9/2000, Neil Blakey-Milner wrote:
 
 >This is a system's administrator's job, surely?  
 
 Sysadmins already have a lot to do manually installing the OS.
 Why make their lives harder?
 
 >A gratuitous change
 >like this by default will confuse and probably irritate many.  
 
 It's not gratuitous. It does not make sense to have EVERY
 FreeBSD server in the same time zone suddenly thrash at the
 same time!
 
 >And I
 >personally wouldn't like to have to document it.
 
 It'd be easy. Simply say that the scripts do a random delay
 (which could, perhaps, be turned off via a variable in rc.conf)
 and then continue.
 
 >One possible solution is to add a 000.time-wait.sh script to your
 >periodic/daily directory, which has "perl -e 'sleep int(rand(180))' in
 >it (overkill, I know).
 >
 >You can change '180' to get a number from a variable in periodic.conf,
 >for example.  It would default to 0, or something.  If you don't want
 >_each_ invocation randmomized, then make the number you get from a
 >variable in periodic.conf be the exact offset.
 
 Why not put that random delay right in the daily/weekly/monthly scripts?
 Or in /etc/crontab, where it's easy to spot?
 
 --Brett
 
 

From: Neil Blakey-Milner <nbm@mithrandr.moria.org>
To: Brett Glass <brett@lariat.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: conf/20498: All FreeBSD systems trigger massive late-night activity at the same times
Date: Wed, 9 Aug 2000 22:39:04 +0200

 On Wed 2000-08-09 (13:45), Brett Glass wrote:
 > >This is a system's administrator's job, surely?  
 > 
 > Sysadmins already have a lot to do manually installing the OS.
 > Why make their lives harder?
 
 It's a policy decision.  You have to make it one way or another.  It is
 just as much work either way.
 
 > >A gratuitous change
 > >like this by default will confuse and probably irritate many.  
 > 
 > It's not gratuitous. It does not make sense to have EVERY
 > FreeBSD server in the same time zone suddenly thrash at the
 > same time!
 
 It's just as reasonable as having them go off on a random time in a
 three hour period.  It's a personal preference.  (for example, I know I
 begin to wonder what's up when one of my machines hasn't sent me mail by
 2:13am)
 
 > >And I
 > >personally wouldn't like to have to document it.
 > 
 > It'd be easy. Simply say that the scripts do a random delay
 > (which could, perhaps, be turned off via a variable in rc.conf)
 > and then continue.
 
 Or, "If you'd like your runs to be at a somewhat random time within
 three hours of 2am, you can set this variable...".
 
 (It would be in periodic.conf)
 
 > >One possible solution is to add a 000.time-wait.sh script to your
 > >periodic/daily directory, which has "perl -e 'sleep int(rand(180))' in
 > >it (overkill, I know).
 > >
 > >You can change '180' to get a number from a variable in periodic.conf,
 > >for example.  It would default to 0, or something.  If you don't want
 > >_each_ invocation randmomized, then make the number you get from a
 > >variable in periodic.conf be the exact offset.
 > 
 > Why not put that random delay right in the daily/weekly/monthly scripts?
 > Or in /etc/crontab, where it's easy to spot?
 
 There is no "daily/weekly/monthly scripts".  It is something run by
 periodic, which means that it takes, in numeric sorted order, all files
 of name "*.sh" which are executable, and runs them one after the other.
 A reasonable expectation of 000.time-wait.sh is that it'd be run first.
 
 mergemaster would want to eat this change whenever /etc/crontab updated.
 Very irritating.
 
 Neil
 -- 
 Neil Blakey-Milner
 Sunesi Clinical Systems
 nbm@mithrandr.moria.org
 
State-Changed-From-To: open->feedback 
State-Changed-By: sheldonh 
State-Changed-When: Thu Aug 10 02:44:29 PDT 2000 
State-Changed-Why:  
The originator reckons that it should be easy to add 
something like "periodic_time_variant" to periodic.conf(8). 
I agree; the maintainer should look forward to patches 
from the originator. 


Responsible-Changed-From-To: freebsd-bugs->brian 
Responsible-Changed-By: sheldonh 
Responsible-Changed-When: Thu Aug 10 02:44:29 PDT 2000 
Responsible-Changed-Why:  
Over to the maintainer, who'll be interested in the patches. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20498 
Responsible-Changed-From-To: brian->freebsd-bugs 
Responsible-Changed-By: brian 
Responsible-Changed-When: Thu Aug 10 03:05:26 PDT 2000 
Responsible-Changed-Why:  
I don't believe it's appropriate to do this in periodic.conf.  It should 
be done in the crontab file to remove the fact that a reboot between 2:00 
and the actual start time would abort the daily run. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20498 
State-Changed-From-To: feedback->closed 
State-Changed-By: sheldonh 
State-Changed-When: Thu Aug 10 03:32:24 PDT 2000 
State-Changed-Why:  
In that case, unless Paul Vixie has some bright ideas, 
the status quo will have to do. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20498 

From: Paul Herman <pherman@frenchfries.net>
To: brian@FreeBSD.ORG
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: conf/20498: All FreeBSD systems trigger massive late-night
 activity at the same times
Date: Thu, 10 Aug 2000 12:42:55 +0200 (CEST)

 On Thu, 10 Aug 2000 brian@FreeBSD.ORG wrote:
 
 > I don't believe it's appropriate to do this in periodic.conf.  It
 > should be done in the crontab file to remove the fact that a
 > reboot between 2:00 and the actual start time would abort the
 > daily run.
 
 Good point.  This seems to be a general problem (not just for
 periodic) and crontab would have the same difficulty, or?  Timestamp
 files (eewww) would theoreticaly be a solution, but doesn't appeal to
 me practicaly.
 
 Not that I am for this feature as part of FreeBSD anyway...
 
 (just someone who reads -bugs, I'll crawl back into -questions now :)
 
 -Paul.
 
 
State-Changed-From-To: closed->open 
State-Changed-By: brian 
State-Changed-When: Thu Aug 10 04:21:46 PDT 2000 
State-Changed-Why:  
I think it might be nice to have a mechanism whereby cron runs  
maybe-periodic every 15 minutes between (say) 2:00 and 23:00.  The  
maybe-periodic script checks a touch file that indicates when the  
last such periodic actually ran (say /var/db/periodic-{dai,week,month}ly.run. 
If the file has a date of yesterday or before (or this day last week  
or month depending on the periodic argument), the script picks a  
random number between 1 and 24 (or 1 and 7 or 1 and 31) .  If the  
file is more than that number of hours (or days) old, the touch-file  
is re-touched and the periodic run is invoked. 

This resolves the random-start-time issue, and also handles systems  
that are shut down every night. 

I'll implement this if nobody objects. 


Responsible-Changed-From-To: freebsd-bugs->brian 
Responsible-Changed-By: brian 
Responsible-Changed-When: Thu Aug 10 04:21:46 PDT 2000 
Responsible-Changed-Why:  
I changed my mind.  This can be resolved with a combination of cron and  
periodic changes 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20498 

From: Gregory Bond <gnb@itga.com.au>
To: brian@FreeBSD.ORG, brett@lariat.org
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: conf/20498: All FreeBSD systems trigger massive late-night activity at the same times 
Date: Fri, 11 Aug 2000 11:06:55 +1000

 > I don't believe it's appropriate to do this in periodic.conf.  It should
 > be done in the crontab file to remove the fact that a reboot between 2:00
 > and the actual start time would abort the daily run.
 
 For the OP, here's another idea:  Change the crontab on every machine, so
 instead of running /etc/periodic, run something like
 	echo periodic daily | at -m now + `jot -r 1 0 240` minutes
 
 + all systems have the same crontab
 + daily runs don't get lost in a reboot
 + all systems will fire up at a different time
 
 
 
State-Changed-From-To: open->closed 
State-Changed-By: brian 
State-Changed-When: Wed May 23 08:19:52 PDT 2001 
State-Changed-Why:  
On retrospect, I don't think it's practical to try to fix this. 
For large FreeBSD installations I think the only real way of avoiding 
this is to locally customise /etc/crontab 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=20498 
>Unformatted:
