From nobody@FreeBSD.org  Thu Jun 27 17:29:53 2002
Return-Path: <nobody@FreeBSD.org>
Received: from www.freebsd.org (www.FreeBSD.org [216.136.204.117])
	by hub.freebsd.org (Postfix) with ESMTP id D74A237B4F3
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 27 Jun 2002 17:29:46 -0700 (PDT)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.12.4/8.12.4) with ESMTP id g5S0SLOT059821
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 27 Jun 2002 17:28:21 -0700 (PDT)
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.4/8.12.4/Submit) id g5S0SL84059820;
	Thu, 27 Jun 2002 17:28:21 -0700 (PDT)
Message-Id: <200206280028.g5S0SL84059820@www.freebsd.org>
Date: Thu, 27 Jun 2002 17:28:21 -0700 (PDT)
From: Guolin Cheng <guolin@alexa.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: /usr/sbin/periodic sends thousands of emails
X-Send-Pr-Version: www-1.0

>Number:         39940
>Category:       bin
>Synopsis:       [patch] /usr/sbin/periodic sends thousands of emails
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jun 27 17:30:01 PDT 2002
>Closed-Date:    Wed Feb 28 07:40:35 GMT 2007
>Last-Modified:  Wed Feb 28 07:40:35 GMT 2007
>Originator:     Guolin Cheng
>Release:        FreeBSD 4.6, but same for all 4.* version
>Organization:
ALexa Internet. Inc.
>Environment:
FreeBSD rontmp.alexa.com 4.6-RELEASE FreeBSD 4.6-RELEASE #0: Fri Jun 21 20:37:42 GMT 2002  root@rontmp.alexa.com:/usr/src/sys/compile/ALEXA  i386


>Description:
 The problems are:

1, /usr/sbin/periodic sends tons of email, when running hourly running scripts whether or not the latter generate outputs. Then email server will be overwhelmed if we have hundreds or thousands FreeBSD boxes.

2, periodic can not tell normal shell scripts from backup files ended with '~' and ',' , which are normally created by CVS/RVS/Perforce. it tries to run all executable files under base directory specified.

3, the coufiguration file */periodic.conf, which contain variable definition entries including <basedir>_{show|output|...}, will confuse Bourne Shell interpreter when the basedir contains character '.', which is normal for directory name but not appropriate for shell variable name substitution.

>How-To-Repeat:
 just create a directory /etc/periodic/hourly, and create a simply output-nothing shell script hahaha.sh under it: 
    
   #! /bin/sh
   # do nothing and output nothing

   # echo "hahahaa... wrong :)"

 Set its executable mode bit and  run the periodic with hourly directory as its argument..

 sh -x /usr/sbin/periodic  hourly

 Then copy the hahaha.sh to hahaha.sh~ and run the same command above, you will find that both shell scripts are processed!

at last change the hourly directory name to cron.hourly name, and it will report problem even at beginning when processing the periodic.conf file!



 
>Fix:
To fix the problem, we need to tell periodic
 1,  skip shell scripts ended with '~' and '.', 
 2,  substitute the '.' character to '_' in the periodic.conf and inform users about that special case, so that no base diretories like cron.daily and cron_daily exists at the same time. or if so, then the setting will overlaps.
 3, send emails only when necessary, normally that means scripts under base directory generating outputs.

The diff between original edited version of /usr/sbin/periodic  are attached.

shell> diff -c /usr/sbin/periodic /var/tmp/periodic
*** /usr/sbin/periodic  Mon Jun 10 21:19:56 2002
--- /var/tmp/periodic   Thu Jun 27 15:10:53 2002
***************
*** 28,33 ****
--- 28,34 ----
  host=`hostname`
  export host
  tmp_output=`mktemp ${TMPDIR:-/tmp}/periodic.XXXXXXXXXX`
+ tmp_email_output=`mktemp ${TMPDIR:-/tmp}/periodic.XXXXXXXXXX`

  # Execute each executable file in the directory list.  If the x bit is not
  # set, assume the user didn't really want us to muck with it (it's a
***************
*** 35,42 ****

  for arg
  do
      # Where's our output going ?
!     eval output=\$${arg##*/}_output
      case "$output" in
      /*) pipe="cat >>$output";;
      "") pipe=cat;;
--- 36,47 ----

  for arg
  do
+     # for special cases that the base directory has a '.' character contained,
+     # and difficult to change, then we can replace '.' with '_' in */period.conf,
+     # to pass the shell parameter name check for */periodic.conf
+     arg_in_conf=`echo $arg | tr '.' '_'`
      # Where's our output going ?
!     eval output=\$${arg_in_conf##*/}_output
      case "$output" in
      /*) pipe="cat >>$output";;
      "") pipe=cat;;
***************
*** 46,52 ****
      success=YES info=YES badconfig=NO # Defaults when ${run}_* aren't YES/NO
      for var in success info badconfig
      do
!         case $(eval echo "\$${arg##*/}_show_$var") in
          [Yy][Ee][Ss]) eval $var=YES;;
          [Nn][Oo])     eval $var=NO;;
          esac
--- 51,57 ----
      success=YES info=YES badconfig=NO # Defaults when ${run}_* aren't YES/NO
      for var in success info badconfig
      do
!         case $(eval echo "\$${arg_in_conf##*/}_show_$var") in
          [Yy][Ee][Ss]) eval $var=YES;;
          [Nn][Oo])     eval $var=NO;;
          esac
***************
*** 72,79 ****
          processed=0
          for dir in $dirlist
          do
!             for file in $dir/*
              do
                  if [ -x $file -a ! -d $file ]
                  then
                      output=TRUE
--- 77,91 ----
          processed=0
          for dir in $dirlist
          do
!             for file in $dir/*[^~,] ; ## Ignore *~ and *, scripts
              do
+             # Don't run *.{rpmsave,rpmorig,rpmnew,swp} scripts
+             [ "${file%.rpmsave}" != "${file}" ] && continue
+             [ "${file%.rpmorig}" != "${file}" ] && continue
+             [ "${file%.rpmnew}" != "${file}" ] && continue
+             [ "${file%.swp}" != "${file}" ] && continue
+             [ "${file%,v}" != "${file}" ] && continue
+
                  if [ -x $file -a ! -d $file ]
                  then
                      output=TRUE
***************
*** 93,106 ****
                  fi
              done
          done
!         if [ $empty = TRUE ]
!         then
!           [ $processed = 1 ] && plural= || plural=s
!           echo "No output from the $processed file$plural processed"
!         else
!           echo ""
!           echo "-- End of $arg output --"
!         fi
!     } | eval $pipe
  done
  rm -f $tmp_output
--- 105,124 ----
                  fi
              done
          done
!       ## If there is no output from scripts processed, then we will not sendmail
!         #if [ $empty = TRUE ]
!         #then
!         #  [ $processed = 1 ] && plural= || plural=s
!         #  echo "No output from the $processed file$plural processed"
!         #else
!         #  echo ""
!         #  echo "-- End of $arg output --"
!         #fi
!     } > $tmp_email_output
!     if [ -s $tmp_email_output ] ;
!       then
!       cat $tmp_email_output | eval $pipe
!     fi
  done
  rm -f $tmp_output
+ rm -f $tmp_email_output


 
>Release-Note:
>Audit-Trail:

From: Peter Pentchev <roam@ringlet.net>
To: Guolin Cheng <guolin@alexa.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/39940: /usr/sbin/periodic sends thousands of emails
Date: Fri, 28 Jun 2002 13:39:54 +0300

 On Thu, Jun 27, 2002 at 05:28:21PM -0700, Guolin Cheng wrote:
 > 
 > >Number:         39940
 > >Category:       bin
 > >Synopsis:       /usr/sbin/periodic sends thousands of emails
 > >Originator:     Guolin Cheng
 > >Release:        FreeBSD 4.6, but same for all 4.* version
 > >Organization:
 > ALexa Internet. Inc.
 > >Environment:
 > FreeBSD rontmp.alexa.com 4.6-RELEASE FreeBSD 4.6-RELEASE #0: Fri Jun 21 20:37:42 GMT 2002  root@rontmp.alexa.com:/usr/src/sys/compile/ALEXA  i386
 > 
 > >Description:
 >  The problems are:
 >
 > 1, /usr/sbin/periodic sends tons of email, when running hourly running
 > scripts whether or not the latter generate outputs. Then email server
 > will be overwhelmed if we have hundreds or thousands FreeBSD boxes.
 
 This can be worked around easily: make your scripts return an
 appropriate exit code to periodic(8) as described in its manual page;
 notably, use an exit code of 0 for scripts that produce no output, and
 set the hourly_show_success variable to 'no' in /etc/periodic.conf.
 Actually, this is the way it is *designed* to work :)
 
 > 2, periodic can not tell normal shell scripts from backup files ended
 > with '~' and ',' , which are normally created by CVS/RVS/Perforce. it
 > tries to run all executable files under base directory specified.
 
 Uhm.. so.. don't leave any? :)  In the case of source-controlled script
 directories, I usually keep my working copies somewhere else, and use
 the 'real' scripts dir only as a checked-out copy; since it is never
 modified, it should never create any conflicts, so no temporary/backup
 files should ever be created.
 
 > 3, the coufiguration file */periodic.conf, which contain variable
 > definition entries including <basedir>_{show|output|...}, will confuse
 > Bourne Shell interpreter when the basedir contains character '.',
 > which is normal for directory name but not appropriate for shell
 > variable name substitution.
 
 One might argue that this may be solved by the "Don't do that"
 approach..  An underscore character would serve just as well as a word
 separator in the directory name.
 
 G'luck,
 Peter
 
 -- 
 Peter Pentchev	roam@ringlet.net	roam@FreeBSD.org
 PGP key:	http://people.FreeBSD.org/~roam/roam.key.asc
 Key fingerprint	FDBA FD79 C26F 3C51 C95E  DF9E ED18 B68D 1619 4553
 I am the meaning of this sentence.
State-Changed-From-To: open->analyzed 
State-Changed-By: vs 
State-Changed-When: Thu Jul 29 09:12:42 GMT 2004 
State-Changed-Why:  
Issue understood. It's a feature. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=39940 
State-Changed-From-To: analyzed->closed 
State-Changed-By: remko 
State-Changed-When: Wed Feb 28 07:40:33 UTC 2007 
State-Changed-Why:  
The issue described and explained by roam should be enough reasoning to 
close the ticket. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=39940 
>Unformatted:
