From nobody@FreeBSD.org  Fri Nov 30 20:16:52 2001
Return-Path: <nobody@FreeBSD.org>
Received: from freefall.freebsd.org (freefall.FreeBSD.org [216.136.204.21])
	by hub.freebsd.org (Postfix) with ESMTP id BD97F37B405
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 30 Nov 2001 20:16:51 -0800 (PST)
Received: (from nobody@localhost)
	by freefall.freebsd.org (8.11.6/8.11.6) id fB14GpG18467;
	Fri, 30 Nov 2001 20:16:51 -0800 (PST)
	(envelope-from nobody)
Message-Id: <200112010416.fB14GpG18467@freefall.freebsd.org>
Date: Fri, 30 Nov 2001 20:16:51 -0800 (PST)
From: Brett Glass <brett@lariat.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: syslogd accumulates "repeated" messages when piping to programs when the program may want to take immediate action
X-Send-Pr-Version: www-1.0

>Number:         32420
>Category:       bin
>Synopsis:       syslogd accumulates "repeated" messages when piping to programs when the program may want to take immediate action
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Nov 30 20:20:01 PST 2001
>Closed-Date:    Mon Jan 7 20:46:30 PST 2002
>Last-Modified:  Thu Jan 31 02:00:00 PST 2002
>Originator:     Brett Glass
>Release:        4.4-RELEASE and all previous
>Organization:
>Environment:
>Description:
FreeBSD's syslogd has a "repeat counting" feature, implemented by
Eric Allman, which is designed to save file space and prevent 
flooding of the console with messages. Unfortunately, because it
also works when messages are piped to a log monitoring program, it
prevents a log monitor from seeing the stream of messages and taking
appropriate action (e.g. blackholing an attacker). While it's a
good idea to do repeat counting when the destination of log messages
is a file, a user, or the console, it should be disabled when the
output is piped.
>How-To-Repeat:

>Fix:
Here's the original code:

                /*
                 * suppress duplicate lines to this file
                 */
                if ((flags & MARK) == 0 && msglen == f->f_prevlen &&
                    !strcmp(msg, f->f_prevline) &&
                    !strcasecmp(from, f->f_prevhost)) {
                        (void)strncpy(f->f_lasttime, timestamp, 15);
                        f->f_prevcount++;

Add f->f_type != F_PIPE to the ANDed conditions in the initial if,
preferably before the string length comparison.

P.S. -- What are all those comments in the source about a
"hollow laugh?"

>Release-Note:
>Audit-Trail:

From: Archie Cobbs <archie@dellroad.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc: brett@lariat.org
Subject: Re: bin/32420: syslogd accumulates "repeated" messages when piping to
 programs when the program may want to take immediate action
Date: Fri, 4 Jan 2002 23:31:39 -0800 (PST)

 Here's a candidate patch.
 
 -Archie
 
 __________________________________________________________________________
 Archie Cobbs     *     Packet Design     *     http://www.packetdesign.com
 
 Index: usr.sbin/syslogd/syslogd.8
 ===================================================================
 RCS file: /home/ncvs/src/usr.sbin/syslogd/syslogd.8,v
 retrieving revision 1.43
 diff -u -r1.43 syslogd.8
 --- syslogd.8	8 Dec 2001 16:17:49 -0000	1.43
 +++ syslogd.8	5 Jan 2002 07:29:17 -0000
 @@ -156,6 +156,12 @@
  Specify one specific IP address or hostname to bind to.
  If a hostname is specified,
  the IPv4 or IPv6 address which corresponds to it is used.
 +.It Fl c
 +Disable the compression of repeated instances of the same line
 +into a single line of the form
 +.Dq last message repeated N times
 +when the output is a pipe to another program.
 +If specified twice, disable this compression in all cases.
  .It Fl d
  Put
  .Nm
 Index: usr.sbin/syslogd/syslogd.c
 ===================================================================
 RCS file: /home/ncvs/src/usr.sbin/syslogd/syslogd.c,v
 retrieving revision 1.93
 diff -u -r1.93 syslogd.c
 --- syslogd.c	27 Nov 2001 20:02:18 -0000	1.93
 +++ syslogd.c	5 Jan 2002 07:29:18 -0000
 @@ -274,6 +274,7 @@
  #endif
  int	send_to_all = 0;	/* send message to all IPv4/IPv6 addresses */
  int	use_bootfile = 0;	/* log entire bootfile for every kern msg */
 +int	no_compress = 0;	/* don't compress messages (1=pipes, 2=all) */
  
  char	bootfile[MAXLINE+1];	/* booted kernel file */
  
 @@ -334,7 +335,7 @@
  	socklen_t len;
  
  	bindhostname = NULL;
 -	while ((ch = getopt(argc, argv, "46Aa:b:df:kl:m:nop:P:suv")) != -1)
 +	while ((ch = getopt(argc, argv, "46Aa:b:cdf:kl:m:nop:P:suv")) != -1)
  		switch (ch) {
  		case '4':
  			family = PF_INET;
 @@ -354,6 +355,9 @@
  		case 'b':
  			bindhostname = optarg;
  			break;
 +		case 'c':
 +			no_compress++;
 +			break;
  		case 'd':		/* debug */
  			Debug++;
  			break;
 @@ -865,7 +869,8 @@
  		/*
  		 * suppress duplicate lines to this file
  		 */
 -		if ((flags & MARK) == 0 && msglen == f->f_prevlen &&
 +		if (no_compress - (f->f_type != F_PIPE) < 1 &&
 +		    (flags & MARK) == 0 && msglen == f->f_prevlen &&
  		    !strcmp(msg, f->f_prevline) &&
  		    !strcasecmp(from, f->f_prevhost)) {
  			(void)strlcpy(f->f_lasttime, timestamp, 16);

From: Brett Glass <brett@lariat.org>
To: Archie Cobbs <archie@dellroad.org>,
	freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: bin/32420: syslogd accumulates "repeated" messages when
  piping to programs when the program may want to take immediate action
Date: Sat, 05 Jan 2002 00:59:36 -0700

 Archie:
 
 It's more complex than I would have made it, but it will
 certainly work.
 
 --Brett
 
 At 12:31 AM 1/5/2002, Archie Cobbs wrote:
   
 >Here's a candidate patch.
 >
 >-Archie
 >
 >__________________________________________________________________________
 >Archie Cobbs     *     Packet Design     *     http://www.packetdesign.com
 >
 >Index: usr.sbin/syslogd/syslogd.8
 >===================================================================
 >RCS file: /home/ncvs/src/usr.sbin/syslogd/syslogd.8,v
 >retrieving revision 1.43
 >diff -u -r1.43 syslogd.8
 >--- syslogd.8   8 Dec 2001 16:17:49 -0000       1.43
 >+++ syslogd.8   5 Jan 2002 07:29:17 -0000
 >@@ -156,6 +156,12 @@
 > Specify one specific IP address or hostname to bind to.
 > If a hostname is specified,
 > the IPv4 or IPv6 address which corresponds to it is used.
 >+.It Fl c
 >+Disable the compression of repeated instances of the same line
 >+into a single line of the form
 >+.Dq last message repeated N times
 >+when the output is a pipe to another program.
 >+If specified twice, disable this compression in all cases.
 > .It Fl d
 > Put
 > .Nm
 >Index: usr.sbin/syslogd/syslogd.c
 >===================================================================
 >RCS file: /home/ncvs/src/usr.sbin/syslogd/syslogd.c,v
 >retrieving revision 1.93
 >diff -u -r1.93 syslogd.c
 >--- syslogd.c   27 Nov 2001 20:02:18 -0000      1.93
 >+++ syslogd.c   5 Jan 2002 07:29:18 -0000
 >@@ -274,6 +274,7 @@
 > #endif
 > int    send_to_all = 0;        /* send message to all IPv4/IPv6 addresses */
 > int    use_bootfile = 0;       /* log entire bootfile for every kern msg */
 >+int    no_compress = 0;        /* don't compress messages (1=pipes, 2=all) */
 > 
 > char   bootfile[MAXLINE+1];    /* booted kernel file */
 > 
 >@@ -334,7 +335,7 @@
 >        socklen_t len;
 > 
 >        bindhostname = NULL;
 >-       while ((ch = getopt(argc, argv, "46Aa:b:df:kl:m:nop:P:suv")) != -1)
 >+       while ((ch = getopt(argc, argv, "46Aa:b:cdf:kl:m:nop:P:suv")) != -1)
 >                switch (ch) {
 >                case '4':
 >                        family = PF_INET;
 >@@ -354,6 +355,9 @@
 >                case 'b':
 >                        bindhostname = optarg;
 >                        break;
 >+               case 'c':
 >+                       no_compress++;
 >+                       break;
 >                case 'd':               /* debug */
 >                        Debug++;
 >                        break;
 >@@ -865,7 +869,8 @@
 >                /*
 >                * suppress duplicate lines to this file
 >                */
 >-               if ((flags & MARK) == 0 && msglen == f->f_prevlen &&
 >+               if (no_compress - (f->f_type != F_PIPE) < 1 &&
 >+                   (flags & MARK) == 0 && msglen == f->f_prevlen &&
 >                    !strcmp(msg, f->f_prevline) &&
 >                    !strcasecmp(from, f->f_prevhost)) {
 >                        (void)strlcpy(f->f_lasttime, timestamp, 16);
 
State-Changed-From-To: open->closed 
State-Changed-By: archie 
State-Changed-When: Mon Jan 7 20:46:30 PST 2002 
State-Changed-Why:  
Implemented via the '-c' flag in -current and -stable. 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=32420 

From: Sheldon Hearn <sheldonh@starjuice.net>
To: Archie Cobbs <archie@dellroad.org>
Cc: bug-followup@freebsd.org
Subject: Re: bin/32420: syslogd accumulates "repeated" messages when piping to programs when the program may want to take immediate action 
Date: Tue, 29 Jan 2002 13:57:14 +0200

 On Fri, 04 Jan 2002 23:50:01 PST, Archie Cobbs wrote:
 
 >  Here's a candidate patch.
 
 Hi Archie,
 
 Sorry to come in late like this, but you moved pretty quickly on this
 one. :-)
 
 Your patch provides an all or nothing solution.  What about systems that
 have syslogd pipe recipients, some of which are programmed to deal with
 aggregated messages and some of which are not?
 
 Wouldn't it be better to extend the ssylog.conf language so that the
 operator can choose _which_ pipe recipients should receive unaggregated
 messages?
 
 Ciao,
 Sheldon.

From: Archie Cobbs <archie@dellroad.org>
To: Sheldon Hearn <sheldonh@starjuice.net>
Cc: bug-followup@freebsd.org
Subject: Re: bin/32420: syslogd accumulates "repeated" messages when piping to
 programs when the program may want to take immediate action
Date: Tue, 29 Jan 2002 10:44:04 -0800 (PST)

 Sheldon Hearn writes:
 > >  Here's a candidate patch.
 > 
 > Hi Archie,
 > 
 > Sorry to come in late like this, but you moved pretty quickly on this
 > one. :-)
 > 
 > Your patch provides an all or nothing solution.  What about systems that
 > have syslogd pipe recipients, some of which are programmed to deal with
 > aggregated messages and some of which are not?
 > 
 > Wouldn't it be better to extend the ssylog.conf language so that the
 > operator can choose _which_ pipe recipients should receive unaggregated
 > messages?
 
 Yes :-)
 
 __________________________________________________________________________
 Archie Cobbs     *     Packet Design     *     http://www.packetdesign.com

From: Sheldon Hearn <sheldonh@starjuice.net>
To: Archie Cobbs <archie@dellroad.org>
Cc: bug-followup@freebsd.org
Subject: Re: bin/32420: syslogd accumulates "repeated" messages [...]
Date: Wed, 30 Jan 2002 16:02:53 +0200

 On Tue, 29 Jan 2002 10:50:02 PST, Archie Cobbs wrote:
 
 >  > Wouldn't it be better to extend the ssylog.conf language so that the
 >  > operator can choose _which_ pipe recipients should receive unaggregated
 >  > messages?
 >  
 >  Yes :-)
 
 Okay, please bug me before MFC'ing this, in case I find time to work on
 a syslog.conf language extension to replace the new option.
 
 Ciao,
 Sheldon.

From: Sheldon Hearn <sheldonh@starjuice.net>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: bin/32420: syslogd accumulates "repeated" messages [...] 
Date: Thu, 31 Jan 2002 11:55:11 +0200

 On Wed, 30 Jan 2002 06:10:02 PST, Sheldon Hearn wrote:
 
 >  Okay, please bug me before MFC'ing this, in case I find time to work on
 >  a syslog.conf language extension to replace the new option.
 
 For the sake of the audit trail, Archie informs me that this has already
 been MFC'd.  It's a real pretty the normal protocol of delaying MFC's
 wasn't followed, because this isn't the best solution, but the best
 solution would duplicate part of the new functionality, and warrant
 deprecation of the new feature.
 
 Ciao,
 Sheldon.
>Unformatted:
