From nobody@FreeBSD.org  Tue May 18 00:24:08 2004
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id B2D9916A4CE
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 18 May 2004 00:24:08 -0700 (PDT)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 72AA143D6B
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 18 May 2004 00:23:57 -0700 (PDT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.12.11/8.12.11) with ESMTP id i4I7NvQ8060213
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 18 May 2004 00:23:57 -0700 (PDT)
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.11/8.12.11/Submit) id i4I7NuYC060140;
	Tue, 18 May 2004 00:23:56 -0700 (PDT)
	(envelope-from nobody)
Message-Id: <200405180723.i4I7NuYC060140@www.freebsd.org>
Date: Tue, 18 May 2004 00:23:56 -0700 (PDT)
From: Pekka Savola <pekkas@netcore.fi>
To: freebsd-gnats-submit@FreeBSD.org
Subject: PATCH: port no-sync option to syslogd from Linux syslogd
X-Send-Pr-Version: www-2.3

>Number:         66790
>Category:       misc
>Synopsis:       PATCH: port no-sync option to syslogd from Linux syslogd
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    dwmalone
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Tue May 18 00:30:21 PDT 2004
>Closed-Date:    Sun May 30 03:05:36 PDT 2004
>Last-Modified:  Sun May 30 03:05:36 PDT 2004
>Originator:     Pekka Savola
>Release:        5.2.1-RELEASE
>Organization:
>Environment:
>Description:
Linux syslogd branch has supported a '-' prefix for files and pipes which causes the syslogd to omit fsync() after each logged line from 1995.  This causes relatively heavy operation if you syslog a LOT of data.

I've ported Martin Schulze's code to BSD syslog.  The code is almost identical.  Linux syslogd is under a BSD license, so this should not be a problem.
>How-To-Repeat:

>Fix:
Patch at http://www.netcore.fi/pekkas/linux//syslogd-freebsd521-nosync.patch

>Release-Note:
>Audit-Trail:

From: "Matthew D. Fuller" <fullermd@over-yonder.net>
To: Pekka Savola <pekkas@netcore.fi>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/66790: PATCH: port no-sync option to syslogd from Linux syslogd
Date: Tue, 18 May 2004 03:05:54 -0500

 On Tue, May 18, 2004 at 12:23:56AM -0700 I heard the voice of
 Pekka Savola, and lo! it spake thus:
 >
 > I've ported Martin Schulze's code to BSD syslog.  The code is almost
 > identical.  Linux syslogd is under a BSD license, so this should not
 > be a problem.
 
 The syslogd on most Linux boxes is actually a derivative of (an old,
 old version of) the BSD syslogd in the first place    8-}
 
 
 -- 
 Matthew Fuller     (MF4839)   |  fullermd@over-yonder.net
 Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
 
 "The only reason I'm burning my candle at both ends, is because I
       haven't figured out how to light the middle yet"

From: Martin Blapp <mb@imp.ch>
To: Pekka Savola <pekkas@netcore.fi>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/66790: PATCH: port no-sync option to syslogd from Linux
 syslogd
Date: Tue, 18 May 2004 17:39:19 +0200 (CEST)

 Hi,
 
 Just tested it out. Works perfect and saves us a lot of IO on
 a busy mailsystem. Everthing is running faster now. We have approx.
 100 new lines per second in one logfile.
 
 IMHO this patch needs to be part of 5.3, definitly.
 
 Thank you for finding and solving this problem.
 
 Martin
 
Responsible-Changed-From-To: freebsd-bugs->dwmalone 
Responsible-Changed-By: dwmalone 
Responsible-Changed-When: Wed May 26 02:34:47 PDT 2004 
Responsible-Changed-Why:  
I have a couple of PRs in this area, so I'll try to distill them into 
a commit. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=66790 

From: David Malone <dwmalone@maths.tcd.ie>
To: freebsd-gnats-submit@FreeBSD.org, pekkas@netcore.fi, mb@imp.ch
Cc:  
Subject: Re: misc/66790: PATCH: port no-sync option to syslogd from Linux syslogd
Date: Sun, 30 May 2004 01:45:08 +0100

 Hi Pekka,
 
 I've just had a look at the patch, and I think it does some bad
 things, but I'd like to see if you agree with my assessment (I'm
 kind of sleepy, so I could have read it completly wrong!)
 
 I believe in FreeBSD, syslogd doesn't fsync() the file after every
 message, just after messages logged by the kernel. This is because
 printsys is the only place where logmsg() is called with the SYNC_FILE
 flag. I think the Linux version of syslogd may differ here, maybe
 for historical reasons (could be async filesystem stuff?)
 
 First, this makes the man page part of the patch wrong. Second, I
 think the patch resluts in fsync() being called after *every* message
 because you've added SYNC_FILE to the flags used by printline(),
 which is the function that calls logmsg() for normal messages. I
 guess this will result in bad performance for anyone not using
 filenames prefixed by "-".
 
 If you agree with this analysis, then I think we can simplify the
 patch to not change the calling of logmsg and make the condition
 for calling fsync something like:
 
 -               } else if (flags & SYNC_FILE)
 +               } else if (flags & f->f_flags & SYNC_FILE)
                         (void)fsync(f->f_file);
 
 (Minor other comment on the patch: I'm not sure why you initialise
 f_flags to zero quite so many times ;-)
 
 I'll try to produce an updated patch tomorrow.
 
 	David.

From: Pekka Savola <pekkas@netcore.fi>
To: David Malone <dwmalone@maths.tcd.ie>
Cc: freebsd-gnats-submit@FreeBSD.org, <mb@imp.ch>
Subject: Re: misc/66790: PATCH: port no-sync option to syslogd from Linux
 syslogd
Date: Sun, 30 May 2004 09:16:01 +0300 (EEST)

 On Sun, 30 May 2004, David Malone wrote:
 > I've just had a look at the patch, and I think it does some bad
 > things, but I'd like to see if you agree with my assessment (I'm
 > kind of sleepy, so I could have read it completly wrong!)
 
 Thanks for looking at it!
  
 > I believe in FreeBSD, syslogd doesn't fsync() the file after every
 > message, just after messages logged by the kernel. This is because
 > printsys is the only place where logmsg() is called with the SYNC_FILE
 > flag. I think the Linux version of syslogd may differ here, maybe
 > for historical reasons (could be async filesystem stuff?)
 
 Yes, this is how it looks like to be, looking it a bit better.  With 
 linux, kernel logging is separated to a separate daemon, and I was 
 looking at syslogd only.
 
 I only tested this with output from ip6fw and it was fine ;-).
 
 > First, this makes the man page part of the patch wrong. Second, I
 > think the patch resluts in fsync() being called after *every* message
 > because you've added SYNC_FILE to the flags used by printline(),
 > which is the function that calls logmsg() for normal messages.  I
 > guess this will result in bad performance for anyone not using
 > filenames prefixed by "-".
 
 Yes, for those messages not logged by the kernel.
 
 (This might actually be a _feature_, to be able to sync after regular 
 log messages as well, but then I think the behaviour should be 
 reversed, like using '+' before the file to add explicit sync to files 
 if required..)
 
 > If you agree with this analysis, then I think we can simplify the
 > patch to not change the calling of logmsg and make the condition
 > for calling fsync something like:
 > 
 > -               } else if (flags & SYNC_FILE)
 > +               } else if (flags & f->f_flags & SYNC_FILE)
 >                         (void)fsync(f->f_file);
 
 Did you mean 'flags &' here?  I guess that would work too, but I used 
 'flags &&' as non-kernel files' flags should be zero.
  
 > (Minor other comment on the patch: I'm not sure why you initialise
 > f_flags to zero quite so many times ;-)
 
 Just to be sure.. it was in the original.  I guess nowadays these are 
 always initialized to zero implicitly.
  
 > I'll try to produce an updated patch tomorrow.
 
 I've also updated my patch at 
 http://www.netcore.fi/pekkas/linux/syslogd-freebsd521-nosync.patch
 which should take these into consideration.
 
 -- 
 Pekka Savola                 "You each name yourselves king, yet the
 Netcore Oy                    kingdom bleeds."
 Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
 
 

From: David Malone <dwmalone@maths.tcd.ie>
To: Pekka Savola <pekkas@netcore.fi>
Cc: freebsd-gnats-submit@FreeBSD.org, mb@imp.ch
Subject: Re: misc/66790: PATCH: port no-sync option to syslogd from Linux syslogd 
Date: Sun, 30 May 2004 09:00:08 +0100

 > I only tested this with output from ip6fw and it was fine ;-).
 
 I had a suspicion that was the case ;-)
 
 > (This might actually be a _feature_, to be able to sync after regular 
 > log messages as well, but then I think the behaviour should be 
 > reversed, like using '+' before the file to add explicit sync to files 
 > if required..)
 
 I was talking about a feature like this to Ian Dowse last night.
 If we added a '+' feature, then we could also make the autosynching
 more clever, so that it only happened when syslogd emptied the
 kernel buffer or something. However, I think these are jobs for
 another day.
 
 > > If you agree with this analysis, then I think we can simplify the
 > > patch to not change the calling of logmsg and make the condition
 > > for calling fsync something like:
 > > 
 > > -               } else if (flags & SYNC_FILE)
 > > +               } else if (flags & f->f_flags & SYNC_FILE)
 > >                         (void)fsync(f->f_file);
 
 > Did you mean 'flags &' here?  I guess that would work too, but I used 
 > 'flags &&' as non-kernel files' flags should be zero.
 
 I actually intended bitwise anding the three to make sure
 the SYNC_FILE bit is set in both variables. Other bits can
 be set in flags, but you're right, it should probably say:
 
 	if ((flags & SYNC_FILE) && (f->flags & SYNC_FILE))
 
 In fact, the second SYNC_FILE should probably be called something
 different 'cos it is a different type of flags variable.
 
 > > (Minor other comment on the patch: I'm not sure why you initialise
 > > f_flags to zero quite so many times ;-)
 
 > Just to be sure.. it was in the original.  I guess nowadays these are 
 > always initialized to zero implicitly.
 
 Sorry - I should have explained myself more clearly. The initialisation
 in the old patch was had been put into this loop:
 
         for (i = 0; i <= LOG_NFACILITIES; i++)
                 f->f_pmask[i] = INTERNAL_NOPRI;
 
 so it set the f_flags to zero 24 times in a row. Initialising it once
 would be no harm, but I see there is a memset just about, so it is
 probably OK to leave out the initialisation anyway.
 
 > > I'll try to produce an updated patch tomorrow.
 
 > I've also updated my patch at 
 > http://www.netcore.fi/pekkas/linux/syslogd-freebsd521-nosync.patch
 > which should take these into consideration.
 
 Thanks - apart from giving the bits in the f_flags variable names
 that are distince from logmsg's flags it looks good. I'll try and
 commit it later today (I may make a few manpage tweaks too, to keep
 the mdoc police happy...)
 
 	David.
State-Changed-From-To: open->closed 
State-Changed-By: dwmalone 
State-Changed-When: Sun May 30 03:04:13 PDT 2004 
State-Changed-Why:  
I've committed a patch that I think deals with all the issues we 
discussed. I've scheduled a MFC for one month, all going well. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=66790 
>Unformatted:
