From scott@fishballoon.org  Fri Jul 25 07:48:05 2003
Return-Path: <scott@fishballoon.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 3E59637B401
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Jul 2003 07:48:05 -0700 (PDT)
Received: from mta06-svc.ntlworld.com (mta06-svc.ntlworld.com [62.253.162.46])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 2630943F75
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Jul 2003 07:48:04 -0700 (PDT)
	(envelope-from scott@fishballoon.org)
Received: from llama.fishballoon.org ([81.104.195.199])
          by mta06-svc.ntlworld.com
          (InterMail vM.4.01.03.37 201-229-121-137-20020806) with ESMTP
          id <20030725144802.OPE16215.mta06-svc.ntlworld.com@llama.fishballoon.org>
          for <FreeBSD-gnats-submit@freebsd.org>;
          Fri, 25 Jul 2003 15:48:02 +0100
Received: from scott by llama.fishballoon.org with local (Exim 4.20)
	id 19g3qo-0002Bp-Kz
	for FreeBSD-gnats-submit@freebsd.org; Fri, 25 Jul 2003 15:47:18 +0100
Message-Id: <E19g3qo-0002Bp-Kz@llama.fishballoon.org>
Date: Fri, 25 Jul 2003 15:47:18 +0100
From: Scott Mitchell <scott+freebsd@fishballoon.org>
Sender: Scott Mitchell <scott@fishballoon.org>
Reply-To: Scott Mitchell <scott+freebsd@fishballoon.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: cvs pserver sig11 crash on 4.8-R
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         54854
>Category:       bin
>Synopsis:       cvs pserver sig11 crash on 4.8-R
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jul 25 07:50:06 PDT 2003
>Closed-Date:    Sat Apr 16 14:47:30 GMT 2005
>Last-Modified:  Sat Apr 16 14:47:30 GMT 2005
>Originator:     Scott Mitchell
>Release:        FreeBSD 4.8-RELEASE i386
>Organization:
>Environment:
FreeBSD pukeko 4.8-RELEASE FreeBSD 4.8-RELEASE #0: Thu Apr  3 10:53:38 GMT 2003     root@freebsd-stable.sentex.ca:/usr/obj/usr/src/sys/GENERIC  i386

>Description:
We recently moved our CVS repository from a 4.6-STABLE machine to a brand
new 4.8 install, on another identical machine.  The server runs cvs in
'pserver' mode, for remote access by various Windows/Solaris/Linux/FreeBSD
clients.

We pretty soon noticed that the cvs server process was occasionally crashing
on sig11 (ie. a segfault).  The only evidence for this was in the message
log - the cvs operations always completed normally on the client side.  This
*never* happened on the old server, so I figured it had to be a hardware
problem on the new machine, or some issue with 4.8.  This was probably
happening about 1 in every 100 times the cvs server was run.

I compiled a debug version of cvs from the 4.8 sources and was able to get a
few cores, once I figured out how to make it actually dump core.  I've
attached the log of a gdb session on one of these -- all the cores I have
show the process crashing in the same place, where it's clearly trying to
follow a NULL pointer.

I've since copied the cvs binary from the 4.6 machine across to the new
server -- we've run with this for the past two weeks and had exactly zero
problems with it.

Unexpected sig11's are often a sign of bad RAM or other hardware trouble,
but I've run numerous buildworlds on this machine with no problems, so I'm
doubtful that this is a hardware issue.  Brian Behlendorf <brian@collab.net>
has reported the same problem, also with no obvious hardware-related cause.

Given that all the cores are the same, and that the only thing we've seen
fail on this machine is the 4.8 cvs code, this smells like a cvs bug to me.
I've no idea if it's in our local extensions or the base cvs code, so I am
reporting the bug to the CVS maintainers as well as here.

I can provide any additional configuration details or more grovelling in the
core dumps on request...

Cheers,

	Scott


----- Attachment #1: gdb.log -----

Script started on Wed Jul 23 11:14:55 2003
pukeko# gdb `which cvs.debug` cvs.debug.81697.core
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called
+at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line
+2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
+/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line
+933 in fill_symbuf

Core was generated by `cvs.debug'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libgnuregex.so.2...done.
Reading symbols from /usr/lib/libmd.so.2...done.
Reading symbols from /usr/lib/libcrypt.so.2...done.
Reading symbols from /usr/lib/libz.so.2...done.
Reading symbols from /usr/lib/libc.so.4...done.
Reading symbols from /usr/libexec/ld-elf.so.1...done.
#0  buf_shutdown (buf=0x0)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/buffer.c:1208
1208        if (buf->shutdown)
(gdb) where
#0  buf_shutdown (buf=0x0)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/buffer.c:1208
#1  0x8087e2b in server_cleanup (sig=0)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/server.c:4892
#2  0x805ec67 in error_exit ()
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/error.c:71
#3  0x805ef27 in error (status=1, errnum=0, 
    message=0x80ab4b9 "received %s signal")
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/error.c:212
#4  0x806daae in main_cleanup (sig=13)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/main.c:395
#5  0x80926e4 in strip_trailing_slashes ()
#6  0xbfbfffac in ?? ()
#7  0x804d85a in buf_send_output (buf=0x80c1040)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/buffer.c:287
#8  0x804d900 in buf_flush (buf=0x80c1040, block=1)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/buffer.c:352
#9  0x8087eb7 in server_cleanup (sig=0)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/server.c:5007
#10 0x80883e2 in server (argc=1, argv=0xbfbffc88)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/server.c:5234
#11 0x806e636 in main (argc=1, argv=0xbfbffc88)
    at /usr/src/gnu/usr.bin/cvs/cvs/../../../../contrib/cvs/src/main.c:1028
#12 0x804a67a in _start ()
(gdb) list
1203    
1204    int
1205    buf_shutdown (buf)
1206         struct buffer *buf;
1207    {
1208        if (buf->shutdown)
1209            return (*buf->shutdown) (buf);
1210        return 0;
1211    }
1212    
(gdb) quit
pukeko# exit

Script done on Wed Jul 23 11:15:28 2003

>How-To-Repeat:
	
>Fix:
Using the cvs binary from 4.6-R is the only workaround I've found so far.

>Release-Note:
>Audit-Trail:

From: Scott Mitchell <scott+freebsd@fishballoon.org>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Mon, 4 Aug 2003 11:09:42 +0100

 OK, I've run memtest86 on this machine for most of the weekend - it got
 through 37 loops of its extended test suite and found zero errors.  So
 I'm as confident as I'm ever going to be that there are no hardware
 issues with the machine.
 
 I've been advised by one of the CVS developers to try a newer version
 of CVS from their repository, without the FreeBSD extensions.  They
 have seen this behaviour before at a few sites, although it mostly seems
 to be related to using older clients with recent servers, which is not
 exactly what I have been seeing (it happens with new clients as well).
 Anyway, I'll try to get a newer release built and see what happens.
 
 	Scott
 

From: Scott Mitchell <scott+freebsd@fishballoon.org>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc: scott+freebsd@fishballoon.org
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Wed, 13 Aug 2003 09:49:58 +0100

 Still seeing these crashes with a stock cvs 1.11.6 (ie. without any of the
 FreeBSD extension).  The stack traces are very similar - it's interesting
 to note that the crash seems to be triggered by bad cleanup from a SIGPIPE,
 which I guess is related to the client-server connection closing down...
 
 Anyway, it looks less likely that this is a FreeBSD problem, although I
 still don't have any kind of solution other than using the binary from 4.6.
 We're planning to swap out the RAM in this machine within the next couple
 of weeks and run all the memory tests again, plus try all the versions of
 cvs we've built, in case this changes anything.
 
 Cheers,
 
 	Scott
 

From: Ruslan Ermilov <ru@FreeBSD.org>
To: Scott Mitchell <scott+freebsd@fishballoon.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Wed, 13 Aug 2003 18:31:23 +0300

 On Wed, Aug 13, 2003 at 02:00:35AM -0700, Scott Mitchell wrote:
 > 
 >  Still seeing these crashes with a stock cvs 1.11.6 (ie. without any of the
 >  FreeBSD extension).  The stack traces are very similar - it's interesting
 >  to note that the crash seems to be triggered by bad cleanup from a SIGPIPE,
 >  which I guess is related to the client-server connection closing down...
 >  
 >  Anyway, it looks less likely that this is a FreeBSD problem, although I
 >  still don't have any kind of solution other than using the binary from 4.6.
 >  We're planning to swap out the RAM in this machine within the next couple
 >  of weeks and run all the memory tests again, plus try all the versions of
 >  cvs we've built, in case this changes anything.
 >  
 See if this patch helps.
 
 %%%
 Index: contrib/cvs/src/server.c
 ===================================================================
 RCS file: /home/ncvs/src/contrib/cvs/src/server.c,v
 retrieving revision 1.13.2.5
 diff -u -p -r1.13.2.5 server.c
 --- contrib/cvs/src/server.c	21 Jan 2003 22:26:46 -0000	1.13.2.5
 +++ contrib/cvs/src/server.c	4 Mar 2003 16:47:06 -0000
 @@ -4889,11 +4889,14 @@ server_cleanup (sig)
  	 * have generated any final output, we shut down BUF_TO_NET.
  	 */
  
 -	status = buf_shutdown (buf_from_net);
 -	if (status != 0)
 -	    error (0, status, "shutting down buffer from client");
 -	buf_free (buf_from_net);
 -	buf_from_net = NULL;
 +	if (buf_from_net != NULL)
 +	{
 +	    status = buf_shutdown (buf_from_net);
 +	    if (status != 0)
 +		error (0, status, "shutting down buffer from client");
 +	    buf_free (buf_from_net);
 +	    buf_from_net = NULL;
 +	}
      }
  
      if (dont_delete_temp)
 %%%
 
 
 Cheers,
 -- 
 Ruslan Ermilov		Sysadmin and DBA,
 ru@sunbay.com		Sunbay Software Ltd,
 ru@FreeBSD.org		FreeBSD committer

From: Scott Mitchell <scott+freebsd@fishballoon.org>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc: scott+freebsd@fishballoon.org
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Mon, 6 Oct 2003 10:04:02 +0100

 Another update: finally got around to swapping out the RAM on the offending
 machine, replaced it all with brand new Crucial DIMMs certified for this
 motherboard.  Still seeing the same crash with the 4.8 cvs, but not with the
 binary from 4.6.
 
 Am now running with Ruslan's patch - no crashes after 3 days and a couple of
 hundred invocations of the cvs server, so I'm hopeful this has solved it.
 
 	Scott
 

From: Scott Mitchell <scott+freebsd@fishballoon.org>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc: scott+freebsd@fishballoon.org, ru@FreeBSD.org
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Wed, 22 Oct 2003 14:14:56 +0100

 Well, the server has been running the patched binary for 3 weeks no with no
 ill effects, so I guess this has fixed (or at least dodged around) the problem.
 
 Obviously I would like a fix for this to go into the official CVS distribution,
 so I don't have to reapply the patch every time I upgrade.  Ruslan, I trust
 you won't object if I forward your patch to the bug-cvs list for their
 consideration?
 
 Cheers,
 
 	Scott
 

From: Ruslan Ermilov <ru@FreeBSD.org>
To: Scott Mitchell <scott+freebsd@fishballoon.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Wed, 22 Oct 2003 18:26:56 +0300

 --eAbsdosE1cNLO4uF
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Wed, Oct 22, 2003 at 02:14:56PM +0100, Scott Mitchell wrote:
 > Well, the server has been running the patched binary for 3 weeks no with =
 no
 > ill effects, so I guess this has fixed (or at least dodged around) the pr=
 oblem.
 >=20
 > Obviously I would like a fix for this to go into the official CVS distrib=
 ution,
 > so I don't have to reapply the patch every time I upgrade.  Ruslan, I tru=
 st
 > you won't object if I forward your patch to the bug-cvs list for their
 > consideration?
 >=20
 I'd appreciate it if you do this, as I don't have time now to check
 if it still applies to the recent CVS sources, and all like that.
 
 
 Cheers,
 --=20
 Ruslan Ermilov		Sysadmin and DBA,
 ru@sunbay.com		Sunbay Software Ltd,
 ru@FreeBSD.org		FreeBSD committer
 
 --eAbsdosE1cNLO4uF
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.2.3 (FreeBSD)
 
 iD8DBQE/lqHAUkv4P6juNwoRAje8AJ0dYLsldRW+/zWgXrlSFa9l40UC8wCdHxNO
 oadB5Mr0Us/hcYajz3cDaJU=
 =FNuk
 -----END PGP SIGNATURE-----
 
 --eAbsdosE1cNLO4uF--

From: Ruslan Ermilov <ru@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Wed, 22 Oct 2003 18:59:22 +0300

 Hi again,
 
 I've just checked -- this has been fixed in CVS repo on August 13,
 on cvs1-11-x-branch branch, in server.c,v 1.284.2.7, and this fix
 is already available in CVS 1.11.7, so the next upgrade of CVS in
 FreeBSD should fix the problem.
 
 
 Cheers,
 -- 
 Ruslan Ermilov		Sysadmin and DBA,
 ru@sunbay.com		Sunbay Software Ltd,
 ru@FreeBSD.org		FreeBSD committer

From: Scott Mitchell <scott+freebsd@fishballoon.org>
To: Ruslan Ermilov <ru@FreeBSD.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: bin/54854: cvs pserver sig11 crash on 4.8-R
Date: Wed, 22 Oct 2003 22:44:26 +0100

 On Wed, Oct 22, 2003 at 06:26:56PM +0300, Ruslan Ermilov wrote:
 > > 
 > I'd appreciate it if you do this, as I don't have time now to check
 > if it still applies to the recent CVS sources, and all like that.
 
 Hi Ruslan,
 
 Just saw your latest followup to the PR - glad to see this has been fixed.
 Maybe I'll drop peter@ an email and see when he's planning the next import
 into FreeBSD.
 
 Thanks again for your help,
 
 	Scott
 
 -- 
 ===========================================================================
 Scott Mitchell           | PGP Key ID | "Eagles may soar, but weasels
 Cambridge, England       | 0x54B171B9 |  don't get sucked into jet engines"
 scott at fishballoon.org | 0xAA775B8B |      -- Anon
State-Changed-From-To: open->feedback 
State-Changed-By: keramida 
State-Changed-When: Fri Apr 15 15:03:20 GMT 2005 
State-Changed-Why:  
Has this been fixed with the import of CVS versions newer than 1.11.6? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=54854 
State-Changed-From-To: feedback->closed 
State-Changed-By: keramida 
State-Changed-When: Sat Apr 16 14:46:34 GMT 2005 
State-Changed-Why:  
Submitter says the problem hasn't surfaced for a long time, 
so it seems that CVS imports of newer CVS releases have fixed it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=54854 
>Unformatted:
