From infofarmer@gmail.com  Sat Nov 19 00:04:34 2005
Return-Path: <infofarmer@gmail.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 4CA2716A41F
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 19 Nov 2005 00:04:34 +0000 (GMT)
	(envelope-from infofarmer@gmail.com)
Received: from proxy.gubkin.ru (proxy.gubkin.ru [193.233.78.245])
	by mx1.FreeBSD.org (Postfix) with ESMTP id D7D3743D45
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 19 Nov 2005 00:04:33 +0000 (GMT)
	(envelope-from infofarmer@gmail.com)
Received: from proxy.gubkin.ru (localhost [127.0.0.1])
	by proxy.gubkin.ru (8.13.4/8.13.4) with ESMTP id jAJ04TZN068039
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 19 Nov 2005 03:04:29 +0300 (MSK)
	(envelope-from infofarmer@gmail.com)
Received: (from sat@localhost)
	by proxy.gubkin.ru (8.13.4/8.13.3/Submit) id jAJ04SvS068038;
	Sat, 19 Nov 2005 03:04:28 +0300 (MSK)
	(envelope-from infofarmer@gmail.com)
Message-Id: <200511190004.jAJ04SvS068038@proxy.gubkin.ru>
Date: Sat, 19 Nov 2005 03:04:28 +0300 (MSK)
From: infofarmer@gmail.com
Reply-To: infofarmer@gmail.com
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: IPSec always causes panics on amd64
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         89261
>Category:       amd64
>Synopsis:       IPSec always causes panics on amd64
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bz
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Nov 19 00:10:27 GMT 2005
>Closed-Date:    Fri Jan 27 21:57:25 GMT 2006
>Last-Modified:  Fri Jan 27 21:57:25 GMT 2006
>Originator:     Andrew Pantyukhin
>Release:        FreeBSD 6.0-RELEASE i386
>Organization:
>Environment:
This bug has been reproduced on 5.4-RELEASE, 6.0-RELEASE and 6.0-STABLE.

	
>Description:
	IPSec key management (setkey) always causes kernel panics
	
>How-To-Repeat:
	Add "options IPSEC" to GENERIC kernel and run "setkey -D"
	Adding other IPSEC options does not help at all.
	
>Fix:

	
Sorry about the acid formatting. I've done this with script(1)

--- ipsec.1.debug begins here ---
Script started on Sat Nov 19 02:43:19 2005
satsmb# kgdb kernel.debug /var/crs[Kash/vmcore.1 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 9: general protection fault while in kernel mode
instruction pointer	= 0x8:0xffffffff804c5d4c
stack pointer	        = 0x10:0xffffffff967808a0
frame pointer	        = 0x10:0xa0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 641 (setkey)
trap number		= 9
panic: general protection fault
Uptime: 3m11s
Dumping 511 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

#0  doadump () at pcpu.h:172
172		__asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:172
#1  0x0000000000000004 in ?? ()
#2  0xffffffff803ba433 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:399
#3  0xffffffff803baa36 in panic (fmt=0xffffff00152be720 "@\023\025")
    at /usr/src/sys/kern/kern_shutdown.c:555
#4  0xffffffff805af57f in trap_fatal (frame=0xffffff00152be720, 
    eva=18446742974561784640) at /usr/src/sys/amd64/amd64/trap.c:655
#5  0xffffffff805afa22 in trap (frame=
      {tf_rdi = -1099013703168, tf_rsi = 4351, tf_rdx = 4351, tf_rcx = -281373226712833, tf_r8 = 0, tf_r9 = -1099156429024, tf_rax = -1770518320, tf_rbx = -1770518145, tf_rbp = 160, tf_r10 = -2139144832, tf_r11 = 1, tf_r12 = 0, tf_r13 = -1099151977264, tf_r14 = 0, tf_r15 = 0, tf_trapno = 9, tf_addr = 0, tf_flags = 4, tf_err = 0, tf_rip = -2142479028, tf_cs = 8, tf_rflags = 66051, tf_rsp = -1770518352, tf_ss = 16}) at /usr/src/sys/amd64/amd64/trap.c:467
#6  0xffffffff8059f0ab in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:168
#7  0xffffffff804c5d4c in keydb_newsecasvar ()
    at /usr/src/sys/netkey/keydb.c:177
#8  0x0000000000001041 in ?? ()
#9  0xffffff00178dc9b0 in ?? ()
#10 0xffffff00156fd4d0 in ?? ()
#11 0xffffff001dadba00 in ?? ()
#12 0x0000000000000000 in ?? ()
Previous frame identical to this frame (corrupt stack?)
(kgdb) up 7
#7  0xffffffff804c5d4c in keydb_newsecasvar ()
    at /usr/src/sys/netkey/keydb.c:177
177				if (q->id < said && said < TAILQ_NEXT(q, tailq)->id)
(kgdb) list
172			said++;
173		TAILQ_FOREACH(q, &satailq, tailq) {
174			if (q->id == said)
175				goto again;
176			if (TAILQ_NEXT(q, tailq)) {
177				if (q->id < said && said < TAILQ_NEXT(q, tailq)->id)
178					break;
179				if (q->id + 1 < TAILQ_NEXT(q, tailq)->id) {
180					said = q->id + 1;
181					break;
(kgdb) pirnt    rint q
$1 = (struct secasvar *) 0xffffffff9678097f
(kgdb) print q->id
$2 = 4351
(kgdb) print q->id     said
$3 = 0
(kgdb) print said    tailq
No symbol "tailq" in current context.
(kgdb) yo  up
#8  0x0000000000001041 in ?? ()
(kgdb) up
#9  0xffffff00178dc9b0 in ?? ()
(kgdb) up
#10 0xffffff00156fd4d0 in ?? ()
(kgdb) up
#11 0xffffff001dadba00 in ?? ()
(kgdb) up
#12 0x0000000000000000 in ?? ()
(kgdb) up
Initial frame selected; you cannot go up.
(kgdb) quit
satsmb# ^Dexit

Script done on Sat Nov 19 02:48:28 2005
--- ipsec.1.debug ends here ---

I will welcome any requests for further investigations.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-amd64->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Fri Dec 2 06:38:03 GMT 2005 
Responsible-Changed-Why:  
I'll work on this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89261 

From: Paul Herman <pherman@frenchfries.net>
To: bug-followup@FreeBSD.org, infofarmer@gmail.com
Cc:  
Subject: Re: amd64/89261: IPSec always causes panics on amd64
Date: Wed, 14 Dec 2005 11:23:13 +0100 (CET)

 Hey guys,
 
 I ran into this just the other day, except I couldn't get a 
 backtrace, according to kgdb my stack was corrupted.  I suspect 
 yours is, too.
 
 Anyway, after a lot of crashes and mucking around, I tracked this 
 down to raw_usend() calling key_output() incorrectly.  Something to 
 do somehow with the compiler not groking the va_list stuff, not 
 sure.
 
 This workaround works for me.  Not sure what the real problem is, 
 though, nor do I have an idea how this came about in the first 
 place.  Compiler flags or compiler bug?
 
 Anyway, hope this helps!
 
 -Paul.
 
 ---------------------------------------------------------------------
 --- sys/netkey/keysock.c.orig	Wed Dec 14 10:39:11 2005
 +++ sys/netkey/keysock.c	Wed Dec 14 10:39:43 2005
 @@ -75,23 +75,11 @@
    * key_output()
    */
   int
 -#if __STDC__
 -key_output(struct mbuf *m, ...)
 -#else
 -key_output(m, va_alist)
 -	struct mbuf *m;
 -	va_dcl
 -#endif
 +key_output(struct mbuf *m, struct socket *so)
   {
   	struct sadb_msg *msg;
   	int len, error = 0;
   	int s;
 -	struct socket *so;
 -	va_list ap;
 -
 -	va_start(ap, m);
 -	so = va_arg(ap, struct socket *);
 -	va_end(ap);
 
   	if (m == 0)
   		panic("key_output: NULL pointer was passed.");
 --- sys/netkey/keysock.h.orig	Wed Dec 14 10:39:15 2005
 +++ sys/netkey/keysock.h	Wed Dec 14 10:39:59 2005
 @@ -71,7 +71,7 @@
 
   extern struct pfkeystat pfkeystat;
 
 -extern int key_output(struct mbuf *, ...);
 +extern int key_output(struct mbuf *, struct socket *);
   extern int key_usrreq(struct socket *,
   	int, struct mbuf *, struct mbuf *, struct mbuf *);
 

From: "Andrew P." <infofarmer@gmail.com>
To: Paul Herman <pherman@frenchfries.net>
Cc: bug-followup@freebsd.org
Subject: Re: amd64/89261: IPSec always causes panics on amd64
Date: Tue, 3 Jan 2006 04:11:26 +0300

 On 12/14/05, Paul Herman <pherman@frenchfries.net> wrote:
 > Hey guys,
 >
 > I ran into this just the other day, except I couldn't get a
 > backtrace, according to kgdb my stack was corrupted.  I suspect
 > yours is, too.
 >
 > Anyway, after a lot of crashes and mucking around, I tracked this
 > down to raw_usend() calling key_output() incorrectly.  Something to
 > do somehow with the compiler not groking the va_list stuff, not
 > sure.
 >
 > This workaround works for me.  Not sure what the real problem is,
 > though, nor do I have an idea how this came about in the first
 > place.  Compiler flags or compiler bug?
 >
 > Anyway, hope this helps!
 >
 > -Paul.
 
 
 I tried it on a recent 6.0-STABLE and it worked! Well,
 at least "setkey -D" doesn't cause a panic anymore.
 I'll test basic IPSec functionality as soon as I can.
 
 Thank you so much!
 
 Have a nice year,
 Andrew P.

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, infofarmer@gmail.com,
	Paul Herman <pherman@frenchfries.net>
Cc:  
Subject: Re: amd64/89261 : IPSec always causes panics on amd64
Date: Tue, 17 Jan 2006 21:50:36 +0000 (UTC)

 Hi,
 
 I got explained why this is a problem on amd64. The patch in the PR
 is the correct solution.
 
 Could you try this patch (which also fixes other parts of the tree):
 http://sources.zabbadoz.net/freebsd/patchset/EXPERIMENTAL/20060113-02-casted-varargs-functions.diff
 
 I'll commit it soon.
State-Changed-From-To: open->patched 
State-Changed-By: bz 
State-Changed-When: Sat Jan 21 10:46:27 UTC 2006 
State-Changed-Why:  
The patch has been applied to HEAD and will be MFCed soon. 
For more information see this commit message: 
http://docs.freebsd.org/cgi/mid.cgi?200601211044.k0LAiZUB066317 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89261 
State-Changed-From-To: patched->closed 
State-Changed-By: bz 
State-Changed-When: Fri Jan 27 21:55:51 UTC 2006 
State-Changed-Why:  
Changes have been MFCed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89261 
>Unformatted:
