From nobody@FreeBSD.org  Sat Jul 25 10:00:37 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 816F2106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 25 Jul 2009 10:00:37 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 6E1A48FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 25 Jul 2009 10:00:37 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n6PA0b96056904
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 25 Jul 2009 10:00:37 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n6PA0bkY056903;
	Sat, 25 Jul 2009 10:00:37 GMT
	(envelope-from nobody)
Message-Id: <200907251000.n6PA0bkY056903@www.freebsd.org>
Date: Sat, 25 Jul 2009 10:00:37 GMT
From: Jrg Ruppe-Tanner <joerg.ruppe.tanner@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: panic after spinloc held too long with FreeBSD 7.2-RELEASE-p2 on CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1618.46-MHz K8-class CPU)
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         137117
>Category:       kern
>Synopsis:       [panic] panic after spinlock held too long with FreeBSD 7.2-RELEASE-p2 on CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1618.46-MHz K8-class CPU)
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    gavin
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Jul 25 10:10:01 UTC 2009
>Closed-Date:    Mon Sep 28 09:01:06 UTC 2009
>Last-Modified:  Mon Sep 28 09:01:06 UTC 2009
>Originator:     Jrg Ruppe-Tanner
>Release:        FreeBSD 7.2-RELEASE-p2 #0
>Organization:
jrtnet.ch
>Environment:
FreeBSD pluto.jrtnet.ch 7.2-RELEASE-p2 FreeBSD 7.2-RELEASE-p2 #0: Wed Jun 24 00:14:35 UTC 2009     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
cat /var/crash/info.0
Dump header from device /dev/mirror/gm0b
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 977420288B (932 MB)
  Blocksize: 512
  Dumptime: Thu Jul 16 09:44:01 2009
  Hostname: pluto.jrtnet.ch
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 7.2-RELEASE-p2 #0: Wed Jun 24 00:14:35 UTC 2009
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
  Panic String: spin lock held too long
  Dump Parity: 32247041
  Bounds: 0
  Dump Status: good

kgdb /boot/kernel/kernel.symbols /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
ssppiinn  lloocckk  00xxffffffffffffffff8800bb33eedd4c00  ((sscchheedd  lloocckk  10))  hheelldd  bbyy  0x0fxffffffffff0f000010714333aa560e 0( t(itdi d 10100005040)5 )t otoo ol olnogng

panic: spin lock held too long
cpuid = 0
Uptime: 12d13h12m53s
Physical memory: 2025 MB
Dumping 932 MB: 917 901 885 869 853 837 821 805 789 773 757 741 725 709 693 677 661 645 629 613 597 581 565 549 533 517 501 485 469 453 437 421 405 389 373 357 341 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5

Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from /boot/kernel/ipdivert.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipdivert.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0x0000000000000004 in ?? ()
#2  0xffffffff8050df79 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#3  0xffffffff8050e382 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:574
#4  0xffffffff8050110c in _mtx_lock_spin_failed (m=Variable "m" is not available.
) at /usr/src/sys/kern/kern_mutex.c:449
#5  0xffffffff805011ae in _mtx_lock_spin (m=0xffffffff80b3edc0, tid=18446742974218086112, opts=Variable "opts" is not available.
) at /usr/src/sys/kern/kern_mutex.c:482
#6  0xffffffff805321be in sched_idletd (dummy=Variable "dummy" is not available.
) at /usr/src/sys/kern/sched_ule.c:801
#7  0xffffffff804ea973 in fork_exit (callout=0xffffffff805320b0 <sched_idletd>, arg=0x0, frame=0xfffffffe80033c80) at /usr/src/sys/kern/kern_fork.c:810
#8  0xffffffff807b748e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:455
#9  0x0000000000000000 in ?? ()
#10 0x0000000000000000 in ?? ()
#11 0x0000000000000001 in ?? ()
#12 0x0000000000000000 in ?? ()
#13 0x0000000000000000 in ?? ()
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---q
Quit
(kgdb)          
>How-To-Repeat:
Still waiting until the system is over a longer time 100% idle.

Workaround is to produce a little load and interupts. 
I do gnu-watch "ntpq -p" and gnu-watch "vmstat 1 2" and this even remote on a another machine.
>Fix:


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: gavin 
State-Changed-When: Mon Jul 27 13:32:23 UTC 2009 
State-Changed-Why:  
To submitter:  Firstly, could you please provide the output of: 

ps -aux -M /var/crash/vmcore.0 

Also, is this something that has only ever happened once, or can you reproduce 
it easily?  Are you in a position where you can get information out of the 
system from the kernel debugger if it happens again?  If so, the output 
of "show alllocks; alltrace; ps; show allpcpu" would be very useful. 

If you don't have interactive access to the debugger, have no way of logging 
the output, or need the achine up as quickly as possible, you should be able 
to make use of the textdump(4) facility, enabled from the command prompt as 
follows: 

ddb script kdb.enter.panic="textdump set; capture on; show allpcpu; bt; ps; alltrace; show alllock; call doadump; reset" 
sysctl debug.ddb.textdump.pending=1 

then please provide the ddb.txt from /var/crash 


Responsible-Changed-From-To: freebsd-bugs->gavin 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Mon Jul 27 13:32:23 UTC 2009 
Responsible-Changed-Why:  
Track 

http://www.freebsd.org/cgi/query-pr.cgi?pr=137117 

From: Joerg Ruppe-Tanner <joerg.ruppe.tanner@gmail.com>
To: gavin@FreeBSD.org
Cc: freebsd-bugs@FreeBSD.org
Subject: Re: kern/137117: [panic] panic after spinlock held too long with
 FreeBSD 7.2-RELEASE-p2 on CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1618.46-MHz
 K8-class CPU)
Date: Mon, 10 Aug 2009 08:42:05 +0200

 This is a multi-part message in MIME format.
 --------------020506040501030203080103
 Content-Type: text/plain; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: RIPEMD160
 
 gavin
 
 For Your Information:
 I have tried this patch
 http://people.freebsd.org/~rink/tmp/ipi_7stable.diff
 And added this Options
 options         KDB                     # Kernel with KDB Framework
 options         KDB_UNATTENDED          # Kernel with KDB Framework
 change the default value of the debug.debugger
 _on_panic sysctl to 0
 options         KDB_TRACE               # Kernel with KDB Framework
 change the default value of the debug.trace_on
 _panic sysctl to 1
 
 But the ule scheduler still make a panic when idle 100% or near 100%.
 And the kernel doesn't write every vmcore :(
 On Productive Maschines i can't aktive the whitness options.
 My solution was to bulid a kernel with sched_4bsd schedular and it works.
 
 If I have time, I make the test on a 3rd Machine and option whitness
 when the Problem is still alive and not fixed elsewhere.
 
 Kind Regards
 
 J=C3=B6rg
 
 
 
 Joerg Ruppe-Tanner wrote:
 > gavin
 >
 > Here the output from /var/crash/vmcore.0
 > [root@pluto ~]# ps -aux -M /var/crash/vmcore.0
 > USER   PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
 > [root@pluto ~]#
 >
 > To reproduce is only waiting.
 > The main problem is that the system only one time create a vmcore
 > file, the other times the only solution was power off the maschine not
 > keyboard interaction.
 > The Problem apears with release 7.2. With release 7.1 there was no
 > problems.
 >
 > I have some Problems with ddb command even with sysctl
 > debug.ddb.textdump.pendinding=3D1.
 > Here are my sysctl debug
 >
 > [root@pluto ~]# sysctl debug
 > debug.acpi.semaphore_debug: 0
 > debug.acpi.suspend_bounce: 0
 > debug.acpi.do_powerstate: 1
 > debug.acpi.acpi_ca_version: 20070320
 > debug.acpi.ec.timeout: 750
 > debug.acpi.ec.polled: 0
 > debug.acpi.ec.burst: 0
 > debug.acpi.batt.batt_sleep_ms: 0
 > debug.firewire_debug: 0
 > debug.fwmem_debug: 0
 > debug.if_fwe_debug: 0
 > debug.if_fwip_debug: 0
 > debug.sbp_debug: 0
 > debug.mddebug: 0
 > debug.elf64_legacy_coredump: 0
 > debug.elf64_trace: 0
 > debug.bootverbose: 0
 > debug.boothowto: 0
 > debug.cpufreq.verbose: 0
 > debug.cpufreq.lowest: 0
 > debug.sizeof.cdev_priv: 376
 > debug.sizeof.cdev: 288
 > debug.sizeof.g_bioq: 72
 > debug.sizeof.g_consumer: 96
 > debug.sizeof.g_provider: 136
 > debug.sizeof.g_geom: 136
 > debug.sizeof.g_class: 136
 > debug.sizeof.kinfo_proc: 1088
 > debug.sizeof.buf: 640
 > debug.sizeof.bio: 216
 > debug.sizeof.proc: 1144
 > debug.sizeof.vnode: 504
 > debug.sizeof.devstat: 288
 > debug.sizeof.namecache: 72
 > debug.to_avg_mpcalls: 2000
 > debug.to_avg_mtxcalls: 0
 > debug.to_avg_gcalls: 1145
 > debug.to_avg_depth: 3279
 > debug.umtx.umtx_pi_allocated: 0
 > debug.kdb.stop_cpus: 1
 > debug.kdb.trap_code: 0
 > debug.kdb.trap: 0
 > debug.kdb.panic: 0
 > debug.kdb.enter: 0
 > debug.kdb.current:
 > debug.kdb.available:
 > debug.rman_debug: 0
 > debug.ttydebug: 0
 > debug.disablefullpath: 0
 > debug.disablecwd: 0
 > debug.vfscache: 1
 > debug.numcachehv: 12695
 > debug.numcache: 71522
 > debug.numneg: 4468
 > debug.ncnegfactor: 16
 > debug.nchash: 131071
 > debug.vnlru_nowhere: 0
 > debug.rush_requests: 0
 > debug.mpsafevfs: 1
 > debug.if_tun_debug: 0
 > debug.nlm_debug: 0
 > debug.collectsnapstats: 0
 > debug.snapdebug: 0
 > debug.dopersistence: 0
 > debug.dir_entry: 2378
 > debug.direct_blk_ptrs: 869
 > debug.inode_bitmap: 2093
 > debug.indir_blk_ptrs: 666
 > debug.sync_limit_hit: 0
 > debug.ino_limit_hit: 0
 > debug.blk_limit_hit: 0
 > debug.ino_limit_push: 0
 > debug.blk_limit_push: 0
 > debug.worklist_push: 0
 > debug.maxindirdeps: 50
 > debug.tickdelay: 2
 > debug.max_softdeps: 400000
 > debug.dobkgrdwrite: 1
 > debug.bigcgs: 0
 > debug.dircheck: 0
 > debug.minidump: 1
 > debug.stop_cpus_with_nmi: 1
 > debug.psm.pkterrthresh: 2
 > debug.psm.usecs: 500000
 > debug.psm.secs: 0
 > debug.psm.errusecs: 0
 > debug.psm.errsecs: 2
 > debug.psm.hz: 20
 > debug.psm.loglevel: 0
 > debug.fdc.settle: 0
 > debug.fdc.spec2: 16
 > debug.fdc.spec1: 175
 > debug.fdc.retries: 10
 > debug.fdc.debugflags: 0
 > debug.fdc.fifo: 8
 > debug.elf32_legacy_coredump: 0
 > debug.elf32_trace: 0
 > debug.nullfs_bug_bypass: 0
 > [root@pluto ~]#    =20
 >
 > Should I boot the kernel in debug modus?  I can't believe.
 >
 > Kind Regards
 >
 > J=C3=B6rg
 >
 >
 >
 >
 > gavin@FreeBSD.org wrote:
 > > Synopsis: [panic] panic after spinlock held too long with FreeBSD
 > 7.2-RELEASE-p2 on CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz
 > (1618.46-MHz K8-class CPU)
 > > State-Changed-From-To: open->feedback
 > > State-Changed-By: gavin
 > > State-Changed-When: Mon Jul 27 13:32:23 UTC 2009
 > > State-Changed-Why:
 > > To submitter:  Firstly, could you please provide the output of:
 >
 > > ps -aux -M /var/crash/vmcore.0
 >
 > > Also, is this something that has only ever happened once, or can you
 > reproduce
 > > it easily?  Are you in a position where you can get information
 > out of the
 > > system from the kernel debugger if it happens again?  If so, the
 > output
 > > of "show alllocks; alltrace; ps; show allpcpu" would be very useful.
 >
 > > If you don't have interactive access to the debugger, have no way of
 > logging
 > > the output, or need the achine up as quickly as possible, you
 > should be
 > able
 > > to make use of the textdump(4) facility, enabled from the command
 > prompt as
 > > follows:
 >
 > > ddb script kdb.enter.panic=3D"textdump set; capture on; show
 > allpcpu; bt;
 > ps; alltrace; show alllock; call doadump; reset"
 > > sysctl debug.ddb.textdump.pending=3D1
 >
 > > then please provide the ddb.txt from /var/crash
 >
 >
 > > Responsible-Changed-From-To: freebsd-bugs->gavin
 > > Responsible-Changed-By: gavin
 > > Responsible-Changed-When: Mon Jul 27 13:32:23 UTC 2009
 > > Responsible-Changed-Why:
 > > Track
 >
 > > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D137117
 >
 >
 >
 
 - --
 
 
 Joerg Ruppe-Tanner
 Schuetzenweg 19
 3294 Bueren an der Aare
 
 email: joerg.ruppe.tanner@gmail.com
 email: joerg.ruppe-tanner@ch-open.ch
 http://www.jrtnet.ch
 Tel.:  (+41) 32 351 5840
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.9 (FreeBSD)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
 
 iEYEAREDAAYFAkp/wTkACgkQaUHqwWEIg1tC0ACfSavYCZvrl3Xn1DEWJYEsB0MH
 qmYAnjcP2mT7p2riSLTlMeQJqbEks6eB
 =3DCRo/
 -----END PGP SIGNATURE-----
 
 
 --------------020506040501030203080103
 Content-Type: text/x-vcard; charset=utf-8;
  name="joerg_ruppe_tanner.vcf"
 Content-Transfer-Encoding: base64
 Content-Disposition: attachment;
  filename="joerg_ruppe_tanner.vcf"
 
 YmVnaW46dmNhcmQNCmZuO3F1b3RlZC1wcmludGFibGU6Sj1DMz1CNnJnIFJ1cHBlLVRhbm5l
 cg0KbjtxdW90ZWQtcHJpbnRhYmxlOlJ1cHBlLVRhbm5lcjtKPUMzPUI2cmcNCmVtYWlsO2lu
 dGVybmV0OmpvZXJnLnJ1cHBlLnRhbm5lckBnbWFpbC5jb20NCngtbW96aWxsYS1odG1sOlRS
 VUUNCnZlcnNpb246Mi4xDQplbmQ6dmNhcmQNCg0K
 --------------020506040501030203080103--

From: Gavin Atkinson <gavin@FreeBSD.org>
To: Joerg Ruppe-Tanner <joerg.ruppe.tanner@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/137117: [panic] panic after spinlock held too long with
	FreeBSD 7.2-RELEASE-p2
Date: Mon, 14 Sep 2009 14:41:06 +0100

 Hi,
 
 Are you able to try this patch?:
 http://www.freebsd.org/~attilio/sched_ule.diff
 
 Thanks,
 
 Gavin
 

From: Joerg Ruppe-Tanner <joerg.ruppe.tanner@gmail.com>
To: Gavin Atkinson <gavin@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/137117: [panic] panic after spinlock held too long with
 FreeBSD 7.2-RELEASE-p2
Date: Tue, 15 Sep 2009 12:48:33 +0200

 This is a multi-part message in MIME format.
 --------------020704060405010705020300
 Content-Type: text/plain; charset=UTF-8
 Content-Transfer-Encoding: 8bit
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: RIPEMD160
 
 Hi Gavin,
 
 Yes, I am able. If I get problems with different sched_ule.c version,
 I will inform You.
 I keep You Informed with the testing.
 
 Kind regards
 
 Jörg
 
 
 
 Gavin Atkinson wrote:
 > Hi,
 >
 > Are you able to try this patch?:
 > http://www.freebsd.org/~attilio/sched_ule.diff
 >
 > Thanks,
 >
 > Gavin
 >
 >
 
 
 - --
 
 
 Joerg Ruppe-Tanner
 Schuetzenweg 19
 3294 Bueren an der Aare
 
 email: joerg.ruppe.tanner@gmail.com
 email: joerg.ruppe-tanner@ch-open.ch
 http://www.jrtnet.ch
 Tel.:  (+41) 32 351 5840
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.10 (FreeBSD)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
 
 iEYEAREDAAYFAkqvcP4ACgkQaUHqwWEIg1tcxwCfQr9mGLf3lmpx/4OQnJ8dCcRo
 16cAnj+VgW5IFeRlxXEovHVA/KlTcLf7
 =PpPw
 -----END PGP SIGNATURE-----
 
 
 --------------020704060405010705020300
 Content-Type: text/x-vcard; charset=utf-8;
  name="joerg_ruppe_tanner.vcf"
 Content-Transfer-Encoding: base64
 Content-Disposition: attachment;
  filename="joerg_ruppe_tanner.vcf"
 
 YmVnaW46dmNhcmQNCmZuO3F1b3RlZC1wcmludGFibGU6Sj1DMz1CNnJnIFJ1cHBlLVRhbm5l
 cg0KbjtxdW90ZWQtcHJpbnRhYmxlOlJ1cHBlLVRhbm5lcjtKPUMzPUI2cmcNCmVtYWlsO2lu
 dGVybmV0OmpvZXJnLnJ1cHBlLnRhbm5lckBnbWFpbC5jb20NCngtbW96aWxsYS1odG1sOlRS
 VUUNCnZlcnNpb246Mi4xDQplbmQ6dmNhcmQNCg0K
 --------------020704060405010705020300--

From: Joerg Ruppe-Tanner <joerg.ruppe.tanner@gmail.com>
To: Gavin Atkinson <gavin@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/137117: [panic] panic after spinlock held too long with
 FreeBSD 7.2-RELEASE-p2
Date: Tue, 22 Sep 2009 10:16:55 +0200

 This is a multi-part message in MIME format.
 --------------020506050001030103060707
 Content-Type: text/plain; charset=UTF-8
 Content-Transfer-Encoding: 8bit
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: RIPEMD160
 
 Hi Gavin
 
 I tried Your Patch. And added in the Kernelconf File
 options         DDB                     # Kernel with DDB Framework
 options         KDB                     # Kernel with KDB Framework
 
 Now this work for three day's without any Problems.(It looks like it
 will be fixed)
 I will try another System and keep You Informed.
 I think, in one week.
 Is that ok for You?
 
 Kind Regards
 
 Jörg
 
 
 Gavin Atkinson wrote:
 > Hi,
 >
 > Are you able to try this patch?:
 > http://www.freebsd.org/~attilio/sched_ule.diff
 >
 > Thanks,
 >
 > Gavin
 >
 >
 
 
 - --
 
 
 Joerg Ruppe-Tanner
 Schuetzenweg 19
 3294 Bueren an der Aare
 
 email: joerg.ruppe.tanner@gmail.com
 email: joerg.ruppe-tanner@ch-open.ch
 http://www.jrtnet.ch
 Tel.:  (+41) 32 351 5840
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.10 (FreeBSD)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
 
 iEYEAREDAAYFAkq4h/IACgkQaUHqwWEIg1vWhwCghIo+D/beT7eTG5Gv5JfkjM4i
 6AQAn1nO/5vHcJDuOzgJp2hZmnE4JU6O
 =q/WA
 -----END PGP SIGNATURE-----
 
 
 --------------020506050001030103060707
 Content-Type: text/x-vcard; charset=utf-8;
  name="joerg_ruppe_tanner.vcf"
 Content-Transfer-Encoding: base64
 Content-Disposition: attachment;
  filename="joerg_ruppe_tanner.vcf"
 
 YmVnaW46dmNhcmQNCmZuO3F1b3RlZC1wcmludGFibGU6Sj1DMz1CNnJnIFJ1cHBlLVRhbm5l
 cg0KbjtxdW90ZWQtcHJpbnRhYmxlOlJ1cHBlLVRhbm5lcjtKPUMzPUI2cmcNCmVtYWlsO2lu
 dGVybmV0OmpvZXJnLnJ1cHBlLnRhbm5lckBnbWFpbC5jb20NCngtbW96aWxsYS1odG1sOlRS
 VUUNCnZlcnNpb246Mi4xDQplbmQ6dmNhcmQNCg0K
 --------------020506050001030103060707--
State-Changed-From-To: feedback->closed 
State-Changed-By: gavin 
State-Changed-When: Mon Sep 28 08:58:50 UTC 2009 
State-Changed-Why:  
Close, submitter is happy that r197223 (also merged to 8.x and 7.x) fixes 
the problem for him. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=137117 
>Unformatted:
