From root@zavijava4.math.ualberta.ca  Fri Jan 16 17:25:15 2009
Return-Path: <root@zavijava4.math.ualberta.ca>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D5592106566B
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 16 Jan 2009 17:25:15 +0000 (UTC)
	(envelope-from root@zavijava4.math.ualberta.ca)
Received: from zavijava4.math.ualberta.ca (zavijava4.math.ualberta.ca [129.128.206.34])
	by mx1.freebsd.org (Postfix) with ESMTP id 670818FC14
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 16 Jan 2009 17:25:15 +0000 (UTC)
	(envelope-from root@zavijava4.math.ualberta.ca)
Received: from zavijava4.math.ualberta.ca (localhost.math.ualberta.ca [127.0.0.1])
	by zavijava4.math.ualberta.ca (8.14.3/8.14.3) with ESMTP id n0GGqE3A008424
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 16 Jan 2009 09:52:14 -0700 (MST)
	(envelope-from root@zavijava4.math.ualberta.ca)
Received: (from root@localhost)
	by zavijava4.math.ualberta.ca (8.14.3/8.14.3/Submit) id n0GGqEWo008423;
	Fri, 16 Jan 2009 09:52:14 -0700 (MST)
	(envelope-from root)
Message-Id: <200901161652.n0GGqEWo008423@zavijava4.math.ualberta.ca>
Date: Fri, 16 Jan 2009 09:52:14 -0700 (MST)
From: <bvowk@math.ualberta.ca>
Reply-To: <bvowk@math.ualberta.ca>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: NFS / rpc.lockd deadlock on 7.1-R
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         130628
>Category:       kern
>Synopsis:       [nfs] NFS / rpc.lockd deadlock on 7.1-R
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jan 16 17:30:01 UTC 2009
>Closed-Date:    
>Last-Modified:  Wed Oct 13 12:10:07 UTC 2010
>Originator:     Barkley Vowk
>Release:        FreeBSD 7.1-RELEASE-p1 amd64
>Organization:
University of Alberta
>Environment:
System: FreeBSD zavijava4.math.ualberta.ca 7.1-RELEASE-p1 FreeBSD 7.1-RELEASE-p1 #10: Wed Jan 14 12:13:19 MST 2009 root@zavijava4.math.ualberta.ca:/usr/obj/usr/src/sys/GENERIC amd64

>Description:
	7.1-R deadlocks when acting as an NFS server for a debian box running 
	2.6.18-6-amd64. The problem can be reproduced in a few minutes with
	dbench running on the linux machine into an nfs mounted directory from
	the 7.1 box. 

>How-To-Repeat:
	mount zj4:/home /mnt ; cd /mnt ; dbench -D . 5

	With a generic kernel, it lasts about 5 seconds. With all the debugging
	options it can last a few minutes. 

	I've included two dumps as requested from the debugging deadlocks 
	section of the handbook. 

	KDB: enter: Break sequence on console
[thread pid 36 tid 100035 ]
Stopped at      kdb_enter_why+0x3d:     movq    $0,0x608808(%rip)
db> ps 
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
  898   894   898     0  S+      select   0xffffffff80bc59e0 top
  894   891   894     0  Ss+     pause    0xffffff000370e5c8 csh
  891   818   891     0  Ss      select   0xffffffff80bc59e0 sshd
  890   874    48     0  S       nanslp   0xffffffff80ade5e8 fsck_ufs
  885     1   885     0  Ss+     ttyin    0xffffff0002f07c10 getty
  884     1   884     0  Ss+     ttyin    0xffffff0002f29810 getty
  883     1   883     0  Ss+     ttyin    0xffffff0002f29c10 getty
  882     1   882     0  Ss+     ttyin    0xffffff0002f2a010 getty
  881     1   881     0  Ss+     ttyin    0xffffff0002f2b410 getty
  880     1   880     0  Ss+     ttyin    0xffffff0002f1d810 getty
  879     1   879     0  Ss+     ttyin    0xffffff0002f2c010 getty
  878     1   878     0  Ss+     ttyin    0xffffff0002f2bc10 getty
  877     1   877     0  Ss+     ttyin    0xffffff0002f1c810 getty
  874   872    48     0  S+      wait     0xffffff0003077000 fsck
  873     1    48     0  S+      piperd   0xffffff000340fc60 logger
  872     1    48     0  S+      wait     0xffffff00030f29b0 sh
  834     1   834     0  Ss      nanslp   0xffffffff80ade5e8 cron
  828     1   828    25  Ss      pause    0xffffff00030abaa0 sendmail
  824     1   824     0  Ss      select   0xffffffff80bc59e0 sendmail
  818     1   818     0  Ss      select   0xffffffff80bc59e0 sshd
  803   802   802   123  S       select   0xffffffff80bc59e0 ntpd
  802     1   802     0  Ss      select   0xffffffff80bc59e0 ntpd
  738     1   738     0  Rs      CPU 3                       rpc.lockd
  725   724   724     0  S       kqread   0xffffff0003c5a000 rpc.statd
  724     1   724     0  Ss      select   0xffffffff80bc59e0 rpc.statd
  711   703   703     0  S       -        0xffffff000328d600 nfsd
  710   703   703     0  S       -        0xffffff0002fbc400 nfsd
  709   703   703     0  S       -        0xffffff000328c600 nfsd
  708   703   703     0  S       -        0xffffff0003149000 nfsd
  707   703   703     0  S       -        0xffffff0003114600 nfsd
  706   703   703     0  S       -        0xffffff0003293600 nfsd
  705   703   703     0  S       -        0xffffff0003112200 nfsd
  704   703   703     0  S       -        0xffffff0003293200 nfsd
  703     1   703     0  Ss      select   0xffffffff80bc59e0 nfsd
  701     1   701     0  Ss      select   0xffffffff80bc59e0 mountd
  680     1   680     0  Ss      select   0xffffffff80bc59e0 rpcbind
  605     1   605     0  Ss      select   0xffffffff80bc59e0 syslogd
  550     1   550     0  Ss      select   0xffffffff80bc59e0 devd
   47     0     0     0  SL      sdflush  0xffffffff80bd7418 [softdepflush]
   46     0     0     0  SL      syncer   0xffffffff80ade1c0 [syncer]
   45     0     0     0  SL      vlruwt   0xffffff0002f734d8 [vnlru]
   44     0     0     0  SL      psleep   0xffffffff80bc63dc [bufdaemon]
   43     0     0     0  SL      pgzero   0xffffffff80bd9264 [pagezero]
   42     0     0     0  SL      psleep   0xffffffff80bd8548 [vmdaemon]
   41     0     0     0  SL      psleep   0xffffffff80bd850c [pagedaemon]
   40     0     0     0  SL      waiting_ 0xffffffff80bca268 [sctp_iterator]
   39     0     0     0  WL                                  [irq1: atkbd0]
   38     0     0     0  WL                                  [swi0: sio]
   37     0     0     0  WL                                  [irq257: bge1]
   36     0     0     0  RL      CPU 2                       [irq256: bge0]
   35     0     0     0  WL                                  [irq24: arcmsr0]
   34     0     0     0  WL                                  [irq16: fxp0]
   33     0     0     0  SL      usbevt   0xffffff00012d1420 [usb2]
   32     0     0     0  SL      usbevt   0xffffffff8103a420 [usb1]
   31     0     0     0  SL      usbtsk   0xffffffff80ad76e8 [usbtask-dr]
   30     0     0     0  SL      usbtsk   0xffffffff80ad76c0 [usbtask-hc]
   29     0     0     0  SL      usbevt   0xffffffff81038420 [usb0]
   28     0     0     0  WL                                  [irq10: ohci0 ohci+]
   27     0     0     0  WL                                  [irq15: ata1]
   26     0     0     0  WL                                  [irq14: ata0]
   25     0     0     0  WL                                  [irq11: atapci0+]
   24     0     0     0  WL                                  [irq9: acpi0]
   23     0     0     0  SL      -        0xffffff000116fd00 [thread taskq]
   22     0     0     0  WL                                  [swi5: +]
    9     0     0     0  SL      -        0xffffff0001251200 [kqueue taskq]
    8     0     0     0  SL      -        0xffffff0001251300 [acpi_task_2]
    7     0     0     0  SL      -        0xffffff0001251300 [acpi_task_1]
    6     0     0     0  SL      -        0xffffff0001251300 [acpi_task_0]
   21     0     0     0  WL                                  [swi2: cambio]
    5     0     0     0  SL      ccb_scan 0xffffffff80aa5260 [xpt_thrd]
   20     0     0     0  WL                                  [swi6: task queue]
   19     0     0     0  WL                                  [swi6: Giant taskq]
   18     0     0     0  SL      -        0xffffffff80ade1c8 [yarrow]
    4     0     0     0  SL      -        0xffffffff80ad8268 [g_down]
    3     0     0     0  SL      -        0xffffffff80ad8260 [g_up]
    2     0     0     0  SL      -        0xffffffff80ad8250 [g_event]
   17     0     0     0  WL                                  [swi3: vm]
   16     0     0     0  RL      CPU 1                       [swi4: clock sio]
   15     0     0     0  WL                                  [swi1: net]
   14     0     0     0  RL      CPU 0                       [idle: cpu0]
   13     0     0     0  RL                                  [idle: cpu1]
   12     0     0     0  RL                                  [idle: cpu2]
   11     0     0     0  RL                                  [idle: cpu3]
    1     0     1     0  SLs     wait     0xffffff00010fd9b0 [init]
   10     0     0     0  SL      audit_wo 0xffffffff80bd67d0 [audit]
    0     0     0     0  SLs     sched    0xffffffff80ad8360 [swapper]
db> 
db> show pcpu
cpuid        = 2
curthread    = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb       = 0xffffffffae62bd40
fpcurthread  = none
idlethread   = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show allpcpu
Current CPU: 2

cpuid        = 0
curthread    = 0xffffff000110c6e0: pid 14 "idle: cpu0"
curpcb       = 0xffffffffac50bd40
fpcurthread  = none
idlethread   = 0xffffff000110c6e0: pid 14 "idle: cpu0"
spin locks held:

cpuid        = 1
curthread    = 0xffffff000110c000: pid 16 "swi4: clock sio"
curpcb       = 0xffffffffac518d40
fpcurthread  = none
idlethread   = 0xffffff0001100000: pid 13 "idle: cpu1"
spin locks held:

cpuid        = 2
curthread    = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb       = 0xffffffffae62bd40
fpcurthread  = none
idlethread   = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391

cpuid        = 3
curthread    = 0xffffff00032baa50: pid 738 "rpc.lockd"
curpcb       = 0xffffffffb0aa3d40
fpcurthread  = none
idlethread   = 0xffffff00011006e0: pid 11 "idle: cpu3"
spin locks held:

db> 
db> show locks
exclusive rw tcpinp r = 0 (0xffffff00034540f0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show alllocks
Process 738 (rpc.lockd) thread 0xffffff00032baa50 (100097)
exclusive sx so_rcv_sx r = 0 (0xffffff00034f5850) locked @ /usr/src/sys/kern/uipc_sockbuf.c:150
exclusive sleep mutex xprt->xp_lock r = 0 (0xffffff00031d1000) locked @ /usr/src/sys/rpc/svc_vc.c:614
Process 36 (irq256: bge0) thread 0xffffff0002e74000 (100035)
exclusive rw tcpinp r = 0 (0xffffff00034540f0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
Process 16 (swi4: clock sio) thread 0xffffff000110c000 (100007)
shared rw tcp r = 0 (0xffffffff80bcab08) locked @ /usr/src/sys/netinet/tcp_timer.c:155
db> 
db> show lockedvnods
Locked vnodes
db> 
db> alltrace

Tracing command top pid 898 tid 100120 td 0xffffff0003a76370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800a8eadc, rsp = 0x7fffffffe9c8, rbp = 0x11 ---

Tracing command csh pid 894 tid 100116 td 0xffffff0003105a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80093419c, rsp = 0x7fffffffe608, rbp = 0x800c3f300 ---

Tracing command sshd pid 891 tid 100115 td 0xffffff0003103000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe4b8, rbp = 0x7fffffffe540 ---

Tracing command fsck_ufs pid 890 tid 100067 td 0xffffff000304d6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80080d8ac, rsp = 0x7fffffffdbf8, rbp = 0x1 ---

Tracing command getty pid 885 tid 100114 td 0xffffff0003103370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
Tracing command getty pid 884 tid 100113 td 0xffffff00031036e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 883 tid 100112 td 0xffffff0003103a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 882 tid 100111 td 0xffffff00030f6000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 881 tid 100104 td 0xffffff00032b9000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 880 tid 100066 td 0xffffff0003055000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---
        
Tracing command getty pid 879 tid 100053 td 0xffffff0003055a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 878 tid 100047 td 0xffffff000304da50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 877 tid 100078 td 0xffffff00030796e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command fsck pid 874 tid 100073 td 0xffffff0003054a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80069ce8c, rsp = 0x7fffffffe378, rbp = 0x3 ---

Tracing command logger pid 873 tid 100051 td 0xffffff0002fa4a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x2bc
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800717b5c, rsp = 0x7fffffffe908, rbp = 0x7fffffffee18 ---

Tracing command sh pid 872 tid 100083 td 0xffffff00031046e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80091be8c, rsp = 0x7fffffffcf98, rbp = 0x30 ---

Tracing command cron pid 834 tid 100072 td 0xffffff00030f6370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8009188ac, rsp = 0x7fffffffec18, rbp = 0x3c ---

Tracing command sendmail pid 828 tid 100056 td 0xffffff0002e74a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1419c, rsp = 0x7fffffffda38, rbp = 0x1 ---

Tracing command sendmail pid 824 tid 100065 td 0xffffff00030f66e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800da5adc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7e0 ---

Tracing command sshd pid 818 tid 100105 td 0xffffff0003104000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe5c8, rbp = 0 ---

Tracing command ntpd pid 803 tid 100055 td 0xffffff0002fa4000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffec38, rbp = 0 ---

Tracing command ntpd pid 802 tid 100099 td 0xffffff00032ba370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
poll() at poll+0x567
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffed38, rbp = 0x323 ---

Tracing command rpc.lockd pid 738 tid 100097 td 0xffffff00032baa50
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffffaba3fff0, rbp = 0xffffffffb0aa3370 ---
_rw_wlock_hard() at _rw_wlock_hard+0x167
_rw_wlock() at _rw_wlock+0xc0
tcp_usr_rcvd() at tcp_usr_rcvd+0x3e
soreceive_generic() at soreceive_generic+0x498
svc_vc_recv() at svc_vc_recv+0x169
svc_run() at svc_run+0x146
nlm_syscall() at nlm_syscall+0x79c
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008a8d7c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee20 ---

Tracing command rpc.statd pid 725 tid 100101 td 0xffffff00032b9a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_kevent() at kern_kevent+0x324
kevent() at kevent+0x90
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800806a2c, rsp = 0x7fffffffe478, rbp = 0x800a8a000 ---

Tracing command rpc.statd pid 724 tid 100049 td 0xffffff0003056000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800821adc, rsp = 0x7fffffffebe8, rbp = 0x800960740 ---

Tracing command nfsd pid 711 tid 100093 td 0xffffff00032bca50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x7 ---

Tracing command nfsd pid 710 tid 100092 td 0xffffff00032bd000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x6 ---

Tracing command nfsd pid 709 tid 100050 td 0xffffff0002fa5000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x5 ---

Tracing command nfsd pid 708 tid 100052 td 0xffffff0002fa46e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x4 ---

Tracing command nfsd pid 707 tid 100060 td 0xffffff0002e74370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x3 ---

Tracing command nfsd pid 706 tid 100103 td 0xffffff00032b9370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x2 ---

Tracing command nfsd pid 705 tid 100074 td 0xffffff00030546e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x1 ---

Tracing command nfsd pid 704 tid 100061 td 0xffffff00030f76e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0 ---

Tracing command nfsd pid 703 tid 100068 td 0xffffff000304d370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800718adc, rsp = 0x7fffffffea38, rbp = 0x5 ---

Tracing command mountd pid 701 tid 100090 td 0xffffff0003079000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082dadc, rsp = 0x7fffffffed28, rbp = 0x7fffffffee20 ---

Tracing command rpcbind pid 680 tid 100091 td 0xffffff00032bd370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8008e4e7c, rsp = 0x7fffffffcb98, rbp = 0x800b0c080 ---

Tracing command syslogd pid 605 tid 100095 td 0xffffff00032bc370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082cadc, rsp = 0x7fffffffddd8, rbp = 0x8 ---

Tracing command devd pid 550 tid 100064 td 0xffffff00030f6a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43fd0c, rsp = 0x7fffffffe918, rbp = 0x7fffffffed40 ---

Tracing command softdepflush pid 47 tid 100046 td 0xffffff0002fa56e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
softdep_flush() at softdep_flush+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae6a1d30, rbp = 0 ---

Tracing command syncer pid 46 tid 100045 td 0xffffff0002fa5a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sched_sync() at sched_sync+0x61a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae69cd30, rbp = 0 ---

Tracing command vnlru pid 45 tid 100044 td 0xffffff0001362a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vnlru_proc() at vnlru_proc+0x696
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae697d30, rbp = 0 ---

Tracing command bufdaemon pid 44 tid 100043 td 0xffffff0002e71000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
buf_daemon() at buf_daemon+0x208
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae692d30, rbp = 0 ---

Tracing command pagezero pid 43 tid 100042 td 0xffffff0002e71370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pagezero() at vm_pagezero+0x73
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae68dd30, rbp = 0 ---

Tracing command vmdaemon pid 42 tid 100041 td 0xffffff0002e716e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae688d30, rbp = 0 ---
        
Tracing command pagedaemon pid 41 tid 100040 td 0xffffff0002e71a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pageout() at vm_pageout+0xaed
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae683d30, rbp = 0 ---

Tracing command sctp_iterator pid 40 tid 100039 td 0xffffff0002e73000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sctp_iterator_thread() at sctp_iterator_thread+0x54
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae67ed30, rbp = 0 ---

Tracing command irq1: atkbd0 pid 39 tid 100038 td 0xffffff0002e73370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae656d30, rbp = 0 ---

Tracing command swi0: sio pid 38 tid 100037 td 0xffffff0002e736e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae651d30, rbp = 0 ---

Tracing command irq257: bge1 pid 37 tid 100036 td 0xffffff0002e73a50
fork_trampoline() at fork_trampoline

Tracing command irq256: bge0 pid 36 tid 100035 td 0xffffff0002e74000
kdb_enter_why() at kdb_enter_why+0x3d
siointr1() at siointr1+0x238
siointr() at siointr+0x2e
intr_execute_handlers() at intr_execute_handlers+0x92
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff804b03ed, rsp = 0xffffffffae62b800, rbp = 0xffffffffae62b840 ---
_mtx_lock_sleep() at _mtx_lock_sleep+0x12d
_mtx_lock_flags() at _mtx_lock_flags+0x111
svc_vc_soupcall() at svc_vc_soupcall+0x22
sowakeup() at sowakeup+0xf8
tcp_do_segment() at tcp_do_segment+0x1e08
tcp_input() at tcp_input+0xa00
ip_input() at ip_input+0xc0
ether_demux() at ether_demux+0x1ed
ether_input() at ether_input+0x19d
bge_intr() at bge_intr+0x49f
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae62bd30, rbp = 0 ---

Tracing command irq24: arcmsr0 pid 35 tid 100034 td 0xffffff00012596e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae60ad30, rbp = 0 ---

Tracing command irq16: fxp0 pid 34 tid 100033 td 0xffffff0001259a50
fork_trampoline() at fork_trampoline

Tracing command usb2 pid 33 tid 100032 td 0xffffff0001360000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5c7d30, rbp = 0 ---

Tracing command usb1 pid 32 tid 100031 td 0xffffff0001360370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5afd30, rbp = 0 ---

Tracing command usbtask-dr pid 31 tid 100030 td 0xffffff00013606e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a6d30, rbp = 0 ---

Tracing command usbtask-hc pid 30 tid 100029 td 0xffffff0001360a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a1d30, rbp = 0 ---

Tracing command usb0 pid 29 tid 100028 td 0xffffff0001362000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae59cd30, rbp = 0 ---

Tracing command irq10: ohci0 ohci+ pid 28 tid 100027 td 0xffffff0001362370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5f5d30, rbp = 0 ---

Tracing command irq15: ata1 pid 27 tid 100026 td 0xffffff00013626e0
fork_trampoline() at fork_trampoline
        
Tracing command irq14: ata0 pid 26 tid 100025 td 0xffffff000111e6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5d7d30, rbp = 0 ---

Tracing command irq11: atapci0+ pid 25 tid 100024 td 0xffffff000111ea50
fork_trampoline() at fork_trampoline

Tracing command irq9: acpi0 pid 24 tid 100023 td 0xffffff0001258000
fork_trampoline() at fork_trampoline

Tracing command thread taskq pid 23 tid 100022 td 0xffffff0001258370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac567d30, rbp = 0 ---

Tracing command swi5: + pid 22 tid 100021 td 0xffffff00012586e0
fork_trampoline() at fork_trampoline

Tracing command kqueue taskq pid 9 tid 100020 td 0xffffff0001258a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac55dd30, rbp = 0 ---

Tracing command acpi_task_2 pid 8 tid 100019 td 0xffffff0001259000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac558d30, rbp = 0 ---

Tracing command acpi_task_1 pid 7 tid 100018 td 0xffffff0001259370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac553d30, rbp = 0 ---

Tracing command acpi_task_0 pid 6 tid 100017 td 0xffffff000110ca50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac54ed30, rbp = 0 ---

Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff000111d000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac549d30, rbp = 0 ---
        
Tracing command xpt_thrd pid 5 tid 100015 td 0xffffff000111d370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
xpt_scanner_thread() at xpt_scanner_thread+0x3a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac544d30, rbp = 0 ---

Tracing command swi6: task queue pid 20 tid 100014 td 0xffffff000111d6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac53fd30, rbp = 0 ---

Tracing command swi6: Giant taskq pid 19 tid 100013 td 0xffffff000111da50
fork_trampoline() at fork_trampoline

Tracing command yarrow pid 18 tid 100012 td 0xffffff000111e000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
random_kthread() at random_kthread+0x1ad
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac535d30, rbp = 0 ---

Tracing command g_down pid 4 tid 100011 td 0xffffff000111e370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_down() at g_io_schedule_down+0x250
g_down_procbody() at g_down_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac52cd30, rbp = 0 ---

Tracing command g_up pid 3 tid 100010 td 0xffffff0001101370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_up() at g_io_schedule_up+0x154
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac527d30, rbp = 0 ---

Tracing command g_event pid 2 tid 100009 td 0xffffff00011016e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_event_procbody() at g_event_procbody+0xa1
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac522d30, rbp = 0 ---

Tracing command swi3: vm pid 17 tid 100008 td 0xffffff0001101a50
fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 16 tid 100007 td 0xffffff000110c000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffffaba35ff0, rbp = 0xffffffffac518ae0 ---
_rw_wlock_hard() at _rw_wlock_hard+0x167
_rw_wlock() at _rw_wlock+0xc0
tcp_timer_delack() at tcp_timer_delack+0x58
softclock() at softclock+0x299
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac518d30, rbp = 0 ---

Tracing command swi1: net pid 15 tid 100006 td 0xffffff000110c370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac513d30, rbp = 0 ---

Tracing command idle: cpu0 pid 14 tid 100005 td 0xffffff000110c6e0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff807685f6, rsp = 0xffffffff80be4070, rbp = 0xffffffffac50bba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac50bd30, rbp = 0 ---

Tracing command idle: cpu1 pid 13 tid 100004 td 0xffffff0001100000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac506b90, rbp = 0xffffffffac506ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac506d30, rbp = 0 ---
        
Tracing command idle: cpu2 pid 12 tid 100003 td 0xffffff0001100370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
critical_exit() at critical_exit+0x91
intr_execute_handlers() at intr_execute_handlers+0xe9
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac501b90, rbp = 0xffffffffac501ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac501d30, rbp = 0 ---

Tracing command idle: cpu3 pid 11 tid 100002 td 0xffffff00011006e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac4fcb90, rbp = 0xffffffffac4fcba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4fcd30, rbp = 0 ---

Tracing command init pid 1 tid 100001 td 0xffffff0001100a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffed60 ---
        
Tracing command audit pid 10 tid 100000 td 0xffffff0001101000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x15e
audit_worker() at audit_worker+0x77
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4f2d30, rbp = 0 ---

Tracing command swapper pid 0 tid 0 td 0xffffffff80ad8840
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
scheduler() at scheduler+0x2cd
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
db> 
db>reboot
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 2


Stopped at      kdb_enter_why+0x3d:     movq    $0,0x608808(%rip)
db> ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
  895     1   895     0  Ss+     ttyin    0xffffff0002f07c10 getty
  894     1   894     0  Ss+     ttyin    0xffffff0002f29810 getty
  893     1   893     0  Ss+     ttyin    0xffffff0002f29c10 getty
  892     1   892     0  Ss+     ttyin    0xffffff0002f2a010 getty
  891     1   891     0  Ss+     ttyin    0xffffff0002f2b410 getty
  890     1   890     0  Ss+     ttyin    0xffffff0002f1d810 getty
  889     1   889     0  Ss+     ttyin    0xffffff0002f2c010 getty
  888     1   888     0  Ss+     ttyin    0xffffff0002f2bc10 getty
  887     1   887     0  Ss+     ttyin    0xffffff0002f1c810 getty
  844     1   844     0  Ss      nanslp   0xffffffff80ade5e8 cron
  838     1   838    25  Ss      pause    0xffffff0002f725c8 sendmail
  832     1   832     0  Ss      select   0xffffffff80bc59e0 sendmail
  826     1   826     0  Ss      select   0xffffffff80bc59e0 sshd
  811   810   810   123  S       select   0xffffffff80bc59e0 ntpd
  810     1   810     0  Ss      select   0xffffffff80bc59e0 ntpd
  746     1   746     0  Rs      CPU 0                       rpc.lockd
  733   732   732     0  S       kqread   0xffffff0003570400 rpc.statd
  732     1   732     0  Ss      select   0xffffffff80bc59e0 rpc.statd
  720   711   711     0  S       -        0xffffff000312a000 nfsd
  718   711   711     0  S       -        0xffffff000339d600 nfsd
  717   711   711     0  S       -        0xffffff0003301000 nfsd
  716   711   711     0  S       -        0xffffff0002fbc800 nfsd
  715   711   711     0  S       -        0xffffff00032ff400 nfsd
  714   711   711     0  S       -        0xffffff00030e2200 nfsd
  713   711   711     0  S       -        0xffffff00032ffc00 nfsd
  712   711   711     0  S       -        0xffffff00030d2c00 nfsd
  711     1   711     0  Ss      select   0xffffffff80bc59e0 nfsd
  709     1   709     0  Ss      select   0xffffffff80bc59e0 mountd
  688     1   688     0  Ss      select   0xffffffff80bc59e0 rpcbind
  613     1   613     0  Ss      select   0xffffffff80bc59e0 syslogd
  558     1   558     0  Ss      select   0xffffffff80bc59e0 devd
   47     0     0     0  SL      sdflush  0xffffffff80bd7418 [softdepflush]
   46     0     0     0  SL      syncer   0xffffffff80ade1c0 [syncer]
   45     0     0     0  SL      vlruwt   0xffffff0002f734d8 [vnlru]
   44     0     0     0  SL      psleep   0xffffffff80bc63dc [bufdaemon]
   43     0     0     0  SL      pgzero   0xffffffff80bd9264 [pagezero]
   42     0     0     0  SL      psleep   0xffffffff80bd8548 [vmdaemon]
   41     0     0     0  SL      psleep   0xffffffff80bd850c [pagedaemon]
   40     0     0     0  SL      waiting_ 0xffffffff80bca268 [sctp_iterator]
   39     0     0     0  WL                                  [irq1: atkbd0]
   38     0     0     0  WL                                  [swi0: sio]
   37     0     0     0  WL                                  [irq257: bge1]
   36     0     0     0  RL      CPU 2                       [irq256: bge0]
   35     0     0     0  WL                                  [irq24: arcmsr0]
   34     0     0     0  WL                                  [irq16: fxp0]
   33     0     0     0  SL      usbevt   0xffffff00012d1420 [usb2]
   32     0     0     0  SL      usbevt   0xffffffff8103a420 [usb1]
   31     0     0     0  SL      usbtsk   0xffffffff80ad76e8 [usbtask-dr]
   30     0     0     0  SL      usbtsk   0xffffffff80ad76c0 [usbtask-hc]
   29     0     0     0  SL      usbevt   0xffffffff81038420 [usb0]
   28     0     0     0  WL                                  [irq10: ohci0 ohci+]
   27     0     0     0  WL                                  [irq15: ata1]
   26     0     0     0  WL                                  [irq14: ata0]
   25     0     0     0  WL                                  [irq11: atapci0+]
   24     0     0     0  WL                                  [irq9: acpi0]
   23     0     0     0  SL      -        0xffffff000116fd00 [thread taskq]
   22     0     0     0  WL                                  [swi5: +]
    9     0     0     0  SL      -        0xffffff0001251200 [kqueue taskq]
    8     0     0     0  SL      -        0xffffff0001251300 [acpi_task_2]
    7     0     0     0  SL      -        0xffffff0001251300 [acpi_task_1]
    6     0     0     0  SL      -        0xffffff0001251300 [acpi_task_0]
   21     0     0     0  WL                                  [swi2: cambio]
    5     0     0     0  SL      ccb_scan 0xffffffff80aa5260 [xpt_thrd]
   20     0     0     0  WL                                  [swi6: task queue]
   19     0     0     0  WL                                  [swi6: Giant taskq]
   18     0     0     0  SL      -        0xffffffff80ade1c8 [yarrow]
    4     0     0     0  SL      -        0xffffffff80ad8268 [g_down]
    3     0     0     0  SL      -        0xffffffff80ad8260 [g_up]
    2     0     0     0  SL      -        0xffffffff80ad8250 [g_event]
   17     0     0     0  WL                                  [swi3: vm]
   16     0     0     0  RL      CPU 1                       [swi4: clock sio]
   15     0     0     0  WL                                  [swi1: net]
   14     0     0     0  RL                                  [idle: cpu0]
   13     0     0     0  RL                                  [idle: cpu1]
   12     0     0     0  RL                                  [idle: cpu2]
   11     0     0     0  RL      CPU 3                       [idle: cpu3]
    1     0     1     0  SLs     wait     0xffffff00010fd9b0 [init]
   10     0     0     0  SL      audit_wo 0xffffffff80bd67d0 [audit]
    0     0     0     0  SLs     sched    0xffffffff80ad8360 [swapper]
db> show pcpu
cpuid        = 2
curthread    = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb       = 0xffffffffae62bd40
fpcurthread  = none
idlethread   = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show allpcpu
Current CPU: 2

cpuid        = 0
curthread    = 0xffffff0003300000: pid 746 "rpc.lockd"
curpcb       = 0xffffffffb0ac1d40
fpcurthread  = none
idlethread   = 0xffffff000110c6e0: pid 14 "idle: cpu0"
spin locks held:

cpuid        = 1
curthread    = 0xffffff000110c000: pid 16 "swi4: clock sio"
curpcb       = 0xffffffffac518d40
fpcurthread  = none
idlethread   = 0xffffff0001100000: pid 13 "idle: cpu1"
spin locks held:

cpuid        = 2
curthread    = 0xffffff0002e74000: pid 36 "irq256: bge0"
curpcb       = 0xffffffffae62bd40
fpcurthread  = none
idlethread   = 0xffffff0001100370: pid 12 "idle: cpu2"
spin locks held:
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391

cpuid        = 3
curthread    = 0xffffff00011006e0: pid 11 "idle: cpu3"
curpcb       = 0xffffffffac4fcd40
fpcurthread  = none
idlethread   = 0xffffff00011006e0: pid 11 "idle: cpu3"
spin locks held:

db> show locks
exclusive rw tcpinp r = 0 (0xffffff0003422cc0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
db> show alllocks
Process 746 (rpc.lockd) thread 0xffffff0003300000 (100103)
exclusive sx so_rcv_sx r = 0 (0xffffff000337c130) locked @ /usr/src/sys/kern/uipc_sockbuf.c:150
exclusive sleep mutex xprt->xp_lock r = 0 (0xffffff0003bc6400) locked @ /usr/src/sys/rpc/svc_vc.c:614
Process 36 (irq256: bge0) thread 0xffffff0002e74000 (100035)
exclusive rw tcpinp r = 0 (0xffffff0003422cc0) locked @ /usr/src/sys/netinet/tcp_input.c:480
exclusive spin mutex sio r = 0 (0xffffffff80bf9380) locked @ /usr/src/sys/dev/sio/sio.c:1391
Process 16 (swi4: clock sio) thread 0xffffff000110c000 (100007)
shared rw tcp r = 0 (0xffffffff80bcab08) locked @ /usr/src/sys/netinet/tcp_timer.c:155
db> show lockedvnods
Locked vnodes
db> alltrace

Tracing command getty pid 895 tid 100139 td 0xffffff00034ce6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 894 tid 100138 td 0xffffff00034cea50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 893 tid 100136 td 0xffffff0003536370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 892 tid 100135 td 0xffffff00031ba6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 891 tid 100132 td 0xffffff000342f370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 890 tid 100057 td 0xffffff0003020a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 889 tid 100130 td 0xffffff000342fa50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 888 tid 100120 td 0xffffff000343e370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command getty pid 887 tid 100127 td 0xffffff000344b6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
ttysleep() at ttysleep+0x25
ttread() at ttread+0x303
giant_read() at giant_read+0x65
devfs_read_f() at devfs_read_f+0x81
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x4c
read() at read+0x54
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829b5c, rsp = 0x7fffffffed88, rbp = 0 ---

Tracing command cron pid 844 tid 100093 td 0xffffff00031e36e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8009188ac, rsp = 0x7fffffffec18, rbp = 0x3c ---

Tracing command sendmail pid 838 tid 100049 td 0xffffff0002fa5370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1419c, rsp = 0x7fffffffda38, rbp = 0x1 ---

Tracing command sendmail pid 832 tid 100131 td 0xffffff000342f6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800da5adc, rsp = 0x7fffffffcf48, rbp = 0x7fffffffd7e0 ---

Tracing command sshd pid 826 tid 100053 td 0xffffff00030236e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801397adc, rsp = 0x7fffffffe5c8, rbp = 0 ---

Tracing command ntpd pid 811 tid 100102 td 0xffffff0003300370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffec38, rbp = 0 ---

Tracing command ntpd pid 810 tid 100126 td 0xffffff000344ba50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
poll() at poll+0x567
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006c063c, rsp = 0x7fffffffed38, rbp = 0x32b ---

Tracing command rpc.lockd pid 746 tid 100103 td 0xffffff0003300000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe97, rsp = 0xffffffff80be4070, rbp = 0xffffffffb0ac1370 ---
_rw_wlock_hard() at _rw_wlock_hard+0x167
_rw_wlock() at _rw_wlock+0xc0
tcp_usr_rcvd() at tcp_usr_rcvd+0x3e
soreceive_generic() at soreceive_generic+0x498
svc_vc_recv() at svc_vc_recv+0x169
svc_run() at svc_run+0x146
nlm_syscall() at nlm_syscall+0x79c
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008a8d7c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee20 ---

Tracing command rpc.statd pid 733 tid 100122 td 0xffffff000343da50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x24c
kern_kevent() at kern_kevent+0x324
kevent() at kevent+0x90
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800806a2c, rsp = 0x7fffffffe478, rbp = 0x800a8a000 ---

Tracing command rpc.statd pid 732 tid 100100 td 0xffffff00030c7a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
kern_select() at kern_select+0x7f0
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800821adc, rsp = 0x7fffffffebe8, rbp = 0x800960740 ---

Tracing command nfsd pid 720 tid 100115 td 0xffffff00030be6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x7 ---

Tracing command nfsd pid 718 tid 100114 td 0xffffff00030bea50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x6 ---

Tracing command nfsd pid 717 tid 100113 td 0xffffff00030c7000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x5 ---

Tracing command nfsd pid 716 tid 100112 td 0xffffff00030c7370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x4 ---

Tracing command nfsd pid 715 tid 100111 td 0xffffff00030c76e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x3 ---

Tracing command nfsd pid 714 tid 100058 td 0xffffff00030206e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x2 ---

Tracing command nfsd pid 713 tid 100099 td 0xffffff00031e3000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0x1 ---

Tracing command nfsd pid 712 tid 100054 td 0xffffff0002fa5000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
nfssvc() at nfssvc+0x2de
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800687d5c, rsp = 0x7fffffffea28, rbp = 0 ---

Tracing command nfsd pid 711 tid 100048 td 0xffffff00030246e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800718adc, rsp = 0x7fffffffea38, rbp = 0x5 ---

Tracing command mountd pid 709 tid 100083 td 0xffffff0002e74a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082dadc, rsp = 0x7fffffffed28, rbp = 0x7fffffffee20 ---

Tracing command rpcbind pid 688 tid 100098 td 0xffffff00031e3370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x16e
poll() at poll+0x1bf
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8008e4e7c, rsp = 0x7fffffffcb98, rbp = 0x800b0c080 ---

Tracing command syslogd pid 613 tid 100087 td 0xffffff00031ba370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082cadc, rsp = 0x7fffffffddd8, rbp = 0x8 ---

Tracing command devd pid 558 tid 100082 td 0xffffff0002fa4000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x15f
kern_select() at kern_select+0x81c
select() at select+0x56
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43fd0c, rsp = 0x7fffffffe918, rbp = 0x7fffffffed40 ---

Tracing command softdepflush pid 47 tid 100046 td 0xffffff0002fa56e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
softdep_flush() at softdep_flush+0x2d7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae6a1d30, rbp = 0 ---

Tracing command syncer pid 46 tid 100045 td 0xffffff0002fa5a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sched_sync() at sched_sync+0x61a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae69cd30, rbp = 0 ---

Tracing command vnlru pid 45 tid 100044 td 0xffffff0001362a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vnlru_proc() at vnlru_proc+0x696
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae697d30, rbp = 0 ---

Tracing command bufdaemon pid 44 tid 100043 td 0xffffff0002e71000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
buf_daemon() at buf_daemon+0x208
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae692d30, rbp = 0 ---

Tracing command pagezero pid 43 tid 100042 td 0xffffff0002e71370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pagezero() at vm_pagezero+0x73
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae68dd30, rbp = 0 ---

Tracing command vmdaemon pid 42 tid 100041 td 0xffffff0002e716e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae688d30, rbp = 0 ---

Tracing command pagedaemon pid 41 tid 100040 td 0xffffff0002e71a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
vm_pageout() at vm_pageout+0xaed
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae683d30, rbp = 0 ---

Tracing command sctp_iterator pid 40 tid 100039 td 0xffffff0002e73000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
sctp_iterator_thread() at sctp_iterator_thread+0x54
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae67ed30, rbp = 0 ---

Tracing command irq1: atkbd0 pid 39 tid 100038 td 0xffffff0002e73370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae656d30, rbp = 0 ---

Tracing command swi0: sio pid 38 tid 100037 td 0xffffff0002e736e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae651d30, rbp = 0 ---

Tracing command irq257: bge1 pid 37 tid 100036 td 0xffffff0002e73a50
fork_trampoline() at fork_trampoline

Tracing command irq256: bge0 pid 36 tid 100035 td 0xffffff0002e74000
kdb_enter_why() at kdb_enter_why+0x3d
siointr1() at siointr1+0x238
siointr() at siointr+0x2e
intr_execute_handlers() at intr_execute_handlers+0x92
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff804b03e9, rsp = 0xffffffffae62b800, rbp = 0xffffffffae62b840 ---
_mtx_lock_sleep() at _mtx_lock_sleep+0x129
_mtx_lock_flags() at _mtx_lock_flags+0x111
svc_vc_soupcall() at svc_vc_soupcall+0x22
sowakeup() at sowakeup+0xf8
tcp_do_segment() at tcp_do_segment+0x1e08
tcp_input() at tcp_input+0xa00
ip_input() at ip_input+0xc0
ether_demux() at ether_demux+0x1ed
ether_input() at ether_input+0x19d
bge_intr() at bge_intr+0x49f
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae62bd30, rbp = 0 ---

Tracing command irq24: arcmsr0 pid 35 tid 100034 td 0xffffff00012596e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae60ad30, rbp = 0 ---

Tracing command irq16: fxp0 pid 34 tid 100033 td 0xffffff0001259a50
fork_trampoline() at fork_trampoline

Tracing command usb2 pid 33 tid 100032 td 0xffffff0001360000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5c7d30, rbp = 0 ---

Tracing command usb1 pid 32 tid 100031 td 0xffffff0001360370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5afd30, rbp = 0 ---

Tracing command usbtask-dr pid 31 tid 100030 td 0xffffff00013606e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a6d30, rbp = 0 ---

Tracing command usbtask-hc pid 30 tid 100029 td 0xffffff0001360a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
usb_task_thread() at usb_task_thread+0x94
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae5a1d30, rbp = 0 ---

Tracing command usb0 pid 29 tid 100028 td 0xffffff0001362000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
usb_event_thread() at usb_event_thread+0xab
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae59cd30, rbp = 0 ---

Tracing command irq10: ohci0 ohci+ pid 28 tid 100027 td 0xffffff0001362370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5f5d30, rbp = 0 ---

Tracing command irq15: ata1 pid 27 tid 100026 td 0xffffff00013626e0
fork_trampoline() at fork_trampoline

Tracing command irq14: ata0 pid 26 tid 100025 td 0xffffff000111e6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac5d7d30, rbp = 0 ---

Tracing command irq11: atapci0+ pid 25 tid 100024 td 0xffffff000111ea50
fork_trampoline() at fork_trampoline

Tracing command irq9: acpi0 pid 24 tid 100023 td 0xffffff0001258000
fork_trampoline() at fork_trampoline

Tracing command thread taskq pid 23 tid 100022 td 0xffffff0001258370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac567d30, rbp = 0 ---

Tracing command swi5: + pid 22 tid 100021 td 0xffffff00012586e0
fork_trampoline() at fork_trampoline

Tracing command kqueue taskq pid 9 tid 100020 td 0xffffff0001258a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac55dd30, rbp = 0 ---

Tracing command acpi_task_2 pid 8 tid 100019 td 0xffffff0001259000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac558d30, rbp = 0 ---

Tracing command acpi_task_1 pid 7 tid 100018 td 0xffffff0001259370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac553d30, rbp = 0 ---

Tracing command acpi_task_0 pid 6 tid 100017 td 0xffffff000110ca50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
taskqueue_thread_loop() at taskqueue_thread_loop+0x7e
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac54ed30, rbp = 0 ---

Tracing command swi2: cambio pid 21 tid 100016 td 0xffffff000111d000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac549d30, rbp = 0 ---

Tracing command xpt_thrd pid 5 tid 100015 td 0xffffff000111d370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_sleep() at _sleep+0x351
xpt_scanner_thread() at xpt_scanner_thread+0x3a
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac544d30, rbp = 0 ---

Tracing command swi6: task queue pid 20 tid 100014 td 0xffffff000111d6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac53fd30, rbp = 0 ---

Tracing command swi6: Giant taskq pid 19 tid 100013 td 0xffffff000111da50
fork_trampoline() at fork_trampoline

Tracing command yarrow pid 18 tid 100012 td 0xffffff000111e000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
random_kthread() at random_kthread+0x1ad
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac535d30, rbp = 0 ---
        
Tracing command g_down pid 4 tid 100011 td 0xffffff000111e370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_down() at g_io_schedule_down+0x250
g_down_procbody() at g_down_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac52cd30, rbp = 0 ---

Tracing command g_up pid 3 tid 100010 td 0xffffff0001101370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_io_schedule_up() at g_io_schedule_up+0x154
g_up_procbody() at g_up_procbody+0x6f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac527d30, rbp = 0 ---

Tracing command g_event pid 2 tid 100009 td 0xffffff00011016e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
g_event_procbody() at g_event_procbody+0xa1
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac522d30, rbp = 0 ---

Tracing command swi3: vm pid 17 tid 100008 td 0xffffff0001101a50
fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 16 tid 100007 td 0xffffff000110c000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff804bbe9b, rsp = 0xffffffffaba35ff0, rbp = 0xffffffffac518ae0 ---
_rw_wlock_hard() at _rw_wlock_hard+0x16b
_rw_wlock() at _rw_wlock+0xc0
tcp_timer_delack() at tcp_timer_delack+0x58
softclock() at softclock+0x299
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac518d30, rbp = 0 ---

Tracing command swi1: net pid 15 tid 100006 td 0xffffff000110c370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ithread_loop() at ithread_loop+0x3c5
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac513d30, rbp = 0 ---

Tracing command idle: cpu0 pid 14 tid 100005 td 0xffffff000110c6e0
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac50bb90, rbp = 0xffffffffac50bba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac50bd30, rbp = 0 ---

Tracing command idle: cpu1 pid 13 tid 100004 td 0xffffff0001100000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac506b90, rbp = 0xffffffffac506ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac506d30, rbp = 0 ---

Tracing command idle: cpu2 pid 12 tid 100003 td 0xffffff0001100370
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
ipi_bitmap_handler() at ipi_bitmap_handler+0x8a
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff807685f6, rsp = 0xffffffffac501b90, rbp = 0xffffffffac501ba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac501d30, rbp = 0 ---
        
Tracing command idle: cpu3 pid 11 tid 100002 td 0xffffff00011006e0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x378
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff807685f6, rsp = 0xffffffffaba3fff0, rbp = 0xffffffffac4fcba0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a4
sched_idletd() at sched_idletd+0x57
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4fcd30, rbp = 0 ---

Tracing command init pid 1 tid 100001 td 0xffffff0001100a50
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_catch_signals() at sleepq_catch_signals+0x2ab
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3cc
wait4() at wait4+0x35
syscall() at syscall+0x1f9
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffed60 ---

Tracing command audit pid 10 tid 100000 td 0xffffff0001101000
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x15e
audit_worker() at audit_worker+0x77
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac4f2d30, rbp = 0 ---

Tracing command swapper pid 0 tid 0 td 0xffffffff80ad8840
sched_switch() at sched_switch+0x2a1
mi_switch() at mi_switch+0x233
sleepq_switch() at sleepq_switch+0xe4
sleepq_timedwait() at sleepq_timedwait+0x3c
_sleep() at _sleep+0x33e
scheduler() at scheduler+0x2cd
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
db> 


>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Fri Jan 16 19:55:38 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s).  Looks like there should be pretty much 
all the info here necessary to figure this out. 
To submitter: are you using TCP or UDP NFS mounts? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=130628 

From: Barkley Vowk <bvowk@math.ualberta.ca>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/130628: [nfs] NFS / rpc.lockd deadlock on 7.1-R
Date: Mon, 19 Jan 2009 10:24:38 -0700 (MST)

 I'm using UDP mounts.

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/130628: commit references a PR
Date: Fri, 15 May 2009 13:59:01 +0000 (UTC)

 Author: dfr
 Date: Fri May 15 13:58:45 2009
 New Revision: 192142
 URL: http://svn.freebsd.org/changeset/base/192142
 
 Log:
   Back port a change to the locking model used to manage active transports from
   FreeBSD-current to avoid a deadlock.
   
   PR:		130628
 
 Modified:
   stable/7/sys/rpc/svc.c
   stable/7/sys/rpc/svc.h
   stable/7/sys/rpc/svc_dg.c
   stable/7/sys/rpc/svc_vc.c
 
 Modified: stable/7/sys/rpc/svc.c
 ==============================================================================
 --- stable/7/sys/rpc/svc.c	Fri May 15 13:26:54 2009	(r192141)
 +++ stable/7/sys/rpc/svc.c	Fri May 15 13:58:45 2009	(r192142)
 @@ -178,18 +178,23 @@ xprt_active(SVCXPRT *xprt)
  }
  
  void
 -xprt_inactive(SVCXPRT *xprt)
 +xprt_inactive_locked(SVCXPRT *xprt)
  {
  	SVCPOOL *pool = xprt->xp_pool;
  
 -	mtx_lock(&pool->sp_lock);
 -
  	if (xprt->xp_active) {
  		TAILQ_REMOVE(&pool->sp_active, xprt, xp_alink);
  		xprt->xp_active = FALSE;
  	}
 -	wakeup(&pool->sp_active);
 +}
  
 +void
 +xprt_inactive(SVCXPRT *xprt)
 +{
 +	SVCPOOL *pool = xprt->xp_pool;
 +
 +	mtx_lock(&pool->sp_lock);
 +	xprt_inactive_locked(xprt);
  	mtx_unlock(&pool->sp_lock);
  }
  
 
 Modified: stable/7/sys/rpc/svc.h
 ==============================================================================
 --- stable/7/sys/rpc/svc.h	Fri May 15 13:26:54 2009	(r192141)
 +++ stable/7/sys/rpc/svc.h	Fri May 15 13:58:45 2009	(r192142)
 @@ -47,6 +47,7 @@
  #include <sys/queue.h>
  #include <sys/_lock.h>
  #include <sys/_mutex.h>
 +#include <sys/_sx.h>
  #endif
  
  /*
 @@ -128,7 +129,7 @@ struct __rpc_svcpool;
   */
  typedef struct __rpc_svcxprt {
  #ifdef _KERNEL
 -	struct mtx	xp_lock;
 +	struct sx	xp_lock;
  	struct __rpc_svcpool *xp_pool;  /* owning pool (see below) */
  	TAILQ_ENTRY(__rpc_svcxprt) xp_link;
  	TAILQ_ENTRY(__rpc_svcxprt) xp_alink;
 @@ -332,6 +333,7 @@ __END_DECLS
  __BEGIN_DECLS
  extern void	xprt_active(SVCXPRT *);
  extern void	xprt_inactive(SVCXPRT *);
 +extern void	xprt_inactive_locked(SVCXPRT *);
  __END_DECLS
  
  #endif
 
 Modified: stable/7/sys/rpc/svc_dg.c
 ==============================================================================
 --- stable/7/sys/rpc/svc_dg.c	Fri May 15 13:26:54 2009	(r192141)
 +++ stable/7/sys/rpc/svc_dg.c	Fri May 15 13:58:45 2009	(r192142)
 @@ -53,6 +53,7 @@ __FBSDID("$FreeBSD$");
  #include <sys/queue.h>
  #include <sys/socket.h>
  #include <sys/socketvar.h>
 +#include <sys/sx.h>
  #include <sys/systm.h>
  #include <sys/uio.h>
  
 @@ -118,7 +119,7 @@ svc_dg_create(SVCPOOL *pool, struct sock
  
  	xprt = mem_alloc(sizeof (SVCXPRT));
  	memset(xprt, 0, sizeof (SVCXPRT));
 -	mtx_init(&xprt->xp_lock, "xprt->xp_lock", NULL, MTX_DEF);
 +	sx_init(&xprt->xp_lock, "xprt->xp_lock");
  	xprt->xp_pool = pool;
  	xprt->xp_socket = so;
  	xprt->xp_p1 = NULL;
 @@ -161,6 +162,9 @@ static enum xprt_stat
  svc_dg_stat(SVCXPRT *xprt)
  {
  
 +	if (soreadable(xprt->xp_socket))
 +		return (XPRT_MOREREQS);
 +
  	return (XPRT_IDLE);
  }
  
 @@ -173,22 +177,17 @@ svc_dg_recv(SVCXPRT *xprt, struct rpc_ms
  	int error, rcvflag;
  
  	/*
 +	 * Serialise access to the socket.
 +	 */
 +	sx_xlock(&xprt->xp_lock);
 +
 +	/*
  	 * The socket upcall calls xprt_active() which will eventually
  	 * cause the server to call us here. We attempt to read a
  	 * packet from the socket and process it. If the read fails,
  	 * we have drained all pending requests so we call
  	 * xprt_inactive().
 -	 *
 -	 * The lock protects us in the case where a new packet arrives
 -	 * on the socket after our call to soreceive fails with
 -	 * EWOULDBLOCK - the call to xprt_active() in the upcall will
 -	 * happen only after our call to xprt_inactive() which ensures
 -	 * that we will remain active. It might be possible to use
 -	 * SOCKBUF_LOCK for this - its not clear to me what locks are
 -	 * held during the upcall.
  	 */
 -	mtx_lock(&xprt->xp_lock);
 -
  	uio.uio_resid = 1000000000;
  	uio.uio_td = curthread;
  	mreq = NULL;
 @@ -196,8 +195,19 @@ svc_dg_recv(SVCXPRT *xprt, struct rpc_ms
  	error = soreceive(xprt->xp_socket, &raddr, &uio, &mreq, NULL, &rcvflag);
  
  	if (error == EWOULDBLOCK) {
 -		xprt_inactive(xprt);
 -		mtx_unlock(&xprt->xp_lock);
 +		/*
 +		 * We must re-test for readability after taking the
 +		 * lock to protect us in the case where a new packet
 +		 * arrives on the socket after our call to soreceive
 +		 * fails with EWOULDBLOCK. The pool lock protects us
 +		 * from racing the upcall after our soreadable() call
 +		 * returns false.
 +		 */
 +		mtx_lock(&xprt->xp_pool->sp_lock);
 +		if (!soreadable(xprt->xp_socket))
 +			xprt_inactive_locked(xprt);
 +		mtx_unlock(&xprt->xp_pool->sp_lock);
 +		sx_xunlock(&xprt->xp_lock);
  		return (FALSE);
  	}
  
 @@ -208,11 +218,11 @@ svc_dg_recv(SVCXPRT *xprt, struct rpc_ms
  		xprt->xp_socket->so_rcv.sb_flags &= ~SB_UPCALL;
  		SOCKBUF_UNLOCK(&xprt->xp_socket->so_rcv);
  		xprt_inactive(xprt);
 -		mtx_unlock(&xprt->xp_lock);
 +		sx_xunlock(&xprt->xp_lock);
  		return (FALSE);
  	}
  
 -	mtx_unlock(&xprt->xp_lock);
 +	sx_xunlock(&xprt->xp_lock);
  
  	KASSERT(raddr->sa_len < xprt->xp_rtaddr.maxlen,
  	    ("Unexpected remote address length"));
 @@ -301,7 +311,7 @@ svc_dg_destroy(SVCXPRT *xprt)
  
  	xprt_unregister(xprt);
  
 -	mtx_destroy(&xprt->xp_lock);
 +	sx_destroy(&xprt->xp_lock);
  	if (xprt->xp_socket)
  		(void)soclose(xprt->xp_socket);
  
 @@ -328,7 +338,5 @@ svc_dg_soupcall(struct socket *so, void 
  {
  	SVCXPRT *xprt = (SVCXPRT *) arg;
  
 -	mtx_lock(&xprt->xp_lock);
  	xprt_active(xprt);
 -	mtx_unlock(&xprt->xp_lock);
  }
 
 Modified: stable/7/sys/rpc/svc_vc.c
 ==============================================================================
 --- stable/7/sys/rpc/svc_vc.c	Fri May 15 13:26:54 2009	(r192141)
 +++ stable/7/sys/rpc/svc_vc.c	Fri May 15 13:58:45 2009	(r192142)
 @@ -54,6 +54,7 @@ __FBSDID("$FreeBSD$");
  #include <sys/queue.h>
  #include <sys/socket.h>
  #include <sys/socketvar.h>
 +#include <sys/sx.h>
  #include <sys/systm.h>
  #include <sys/uio.h>
  #include <netinet/tcp.h>
 @@ -142,7 +143,7 @@ svc_vc_create(SVCPOOL *pool, struct sock
  	}
  
  	xprt = mem_alloc(sizeof(SVCXPRT));
 -	mtx_init(&xprt->xp_lock, "xprt->xp_lock", NULL, MTX_DEF);
 +	sx_init(&xprt->xp_lock, "xprt->xp_lock");
  	xprt->xp_pool = pool;
  	xprt->xp_socket = so;
  	xprt->xp_p1 = NULL;
 @@ -219,7 +220,7 @@ svc_vc_create_conn(SVCPOOL *pool, struct
  	cd->strm_stat = XPRT_IDLE;
  
  	xprt = mem_alloc(sizeof(SVCXPRT));
 -	mtx_init(&xprt->xp_lock, "xprt->xp_lock", NULL, MTX_DEF);
 +	sx_init(&xprt->xp_lock, "xprt->xp_lock");
  	xprt->xp_pool = pool;
  	xprt->xp_socket = so;
  	xprt->xp_p1 = cd;
 @@ -255,9 +256,9 @@ svc_vc_create_conn(SVCPOOL *pool, struct
  	 * Throw the transport into the active list in case it already
  	 * has some data buffered.
  	 */
 -	mtx_lock(&xprt->xp_lock);
 +	sx_xlock(&xprt->xp_lock);
  	xprt_active(xprt);
 -	mtx_unlock(&xprt->xp_lock);
 +	sx_xunlock(&xprt->xp_lock);
  
  	return (xprt);
  cleanup_svc_vc_create:
 @@ -347,22 +348,27 @@ svc_vc_rendezvous_recv(SVCXPRT *xprt, st
  	 * connection from the socket and turn it into a new
  	 * transport. If the accept fails, we have drained all pending
  	 * connections so we call xprt_inactive().
 -	 *
 -	 * The lock protects us in the case where a new connection arrives
 -	 * on the socket after our call to accept fails with
 -	 * EWOULDBLOCK - the call to xprt_active() in the upcall will
 -	 * happen only after our call to xprt_inactive() which ensures
 -	 * that we will remain active. It might be possible to use
 -	 * SOCKBUF_LOCK for this - its not clear to me what locks are
 -	 * held during the upcall.
  	 */
 -	mtx_lock(&xprt->xp_lock);
 +	sx_xlock(&xprt->xp_lock);
  
  	error = svc_vc_accept(xprt->xp_socket, &so);
  
  	if (error == EWOULDBLOCK) {
 -		xprt_inactive(xprt);
 -		mtx_unlock(&xprt->xp_lock);
 +		/*
 +		 * We must re-test for new connections after taking
 +		 * the lock to protect us in the case where a new
 +		 * connection arrives after our call to accept fails
 +		 * with EWOULDBLOCK. The pool lock protects us from
 +		 * racing the upcall after our TAILQ_EMPTY() call
 +		 * returns false.
 +		 */
 +		ACCEPT_LOCK();
 +		mtx_lock(&xprt->xp_pool->sp_lock);
 +		if (TAILQ_EMPTY(&xprt->xp_socket->so_comp))
 +			xprt_inactive_locked(xprt);
 +		mtx_unlock(&xprt->xp_pool->sp_lock);
 +		ACCEPT_UNLOCK();
 +		sx_xunlock(&xprt->xp_lock);
  		return (FALSE);
  	}
  
 @@ -373,11 +379,11 @@ svc_vc_rendezvous_recv(SVCXPRT *xprt, st
  		xprt->xp_socket->so_rcv.sb_flags &= ~SB_UPCALL;
  		SOCKBUF_UNLOCK(&xprt->xp_socket->so_rcv);
  		xprt_inactive(xprt);
 -		mtx_unlock(&xprt->xp_lock);
 +		sx_xunlock(&xprt->xp_lock);
  		return (FALSE);
  	}
  
 -	mtx_unlock(&xprt->xp_lock);
 +	sx_xunlock(&xprt->xp_lock);
  
  	sa = 0;
  	error = soaccept(so, &sa);
 @@ -422,7 +428,7 @@ svc_vc_destroy_common(SVCXPRT *xprt)
  
  	xprt_unregister(xprt);
  
 -	mtx_destroy(&xprt->xp_lock);
 +	sx_destroy(&xprt->xp_lock);
  	if (xprt->xp_socket)
  		(void)soclose(xprt->xp_socket);
  
 @@ -483,21 +489,29 @@ svc_vc_stat(SVCXPRT *xprt)
  
  	/*
  	 * Return XPRT_MOREREQS if we have buffered data and we are
 -	 * mid-record or if we have enough data for a record marker.
 +	 * mid-record or if we have enough data for a record
 +	 * marker. Since this is only a hint, we read mpending and
 +	 * resid outside the lock. We do need to take the lock if we
 +	 * have to traverse the mbuf chain.
  	 */
  	if (cd->mpending) {
  		if (cd->resid)
  			return (XPRT_MOREREQS);
  		n = 0;
 +		sx_xlock(&xprt->xp_lock);
  		m = cd->mpending;
  		while (m && n < sizeof(uint32_t)) {
  			n += m->m_len;
  			m = m->m_next;
  		}
 +		sx_xunlock(&xprt->xp_lock);
  		if (n >= sizeof(uint32_t))
  			return (XPRT_MOREREQS);
  	}
  
 +	if (soreadable(xprt->xp_socket))
 +		return (XPRT_MOREREQS);
 +
  	return (XPRT_IDLE);
  }
  
 @@ -509,6 +523,12 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  	struct mbuf *m;
  	int error, rcvflag;
  
 +	/*
 +	 * Serialise access to the socket and our own record parsing
 +	 * state.
 +	 */
 +	sx_xlock(&xprt->xp_lock);
 +
  	for (;;) {
  		/*
  		 * If we have an mbuf chain in cd->mpending, try to parse a
 @@ -584,6 +604,7 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  				 */
  				xdrmbuf_create(&xprt->xp_xdrreq, cd->mreq, XDR_DECODE);
  				cd->mreq = NULL;
 +				sx_xunlock(&xprt->xp_lock);
  				if (! xdr_callmsg(&xprt->xp_xdrreq, msg)) {
  					XDR_DESTROY(&xprt->xp_xdrreq);
  					return (FALSE);
 @@ -602,17 +623,7 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  		 * the result in cd->mpending. If the read fails,
  		 * we have drained both cd->mpending and the socket so
  		 * we can call xprt_inactive().
 -		 *
 -		 * The lock protects us in the case where a new packet arrives
 -		 * on the socket after our call to soreceive fails with
 -		 * EWOULDBLOCK - the call to xprt_active() in the upcall will
 -		 * happen only after our call to xprt_inactive() which ensures
 -		 * that we will remain active. It might be possible to use
 -		 * SOCKBUF_LOCK for this - its not clear to me what locks are
 -		 * held during the upcall.
  		 */
 -		mtx_lock(&xprt->xp_lock);
 -
  		uio.uio_resid = 1000000000;
  		uio.uio_td = curthread;
  		m = NULL;
 @@ -621,8 +632,20 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  		    &rcvflag);
  
  		if (error == EWOULDBLOCK) {
 -			xprt_inactive(xprt);
 -			mtx_unlock(&xprt->xp_lock);
 +			/*
 +			 * We must re-test for readability after
 +			 * taking the lock to protect us in the case
 +			 * where a new packet arrives on the socket
 +			 * after our call to soreceive fails with
 +			 * EWOULDBLOCK. The pool lock protects us from
 +			 * racing the upcall after our soreadable()
 +			 * call returns false.
 +			 */
 +			mtx_lock(&xprt->xp_pool->sp_lock);
 +			if (!soreadable(xprt->xp_socket))
 +				xprt_inactive_locked(xprt);
 +			mtx_unlock(&xprt->xp_pool->sp_lock);
 +			sx_xunlock(&xprt->xp_lock);
  			return (FALSE);
  		}
  
 @@ -634,7 +657,7 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  			SOCKBUF_UNLOCK(&xprt->xp_socket->so_rcv);
  			xprt_inactive(xprt);
  			cd->strm_stat = XPRT_DIED;
 -			mtx_unlock(&xprt->xp_lock);
 +			sx_xunlock(&xprt->xp_lock);
  			return (FALSE);
  		}
  
 @@ -642,8 +665,9 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  			/*
  			 * EOF - the other end has closed the socket.
  			 */
 +			xprt_inactive(xprt);
  			cd->strm_stat = XPRT_DIED;
 -			mtx_unlock(&xprt->xp_lock);
 +			sx_xunlock(&xprt->xp_lock);
  			return (FALSE);
  		}
  
 @@ -651,8 +675,6 @@ svc_vc_recv(SVCXPRT *xprt, struct rpc_ms
  			m_last(cd->mpending)->m_next = m;
  		else
  			cd->mpending = m;
 -
 -		mtx_unlock(&xprt->xp_lock);
  	}
  }
  
 @@ -739,9 +761,7 @@ svc_vc_soupcall(struct socket *so, void 
  {
  	SVCXPRT *xprt = (SVCXPRT *) arg;
  
 -	mtx_lock(&xprt->xp_lock);
  	xprt_active(xprt);
 -	mtx_unlock(&xprt->xp_lock);
  }
  
  #if 0
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: Burt Rosenberg <burt@cs.miami.edu>
To: bug-followup@FreeBSD.org, Joe Marcus Clarke <marcus@marcuscom.com>
Cc: bvowk@math.ualberta.ca
Subject: Re: kern/130628: [nfs] NFS / rpc.lockd deadlock on 7.1-R
Date: Thu, 3 Sep 2009 14:40:24 -0400

 --000e0cd518fc7a0bcd0472b0b7ce
 Content-Type: multipart/alternative; boundary=000e0cd518fc7a0bbf0472b0b7cc
 
 --000e0cd518fc7a0bbf0472b0b7cc
 Content-Type: text/plain; charset=ISO-8859-1
 
 It seems that :
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/130628
 
 appears in 7.2-R-p3; With this kernel, against Fedora 8 distros:
 
 Linux prism09.cs.miami.edu 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 18:59:49 EST
 2008 x86_64 x86_64 x86_64 GNU/Linux
 
 which are using NFS (tcp) to mount homedirs form the freebsd server to the
 fedora client,
 server will become unresponsive from the network during graphical login of a
 client.
 
 Applying the patch given in the article
 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/130628 seems at present to
 fix the problem. Under a 7.2-R-p3, we can manifest the problem in a few
 minutes, and under said kernel with patches as described in the article, and
 as provided by diffs against the current source, we have not yet seen the
 problem.
 
 When the problem appears, the sever cannot be pinged, an other network
 connections are halted.
 
 On the server, for instance, top shows:
 
 Proc, state, pri
 --------------------
 pc.lockd   *tcpin   -68
 nfsd          -       4
 rpcbind     select   44
 ntpd        select   44
 nfsd        select   44
 ... etc...
 
 
 Also,
 
 ./lockd restart
 Stopping lockd.
 Waiting for PIDS: 1114, 1114, 1114, 1114,....
 
 kill -9 1114 also ineffective.
 
 So it seems to be something spinning in lockd.
 
 I think this is a serious issue and would like to see it resolved. Our setup
 is available if you would like to send instrumented code. I attach diffs.
 
 --000e0cd518fc7a0bbf0472b0b7cc
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 It seems that :<br>=A0<br> <a href=3D"http://www.freebsd.org/cgi/query-pr.c=
 gi?pr=3Dkern/130628">http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/1306=
 28</a><br><br>appears in 7.2-R-p3; With this kernel, against Fedora 8 distr=
 os:<br>
 <br>Linux <a href=3D"http://prism09.cs.miami.edu/" target=3D"_blank">prism0=
 9.cs.miami.edu</a> 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 18:59:49 EST 2008 x86_=
 64 x86_64 x86_64 GNU/Linux<br><br>which are using NFS (tcp) to mount homedi=
 rs form the freebsd server to the fedora client, <br>
 server will become unresponsive from the network during graphical login of =
 a client.<br><br>Applying the patch given in the article <a href=3D"http://=
 www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/130628">http://www.freebsd.org/c=
 gi/query-pr.cgi?pr=3Dkern/130628</a> seems at present to fix the problem. U=
 nder a 7.2-R-p3, we can manifest the problem in a few minutes, and under sa=
 id kernel with patches as described in the article, and as provided by diff=
 s against the current source, we have not yet seen the problem.<br>
 <br>When the problem appears, the sever cannot be pinged, an other network =
 connections are halted. <br><br>On the server, for instance, top shows:<br>=
 <br style=3D"font-family: courier new,monospace;"><span style=3D"font-famil=
 y: courier new,monospace;">Proc, state, pri</span><br style=3D"font-family:=
  courier new,monospace;">
 <span style=3D"font-family: courier new,monospace;">--------------------</s=
 pan><br style=3D"font-family: courier new,monospace;"><span style=3D"font-f=
 amily: courier new,monospace;">pc.lockd=A0=A0 *tcpin=A0=A0 -68 </span><br s=
 tyle=3D"font-family: courier new,monospace;">
 <span style=3D"font-family: courier new,monospace;">nfsd=A0=A0=A0=A0=A0=A0=
 =A0=A0=A0 -=A0=A0=A0=A0=A0=A0 4</span><br style=3D"font-family: courier new=
 ,monospace;"><span style=3D"font-family: courier new,monospace;">rpcbind=A0=
 =A0=A0=A0 select=A0=A0 44</span><br style=3D"font-family: courier new,monos=
 pace;">
 <span style=3D"font-family: courier new,monospace;">ntpd=A0=A0=A0=A0=A0=A0=
 =A0 select=A0=A0 44</span><br style=3D"font-family: courier new,monospace;"=
 ><span style=3D"font-family: courier new,monospace;">nfsd=A0=A0=A0=A0=A0=A0=
 =A0 select=A0=A0 44</span><br style=3D"font-family: courier new,monospace;"=
 >
 <span style=3D"font-family: courier new,monospace;">... etc...</span><br><b=
 r><br>Also,<br><br><span style=3D"font-family: courier new,monospace;">./lo=
 ckd restart</span><br style=3D"font-family: courier new,monospace;"><span s=
 tyle=3D"font-family: courier new,monospace;">Stopping lockd.</span><br styl=
 e=3D"font-family: courier new,monospace;">
 <span style=3D"font-family: courier new,monospace;">Waiting for PIDS: 1114,=
  1114, 1114, 1114,....</span><br style=3D"font-family: courier new,monospac=
 e;"><br>kill -9 1114 also ineffective.<br><br>So it seems to be something s=
 pinning in lockd.<br>
 <br>I think this is a serious issue and would like to see it resolved. Our =
 setup is available if you would like to send instrumented code. I attach di=
 ffs.<br><br><br><br>
 
 --000e0cd518fc7a0bbf0472b0b7cc--
 --000e0cd518fc7a0bcd0472b0b7ce
 Content-Type: application/octet-stream; name="svc.c.diff"
 Content-Disposition: attachment; filename="svc.c.diff"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_fz5u5alz0
 
 MTgxYzE4MQo8IHhwcnRfaW5hY3RpdmUoU1ZDWFBSVCAqeHBydCkKLS0tCj4geHBydF9pbmFjdGl2
 ZV9sb2NrZWQoU1ZDWFBSVCAqeHBydCkKMTg1LDE4NmQxODQKPCAJbXR4X2xvY2soJnBvb2wtPnNw
 X2xvY2spOwo8IAoxOTFjMTg5CjwgCXdha2V1cCgmcG9vbC0+c3BfYWN0aXZlKTsKLS0tCj4gfQox
 OTJhMTkxLDE5Nwo+IHZvaWQKPiB4cHJ0X2luYWN0aXZlKFNWQ1hQUlQgKnhwcnQpCj4gewo+IAlT
 VkNQT09MICpwb29sID0geHBydC0+eHBfcG9vbDsKPiAKPiAJbXR4X2xvY2soJnBvb2wtPnNwX2xv
 Y2spOwo+IAl4cHJ0X2luYWN0aXZlX2xvY2tlZCh4cHJ0KTsK
 --000e0cd518fc7a0bcd0472b0b7ce
 Content-Type: application/octet-stream; name="svc.h.diff"
 Content-Disposition: attachment; filename="svc.h.diff"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_fz5u5am51
 
 NDlhNTAKPiAjaW5jbHVkZSA8c3lzL19zeC5oPgoxMzFjMTMyCjwgCXN0cnVjdCBtdHgJeHBfbG9j
 azsKLS0tCj4gCXN0cnVjdCBzeCAgICAgICB4cF9sb2NrOwozMzRhMzM2Cj4gZXh0ZXJuIHZvaWQg
 ICAgIHhwcnRfaW5hY3RpdmVfbG9ja2VkKFNWQ1hQUlQgKik7Cg==
 --000e0cd518fc7a0bcd0472b0b7ce
 Content-Type: application/octet-stream; name="svc_dg.c.diff"
 Content-Disposition: attachment; filename="svc_dg.c.diff"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_fz5u5am92
 
 NTVhNTYKPiAjaW5jbHVkZSA8c3lzL3N4Lmg+CjEyMWMxMjIKPCAJbXR4X2luaXQoJnhwcnQtPnhw
 X2xvY2ssICJ4cHJ0LT54cF9sb2NrIiwgTlVMTCwgTVRYX0RFRik7Ci0tLQo+IAlzeF9pbml0KCZ4
 cHJ0LT54cF9sb2NrLCAieHBydC0+eHBfbG9jayIpOwoxNjNhMTY1LDE2Nwo+IAlpZiAoc29yZWFk
 YWJsZSh4cHJ0LT54cF9zb2NrZXQpKQo+IAkJZXR1cm4gKFhQUlRfTU9SRVJFUVMpOwo+IAoxNzRh
 MTc5LDE4Mwo+IAkvKiAKPiAJICogU2VyaWFsaXNlIGFjY2VzcyB0byB0aGUgc29ja2V0Lgo+IAkg
 Ki8KPiAJc3hfeGxvY2soJnhwcnQtPnhwX2xvY2spOwo+IAoxOTAsMTkxZDE5OAo8IAltdHhfbG9j
 aygmeHBydC0+eHBfbG9jayk7CjwgCjE5OSwyMDBjMjA2LDIxMAo8IAkJeHBydF9pbmFjdGl2ZSh4
 cHJ0KTsKPCAJCW10eF91bmxvY2soJnhwcnQtPnhwX2xvY2spOwotLS0KPiAJCW10eF9sb2NrKCZ4
 cHJ0LT54cF9wb29sLT5zcF9sb2NrKTsKPiAJCWlmICghc29yZWFkYWJsZSh4cHJ0LT54cF9zb2Nr
 ZXQpKQo+IAkJCXhwcnRfaW5hY3RpdmVfbG9ja2VkKHhwcnQpOwo+IAkJbXR4X3VubG9jaygmeHBy
 dC0+eHBfcG9vbC0+c3BfbG9jayk7Cj4gCQlzeF94dW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKMjEx
 YzIyMQo8IAkJbXR4X3VubG9jaygmeHBydC0+eHBfbG9jayk7Ci0tLQo+IAkJc3hfeHVubG9jaygm
 eHBydC0+eHBfbG9jayk7CjIxNWMyMjUKPCAJbXR4X3VubG9jaygmeHBydC0+eHBfbG9jayk7Ci0t
 LQo+IAlzeF94dW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKMzA0YzMxNAo8IAltdHhfZGVzdHJveSgm
 eHBydC0+eHBfbG9jayk7Ci0tLQo+IAlzeF9kZXN0cm95KCZ4cHJ0LT54cF9sb2NrKTsKMzMxZDM0
 MAo8IAltdHhfbG9jaygmeHBydC0+eHBfbG9jayk7CjMzM2QzNDEKPCAJbXR4X3VubG9jaygmeHBy
 dC0+eHBfbG9jayk7Cg==
 --000e0cd518fc7a0bcd0472b0b7ce
 Content-Type: application/octet-stream; name="svc_vc.c.diff"
 Content-Disposition: attachment; filename="svc_vc.c.diff"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_fz5u5amc3
 
 NTZhNTcKPiAjaW5jbHVkZSA8c3lzL3N4Lmg+CjE0NWMxNDYKPCAJbXR4X2luaXQoJnhwcnQtPnhw
 X2xvY2ssICJ4cHJ0LT54cF9sb2NrIiwgTlVMTCwgTVRYX0RFRik7Ci0tLQo+IAlzeF9pbml0KCZ4
 cHJ0LT54cF9sb2NrLCAieHBydC0+eHBfbG9jayIpOwoyMjJjMjIzLDIyNAo8IAltdHhfaW5pdCgm
 eHBydC0+eHBfbG9jaywgInhwcnQtPnhwX2xvY2siLCBOVUxMLCBNVFhfREVGKTsKLS0tCj4gCXN4
 X2luaXQoJnhwcnQtPnhwX2xvY2ssICJ4cHJ0LT54cF9sb2NrIik7Cj4gCjI1OGMyNjAKPCAJbXR4
 X2xvY2soJnhwcnQtPnhwX2xvY2spOwotLS0KPiAJc3hfeGxvY2soJnhwcnQtPnhwX2xvY2spOwoy
 NjBjMjYyLDI2Mwo8IAltdHhfdW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKLS0tCj4gCXN4X3h1bmxv
 Y2soJnhwcnQtPnhwX2xvY2spOwo+IAozNTljMzYyCjwgCW10eF9sb2NrKCZ4cHJ0LT54cF9sb2Nr
 KTsKLS0tCj4gCXN4X3hsb2NrKCZ4cHJ0LT54cF9sb2NrKTsKMzY0LDM2NWMzNjcsMzczCjwgCQl4
 cHJ0X2luYWN0aXZlKHhwcnQpOwo8IAkJbXR4X3VubG9jaygmeHBydC0+eHBfbG9jayk7Ci0tLQo+
 IAkJQUNDRVBUX0xPQ0soKTsKPiAJCW10eF9sb2NrKCZ4cHJ0LT54cF9wb29sLT5zcF9sb2NrKTsK
 PiAJCWlmIChUQUlMUV9FTVBUWSgmeHBydC0+eHBfc29ja2V0LT5zb19jb21wKSkKPiAJCQl4cHJ0
 X2luYWN0aXZlX2xvY2tlZCh4cHJ0KTsKPiAJCW10eF91bmxvY2soJnhwcnQtPnhwX3Bvb2wtPnNw
 X2xvY2spOwo+IAkJQUNDRVBUX1VOTE9DSygpOwo+IAkJc3hfeHVubG9jaygmeHBydC0+eHBfbG9j
 ayk7CjM3NmMzODQKPCAJCW10eF91bmxvY2soJnhwcnQtPnhwX2xvY2spOwotLS0KPiAJCXN4X3h1
 bmxvY2soJnhwcnQtPnhwX2xvY2spOwozODBjMzg4CjwgCW10eF91bmxvY2soJnhwcnQtPnhwX2xv
 Y2spOwotLS0KPiAJc3hfeHVubG9jaygmeHBydC0+eHBfbG9jayk7CjQyNWM0MzMKPCAJbXR4X2Rl
 c3Ryb3koJnhwcnQtPnhwX2xvY2spOwotLS0KPiAJc3hfZGVzdHJveSgmeHBydC0+eHBfbG9jayk7
 CjQ5MWE1MDAKPiAJCXN4X3hsb2NrKCZ4cHJ0LT54cF9sb2NrKTsKNDk2YTUwNgo+IAkJc3hfeHVu
 bG9jaygmeHBydC0+eHBfbG9jayk7CjUwMGE1MTEsNTEzCj4gCWlmIChzb3JlYWRhYmxlKHhwcnQt
 PnhwX3NvY2tldCkpCj4gCQlyZXR1cm4gKFhQUlRfTU9SRVJFUVMpOwo+IAo1MTFhNTI1LDUyNgo+
 IAlzeF94bG9jaygmeHBydC0+eHBfbG9jayk7Cj4gCjU4NmE2MDIKPiAJCQkJc3hfeHVubG9jaygm
 eHBydC0+eHBfbG9jayk7CjYxNGQ2MjkKPCAJCW10eF9sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKNjI0
 LDYyNWM2MzksNjQzCjwgCQkJeHBydF9pbmFjdGl2ZSh4cHJ0KTsKPCAJCQltdHhfdW5sb2NrKCZ4
 cHJ0LT54cF9sb2NrKTsKLS0tCj4gCQkJbXR4X2xvY2soJnhwcnQtPnhwX3Bvb2wtPnNwX2xvY2sp
 Owo+IAkJCWlmICghc29yZWFkYWJsZSh4cHJ0LT54cF9zb2NrZXQpKQo+IAkJCQl4cHJ0X2luYWN0
 aXZlX2xvY2tlZCh4cHJ0KTsKPiAJCQltdHhfdW5sb2NrKCZ4cHJ0LT54cF9wb29sLT5zcF9sb2Nr
 KTsKPiAJCQlzeF94dW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKNjM3YzY1NQo8IAkJCW10eF91bmxv
 Y2soJnhwcnQtPnhwX2xvY2spOwotLS0KPiAJCQlzeF94dW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsK
 NjQ0YTY2Mwo+IAkJCXhwcnRfaW5hY3RpdmUoeHBydCk7CjY0NmM2NjUKPCAJCQltdHhfdW5sb2Nr
 KCZ4cHJ0LT54cF9sb2NrKTsKLS0tCj4gCQkJc3hfeHVubG9jaygmeHBydC0+eHBfbG9jayk7CjY1
 NCw2NTVkNjcyCjwgCjwgCQltdHhfdW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKNzQyZDc1OAo8IAlt
 dHhfbG9jaygmeHBydC0+eHBfbG9jayk7Cjc0NGQ3NTkKPCAJbXR4X3VubG9jaygmeHBydC0+eHBf
 bG9jayk7Cg==
 --000e0cd518fc7a0bcd0472b0b7ce--

From: Burt Rosenberg <burt@cs.miami.edu>
To: bug-followup@freebsd.org, Joe Marcus Clarke <marcus@marcuscom.com>
Cc:  
Subject: Re: kern/130628: [nfs] NFS / rpc.lockd deadlock on 7.1-R
Date: Wed, 14 Oct 2009 10:31:45 -0400

 --000e0cd6c8b6adc3e40475e605f0
 Content-Type: text/plain; charset=ISO-8859-1
 
 The patch which helped, but did not entirely fix the lock is not in 7.2-p4,
 i386.
 
 Furthermore, we now have a deadlock on an NFS mount between a free bsd
 7.2-p3 and a Linux 2.6.18-164.el5 SMP i686 athlon i386,
 
 in this situation there is a  cisco ASA 5220 between linux and freebsd
 boxes, and we run tcp nfs.
 
 
 
 On Thu, Sep 3, 2009 at 2:40 PM, Burt Rosenberg <burt@cs.miami.edu> wrote:
 
 > It seems that :
 >
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/130628
 >
 > appears in 7.2-R-p3; With this kernel, against Fedora 8 distros:
 >
 > Linux prism09.cs.miami.edu 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 18:59:49 EST
 > 2008 x86_64 x86_64 x86_64 GNU/Linux
 >
 > which are using NFS (tcp) to mount homedirs form the freebsd server to the
 > fedora client,
 > server will become unresponsive from the network during graphical login of
 > a client.
 >
 > Applying the patch given in the article
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/130628 seems at present to
 > fix the problem. Under a 7.2-R-p3, we can manifest the problem in a few
 > minutes, and under said kernel with patches as described in the article, and
 > as provided by diffs against the current source, we have not yet seen the
 > problem.
 >
 > When the problem appears, the sever cannot be pinged, an other network
 > connections are halted.
 >
 > On the server, for instance, top shows:
 >
 > Proc, state, pri
 > --------------------
 > pc.lockd   *tcpin   -68
 > nfsd          -       4
 > rpcbind     select   44
 > ntpd        select   44
 > nfsd        select   44
 > ... etc...
 >
 >
 > Also,
 >
 > ./lockd restart
 > Stopping lockd.
 > Waiting for PIDS: 1114, 1114, 1114, 1114,....
 >
 > kill -9 1114 also ineffective.
 >
 > So it seems to be something spinning in lockd.
 >
 > I think this is a serious issue and would like to see it resolved. Our
 > setup is available if you would like to send instrumented code. I attach
 > diffs.
 >
 >
 >
 >
 
 --000e0cd6c8b6adc3e40475e605f0
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 The patch which helped, but did not entirely fix the lock is not in 7.2-p4,=
  i386.<br><br>Furthermore, we now have a deadlock on an NFS mount between a=
  free bsd 7.2-p3 and a Linux  2.6.18-164.el5 SMP i686 athlon i386, <br><br>
 in this situation there is a=A0 cisco ASA 5220 between linux and freebsd bo=
 xes, and we run tcp nfs.<br><br><br><br><div class=3D"gmail_quote">On Thu, =
 Sep 3, 2009 at 2:40 PM, Burt Rosenberg <span dir=3D"ltr">&lt;<a href=3D"mai=
 lto:burt@cs.miami.edu">burt@cs.miami.edu</a>&gt;</span> wrote:<br>
 <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">It seems that :<b=
 r>=A0<br> <a href=3D"http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/1306=
 28" target=3D"_blank">http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/130=
 628</a><br>
 <br>appears in 7.2-R-p3; With this kernel, against Fedora 8 distros:<br>
 <br>Linux <a href=3D"http://prism09.cs.miami.edu/" target=3D"_blank">prism0=
 9.cs.miami.edu</a> 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 18:59:49 EST 2008 x86_=
 64 x86_64 x86_64 GNU/Linux<br><br>which are using NFS (tcp) to mount homedi=
 rs form the freebsd server to the fedora client, <br>
 
 server will become unresponsive from the network during graphical login of =
 a client.<br><br>Applying the patch given in the article <a href=3D"http://=
 www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/130628" target=3D"_blank">http:/=
 /www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/130628</a> seems at present to =
 fix the problem. Under a 7.2-R-p3, we can manifest the problem in a few min=
 utes, and under said kernel with patches as described in the article, and a=
 s provided by diffs against the current source, we have not yet seen the pr=
 oblem.<br>
 
 <br>When the problem appears, the sever cannot be pinged, an other network =
 connections are halted. <br><br>On the server, for instance, top shows:<br>=
 <br style=3D"font-family: courier new,monospace;"><span style=3D"font-famil=
 y: courier new,monospace;">Proc, state, pri</span><br style=3D"font-family:=
  courier new,monospace;">
 
 <span style=3D"font-family: courier new,monospace;">--------------------</s=
 pan><br style=3D"font-family: courier new,monospace;"><span style=3D"font-f=
 amily: courier new,monospace;">pc.lockd=A0=A0 *tcpin=A0=A0 -68 </span><br s=
 tyle=3D"font-family: courier new,monospace;">
 
 <span style=3D"font-family: courier new,monospace;">nfsd=A0=A0=A0=A0=A0=A0=
 =A0=A0=A0 -=A0=A0=A0=A0=A0=A0 4</span><br style=3D"font-family: courier new=
 ,monospace;"><span style=3D"font-family: courier new,monospace;">rpcbind=A0=
 =A0=A0=A0 select=A0=A0 44</span><br style=3D"font-family: courier new,monos=
 pace;">
 
 <span style=3D"font-family: courier new,monospace;">ntpd=A0=A0=A0=A0=A0=A0=
 =A0 select=A0=A0 44</span><br style=3D"font-family: courier new,monospace;"=
 ><span style=3D"font-family: courier new,monospace;">nfsd=A0=A0=A0=A0=A0=A0=
 =A0 select=A0=A0 44</span><br style=3D"font-family: courier new,monospace;"=
 >
 
 <span style=3D"font-family: courier new,monospace;">... etc...</span><br><b=
 r><br>Also,<br><br><span style=3D"font-family: courier new,monospace;">./lo=
 ckd restart</span><br style=3D"font-family: courier new,monospace;"><span s=
 tyle=3D"font-family: courier new,monospace;">Stopping lockd.</span><br styl=
 e=3D"font-family: courier new,monospace;">
 
 <span style=3D"font-family: courier new,monospace;">Waiting for PIDS: 1114,=
  1114, 1114, 1114,....</span><br style=3D"font-family: courier new,monospac=
 e;"><br>kill -9 1114 also ineffective.<br><br>So it seems to be something s=
 pinning in lockd.<br>
 
 <br>I think this is a serious issue and would like to see it resolved. Our =
 setup is available if you would like to send instrumented code. I attach di=
 ffs.<br><br><br><br>
 </blockquote></div><br>
 
 --000e0cd6c8b6adc3e40475e605f0--

From: Eric Crist <ecrist@secure-computing.net>
To: bug-followup@FreeBSD.org,
 bvowk@math.ualberta.ca
Cc:  
Subject: Re: kern/130628: [nfs] NFS / rpc.lockd deadlock on 7.1-R
Date: Wed, 13 Oct 2010 06:51:55 -0500

 Is there any follow-up to this bug?  We're still seeing this (or a =
 similar issue) in 8.1-RELEASE.=
>Unformatted:
