From eugenea@go2net.com  Wed Feb 21 14:38:44 2001
Return-Path: <eugenea@go2net.com>
Received: from hunches.go2net.com (natbox.go2net.com [209.191.181.146])
	by hub.freebsd.org (Postfix) with ESMTP id AFD0937B4EC
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 21 Feb 2001 14:38:44 -0800 (PST)
	(envelope-from eugenea@go2net.com)
Received: from eugenea by hunches.go2net.com with local-smtp (Exim 3.12 #1 (Debian))
	id 14VhuG-0004XJ-00; Wed, 21 Feb 2001 14:38:44 -0800
Message-Id: <Pine.LNX.3.96.1010221143653.17437A-100000@hunches.go2net.com>
Date: Wed, 21 Feb 2001 14:38:44 -0800 (PST)
From: Yevgeniy Aleynikov <eugenea@go2net.com>
To: FreeBSD-gnats-submit@freebsd.org
Subject: Kernel trap 12 in camisr

>Number:         25264
>Category:       kern
>Synopsis:       Kernel trap 12 in camisr
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 21 14:40:00 PST 2001
>Closed-Date:    Fri Mar 23 21:40:06 PST 2001
>Last-Modified:  Fri Mar 23 21:40:31 PST 2001
>Originator:     Eugene Aleynikov
>Release:        FreeBSD 4.2-RELEASE i386
>Organization:
Go2net
>Environment:

Standard BSDi X-treme server - Intel L440GX+ m/b
dmesg.boot:
Copyright (c) 1992-2000 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.2-RELEASE #5: Tue Feb 20 12:37:46 PST 2001
    user@domain.com:/usr/src/sys/compile/HMV
Timecounter "i8254"  frequency 1193182 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (796.54-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x683  Stepping = 3

Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory  = 536805376 (524224K bytes)
avail memory = 519602176 (507424K bytes)
Programming 24 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
FreeBSD/SMP: Multiprocessor motherboard
 cpu0 (BSP): apic id:  1, version: 0x00040011, at 0xfee00000
 cpu1 (AP):  apic id:  0, version: 0x00040011, at 0xfee00000
 io0 (APIC): apic id:  2, version: 0x00170011, at 0xfec00000
Preloaded elf kernel "kernel" at 0xc02ab000.
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82443GX host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pcib2: <Intel 82443GX (440 GX) PCI-PCI (AGP) bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib2
pcib3: <PCI to PCI bridge (vendor=1011 device=0023)> at device 15.0 on
pci1
pci2: <PCI bus> on pcib3
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x3000-0x303f mem
0xf4200000-0xf42f
ffff,0xf4300000-0xf4300fff irq 20 at device 4.0 on pci2
fxp0: Ethernet address 00:02:b3:08:76:45
ahc0: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0x2000-0x20ff mem
0xf4100000
-0xf4100fff irq 19 at device 12.0 on pci0
aic7896/97: Wide Channel A, SCSI Id=7, 32/255 SCBs
ahc1: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0x2400-0x24ff mem
0xf4101000
-0xf4101fff irq 19 at device 12.1 on pci0
aic7896/97: Wide Channel B, SCSI Id=7, 32/255 SCBs
fxp1: <Intel Pro 10/100B/100+ Ethernet> port 0x2800-0x283f mem
0xf4000000-0xf40f
ffff,0xf4102000-0xf4102fff irq 21 at device 14.0 on pci0
fxp1: Ethernet address 00:d0:b7:a9:c4:c6
isab0: <Intel 82371AB PCI to ISA bridge> at device 18.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0x2860-0x286f at device 18.1
on pci
0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <Intel 82371AB/EB (PIIX4) USB controller> at 18.2 irq 21
Timecounter "PIIX"  frequency 3579545 Hz
chip1: <Intel 82371AB Power management controller> port 0x1040-0x104f at
device 
18.3 on pci0
pci0: <Cirrus Logic GD5480 SVGA controller> at 20.0
pcib1: <Intel 82443GX host to AGP bridge> on motherboard
pci3: <PCI bus> on pcib1
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> on isa0
sc0: VGA <16 virtual consoles, flags=0x0>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
IP packet filtering initialized, divert disabled, rule-based forwarding
enabled,
 default to accept, logging limited to 100 packets/entry by default
DUMMYNET initialized (000608)
Waiting 15 seconds for SCSI devices to settle
SMP: AP CPU #1 Launched!
sa0 at ahc1 bus 0 target 3 lun 0
sa0: <HP C1537A L005> Removable Sequential Access SCSI-2 device 
sa0: 10.000MB/s transfers (10.000MHz, offset 32)
da1 at ahc0 bus 0 target 2 lun 0
da1: <WSI FLASHDISK 0223> Fixed Direct Access SCSI-2 device 
da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing
Enabled
da1: 35003MB (71686272 512 byte sectors: 255H 63S/T 4462C)
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST39204LC 0006> Fixed Direct Access SCSI-3 device 
da0: 80.000MB/s transfers (40.000MHz, offset 63, 16bit), Tagged Queueing
Enabled
da0: 8750MB (17921835 512 byte sectors: 255H 63S/T 1115C)
Mounting root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted

------------
Kernel config:
makeoptions     DEBUG=-g                #Build kernel with gdb(1) debug
symbols

machine         i386
#cpu            I586_CPU
cpu             I686_CPU
ident           "HMV"
# This may be set way to high.  It might we worth trying it lower if we
# are allocating too much of our resources for the kernel.
maxusers        256

options         INET                    #InterNETworking
options         FFS                     #Berkeley Fast Filesystem
options         FFS_ROOT                #FFS usable as root device [keep
this!]
# following available as modules
#options        MFS                     #Memory Filesystem
#options        UNION                   #Union filesystem
#options        MFS_ROOT                #MFS usable as root device, "MFS"
req'ed
#options        NFS                     #Network Filesystem
#options        NFS_ROOT                #NFS usable as root device, "NFS"
req'ed
#options        MSDOSFS                 #MSDOS Filesystem
#options        "CD9660"                #ISO 9660 Filesystem
options         PROCFS                  #Process filesystem
options         "COMPAT_43"             #Compatible with BSD 4.3 [KEEP
THIS!]
options         SCSI_DELAY=15000        #Be pessimistic about Joe SCSI
device
options         UCONSOLE                #Allow users to grab the console
options         USERCONFIG              #boot -c editor
options         VISUAL_USERCONFIG       #visual boot -c editor
options         KTRACE                  #ktrace(1) syscall trace support
options         SYSVSHM                 #SYSV-style shared memory
options         SYSVMSG                 #SYSV-style message queues
options         SYSVSEM                 #SYSV-style semaphores
options         P1003_1B                #Posix P1003_1B real-time
extensions
options         _KPOSIX_PRIORITY_SCHEDULING
#options                ICMP_BANDLIM            #Rate limit bad replies
options         PERFMON

# To make an SMP kernel, the next two are needed
options         SMP                     # Symmetric MultiProcessor Kernel
options         APIC_IO                 # Symmetric (APIC) I/O

device          isa
device          pci

# Floppy drives
device          fdc0    at isa? port IO_FD1 irq 6 drq 2
device          fd0     at fdc0 drive 0
device          fd1     at fdc0 drive 1

# ATA and ATAPI devices
device          ata0    at isa? port IO_WD1 irq 14
device          ata1    at isa? port IO_WD2 irq 15
device          ata
#device         atadisk                 # ATA disk drives
device          atapicd                 # ATAPI CDROM drives
device          atapifd                 # ATAPI floppy drives
device          atapist                 # ATAPI tape drives
options         ATA_STATIC_ID           #Static device numbering
#options        ATA_ENABLE_ATAPI_DMA    #Enable DMA on ATAPI devices

# SCSI Controllers
# A single entry for any of these devices (ncr, ahb, ahc) is
# sufficient for any number of installed devices.
device  ahc             # AHA2940 and onboard AIC7xxx devices
device  bt0     at isa?

# SCSI peripherals
# Only one of each of these is needed, they are dynamically allocated.
device  scbus           # SCSI bus (required)
device          da              # Direct Access (disks)
device          sa              # Sequential Access (tape etc)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI)

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc0 at isa? port IO_KBD
device          atkbd0  at atkbdc? irq 1
# flags 0x1
device          psm0    at atkbdc? irq 12

device          vga0    at isa?

# splash screen/screen saver
#pseudo-device  splash

# syscons is the default console driver, resembling an SCO console
#device         sc0     at isa? flags 0x100
device          sc0     at isa?
options         SC_DISABLE_DDBKEY       # disable `debug' key
options         SC_DISABLE_REBOOT       # disable reboot key sequence
#options        SC_MOUSE_CHAR=0x3       # char code for text mode mouse
cursor
options         SC_NO_CUTPASTE
options         SC_NO_FONT_LOADING
options         SC_NO_SYSMOUSE

# Floating point support - do not disable.
device          npx0    at nexus? port IO_NPX irq 13

# Serial (COM) ports
device          sio0    at isa? port IO_COM1 flags 0x10 irq 4
device          sio1    at isa? port IO_COM2 irq 3
#device         sio2    at isa? disable port IO_COM3 tty irq 5
#device         sio3    at isa? disable port IO_COM4 tty irq 9

# Parallel port
device          ppc0    at isa? irq 7
device          ppbus           # Parallel port bus (required)
#device         lpt             # Printer
#device         plip            # TCP/IP over parallel
#device         ppi             # Parallel port interface device
#device         vpo             # Requires scbus and da

# PCI Ethernet NICs.
device          fxp             # Intel EtherExpress PRO/100B (82557,
82558)

# Pseudo devices - the number indicates how many units to allocated.
pseudo-device   loop            # Network loopback
pseudo-device   ether           # Ethernet support
# Jack this up just in case we have lots of automated stuff doing work.
pseudo-device   pty     # Pseudo-ttys (telnet etc)
pseudo-device   gzip            # Exec gzipped a.out's

# The `bpf' pseudo-device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# The number of devices determines the maximum number of
# simultaneous BPF clients programs runnable.
pseudo-device   bpf     #Berkeley packet filter

# We need to service many network requests.  This should probably be tuned
# better to more closely fit our needs.
options                 NMBCLUSTERS=12288

# Set the number of PV entries per process.  Increasing this can
# stop panics related to heavy use of shared memory. However, that can
# (combined with large amounts of physical memory) cause panics at
# boot time due the kernel running out of VM space.
#
# If you're tweaking this, you might also want to increase the sysctls
# "vm.v_free_min", "vm.v_free_reserved", and "vm.v_free_target".
#
# The value below is the one more than the default.
#
#options        PMAP_SHPGPERPROC=201
options         PMAP_SHPGPERPROC=401


options         QUOTA
options         IPFIREWALL              #firewall
options         IPFIREWALL_VERBOSE      #print information about
options         IPFIREWALL_FORWARD      #enable transparent proxy support
options         IPFIREWALL_VERBOSE_LIMIT=100    #limit verbosity
options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by
default
options         DUMMYNET

#kldstat
Id Refs Address    Size     Name
 1    2 0xc0100000 1a9ac0   kernel
 2    1 0xc35b5000 4000     null.ko
(there's one nullfs mount)

>Description:

Several times a day kernel crashes with following diag.
Cant find coredump because SCSI system is not functioning during crash.

login: panic: lockmgr: pid 64908, not exclusive lock holder 380 unlocking
mp_lock = 00000001; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0

syncing disks... 203 204 204 204 204 204 204 204 204 204 204 204 204 204
204 204
 204 204 204 204 
giving up on 202 buffers
Uptime: 15h47m27s
(da1:ahc0:0:2:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 
(da1:ahc0:0:2:0): ILLEGAL REQUEST asc:20,0
(da1:ahc0:0:2:0): Invalid command operation code

dumping to dev #da/0x20001, offset 1048704
dump 

Fatal trap 12: page fault while in kernel mode
mp_lock = 00000002; cpuid = 0; lapic.id = 01000000
fault virtual address   = 0x0
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xc0121968
stack pointer           = 0x10:0xdbc0095c
frame pointer           = 0x10:0xdbc0096c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 64908 (httpd)
interrupt mask          = net tty bio cam  <- SMP: XXX
trap number             = 12
panic: page fault
mp_lock = 00000002; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 15h47m28s


Fatal trap 12: page fault while in kernel mode
mp_lock = 00000003; cpuid = 0; lapic.id = 01000000
fault virtual address   = 0x10
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc0121a84
stack pointer           = 0x10:0xdbc00578
frame pointer           = 0x10:0xdbc0058c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 64908 (httpd)
interrupt mask          = net tty bio cam  <- SMP: XXX
trap number             = 12
panic: page fault
mp_lock = 00000003; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 15h47m28s


Fatal trap 12: page fault while in kernel mode
mp_lock = 00000004; cpuid = 0; lapic.id = 01000000
fault virtual address   = 0x10
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc0121a84
stack pointer           = 0x10:0xdbc00194
frame pointer           = 0x10:0xdbc001a8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 64908 (httpd)
interrupt mask          = net tty bio cam  <- SMP: XXX
trap number             = 12
panic: page fault
mp_lock = 00000004; cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 15h47m28s

--------skipped several traps with the same instruction pointer-----

(kgdb) x/i 0xc0121968
0xc0121968 <camisr+200>:        mov    %eax,(%edx)

(kgdb) list *0xc0121968
0xc0121968 is in camisr (../../cam/cam_queue.h:224).
219
220     static __inline void
221     cam_ccbq_ccb_done(struct cam_ccbq *ccbq, union ccb *done_ccb)
222     {
223             TAILQ_REMOVE(&ccbq->active_ccbs, &done_ccb->ccb_h,
224                          xpt_links.tqe);
225             ccbq->dev_active--;
226             ccbq->dev_openings++;
227             ccbq->held++;
228     }

(kgdb) x/i 0xc0121a84
0xc0121a84 <camisr+484>:        pushl  (%eax)

(kgdb) list *0xc0121a84
0xc0121a84 is in camisr (../../cam/cam_xpt.c:6332).
6327                    } else if (runq) {
6328                            xpt_run_dev_sendq(ccb_h->path->bus);
6329                    }
6330
6331                    /* Call the peripheral driver's callback */
6332                    (*ccb_h->cbfcnp)(ccb_h->path->periph, (union ccb
*)ccb_h);
6333
6334                    /* Raise IPL for while test */
6335                    s = splcam();
6336            }

----------

There's another server also that crashes similar way.
SCSI is terminated propertly. SCSI tape (SE device) is on the second 
SCSI bus alone.

>How-To-Repeat:

    Just keep it running under heavy user load (web+cgi).

>Fix:

None


>Release-Note:
>Audit-Trail:

From: Eugene Aleynikov <eugenea@infospace.com>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/25264: Kernel trap 12 in camisr
Date: Thu, 22 Feb 2001 14:01:02 -0800

 Another trap:
  panic: lockmgr: pid 44687, not exclusive lock holder 374 unlocking
 mp_lock = 01000001; cpuid = 1; lapic.id = 00000000
 boot() called on cpu#1
 
 syncing disks... panic: rslock: cpu: 1, addr: 0xc3d25e00, lock:
 0x01000001
 mp_lock = 01000001; cpuid = 1; lapic.id = 00000000
 boot() called on cpu#1
 Uptime: 3h57m11s
 (da1:ahc0:0:2:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 
 (da1:ahc0:0:2:0): ILLEGAL REQUEST asc:20,0
 (da1:ahc0:0:2:0): Invalid command operation code
 
 dumping to dev #da/0x20001, offset 1048704
 dump 
 
 Fatal trap 12: page fault while in kernel mode
 mp_lock = 01000002; cpuid = 1; lapic.id = 00000000
 
 fault virtual address   = 0x0
 fault code              = supervisor write, page not present
 instruction pointer     = 0x8:0xc0121968
 
 stack pointer           = 0x10:0xdbbc6800
 frame pointer           = 0x10:0xdbbc6810
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 44687 (proftpd)
 interrupt mask          = net tty bio cam  <- SMP: XXX
 trap number             = 12
 panic: page fault
 mp_lock = 01000002; cpuid = 1; lapic.id = 00000000
 boot() called on cpu#1
 Uptime: 3h57m11s
 
 
 Fatal trap 12: page fault while in kernel mode
 mp_lock = 01000003; cpuid = 1; lapic.id = 00000000
 fault virtual address   = 0x10
 fault code              = supervisor read, page not present
 instruction pointer     = 0x8:0xc0121a84
 stack pointer           = 0x10:0xdbbc641c
 frame pointer           = 0x10:0xdbbc6430
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 44687 (proftpd)
 interrupt mask          = net tty bio cam  <- SMP: XXX
 trap number             = 12
 panic: page fault
 mp_lock = 01000003; cpuid = 1; lapic.id = 00000000
 boot() called on cpu#1
 Uptime: 3h57m11s
 
 
 0xc0121968 is in probedone (../../cam/cam_xpt.c:5619).
 5614     path->device->serial_num_len =
 5615         serial_buf->length;
 5616     path->device->serial_num[serial_buf->length]
 5617         = '\0';
 5618    }
 5619   } else if (cam_periph_error(done_ccb, 0,
 5620          SF_RETRY_UA|SF_NO_PRINT,
 5621          &softc->saved_ccb) == ERESTART) {
 5622    return;
 5623   } else if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) != 0) {
 (kgdb) list *0xc0121a84
 0xc0121a84 is in probedone (../../cam/cam_xpt.c:5693).
 5688    /* Don't wedge the queue */
 5689    xpt_release_devq(done_ccb->ccb_h.path, /*count*/1,
 5690       /*run_queue*/TRUE);
 5691   }
 5692
 5693   path->device->flags &= ~CAM_DEV_UNCONFIGURED;
 5694
 5695   if ((softc->flags & PROBE_NO_ANNOUNCE) == 0) {
 5696    /* Inform the XPT that a new device has been found */
 5697    done_ccb->ccb_h.func_code = XPT_GDEV_TYPE;

From: Eugene Aleynikov <eugenea@infospace.com>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/25264: Kernel trap 12 in camisr
Date: Mon, 26 Feb 2001 10:06:52 -0800

 After getting rid of nullfs problem was fixed.
 This is nullfs issue so we can close this problem.
 
 Nullfs status probably should me moved to broken.
State-Changed-From-To: open->closed 
State-Changed-By: kris 
State-Changed-When: Fri Mar 23 21:40:06 PST 2001 
State-Changed-Why:  
Submitter reports problem was actually due to using 
nullfs 

http://www.freebsd.org/cgi/query-pr.cgi?pr=25264 
>Unformatted:
