From nobody@FreeBSD.org  Sat Jan 24 12:11:12 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id C07A810656BF
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 24 Jan 2009 12:11:12 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id AD1298FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 24 Jan 2009 12:11:12 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n0OCBCBv025562
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 24 Jan 2009 12:11:12 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n0OCBCxZ025561;
	Sat, 24 Jan 2009 12:11:12 GMT
	(envelope-from nobody)
Message-Id: <200901241211.n0OCBCxZ025561@www.freebsd.org>
Date: Sat, 24 Jan 2009 12:11:12 GMT
From: Elmar Stellnberger <estellnb@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: unexpected crash due to page fault
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         130941
>Category:       kern
>Synopsis:       [panic] [cd9660] [geom] unexpected crash due to page fault
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    gavin
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Jan 24 12:20:01 UTC 2009
>Closed-Date:    
>Last-Modified:  Sat Mar  7 12:50:03 UTC 2009
>Originator:     Elmar Stellnberger
>Release:        FreeBSD 7.1
>Organization:
>Environment:
FreeBSD scaleo.studiob 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Thu Jan  1 14:37:25 UTC 2009     root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
sudden and unexpected reboot;
/var/log/messages:
Jan 22 19:28:01 scaleo kernel: WARNING: / was not properly dismounted
Jan 22 19:28:01 scaleo kernel: WARNING: /usr was not properly dismounted
Jan 22 19:28:01 scaleo savecore: reboot after panic: page fault
Jan 22 19:28:01 scaleo savecore: writing core to vmcore.0

has now occured two times when working with FreeBSD

p.s.: how can I post a core file bz2 when i submit a bug via the web
form http://www.freebsd.org/send-pr.html (too large for an email as well)?
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: gavin 
State-Changed-When: Sat Jan 24 19:03:37 UTC 2009 
State-Changed-Why:  
To submitter:  Sending the core file (or even providing it for download) is 
not hugely useful, as getting useful information out of it is easiest done 
on the system with the issues.  Can you please follow the instructions at 
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html 
(specifically, obtaining the backtrace) and provide the result? 

Also, please give some details about this system: what are it's main roles 
(desktop, web server, dns server etc), has it been stable with previous 
versins of FreeBSD, and have any hardware changes been made recently, etc? 


Responsible-Changed-From-To: freebsd-bugs->gavin 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Sat Jan 24 19:03:37 UTC 2009 
Responsible-Changed-Why:  
Track 

http://www.freebsd.org/cgi/query-pr.cgi?pr=130941 

From: linimon@lonesome.com (Mark Linimon)
To: Elmar Stellnberger <estellnb@gmail.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/130941: unexpected crash due to page fault
Date: Sat, 24 Jan 2009 13:03:39 -0600

 > p.s.: how can I post a core file bz2 when i submit a bug via the web
 > form http://www.freebsd.org/send-pr.html (too large for an email as well)?
 
 Please post it somewhere on the web and submit a followup with a URL
 to it.  Thanks.
 
 mcl

From: Elmar Stellnberger <estellnb@googlemail.com>
To: gavin@FreeBSD.org, bug-followup@freebsd.org
Cc:  
Subject: Re: kern/130941: [panic] unexpected crash due to page fault
Date: Tue, 27 Jan 2009 21:53:00 +0000

 gdb seems to have problems reading the core file:
 /var/crash/vmcore.0": not in executable format: File format not recognized
 
 Besides this there is no file called kernel.debug on my machine although
 the kernel sources seem to be installed at /usr/src/sys/i386
 
 How can I post a message to this bug without sending an email?
 
 gavin@FreeBSD.org wrote:
 > Synopsis: [panic] unexpected crash due to page fault
 >
 > State-Changed-From-To: open->feedback
 > State-Changed-By: gavin
 > State-Changed-When: Sat Jan 24 19:03:37 UTC 2009
 > State-Changed-Why: 
 > To submitter:  Sending the core file (or even providing it for download) is
 > not hugely useful, as getting useful information out of it is easiest done
 > on the system with the issues.  Can you please follow the instructions at
 > http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html
 > (specifically, obtaining the backtrace) and provide the result?
 >
 > Also, please give some details about this system: what are it's main roles
 > (desktop, web server, dns server etc), has it been stable with previous
 > versins of FreeBSD, and have any hardware changes been made recently, etc?
 >
 >
 > Responsible-Changed-From-To: freebsd-bugs->gavin
 > Responsible-Changed-By: gavin
 > Responsible-Changed-When: Sat Jan 24 19:03:37 UTC 2009
 > Responsible-Changed-Why: 
 > Track
 >
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=130941
 >
 >   
 

From: Gavin Atkinson <gavin@FreeBSD.org>
To: Elmar Stellnberger <estellnb@googlemail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/130941: [panic] unexpected crash due to page fault
Date: Tue, 27 Jan 2009 23:04:17 +0000 (GMT)

 On Tue, 27 Jan 2009, Elmar Stellnberger wrote:
 > gdb seems to have problems reading the core file:
 > /var/crash/vmcore.0": not in executable format: File format not recognized
 
 Can you try "kgdb /boot/kernel/kernel.symbols /var/crash/vmcore.0"
 Or if kernel.symbols doesn't exist,
 "kgdb /boot/kernel/kernel /var/crash/vmcore.0"
 
 > Besides this there is no file called kernel.debug on my machine although
 > the kernel sources seem to be installed at /usr/src/sys/i386
 
 That's expected, if it's not been compiled from source with debugging 
 enabled.
 
 Gavin

From: Elmar Stellnberger <estellnb@googlemail.com>
To: Gavin Atkinson <gavin@FreeBSD.org>, bug-followup@freebsd.org
Cc:  
Subject: Re: kern/130941: [panic] unexpected crash due to page fault
Date: Wed, 28 Jan 2009 15:53:54 +0000

 This is a multi-part message in MIME format.
 --------------090302090707070603070200
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 Gavin Atkinson wrote:
 > On Tue, 27 Jan 2009, Elmar Stellnberger wrote:
 >> gdb seems to have problems reading the core file:
 >> /var/crash/vmcore.0": not in executable format: File format not 
 >> recognized
 >
 > Can you try "kgdb /boot/kernel/kernel.symbols /var/crash/vmcore.0"
 > Or if kernel.symbols doesn't exist,
 > "kgdb /boot/kernel/kernel /var/crash/vmcore.0"
 >
 >> Besides this there is no file called kernel.debug on my machine although
 >> the kernel sources seem to be installed at /usr/src/sys/i386
 >
 > That's expected, if it's not been compiled from source with debugging 
 > enabled.
 >
 > Gavin
 >
 
 
 --------------090302090707070603070200
 Content-Type: text/plain;
  name="backtrace.txt"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline;
  filename="backtrace.txt"
 
  
 (kgdb) backtrace
 #0  doadump () at pcpu.h:196
 #1  0xc0790ea7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
 #2  0xc0791179 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:574
 #3  0xc0aa338c in trap_fatal (frame=0xe7e419e4, eva=851974)
     at /usr/src/sys/i386/i386/trap.c:939
 #4  0xc0aa3610 in trap_pfault (frame=0xe7e419e4, usermode=0, eva=851974)
     at /usr/src/sys/i386/i386/trap.c:852
 #5  0xc0aa3fcc in trap (frame=0xe7e419e4) at /usr/src/sys/i386/i386/trap.c:530
 #6  0xc0a89e3b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
 #7  0xc0737c89 in g_io_request (bp=0xc6005528, cp=0xcac4cd00)
     at /usr/src/sys/geom/geom_io.c:364
 #8  0xc073c286 in g_vfs_strategy (bo=0xc8dbeec4, bp=0xd91e1c14)
     at /usr/src/sys/geom/geom_vfs.c:107
 #9  0xc074cfd0 in cd9660_strategy (ap=0xe7e41aa8)
     at /usr/src/sys/fs/cd9660/cd9660_vnops.c:758
 #10 0xc0ab90dd in VOP_STRATEGY_APV (vop=0xc0bd6100, a=0xe7e41aa8) at vnode_if.c:1771
 #11 0xc07f5b6e in bufstrategy (bo=0xc93582e8, bp=0xd91e1c14) at vnode_if.h:935
 #12 0xc07fca7a in breadn (vp=0xc9358228, blkno=0, size=2048, rablkno=0x0, rabsize=0x0,
     cnt=0, cred=0x0, bpp=0xe7e41b3c) at buf.h:429
 #13 0xc07fcb2c in bread (vp=0xc9358228, blkno=Unhandled dwarf expression opcode 0x93
 ) at /usr/src/sys/kern/vfs_bio.c:734
 #14 0xc0748ee0 in cd9660_blkatoff (vp=0xc9358228, offset=0, res=0x0, bpp=0xe7e41bc0)
     at /usr/src/sys/fs/cd9660/cd9660_lookup.c:407
 #15 0xc074d82d in cd9660_readdir (ap=0xe7e41c00)
     at /usr/src/sys/fs/cd9660/cd9660_vnops.c:516
 #16 0xc0ab8352 in VOP_READDIR_APV (vop=0xc0bd6100, a=0xe7e41c00) at vnode_if.c:1407
 #17 0xc081ab40 in kern_getdirentries (td=0xc6525230, fd=5,
     buf=0x2832c000 <Address 0x2832c000 out of bounds>, count=4096, basep=0xe7e41c74)
     at vnode_if.h:747
 ---Type <return> to continue, or q <return> to quit---
 #18 0xc081ad51 in getdirentries (td=0xc6525230, uap=0xe7e41cfc)
     at /usr/src/sys/kern/vfs_syscalls.c:3767
 #19 0xc0aa3965 in syscall (frame=0xe7e41d38) at /usr/src/sys/i386/i386/trap.c:1090
 #20 0xc0a89ea0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
 #21 0x00000033 in ?? ()
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) list
 255             call    syscall
 256             add     $4, %esp
 257             MEXITCOUNT
 258             jmp     doreti
 259
 260     ENTRY(fork_trampoline)
 261             pushl   %esp                    /* trapframe pointer */
 262             pushl   %ebx                    /* arg1 */
 263             pushl   %esi                    /* function */
 264             call    fork_exit
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #1  0xc0790ea7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
 (kgdb) list -
 408             /*
 409              * Ok, now do things that assume all filesystem activity has
 410              * been completed.
 411              */
 412             EVENTHANDLER_INVOKE(shutdown_post_sync, howto);
 413
 414             /* XXX This doesn't disable interrupts any more.  Reconsider? */
 415             splhigh();
 416
 417             if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping)
 418                     doadump();
 (kgdb) list
 413
 414             /* XXX This doesn't disable interrupts any more.  Reconsider? */
 415             splhigh();
 416
 417             if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping)
 418                     doadump();
 419
 420             /* Now that we're going to really halt the system... */
 421             EVENTHANDLER_INVOKE(shutdown_final, howto);
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #2  0xc0791179 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:574
 574             boot(bootopt);
 (kgdb) list -
 559             /* See if the user aborted the panic, in which case we continue. */
 560             if (panicstr == NULL) {
 561     #ifdef SMP
 562                     atomic_store_rel_int(&panic_cpu, NOCPU);
 563     #endif
 564                     return;
 565             }
 566     #endif
 567     #endif
 568             /*thread_lock(td); */
 (kgdb) list
 569             td->td_flags |= TDF_INPANIC;
 570             /* thread_unlock(td); */
 571             if (!sync_on_panic)
 572                     bootopt |= RB_NOSYNC;
 573             critical_exit();
 574             boot(bootopt);
 575     }
 576
 577     /*
 578      * Support for poweroff delay.
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #3  0xc0aa338c in trap_fatal (frame=0xe7e419e4, eva=851974) at /usr/src/sys/i386/i386/trap.c:939
 939                     panic("%s", trap_msg[type]);
 (kgdb) l -
 929                     frame->tf_err = eva;    /* smuggle fault address to ddb */
 930                     if (kdb_trap(type, 0, frame)) {
 931                             frame->tf_err = code;   /* restore error code */
 932                             return;
 933                     }
 934                     frame->tf_err = code;           /* restore error code */
 935             }
 936     #endif
 937             printf("trap number             = %d\n", type);
 938             if (type <= MAX_TRAP_MSG)
 (kgdb) l
 939                     panic("%s", trap_msg[type]);
 940             else
 941                     panic("unknown/reserved trap");
 942     }
 943
 944     /*
 945      * Double fault handler. Called when a fault occurs while writing
 946      * a frame for a trap/exception onto the stack. This usually occurs
 947      * when the stack overflows (such is the case with infinite recursion,
 948      * for example).
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #4  0xc0aa3610 in trap_pfault (frame=0xe7e419e4, usermode=0, eva=851974) at /usr/src/sys/i386/i386/trap.c:852
 852                     trap_fatal(frame, eva);
 (kgdb) l -
 842             }
 843             if (rv == KERN_SUCCESS)
 844                     return (0);
 845     nogo:
 846             if (!usermode) {
 847                     if (td->td_intr_nesting_level == 0 &&
 848                         PCPU_GET(curpcb)->pcb_onfault != NULL) {
 849                             frame->tf_eip = (int)PCPU_GET(curpcb)->pcb_onfault;
 850                             return (0);
 851                     }
 (kgdb) l
 852                     trap_fatal(frame, eva);
 853                     return (-1);
 854             }
 855
 856             return((rv == KERN_PROTECTION_FAILURE) ? SIGBUS : SIGSEGV);
 857     }
 858
 859     static void
 860     trap_fatal(frame, eva)
 861             struct trapframe *frame;
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #5  0xc0aa3fcc in trap (frame=0xe7e419e4) at /usr/src/sys/i386/i386/trap.c:530
 530                             (void) trap_pfault(frame, FALSE, eva);
 (kgdb) l -
 520                             i = SIGFPE;
 521                             break;
 522                     }
 523             } else {
 524                     /* kernel trap */
 525
 526                     KASSERT(cold || td->td_ucred != NULL,
 527                         ("kernel trap doesn't have ucred"));
 528                     switch (type) {
 529                     case T_PAGEFLT:                 /* page fault */
 (kgdb) l
 530                             (void) trap_pfault(frame, FALSE, eva);
 531                             goto out;
 532
 533                     case T_DNA:
 534     #ifdef DEV_NPX
 535                             /*
 536                              * The kernel is apparently using npx for copying.
 537                              * XXX this should be fatal unless the kernel has
 538                              * registered such use.
 539                              */
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #6  0xc0a89e3b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
 159             call    trap
 (kgdb) l -
 149     alltraps:
 150             pushal
 151             pushl   %ds
 152             pushl   %es
 153             pushl   %fs
 154     alltraps_with_regs_pushed:
 155             SET_KERNEL_SREGS
 156             FAKE_MCOUNT(TF_EIP(%esp))
 157     calltrap:
 158             pushl   %esp
 (kgdb) l
 159             call    trap
 160             add     $4, %esp
 161
 162             /*
 163              * Return via doreti to handle ASTs.
 164              */
 165             MEXITCOUNT
 166             jmp     doreti
 167
 168     /*
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #7  0xc0737c89 in g_io_request (bp=0xc6005528, cp=0xcac4cd00) at /usr/src/sys/geom/geom_io.c:364
 364             g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
 Current language:  auto; currently c
 (kgdb) l -
 354             }
 355             if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) {
 356                     KASSERT(bp->bio_offset % cp->provider->sectorsize == 0,
 357                         ("wrong offset %jd for sectorsize %u",
 358                         bp->bio_offset, cp->provider->sectorsize));
 359                     KASSERT(bp->bio_length % cp->provider->sectorsize == 0,
 360                         ("wrong length %jd for sectorsize %u",
 361                         bp->bio_length, cp->provider->sectorsize));
 362             }
 363
 (kgdb) l
 364             g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
 365                 bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd);
 366
 367             bp->bio_from = cp;
 368             bp->bio_to = pp;
 369             bp->bio_error = 0;
 370             bp->bio_completed = 0;
 371
 372             KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
 373                 ("Bio already on queue bp=%p", bp));
 
 (kgdb) p cp->geom->name
 $1 = 0xc0b3c773 "vfslock"
 
 (kgdb) p pp->name
 Cannot access memory at address 0xd0006
 
 (kgdb) p bp->bio_cmd
 $2 = 1 '\001'
 
 (kgdb) info locals
 pp = (struct g_provider *) 0xd0006
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #8  0xc073c286 in g_vfs_strategy (bo=0xc8dbeec4, bp=0xd91e1c14) at /usr/src/sys/geom/geom_vfs.c:107
 107             g_io_request(bip, cp);
 (kgdb) l -
 97              cp = bo->bo_private;
 98              G_VALID_CONSUMER(cp);
 99
 100             bip = g_alloc_bio();
 101             bip->bio_cmd = bp->b_iocmd;
 102             bip->bio_offset = bp->b_iooffset;
 103             bip->bio_data = bp->b_data;
 104             bip->bio_done = g_vfs_done;
 105             bip->bio_caller2 = bp;
 106             bip->bio_length = bp->b_bcount;
 (kgdb) l
 107             g_io_request(bip, cp);
 108     }
 109
 110     static void
 111     g_vfs_orphan(struct g_consumer *cp)
 112     {
 113
 114             /*
 115              * Don't do anything here yet.
 116              *
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) up
 #9  0xc074cfd0 in cd9660_strategy (ap=0xe7e41aa8) at /usr/src/sys/fs/cd9660/cd9660_vnops.c:758
 758             BO_STRATEGY(bo, bp);
 (kgdb) l -
 748                         (ip->i_mnt->im_bshift - DEV_BSHIFT);
 749                     if ((long)bp->b_blkno == -1)    /* XXX: cut&paste junk ? */
 750                             clrbuf(bp);
 751             }
 752             if ((long)bp->b_blkno == -1) {  /* XXX: cut&paste junk ? */
 753                     bufdone(bp);
 754                     return (0);
 755             }
 756             bp->b_iooffset = dbtob(bp->b_blkno);
 757             bo = ip->i_mnt->im_bo;
 (kgdb) l
 758             BO_STRATEGY(bo, bp);
 759             return (0);
 760     }
 761
 762     /*
 763      * Return POSIX pathconf information applicable to cd9660 filesystems.
 764      */
 765     static int
 766     cd9660_pathconf(ap)
 767             struct vop_pathconf_args /* {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 --------------090302090707070603070200--

From: Gavin Atkinson <gavin@FreeBSD.org>
To: Elmar Stellnberger <estellnb@googlemail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/130941: [panic] unexpected crash due to page fault
Date: Fri, 06 Mar 2009 16:20:12 +0000

 Apologies for the long delay in responding.  The debugging info you have
 provided so far is great.  Could you do a couple more things please?
 
 On Wed, 2009-01-28 at 15:53 +0000, Elmar Stellnberger wrote:
 
 > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 > 
 > (kgdb) up
 > #8  0xc073c286 in g_vfs_strategy (bo=0xc8dbeec4, bp=0xd91e1c14) at /usr/src/sys/geom/geom_vfs.c:107
 > 107             g_io_request(bip, cp);
 > (kgdb) l -
 > 97              cp = bo->bo_private;
 > 98              G_VALID_CONSUMER(cp);
 > 99
 > 100             bip = g_alloc_bio();
 > 101             bip->bio_cmd = bp->b_iocmd;
 > 102             bip->bio_offset = bp->b_iooffset;
 > 103             bip->bio_data = bp->b_data;
 > 104             bip->bio_done = g_vfs_done;
 > 105             bip->bio_caller2 = bp;
 > 106             bip->bio_length = bp->b_bcount;
 > (kgdb) l
 > 107             g_io_request(bip, cp);
 > 108     }
 > 109
 > 110     static void
 > 111     g_vfs_orphan(struct g_consumer *cp)
 > 112     {
 > 113
 > 114             /*
 > 115              * Don't do anything here yet.
 > 116              *
 
 Here, please show the output of
 
 p (struct g_consumer) *cp
 p (struct bio) *bip
 
 Thanks,
 
 Gavin

From: Elmar Stellnberger <estellnb@gmail.com>
To: Gavin Atkinson <gavin@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/130941: [panic] unexpected crash due to page fault
Date: Sat, 07 Mar 2009 13:22:28 +0000

 This is a multi-part message in MIME format.
 --------------030206000305030302030704
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 Gavin Atkinson wrote:
 > Apologies for the long delay in responding.  The debugging info you have
 > provided so far is great.  Could you do a couple more things please?
 > 
 > On Wed, 2009-01-28 at 15:53 +0000, Elmar Stellnberger wrote:
 > 
 >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 >>
 >> (kgdb) up
 >> #8  0xc073c286 in g_vfs_strategy (bo=0xc8dbeec4, bp=0xd91e1c14) at /usr/src/sys/geom/geom_vfs.c:107
 >> 107             g_io_request(bip, cp);
 >> (kgdb) l -
 >> 97              cp = bo->bo_private;
 >> 98              G_VALID_CONSUMER(cp);
 >> 99
 >> 100             bip = g_alloc_bio();
 >> 101             bip->bio_cmd = bp->b_iocmd;
 >> 102             bip->bio_offset = bp->b_iooffset;
 >> 103             bip->bio_data = bp->b_data;
 >> 104             bip->bio_done = g_vfs_done;
 >> 105             bip->bio_caller2 = bp;
 >> 106             bip->bio_length = bp->b_bcount;
 >> (kgdb) l
 >> 107             g_io_request(bip, cp);
 >> 108     }
 >> 109
 >> 110     static void
 >> 111     g_vfs_orphan(struct g_consumer *cp)
 >> 112     {
 >> 113
 >> 114             /*
 >> 115              * Don't do anything here yet.
 >> 116              *
 > 
 > Here, please show the output of
 > 
 > p (struct g_consumer) *cp
 > p (struct bio) *bip
 > 
 > Thanks,
 > 
 > Gavin
 > 
 
 
 --------------030206000305030302030704
 Content-Type: text/plain;
  name="backtrace2.txt"
 Content-Transfer-Encoding: 8bit
 Content-Disposition: inline;
  filename="backtrace2.txt"
 
 
  
 Fatal trap 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address   = 0xd0006
 fault code              = supervisor read, page not present
 instruction pointer     = 0x20:0xc0737c89
 stack pointer           = 0x28:0xe7e41a24
 frame pointer           = 0x28:0xe7e41a50
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 30666 (gfind)
 trap number             = 12
 panic: page fault
 cpuid = 0
 Uptime: 1h24m58s
 Physical memory: 2034 MB
 Dumping 229 MB: 214 198 182 166 150 134 118 102 86 70 54 38 22 6
 
 
 #6  0xc0a89e3b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
 #7  0xc0737c89 in g_io_request (bp=0xc6005528, cp=0xcac4cd00) at /usr/src/sys/geom/geom_io.c:364
 #8  0xc073c286 in g_vfs_strategy (bo=0xc8dbeec4, bp=0xd91e1c14) at /usr/src/sys/geom/geom_vfs.c:107
 #9  0xc074cfd0 in cd9660_strategy (ap=0xe7e41aa8) at /usr/src/sys/fs/cd9660/cd9660_vnops.c:758
 #10 0xc0ab90dd in VOP_STRATEGY_APV (vop=0xc0bd6100, a=0xe7e41aa8) at vnode_if.c:1771
 #11 0xc07f5b6e in bufstrategy (bo=0xc93582e8, bp=0xd91e1c14) at vnode_if.h:935
 #12 0xc07fca7a in breadn (vp=0xc9358228, blkno=0, size=2048, rablkno=0x0, rabsize=0x0, cnt=0, cred=0x0, bpp=0xe7e41b3c)
     at buf.h:429
 #13 0xc07fcb2c in bread (vp=0xc9358228, blkno=Unhandled dwarf expression opcode 0x93
 ) at /usr/src/sys/kern/vfs_bio.c:734
 #14 0xc0748ee0 in cd9660_blkatoff (vp=0xc9358228, offset=0, res=0x0, bpp=0xe7e41bc0)
     at /usr/src/sys/fs/cd9660/cd9660_lookup.c:407
 #15 0xc074d82d in cd9660_readdir (ap=0xe7e41c00) at /usr/src/sys/fs/cd9660/cd9660_vnops.c:516
 #16 0xc0ab8352 in VOP_READDIR_APV (vop=0xc0bd6100, a=0xe7e41c00) at vnode_if.c:1407
 #17 0xc081ab40 in kern_getdirentries (td=0xc6525230, fd=5, buf=0x2832c000 <Address 0x2832c000 out of bounds>, count=4096,
     basep=0xe7e41c74) at vnode_if.h:747
 #18 0xc081ad51 in getdirentries (td=0xc6525230, uap=0xe7e41cfc) at /usr/src/sys/kern/vfs_syscalls.c:3767
 #19 0xc0aa3965 in syscall (frame=0xe7e41d38) at /usr/src/sys/i386/i386/trap.c:1090
 #20 0xc0a89ea0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
 #21 0x00000033 in ?? ()
 
 
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) frame 7
 #7  0xc0737c89 in g_io_request (bp=0xc6005528, cp=0xcac4cd00) at /usr/src/sys/geom/geom_io.c:364
 364             g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
 
 
 322     g_io_request(struct bio *bp, struct g_consumer *cp)
 323     {
 (kgdb) l
 324             struct g_provider *pp;
 325
 326             KASSERT(cp != NULL, ("NULL cp in g_io_request"));
 327             KASSERT(bp != NULL, ("NULL bp in g_io_request"));
 328             pp = cp->provider;
 
 329             KASSERT(pp != NULL, ("consumer not attached in g_io_request"));
 330     #ifdef DIAGNOSTIC
 331             KASSERT(bp->bio_driver1 == NULL,
 332                 ("bio_driver1 used by the consumer (geom %s)", cp->geom->name));
 333             KASSERT(bp->bio_driver2 == NULL,
 (kgdb) l
 334                 ("bio_driver2 used by the consumer (geom %s)", cp->geom->name));
 335             KASSERT(bp->bio_pflags == 0,
 336                 ("bio_pflags used by the consumer (geom %s)", cp->geom->name));
 337             /*
 338              * Remember consumer's private fields, so we can detect if they were
 339              * modified by the provider.
 340              */
 341             bp->_bio_caller1 = bp->bio_caller1;
 342             bp->_bio_caller2 = bp->bio_caller2;
 343             bp->_bio_cflags = bp->bio_cflags;
 (kgdb) l
 344     #endif
 345
 346             if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_GETATTR)) {
 347                     KASSERT(bp->bio_data != NULL,
 348                         ("NULL bp->data in g_io_request(cmd=%hhu)", bp->bio_cmd));
 349             }
 350             if (bp->bio_cmd & (BIO_DELETE|BIO_FLUSH)) {
 351                     KASSERT(bp->bio_data == NULL,
 352                         ("non-NULL bp->data in g_io_request(cmd=%hhu)",
 353                         bp->bio_cmd));
 (kgdb) l
 354             }
 355             if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) {
 356                     KASSERT(bp->bio_offset % cp->provider->sectorsize == 0,
 357                         ("wrong offset %jd for sectorsize %u",
 358                         bp->bio_offset, cp->provider->sectorsize));
 359                     KASSERT(bp->bio_length % cp->provider->sectorsize == 0,
 360                         ("wrong length %jd for sectorsize %u",
 361                         bp->bio_length, cp->provider->sectorsize));
 362             }
 363
 (kgdb) l
 364             g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
 365                 bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd);
 (kgdb) p cp->geom->name
 $1 = 0xc0b3c773 "vfslock"
 
 (kgdb) p pp->name
 Cannot access memory at address 0xd0006
 
 (kgdb) p bp->bio_cmd
 $2 = 1 '\001'
 
 (kgdb) info locals
 pp = (struct g_provider *) 0xd0006
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) frame 8
 
 91      void
 92      g_vfs_strategy(struct bufobj *bo, struct buf *bp)
 93      {
 94              struct g_consumer *cp;
 95              struct bio *bip;
 96
 (kgdb) l
 97              cp = bo->bo_private;
 98              G_VALID_CONSUMER(cp);
 99
 100             bip = g_alloc_bio();
 101             bip->bio_cmd = bp->b_iocmd;
 102             bip->bio_offset = bp->b_iooffset;
 103             bip->bio_data = bp->b_data;
 104             bip->bio_done = g_vfs_done;
 105             bip->bio_caller2 = bp;
 106             bip->bio_length = bp->b_bcount;
 (kgdb) l
 107             g_io_request(bip, cp);
 108     }
 
 (kdbg) info locals
 cp = (struct g_consumer *) 0xcac4cd00
 bip = Variable "bip" is not available.
 
 
 (kdbg)  p *cp
 (kdbg)  p (struct g_consumer) *cp
 $4 = {geom = 0xc5611b30, consumer = {le_next = 0x0, le_prev = 0xd93a6398}, provider = 0xd0006, consumers = {
     le_next = 0xc6782c00, le_prev = 0xc652cd10}, acr = 7, acw = 3211872, ace = 0, spoiled = 0, stat = 0x0, nstart = 0,
   nend = 3404908096, private = 0xc6782c00, index = 3401582104}
 
 (kgdb) p *((struct g_consumer *)bo->bo_private)
 $15 = {geom = 0xc5611b30, consumer = {le_next = 0x0, le_prev = 0xd93a6398}, provider = 0xd0006, consumers = {
     le_next = 0xc6782c00, le_prev = 0xc652cd10}, acr = 7, acw = 3211872, ace = 0, spoiled = 0, stat = 0x0, nstart = 0,
   nend = 3404908096, private = 0xc6782c00, index = 3401582104}
 
 (kgdb) p cp->provider
 $17 = (struct g_provider *) 0xd0006
 
 (kgdb) p *(cp->provider)
 Cannot access memory at address 0xd0006
 
 
 
 
 
 (kgdb) p (struct bio) *bip
 Variable "bip" is not available.
 
 
 (kgdb) down
 #7  0xc0737c89 in g_io_request (bp=0xc6005528, cp=0xcac4cd00) at /usr/src/sys/geom/geom_io.c:364
 364             g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
 
 (kgdb) p (struct bio) *bp
 $5 = {bio_cmd = 1 '\001', bio_flags = 0 '\0', bio_cflags = 0 '\0', bio_pflags = 0 '\0', bio_dev = 0x0, bio_disk = 0x0,
   bio_offset = 165888, bio_bcount = 0, bio_data = 0xda943000 "\212��\211</b>", bio_error = 0, bio_resid = 0,
   bio_done = 0xc073c290 <g_vfs_done>, bio_driver1 = 0x0, bio_driver2 = 0x0, bio_caller1 = 0x0, bio_caller2 = 0xd91e1c14,
   bio_queue = {tqe_next = 0x0, tqe_prev = 0x0}, bio_attribute = 0x0, bio_from = 0x0, bio_to = 0x0, bio_length = 2048,
   bio_completed = 0, bio_children = 0, bio_inbed = 0, bio_parent = 0x0, bio_t0 = {sec = 0, frac = 0}, bio_task = 0,
   bio_task_arg = 0x0, bio_pblkno = 0}
 
 (kdbg)  p (struct g_consumer) *cp
 --- as before ---
 
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
 
 (kdbg) frame 8
 
 (kgdb) p (struct bufobj) *bo
 $10 = {bo_mtx = 0xc8dbee8c, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xc8dbeec8}, bv_root = 0x0, bv_cnt = 0},
   bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xc8dbeed8}, bv_root = 0x0, bv_cnt = 0}, bo_numoutput = 0,
   bo_flag = 0, bo_ops = 0xc0bd5404, bo_bsize = 2048, bo_object = 0xc9f40a2c, bo_synclist = {le_next = 0x0,
     le_prev = 0x0}, bo_private = 0xcac4cd00, __bo_vnode = 0xc8dbee04}
 
 (kgdb) p bo->bo_private
 $12 = (void *) 0xcac4cd00
 
 (kgdb) p cp
 $13 = (struct g_consumer *) 0xcac4cd00
 
 (kgdb) p (struct buf) *bp
 $8 = {b_bufobj = 0xc93582e8, b_bcount = 2048, b_caller1 = 0x0, b_data = 0xda943000 "\212��\211</b>", b_error = 0,
   b_iocmd = 1 '\001', b_ioflags = 0 '\0', b_iooffset = 165888, b_resid = 0, b_iodone = 0, b_blkno = 324, b_offset = 0,
   b_bobufs = {tqe_next = 0x0, tqe_prev = 0xc93582ec}, b_left = 0x0, b_right = 0x0, b_vflags = 0, b_freelist = {
     tqe_next = 0xd93c52bc, tqe_prev = 0xc0c4a068}, b_qindex = 0, b_flags = 570425344, b_xflags = 2 '\002', b_lock = {
     lk_object = {lo_name = 0xc0b3ba9b "bufwait", lo_type = 0xc0b3ba9b "bufwait", lo_flags = 70844416, lo_witness_data = {
         lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, lk_interlock = 0xc0c3df28, lk_flags = 262144,
     lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 1, lk_prio = 80, lk_timo = 0, lk_lockholder = 0xc6525230,
     lk_newlock = 0x0}, b_bufsize = 2048, b_runningbufspace = 0, b_kvabase = 0xda943000 "\212��\211</b>",
   b_kvasize = 16384, b_lblkno = 0, b_vp = 0xc9358228, b_dirtyoff = 0, b_dirtyend = 0, b_rcred = 0x0, b_wcred = 0x0,
   b_saveaddr = 0xda943000, b_pager = {pg_reqpage = 0}, b_cluster = {cluster_head = {tqh_first = 0xd9268360,
       tqh_last = 0xd91833d8}, cluster_entry = {tqe_next = 0xd9268360, tqe_prev = 0xd91833d8}}, b_pages = {0xc34a4da8,
     0x0 <repeats 31 times>}, b_npages = 1, b_dep = {lh_first = 0x0}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0,
   b_fsprivate3 = 0x0, b_pin_count = 0}
 
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 (kgdb) frame 9
 #9  0xc074cfd0 in cd9660_strategy (ap=0xe7e41aa8) at /usr/src/sys/fs/cd9660/cd9660_vnops.c:758
 758             BO_STRATEGY(bo, bp);
 
 727     /*
 (kgdb) l
 728      * Calculate the logical to physical mapping if not done already,
 729      * then call the device strategy routine.
 730      */
 731     static int
 732     cd9660_strategy(ap)
 733             struct vop_strategy_args /* {
 734                     struct buf *a_vp;
 735                     struct buf *a_bp;
 736             } */ *ap;
 737     {
 (kgdb) l
 738             struct buf *bp = ap->a_bp;
 739             struct vnode *vp = ap->a_vp;
 740             struct iso_node *ip;
 741             struct bufobj *bo;
 742
 743             ip = VTOI(vp);
 744             if (vp->v_type == VBLK || vp->v_type == VCHR)
 745                     panic("cd9660_strategy: spec");
 746             if (bp->b_blkno == bp->b_lblkno) {
 747                     bp->b_blkno = (ip->iso_start + bp->b_lblkno) <<
 (kgdb) l
 748                         (ip->i_mnt->im_bshift - DEV_BSHIFT);
 749                     if ((long)bp->b_blkno == -1)    /* XXX: cut&paste junk ? */
 750                             clrbuf(bp);
 751             }
 752             if ((long)bp->b_blkno == -1) {  /* XXX: cut&paste junk ? */
 753                     bufdone(bp);
 754                     return (0);
 755             }
 756             bp->b_iooffset = dbtob(bp->b_blkno);
 757             bo = ip->i_mnt->im_bo;
 (kgdb) l
 758             BO_STRATEGY(bo, bp);
 759             return (0);
 760     }
 
 (kgdb) p ip->i_mnt->im_bo
 $18 = (struct bufobj *) 0xc8dbeec4
 
 (kgdb) p (struct iso_mnt)*ip->i_mnt
 $20 = {im_flags = 0, im_mountp = 0xc5611000, im_dev = 0xc557d300, im_devvp = 0xc8dbee04, im_cp = 0xc7415600,
   im_bo = 0xc8dbeec4, logical_block_size = 2048, im_bshift = 11, im_bmask = 2047, volume_space_size = 1093494,
   root = "\"\000Q\000\000\000\000\000\000Q\000\030\000\000\000\000\030\000m\001\001\025-8\000\002\000\000\001\000\000\001\001", root_extent = 81, root_size = 6144, iso_ftype = ISO_FTYPE_RRIP, rr_skip = 0, rr_skip0 = 0, joliet_level = 0,
   im_d2l = 0x0, im_l2d = 0x0}
 
 (kgdb) p ip->i_mnt
 $24 = (struct iso_mnt *) 0xcab64d00
 
 (kgdb) p *ip
 $23 = {i_vnode = 0xc9358228, i_flag = 0, i_number = 165888, i_mnt = 0xcab64d00, i_lockf = 0x0, i_endoff = 0,
   i_diroff = 0, i_offset = 6144, i_ino = 170440, iso_extent = 81, i_size = 6144, iso_start = 81, inode = {iso_atime = {
       tv_sec = 1230846886, tv_nsec = 0}, iso_mtime = {tv_sec = 1230846356, tv_nsec = 0}, iso_ctime = {
       tv_sec = 1230846356, tv_nsec = 0}, iso_mode = 16749, iso_uid = 0, iso_gid = 0, iso_links = 21, iso_rdev = 0}}
 
 
 (kgdb) p *ap->a_vp
 $29 = {v_type = VDIR, v_tag = 0xc0b33da2 "isofs", v_op = 0xc0bd6100, v_data = 0xcacd9280, v_mount = 0xc5611000,
   v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0xc5611068}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0,
     vu_fifoinfo = 0x0, vu_yield = 0}, v_hashlist = {le_next = 0x0, le_prev = 0xcadea8c0}, v_hash = 165888, v_cache_src = {
     lh_first = 0xc9da18c4}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xc9358258}, v_dd = 0x0, v_cstart = 0,
   v_lasta = 0, v_lastw = 0, v_clen = 0, v_lock = {lk_object = {lo_name = 0xc0b33da2 "isofs",
       lo_type = 0xc0b33da2 "isofs", lo_flags = 70844416, lo_witness_data = {lod_list = {stqe_next = 0x0},
         lod_witness = 0x0}}, lk_interlock = 0xc0c3d7f0, lk_flags = 262272, lk_sharecount = 0, lk_waitcount = 0,
     lk_exclusivecount = 1, lk_prio = 80, lk_timo = 51, lk_lockholder = 0xc6525230, lk_newlock = 0x0}, v_interlock = {
     lock_object = {lo_name = 0xc0b3d061 "vnode interlock", lo_type = 0xc0b3d061 "vnode interlock", lo_flags = 16973824,
       lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0},
   v_vnlock = 0xc9358280, v_holdcnt = 5, v_usecount = 2, v_iflag = 0, v_vflag = 1, v_writecount = 0, v_freelist = {
     tqe_next = 0xc720133c, tqe_prev = 0xc8c32958}, v_bufobj = {bo_mtx = 0xc93582b0, bo_clean = {bv_hd = {
         tqh_first = 0xd91e1c14, tqh_last = 0xd91e1c4c}, bv_root = 0xd91e1c14, bv_cnt = 1}, bo_dirty = {bv_hd = {
         tqh_first = 0x0, tqh_last = 0xc93582fc}, bv_root = 0x0, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0,
     bo_ops = 0xc0be41a0, bo_bsize = 2048, bo_object = 0xc9095744, bo_synclist = {le_next = 0x0, le_prev = 0x0},
     bo_private = 0xc9358228, __bo_vnode = 0xc9358228}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0}
 
 (kgdb) p *ap
 $30 = {a_gen = {a_desc = 0xc0c0e740}, a_vp = 0xc9358228, a_bp = 0xd91e1c14}
 (
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
 (kgdb) frame 10
 #10 0xc0ab90dd in VOP_STRATEGY_APV (vop=0xc0bd6100, a=0xe7e41aa8) at vnode_if.c:1771
 1771    vnode_if.c: No such file or directory.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 --------------030206000305030302030704--
>Unformatted:
