From postmaster@console.kharkov.ukrpack.net  Tue Apr 15 23:25:05 2003
Return-Path: <postmaster@console.kharkov.ukrpack.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 243F237B401
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 15 Apr 2003 23:25:05 -0700 (PDT)
Received: from mail.ic.kharkov.ua (mail.ic.kharkov.ua [212.1.112.3])
	by mx1.FreeBSD.org (Postfix) with SMTP id A7AE543F93
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 15 Apr 2003 23:25:02 -0700 (PDT)
	(envelope-from postmaster@console.kharkov.ukrpack.net)
Received: (qmail 72953 invoked by uid 1001); 16 Apr 2003 06:25:36 -0000
Message-Id: <20030416062536.72952.qmail@console.kharkov.ukrpack.net>
Date: 16 Apr 2003 06:25:36 -0000
From: "Aleksey Ovcharenko" <postmaster@console.kharkov.ukrpack.net>
Reply-To: Aleksey Ovcharenko <alexovch@ic.kharkov.ua>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: kernel panic: ufsdirhash_lookup: bad offset in hash array
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         51016
>Category:       kern
>Synopsis:       kernel panic: ufsdirhash_lookup: bad offset in hash array
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Apr 15 23:30:12 PDT 2003
>Closed-Date:    Tue Oct 21 02:40:55 PDT 2003
>Last-Modified:  Tue Oct 21 02:40:55 PDT 2003
>Originator:     Aleksey Ovcharenko
>Release:        FreeBSD 4.7-RELEASE-p7 i386
>Organization:
>Environment:
System: FreeBSD console.ic.kharkov.ua 4.7-RELEASE-p7 FreeBSD 4.7-RELEASE-p7 #2: Mon Mar 17 08:27:16 EET 2003 root@console.ic.kharkov.ua:/usr/obj/usr/src/sys/INFOCOM i386


	
>Description:
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...(no debugging symbols found)...
IdlePTD at phsyical address 0x00324000
initial pcb at physical address 0x00291bc0
panicstr: ufsdirhash_lookup: bad offset in hash array
panic messages:
---
panic: ufsdirhash_lookup: bad offset in hash array

syncing disks... 6 5 5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
done
Uptime: 1d16h21m18s

dumping to dev #ad/0x20001, offset 1051136
dump ata0: resetting devices .. done
510 509 508 507 506 505 504 503 502 501 500 499 498 497 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480 479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460 459 458 457 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440 439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420 419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400 399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 380 379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360 359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340 339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322 321 320 319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300 299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280 279 278 277 276 275 274 273 272 271 270 269 268 267 266 265 264 26
 3 262 261 260 259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0  0xc014ec8e in dumpsys ()
(kgdb)
>How-To-Repeat:
Use fetchnews from leafnode+-2.14. It cause a panic, but not alltime.
>Fix:

	


>Release-Note:
>Audit-Trail:

From: "Ted Mittelstaedt" <tedm@toybox.placo.com>
To: <freebsd-gnats-submit@FreeBSD.org>, <alexovch@ic.kharkov.ua>
Cc:  
Subject: Re: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Fri, 9 May 2003 23:56:31 -0700

 Did you install leafnode+ from the FreeBSD ports collection?  Does
 the leafnode+ port maintainer confirm this problem?  This should be
 probably be reclassed as a ports bug I would think.
 
 Ted Mittelstaedt                                       tedm@toybox.placo.com
 

From: "Ted Mittelstaedt" <tedm@toybox.placo.com>
To: <alexovch@ic.kharkov.ua>
Cc: <freebsd-gnats-submit@FreeBSD.org>
Subject: RE: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Sat, 10 May 2003 11:44:45 -0700

 >If some port can cause a system crash it's probably a kernel 
 >security bug and 
 >it's have to be fixed imho.
 
 This is true if the port crashes on every FreeBSD system it
 is tried on.  However if your system is the only one that is
 crashing then it's most likely something specific to your
 system.  Many times hardware problems with hardware, or device
 driver bugs, can hand back garbage to the kernel and cause a
 panic.  You should include a dmesg printout so we can see the
 kind of hardware your running.
 
 Since the leafnode+ maintainer obviously is running
 leafnode he can confirm if there's any history of crashing.
 
 I've also compiled and installed leafnode myself with no ill
 effects, although I don't currently use it, I use NewsCache.
 
 Note that you need to CC freebsd-gnats-submit@FreeBSD.org
 and leave the subject line intact in order for your responses
 to be made part of the audit trail.
 
 Ted Mittelstaedt                                       tedm@toybox.placo.com
 
 

From: Valentin Nechayev <netch@segfault.kiev.ua>
To: Ted Mittelstaedt <tedm@toybox.placo.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Sun, 11 May 2003 11:13:49 +0300

  Sat, May 10, 2003 at 11:50:12, tedm (Ted Mittelstaedt) wrote about "RE: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array": 
 
  >>If some port can cause a system crash it's probably a kernel 
  >>security bug and 
  >>it's have to be fixed imho.
 
 In any case it can't be classified as port bug, as you suggested
 in previous letter, unless program has direct access to kernel memory or
 raw disks. I don't believe he run leafnode as root.
 
 TM>  This is true if the port crashes on every FreeBSD system it
 TM>  is tried on.
 
 Totally wrong. Problem can be triggered with specific config or specific
 running history. It even can be irreproduceable, if it can be diagnosed.
 If it is reproduceable, it can be diagnosed much easier.
 
 TM>  However if your system is the only one that is
 TM>  crashing then it's most likely something specific to your
 TM>  system.
 
 This is also possible but you haven't *any* real arguments "pro" it.
 
 
 -netch-

From: "Ted Mittelstaedt" <tedm@toybox.placo.com>
To: "Valentin Nechayev" <netch@segfault.kiev.ua>
Cc: <freebsd-gnats-submit@freebsd.org>
Subject: RE: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Sun, 11 May 2003 21:04:39 -0700

 Hi Valentin,
 
   As previously stated I have run leafnode with no ill effects.  Do
 you run leafnode or have you seen this same panic elsewhere?  If not,
 could you please install leafnode and see if you can get it to panic
 your system so that we can get some confirmation that this is
 reproducible on different systems?
 
 Thanks!!
 
 Ted Mittelstaedt                                       tedm@toybox.placo.com
 

From: Valentin Nechayev <netch@netch.kiev.ua>
To: Ted Mittelstaedt <tedm@toybox.placo.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Mon, 12 May 2003 08:59:18 +0300

  Sun, May 11, 2003 at 21:04:39, tedm wrote about "RE: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array": 
 
 >   As previously stated I have run leafnode with no ill effects.  Do
 > you run leafnode or have you seen this same panic elsewhere?  If not,
 > could you please install leafnode and see if you can get it to panic
 > your system so that we can get some confirmation that this is
 > reproducible on different systems?
 
 I don't use leafnode and don't want to use it (I prefer INN and mailnews gate).
 Sorry, but there are no reason for me to run it, and even whether I go
 to install it, configuration possibly will be quite different from one
 of PR initiator.
 The only thing I wanted to see is avoidness of "You are the only who
 sees this bug, so it doesn't exist". You are right in intention to get
 leafnode config and other important details.
 
 
 -netch-
State-Changed-From-To: open->feedback 
State-Changed-By: dwmalone 
State-Changed-When: Mon May 26 04:06:09 PDT 2003 
State-Changed-Why:  
See if Aleksey can either get us more debugging info or check to see that 
his hardware is OK. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=51016 

From: David Malone <dwmalone@maths.tcd.ie>
To: freebsd-gnats-submit@FreeBSD.org
Cc: alexovch@ic.kharkov.ua
Subject: Re: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Mon, 26 May 2003 12:05:42 +0100

 Dear Aleksey,
 
 The we've suaully found that panics in ufsdirhash_lookup are caused
 by bad memory or bad processors. While it is possible there is a
 bug in dirhash, it seems more likely that hardware is at fault. The
 intermittent nature of your problem would also reenforce this.
 
 If you can run gdb -k again and get a back trace, by typing "bt"
 this might help us see what's going on.
 
 	David.

From: Aleksey Ovcharenko <alexovch@ic.kharkov.ua>
To: freebsd-gnats-submit@FreeBSD.org
Cc: David Malone <dwmalone@maths.tcd.ie>
Subject: Re: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Wed, 4 Jun 2003 12:39:55 +0259

 On Monday 26 May 2003 14:05, David Malone wrote:
 > Dear Aleksey,
 >
 > The we've suaully found that panics in ufsdirhash_lookup are caused
 > by bad memory or bad processors. While it is possible there is a
 > bug in dirhash, it seems more likely that hardware is at fault. The
 > intermittent nature of your problem would also reenforce this.
 >
 > If you can run gdb -k again and get a back trace, by typing "bt"
 > this might help us see what's going on.
 I send you this information some days ago, but still no response.
 
 Resending it in case it losts.
 
 (kgdb) bt
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 #1  0xc014ea58 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
 #2  0xc014ee8c in poweroff_wait (junk=0xc0259b40, howto=0) at /usr/src/sys/kern/kern_shutdown.c:595
 #3  0xc01e8a0f in ufsdirhash_lookup (ip=0xcd473000, name=0xe6b37800 "1051794533.99778.console.ic.kharkov.ua", namelen=38, offp=0xcd473064,
     bpp=0xe6ec6d5c, prevoffp=0x0) at /usr/src/sys/ufs/ufs/ufs_dirhash.c:361
 #4  0xc01e29bb in ufs_lookup (ap=0xe6ec6db8) at /usr/src/sys/ufs/ufs/ufs_lookup.c:212
 #5  0xc01e8241 in ufs_vnoperate (ap=0xe6ec6db8) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
 #6  0xc0178772 in vfs_cache_lookup (ap=0xe6ec6e10) at vnode_if.h:77
 #7  0xc01e8241 in ufs_vnoperate (ap=0xe6ec6e10) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
 #8  0xc017b70d in lookup (ndp=0xe6ec6e8c) at vnode_if.h:52
 #9  0xc017b1f8 in namei (ndp=0xe6ec6e8c) at /usr/src/sys/kern/vfs_lookup.c:153
 #10 0xc0180fc5 in lstat (p=0xe6e268e0, uap=0xe6ec6f80) at /usr/src/sys/kern/vfs_syscalls.c:1823
 #11 0xc022bd21 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134547712, tf_esi = 134547816, tf_ebp = -1077937680,
       tf_isp = -420712492, tf_ebx = 672177004, tf_edx = 3, tf_ecx = 134547776, tf_eax = 190, tf_trapno = 12, tf_err = 2, tf_eip = 671854364,
       tf_cs = 31, tf_eflags = 659, tf_esp = -1077937820, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
 #12 0xc021ffe5 in Xint0x80_syscall ()
 #13 0x280b756d in ?? ()
 #14 0x280b6dea in ?? ()
 #15 0x8048e62 in ?? ()
 #16 0x80488f1 in ?? ()
 (kgdb)
 
 -- 
 Sincerely Yours, Aleksey Ovcharenko
State-Changed-From-To: feedback->open 
State-Changed-By: kris 
State-Changed-When: Fri Aug 15 20:58:21 PDT 2003 
State-Changed-Why:  
Requested feedback received 

http://www.freebsd.org/cgi/query-pr.cgi?pr=51016 

From: Aleksey Ovcharenko <alexovch@ic.kharkov.ua>
To: freebsd-bugs@FreeBSD.org
Cc: Kris Kennaway <kris@FreeBSD.org>,
	<freebsd-gnats-submit@FreeBSD.org>
Subject: Re: kern/51016: kernel panic: ufsdirhash_lookup: bad offset in hash array
Date: Tue, 9 Sep 2003 18:22:23 +0300

 Hi, again!
 
 I have tested my pc hardware on very different crash-tests (the only for 
 Windows, don't have it for FreeBSD) and everytihing was working fine 9 days 
 non-stop at 40+ Celsium degree.
 
 But after switching to FreeBSD crashes have begone again. Can't ever reproduce 
 it, just happened suddenly.
 
 I changed my drive cable to UDMA-33 - didn't help, crash was this message like 
 "panic on no-panic entry" etc.
 
 Then I change motheboard, chipset etc., nope, crashes again and again.
 Here we go:
 
 IdlePTD at phsyical address 0x00329000
 initial pcb at physical address 0x00297460
 panicstr: general protection fault
 panic messages:
 ---
 Fatal trap 9: general protection fault while in kernel mode
 instruction pointer     = 0x8:0xc01f9d2f
 stack pointer           = 0x10:0xe71bdcb0
 frame pointer           = 0x10:0xe71bdcb4
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 3746 (find)
 interrupt mask          = none
 trap number             = 9
 panic: general protection fault
 ---
 syncing disks... 48 22 22 22
 done
 Uptime: 16h18m38s
 
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 487             if (dumping++) {
 (kgdb) where
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 #1  0xc014f830 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
 #2  0xc014fc64 in poweroff_wait (junk=0xc026a4ac, howto=-1071210555) at 
 /usr/src/sys/kern/kern_shutdown.c:595
 #3  0xc022db0b in trap_fatal (frame=0xe71bdc70, eva=0) at 
 /usr/src/sys/i386/i386/trap.c:974
 #4  0xc022d4e7 in trap (frame={tf_fs = -851116016, tf_es = 16, tf_ds = 16, 
 tf_edi = 512, tf_esi = -848367104, tf_ebp = -417604428,
       tf_isp = -417604452, tf_ebx = -854001920, tf_edx = -1, tf_ecx = 4, 
 tf_eax = 372, tf_trapno = 9, tf_err = 0, tf_eip = -1071669969,
       tf_cs = 8, tf_eflags = 66050, tf_esp = 0, tf_ss = -417604360}) at 
 /usr/src/sys/i386/i386/trap.c:636
 #5  0xc01f9d2f in zalloc (z=0xcd18f700) at /usr/src/sys/vm/vm_zone.c:70
 #6  0xc01e9c1b in ufsdirhash_build (ip=0xcd45b100) at 
 /usr/src/sys/ufs/ufs/ufs_dirhash.c:166
 #7  0xc01e41da in ufs_lookup (ap=0xe71bddb8) at 
 /usr/src/sys/ufs/ufs/ufs_lookup.c:196
 #8  0xc01e99d1 in ufs_vnoperate (ap=0xe71bddb8) at 
 /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
 #9  0xc0179d02 in vfs_cache_lookup (ap=0xe71bde10) at vnode_if.h:77
 #10 0xc01e99d1 in ufs_vnoperate (ap=0xe71bde10) at 
 /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
 #11 0xc017ccf5 in lookup (ndp=0xe71bde8c) at vnode_if.h:52
 #12 0xc017c7e0 in namei (ndp=0xe71bde8c) at /usr/src/sys/kern/vfs_lookup.c:153
 #13 0xc0182619 in lstat (p=0xe6f728a0, uap=0xe71bdf80) at 
 /usr/src/sys/kern/vfs_syscalls.c:1824
 #14 0xc022dd71 in syscall2 (frame={tf_fs = 134545455, tf_es = 47, tf_ds = 
 -1078001617, tf_edi = 134576128, tf_esi = 134576200,
       tf_ebp = -1077937436, tf_isp = -417603628, tf_ebx = 672089996, tf_edx = 
 3, tf_ecx = 134576192, tf_eax = 190, tf_trapno = 7,
       tf_err = 2, tf_eip = 671765056, tf_cs = 31, tf_eflags = 659, tf_esp = 
 -1077937576, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
 #15 0xc0221e35 in Xint0x80_syscall ()
 #16 0x280a1871 in ?? ()
 #17 0x280a10ee in ?? ()
 #18 0x8049732 in ?? ()
 #19 0x804b9f8 in ?? ()
 #20 0x80493ce in ?? ()
 (kgdb) up 6
 #6  0xc01e9c1b in ufsdirhash_build (ip=0xcd45b100) at 
 /usr/src/sys/ufs/ufs/ufs_dirhash.c:166
 166                     if ((dh->dh_hash[i] = zalloc(ufsdirhash_zone)) == 
 NULL)
 (kgdb) list
 161             MALLOC(dh->dh_blkfree, u_int8_t *, nblocks * 
 sizeof(dh->dh_blkfree[0]),
 162                 M_DIRHASH, M_NOWAIT);
 163             if (dh->dh_hash == NULL || dh->dh_blkfree == NULL)
 164                     goto fail;
 165             for (i = 0; i < narrays; i++) {
 166                     if ((dh->dh_hash[i] = zalloc(ufsdirhash_zone)) == 
 NULL)
 167                             goto fail;
 168                     for (j = 0; j < DH_NBLKOFF; j++)
 169                             dh->dh_hash[i][j] = DIRHASH_EMPTY;
 170             }
 (kgdb) up 5
 #11 0xc017ccf5 in lookup (ndp=0xe71bde8c) at vnode_if.h:52
 52              rc = VCALL(dvp, VOFFSET(vop_lookup), &a);
 (kgdb) list
 47              int rc;
 48              a.a_desc = VDESC(vop_lookup);
 49              a.a_dvp = dvp;
 50              a.a_vpp = vpp;
 51              a.a_cnp = cnp;
 52              rc = VCALL(dvp, VOFFSET(vop_lookup), &a);
 53              return (rc);
 54      }
 55      struct vop_cachedlookup_args {
 56              struct vnodeop_desc *a_desc;
 (kgdb)
 
 The only way to stop those crashes is to turn off dirhash, so it sounds like 
 kernel bug. I'll glad to help to find it, just tell me how.
 
 -- 
 Sincerely Yours, Aleksey Ovcharenko
State-Changed-From-To: open->closed 
State-Changed-By: dwmalone 
State-Changed-When: Tue Oct 21 02:37:52 PDT 2003 
State-Changed-Why:  
Based on the core file that I examined for Aleksey and some testing, 
we're now pretty cetrain that this problem was caused by ipfw2 
writing to memory it had already freed. Upgrading to revieion 
1.6.2.18 of ip_fw2.c seems to have fixed the problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=51016 
>Unformatted:
