From nobody@FreeBSD.org  Mon Apr  1 07:47:42 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	by hub.freebsd.org (Postfix) with ESMTP id 23A7EF8D
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  1 Apr 2013 07:47:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 1519762F
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  1 Apr 2013 07:47:42 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r317leLe042588
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 1 Apr 2013 07:47:40 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r317leus042587;
	Mon, 1 Apr 2013 07:47:40 GMT
	(envelope-from nobody)
Message-Id: <201304010747.r317leus042587@red.freebsd.org>
Date: Mon, 1 Apr 2013 07:47:40 GMT
From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: freebsd-gnats-submit@FreeBSD.org
Subject: zfs livelock (deadlock) with high write-to-disk load
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         177536
>Category:       kern
>Synopsis:       [zfs] zfs livelock (deadlock) with high write-to-disk load
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 01 07:50:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Thu Aug 15 05:40:00 UTC 2013
>Originator:     Martin Birgmeier
>Release:        9.1
>Organization:
MBi at home
>Environment:
FreeBSD hal.xyzzy 9.1-RELEASE FreeBSD 9.1-RELEASE #1: Fri Jan  4 12:37:44 CET 2013     root@v904.xyzzy:/usr/obj/.../hal/z/SRC/FreeBSD/release/9.1.0/sys/XYZZY_SMP  amd64
>Description:
I am using a program whose purpose is to efficiently copy disk partitions to image files stored on zfs. Multiple backups in time are kept using snapshots of the file system on which these image files reside.

The program uses mmap(2) to map an image file into memory, then reads the disk partition to be backup up in fixed-size blocks (default 128k). For each block, it first compares the already stored data with the new one using memcmp(3), and does a write using memcpy(3) only if there is a mismatch. In this way, only the blocks which have changed are really written, keeping the zfs snapshots small.

The files mapped in this way are typically several 100GB in size.

Problem: Every now and then, the above operation results in a live lock (deadlock) of the zfs server. Specifically today, the following could be observed:
- the program described above did not proceed any further after having read/written several dozen GB of data
- an 'ls -l' in the directory holding the image file was stuck
- there was constant low-level disk activity, but obviously no progress regarding writing data

Some environmental information:
- The pool consists of 6 partitions on 6 SATA disks in raidz2 configuration
- The pool has been created on 2010-10-22 and still has version 14
- / is an UFS partition on one of the SATA disks
- The server has 16 GB RAM
- There are no indications at all of any hardware disk failures
- other zfs file systems on the same pool could still be read/written to
- the machines responded quite normally to many requests including remote login etc.
- a VBoxHeadless session whose host user data was exclusively on the UFS root, with the vbox disk pointing to another raw partition on one of the disks was stuck
- From previous experience I knew that I had to hard reset the machine, since a shutdown would result in zfs sync (?) continuing forever after the UFS sync had already finished (by the way, it was a bad idea not to do a shutdown today, as I destroyed my .zhistory residing on the root UFS partition that way)
- Even if the problem does not occur, when the program described above has finished comparing/copying all blocks, several more minutes pass before the program finally finishes. During that time there is high disk activity.
- Under FreeBSD 8.2, the problem did not occur, and neither the symptoms described in the previous point (long time to program finish).

>How-To-Repeat:
The problem cannot be easily reproduced; however, from past experience it seems that the following increases the chances of it happening:
- a high write load
- after that write load has already lasted for a while, another rather high read load (possibly on another zfs file system of the same zpool)
- maybe activity on an unrelated partition (non-ZFS) on the disks which also carry the partitions used for the zpool

Unfortunately, most of this is just guesses and hopefully not too misleading.

>Fix:


>Release-Note:
>Audit-Trail:

From: Andriy Gapon <avg@FreeBSD.org>
To: bug-followup@FreeBSD.org, Martin.Birgmeier@aon.at
Cc:  
Subject: Re: kern/177536: zfs livelock (deadlock) with high write-to-disk
 load
Date: Mon, 01 Apr 2013 23:51:14 +0300

 Two points:
 - the problem may be (partially) fixed in stable/9
 - https://wiki.freebsd.org/AvgZfsDeadlockDebug
 
 -- 
 Andriy Gapon

From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/177536: zfs livelock (deadlock) with high write-to-disk
 load
Date: Tue, 02 Apr 2013 20:07:05 +0200

 Thanks for the pointer - which I knew about, but unfortunately found out
 too late that the kernel needs to be compiled with options STACK and/or
 options DDB/KDB for this to work.
 
 Could this information be added to that page?
 
 I already recompiled the kernel, but are ambivalent about whether I
 should now hope to see the deadlock again... ;-)
 
 One more thing: Could you point me to the SVN revision in head and/or
 stable/9 which presumably contains the (partial) fix?
 
 Regards,
 
 Martin
 
 On 04/01/13 22:51, Andriy Gapon wrote:
 > Two points:
 > - the problem may be (partially) fixed in stable/9
 > - https://wiki.freebsd.org/AvgZfsDeadlockDebug
 >
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Apr 3 20:14:16 UTC 2013 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=177536 

From: Andriy Gapon <avg@FreeBSD.org>
To: Martin Birgmeier <Martin.Birgmeier@aon.at>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/177536: zfs livelock (deadlock) with high write-to-disk
 load
Date: Thu, 04 Apr 2013 18:31:39 +0300

 on 02/04/2013 21:07 Martin Birgmeier said the following:
 > Thanks for the pointer - which I knew about, but unfortunately found out
 > too late that the kernel needs to be compiled with options STACK and/or
 > options DDB/KDB for this to work.
 > 
 > Could this information be added to that page?
 
 Well, I do not see any reason for people to not include at least STACK in their
 kernel.  It's in GENERIC too.  you are the first one to run into this kind of an
 issue :-)
 
 > I already recompiled the kernel, but are ambivalent about whether I
 > should now hope to see the deadlock again... ;-)
 > 
 > One more thing: Could you point me to the SVN revision in head and/or
 > stable/9 which presumably contains the (partial) fix?
 
 I believe that with r244626 from 2012-12-23 you should not experience hang on
 shutdown.
 
 > On 04/01/13 22:51, Andriy Gapon wrote:
 >> Two points:
 >> - the problem may be (partially) fixed in stable/9
 >> - https://wiki.freebsd.org/AvgZfsDeadlockDebug
 >>
 > 
 
 
 -- 
 Andriy Gapon

From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: bug-followup@FreeBSD.org, Andriy Gapon <avg@FreeBSD.org>
Cc:  
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Sat, 27 Apr 2013 11:40:16 +0200

 So it happened again... same system (9.1.0 release), except that the
 kernel has been recompiled with options DDB, KDB, and STACK.
 
 I ran procstat -kk -a (twice). Output can be found in
 http://members.aon.at/xyzzy/procstat.-kk.-a.1.gz and
 http://members.aon.at/xyzzy/procstat.-kk.-a.2.gz, respectively. I also
 started kgdb in script(1), executing "thread apply all bt" in it. Output
 can be found in http://members.aon.at/xyzzy/kgdb.thread.apply.all.bt.gz.
 
 More info on the "test case":
 - As described in the initial report, / is a UFS GPT partition on one of
 6 SATA disks. There exists a zpool "hal.1" on one (other) GPT partition
 on each of these disks.
 - VirtualBox is run by a user whose home dir is on one of the zfs file
 systems.
 - First, a big write load to another zfs file system of the same zpool
 was started (160 GB copy from a remote machine).
 - Then, 3 VBoxHeadless instances were started.
 ==> livelock on zfs
 - procstat run twice, then script + kgdb
 - copied output to another machine
 - shutdown the hung machine (via "shutdown -p")
 ==> "some processes would not die"
 ==> "syncing disks" executes until all zeros, then the system just sits
 there with continuous disk activity (obviously from zfs), shutdown does
 not proceed further
 - hard reset
 - on reboot: UFS file system check (no errors), ZFS starts fine and
 seems mostly unaffected (except of course that the 160 GB copy is truncated)
 
 An analysis would be appreciated, and also a hint whether I should
 switch to stable/9 instead.
 
 Regards,
 
 Martin
 

From: "Christopher D. Harrison" <harrison@biostat.wisc.edu>
To: bug-followup@FreeBSD.org, Martin.Birgmeier@aon.at
Cc:  
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Wed, 31 Jul 2013 11:36:26 -0500

 I have also been hitting this bug with 9.1-RELEASE.    My debugging has 
 got me to the point where I am almost certain the problem is in the MPS 
 driver in 9.1-RELEASE.   9.0-RELEASE would not deadlock under high I/O 
 but 9.1-RELEASE running the mps driver seems too.
 Martin:
 What device are your drives connected too?   Is it an LSI HBA?
 
 My device is:
 mps0@pci0:5:0:0:    class=0x010700 card=0x30c01000 chip=0x00641000 
 rev=0x02 hdr=0x00
      vendor     = 'LSI Logic / Symbios Logic'
      device     = 'SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor]'
      class      = mass storage
      subclass   = SAS
 igb2@pci0:4:0:0:    class=0x020000 card=0x10c915d9 chip=0x10c98086 
 rev=0x01 hdr=0x00
 
 I have 6 systems in production which experience this deadlock issue.   
 The deadlock appears most frequently under high load but I have 
 experienced it on systems which are idle too.
 
      -C
 

From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: "Christopher D. Harrison" <harrison@biostat.wisc.edu>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Wed, 31 Jul 2013 21:56:20 +0200

 No, this is not an LSI HBA, but rather
 
 atapci0: <JMicron JMB361 UDMA133 controller> port
 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f
 mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3
 atapci1: <AHCI SATA controller> at channel -1 on atapci0
 atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported
 ata2: <ATA channel> on atapci1
 ata3: <ATA channel> on atapci1
 ata4: <ATA channel> at channel 0 on atapci0
 atapci2: <ATI IXP700/800 SATA300 controller> port
 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f
 mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0
 atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported
 ata5: <ATA channel> at channel 0 on atapci2
 ata6: <ATA channel> at channel 1 on atapci2
 ata7: <ATA channel> at channel 2 on atapci2
 ata8: <ATA channel> at channel 3 on atapci2
 ata9: <ATA channel> at channel 4 on atapci2
 ata10: <ATA channel> at channel 5 on atapci2
 ada0 at ata5 bus 0 scbus3 target 0 lun 0
 ada1 at ata6 bus 0 scbus4 target 0 lun 0
 ada2 at ata7 bus 0 scbus5 target 0 lun 0
 ada3 at ata8 bus 0 scbus6 target 0 lun 0
 ada4 at ata9 bus 0 scbus7 target 0 lun 0
 ada5 at ata10 bus 0 scbus8 target 0 lun 0
 
 (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.)
 
 Regards,
 
 Martin
 

From: "Christopher D. Harrison" <harrison@biostat.wisc.edu>
To: Martin Birgmeier <Martin.Birgmeier@aon.at>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Wed, 31 Jul 2013 15:32:02 -0500

 what raidz level are you using: raidz2, raid0, raid1
 also what are the pool config (see below, my pool is called z):
 
 MOS Configuration:
          version: 28
          name: 'z'
          state: 0
          txg: 214028175
          pool_guid: 5334451366918939808
          hostid: 4266313884
          hostname: 'pisces.biostat.wisc.edu'
          hole_array[0]: 3
          vdev_children: 4
          vdev_tree:
              type: 'root'
              id: 0
              guid: 5334451366918939808
              children[0]:
                  type: 'raidz'
                  id: 0
                  guid: 10192098197416954465
                  nparity: 2
                  metaslab_array: 34
                  metaslab_shift: 37
                  ashift: 9
                  asize: 16003083796480
                  is_log: 0
                  create_txg: 4
                  children[0]:
                      type: 'disk'
                      id: 0
                      guid: 5442684633350386182
                      path: '/dev/da3p1'
                      phys_path: '/dev/da3p1'
                      whole_disk: 1
                      DTL: 2318341
                      create_txg: 4
                  children[1]:
                      type: 'disk'
                      id: 1
                      guid: 4184769488943604229
                      path: '/dev/da2p1'
                      phys_path: '/dev/da2p1'
                      whole_disk: 1
                      DTL: 2318340
                      create_txg: 4
                  children[2]:
                      type: 'disk'
                      id: 2
                      guid: 6673578809124996458
                      path: '/dev/da1p1'
                      phys_path: '/dev/da1p1'
                      whole_disk: 1
                      DTL: 2318358
                      create_txg: 4
                  children[3]:
                      type: 'disk'
                      id: 3
                      guid: 14565784372994613264
                      path: '/dev/da0p1'
                      phys_path: '/dev/da0p1'
                      whole_disk: 1
                      DTL: 2318357
                      create_txg: 4
                  children[4]:
                      type: 'disk'
                      id: 4
                      guid: 17127372890035360647
                      path: '/dev/da6p1'
                      phys_path: '/dev/da6p1'
                      whole_disk: 1
                      DTL: 2318356
                      create_txg: 4
                  children[5]:
                      type: 'disk'
                      id: 5
                      guid: 5667576937780231702
                      path: '/dev/label/disk05'
                      phys_path: '/dev/label/disk05'
                      whole_disk: 1
                      DTL: 26389621
                      create_txg: 4
                  children[6]:
                      type: 'disk'
                      id: 6
                      guid: 16545540667878636920
                      path: '/dev/da5p1'
                      phys_path: '/dev/da5p1'
                      whole_disk: 1
                      DTL: 2318354
                      create_txg: 4
                  children[7]:
                      type: 'disk'
                      id: 7
                      guid: 16180883375304336701
                      path: '/dev/da4p1'
                      phys_path: '/dev/da4p1'
                      whole_disk: 1
                      DTL: 2318353
                      create_txg: 4
              children[1]:
                  type: 'raidz'
                  id: 1
                  guid: 7000405778472969514
                  nparity: 2
                  metaslab_array: 32
                  metaslab_shift: 37
                  ashift: 9
                  asize: 16003083796480
                  is_log: 0
                  create_txg: 4
                  children[0]:
                      type: 'disk'
                      id: 0
                      guid: 11848332905648146086
                      path: '/dev/da10p1'
                      phys_path: '/dev/da10p1'
                      whole_disk: 1
                      DTL: 2318352
                      create_txg: 4
                  children[1]:
                      type: 'disk'
                      id: 1
                      guid: 17180048053682905420
                      path: '/dev/da9p1'
                      phys_path: '/dev/da9p1'
                      whole_disk: 1
                      DTL: 2318351
                      create_txg: 4
                  children[2]:
                      type: 'disk'
                      id: 2
                      guid: 14648362260613207535
                      path: '/dev/da8p1'
                      phys_path: '/dev/da8p1'
                      whole_disk: 1
                      DTL: 2318350
                      create_txg: 4
                  children[3]:
                      type: 'disk'
                      id: 3
                      guid: 2512073543886070959
                      path: '/dev/da7p1'
                      phys_path: '/dev/da7p1'
                      whole_disk: 1
                      DTL: 2318349
                      create_txg: 4
                  children[4]:
                      type: 'disk'
                      id: 4
                      guid: 9213113982598250625
                      path: '/dev/da14p1'
                      phys_path: '/dev/da14p1'
                      whole_disk: 1
                      DTL: 2318348
                      create_txg: 4
                  children[5]:
                      type: 'disk'
                      id: 5
                      guid: 14965308070554000358
                      path: '/dev/da13p1'
                      phys_path: '/dev/da13p1'
                      whole_disk: 1
                      DTL: 2318347
                      create_txg: 4
                  children[6]:
                      type: 'disk'
                      id: 6
                      guid: 8812761598924675500
                      path: '/dev/da12p1'
                      phys_path: '/dev/da12p1'
                      whole_disk: 1
                      DTL: 2318346
                      create_txg: 4
                  children[7]:
                      type: 'disk'
                      id: 7
                      guid: 15539718945154864246
                      path: '/dev/da11p1'
                      phys_path: '/dev/da11p1'
                      whole_disk: 1
                      DTL: 2318345
                      create_txg: 4
              children[2]:
                  type: 'raidz'
                  id: 2
                  guid: 12675549095184402399
                  nparity: 2
                  metaslab_array: 30
                  metaslab_shift: 37
                  ashift: 9
                  asize: 16003083796480
                  is_log: 0
                  create_txg: 4
                  children[0]:
                      type: 'disk'
                      id: 0
                      guid: 9157198169193782917
                      path: '/dev/da16p1'
                      phys_path: '/dev/da16p1'
                      whole_disk: 1
                      DTL: 35652327
                      create_txg: 4
                  children[1]:
                      type: 'disk'
                      id: 1
                      guid: 14494767130322696735
                      path: '/dev/da17p1'
                      phys_path: '/dev/da17p1'
                      whole_disk: 1
                      DTL: 2318365
                      create_txg: 4
                  children[2]:
                      type: 'disk'
                      id: 2
                      guid: 17589735305722725561
                      path: '/dev/da18p1'
                      phys_path: '/dev/da18p1'
                      whole_disk: 1
                      DTL: 2318364
                      create_txg: 4
                  children[3]:
                      type: 'disk'
                      id: 3
                      guid: 14307956723966398718
                      path: '/dev/da19p1'
                      phys_path: '/dev/da19p1'
                      whole_disk: 1
                      DTL: 2318363
                      create_txg: 4
                  children[4]:
                      type: 'disk'
                      id: 4
                      guid: 15965579855708087029
                      path: '/dev/da20p1'
                      phys_path: '/dev/da20p1'
                      whole_disk: 1
                      DTL: 2318362
                      create_txg: 4
                  children[5]:
                      type: 'disk'
                      id: 5
                      guid: 760237787742339099
                      path: '/dev/da21p1'
                      phys_path: '/dev/da21p1'
                      whole_disk: 1
                      DTL: 2318361
                      create_txg: 4
                  children[6]:
                      type: 'disk'
                      id: 6
                      guid: 2500372020986714069
                      path: '/dev/da22p1'
                      phys_path: '/dev/da22p1'
                      whole_disk: 1
                      DTL: 2318360
                      create_txg: 4
                  children[7]:
                      type: 'disk'
                      id: 7
                      guid: 4544065732274116075
                      path: '/dev/label/disk24'
                      phys_path: '/dev/label/disk24'
                      whole_disk: 1
                      DTL: 5572568
                      create_txg: 4
              children[3]:
                  type: 'hole'
                  id: 3
                  guid: 0
                  metaslab_array: 0
                  metaslab_shift: 17
                  ashift: 0
                  asize: 0
                  is_log: 0
                  is_hole: 1
 
 On 07/31/13 14:56, Martin Birgmeier wrote:
 > No, this is not an LSI HBA, but rather
 >
 > atapci0:<JMicron JMB361 UDMA133 controller>  port
 > 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f
 > mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3
 > atapci1:<AHCI SATA controller>  at channel -1 on atapci0
 > atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported
 > ata2:<ATA channel>  on atapci1
 > ata3:<ATA channel>  on atapci1
 > ata4:<ATA channel>  at channel 0 on atapci0
 > atapci2:<ATI IXP700/800 SATA300 controller>  port
 > 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f
 > mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0
 > atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported
 > ata5:<ATA channel>  at channel 0 on atapci2
 > ata6:<ATA channel>  at channel 1 on atapci2
 > ata7:<ATA channel>  at channel 2 on atapci2
 > ata8:<ATA channel>  at channel 3 on atapci2
 > ata9:<ATA channel>  at channel 4 on atapci2
 > ata10:<ATA channel>  at channel 5 on atapci2
 > ada0 at ata5 bus 0 scbus3 target 0 lun 0
 > ada1 at ata6 bus 0 scbus4 target 0 lun 0
 > ada2 at ata7 bus 0 scbus5 target 0 lun 0
 > ada3 at ata8 bus 0 scbus6 target 0 lun 0
 > ada4 at ata9 bus 0 scbus7 target 0 lun 0
 > ada5 at ata10 bus 0 scbus8 target 0 lun 0
 >
 > (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.)
 >
 > Regards,
 >
 > Martin
 >
 

From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: "Christopher D. Harrison" <harrison@biostat.wisc.edu>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Thu, 01 Aug 2013 07:20:50 +0200

 It is raidz2, using gpt partitions each covering half of six 2 TB disks.
 The remainder of these disks are mostly unused, with one containing a
 small UFS root for booting, and the others some small partitions which
 are used for various VirtualBox instances.
 
 Regards,
 
 Martin
 
 On 07/31/13 22:32, Christopher D. Harrison wrote:
 > what raidz level are you using: raidz2, raid0, raid1
 > also what are the pool config (see below, my pool is called z):
 >
 > MOS Configuration:
 >         version: 28
 >         name: 'z'
 >         state: 0
 >         txg: 214028175
 >         pool_guid: 5334451366918939808
 >         hostid: 4266313884
 >         hostname: 'pisces.biostat.wisc.edu'
 >         hole_array[0]: 3
 >         vdev_children: 4
 >         vdev_tree:
 >             type: 'root'
 >             id: 0
 >             guid: 5334451366918939808
 >             children[0]:
 >                 type: 'raidz'
 >                 id: 0
 >                 guid: 10192098197416954465
 >                 nparity: 2
 >                 metaslab_array: 34
 >                 metaslab_shift: 37
 >                 ashift: 9
 >                 asize: 16003083796480
 >                 is_log: 0
 >                 create_txg: 4
 >                 children[0]:
 >                     type: 'disk'
 >                     id: 0
 >                     guid: 5442684633350386182
 >                     path: '/dev/da3p1'
 >                     phys_path: '/dev/da3p1'
 >                     whole_disk: 1
 >                     DTL: 2318341
 >                     create_txg: 4
 >                 children[1]:
 >                     type: 'disk'
 >                     id: 1
 >                     guid: 4184769488943604229
 >                     path: '/dev/da2p1'
 >                     phys_path: '/dev/da2p1'
 >                     whole_disk: 1
 >                     DTL: 2318340
 >                     create_txg: 4
 >                 children[2]:
 >                     type: 'disk'
 >                     id: 2
 >                     guid: 6673578809124996458
 >                     path: '/dev/da1p1'
 >                     phys_path: '/dev/da1p1'
 >                     whole_disk: 1
 >                     DTL: 2318358
 >                     create_txg: 4
 >                 children[3]:
 >                     type: 'disk'
 >                     id: 3
 >                     guid: 14565784372994613264
 >                     path: '/dev/da0p1'
 >                     phys_path: '/dev/da0p1'
 >                     whole_disk: 1
 >                     DTL: 2318357
 >                     create_txg: 4
 >                 children[4]:
 >                     type: 'disk'
 >                     id: 4
 >                     guid: 17127372890035360647
 >                     path: '/dev/da6p1'
 >                     phys_path: '/dev/da6p1'
 >                     whole_disk: 1
 >                     DTL: 2318356
 >                     create_txg: 4
 >                 children[5]:
 >                     type: 'disk'
 >                     id: 5
 >                     guid: 5667576937780231702
 >                     path: '/dev/label/disk05'
 >                     phys_path: '/dev/label/disk05'
 >                     whole_disk: 1
 >                     DTL: 26389621
 >                     create_txg: 4
 >                 children[6]:
 >                     type: 'disk'
 >                     id: 6
 >                     guid: 16545540667878636920
 >                     path: '/dev/da5p1'
 >                     phys_path: '/dev/da5p1'
 >                     whole_disk: 1
 >                     DTL: 2318354
 >                     create_txg: 4
 >                 children[7]:
 >                     type: 'disk'
 >                     id: 7
 >                     guid: 16180883375304336701
 >                     path: '/dev/da4p1'
 >                     phys_path: '/dev/da4p1'
 >                     whole_disk: 1
 >                     DTL: 2318353
 >                     create_txg: 4
 >             children[1]:
 >                 type: 'raidz'
 >                 id: 1
 >                 guid: 7000405778472969514
 >                 nparity: 2
 >                 metaslab_array: 32
 >                 metaslab_shift: 37
 >                 ashift: 9
 >                 asize: 16003083796480
 >                 is_log: 0
 >                 create_txg: 4
 >                 children[0]:
 >                     type: 'disk'
 >                     id: 0
 >                     guid: 11848332905648146086
 >                     path: '/dev/da10p1'
 >                     phys_path: '/dev/da10p1'
 >                     whole_disk: 1
 >                     DTL: 2318352
 >                     create_txg: 4
 >                 children[1]:
 >                     type: 'disk'
 >                     id: 1
 >                     guid: 17180048053682905420
 >                     path: '/dev/da9p1'
 >                     phys_path: '/dev/da9p1'
 >                     whole_disk: 1
 >                     DTL: 2318351
 >                     create_txg: 4
 >                 children[2]:
 >                     type: 'disk'
 >                     id: 2
 >                     guid: 14648362260613207535
 >                     path: '/dev/da8p1'
 >                     phys_path: '/dev/da8p1'
 >                     whole_disk: 1
 >                     DTL: 2318350
 >                     create_txg: 4
 >                 children[3]:
 >                     type: 'disk'
 >                     id: 3
 >                     guid: 2512073543886070959
 >                     path: '/dev/da7p1'
 >                     phys_path: '/dev/da7p1'
 >                     whole_disk: 1
 >                     DTL: 2318349
 >                     create_txg: 4
 >                 children[4]:
 >                     type: 'disk'
 >                     id: 4
 >                     guid: 9213113982598250625
 >                     path: '/dev/da14p1'
 >                     phys_path: '/dev/da14p1'
 >                     whole_disk: 1
 >                     DTL: 2318348
 >                     create_txg: 4
 >                 children[5]:
 >                     type: 'disk'
 >                     id: 5
 >                     guid: 14965308070554000358
 >                     path: '/dev/da13p1'
 >                     phys_path: '/dev/da13p1'
 >                     whole_disk: 1
 >                     DTL: 2318347
 >                     create_txg: 4
 >                 children[6]:
 >                     type: 'disk'
 >                     id: 6
 >                     guid: 8812761598924675500
 >                     path: '/dev/da12p1'
 >                     phys_path: '/dev/da12p1'
 >                     whole_disk: 1
 >                     DTL: 2318346
 >                     create_txg: 4
 >                 children[7]:
 >                     type: 'disk'
 >                     id: 7
 >                     guid: 15539718945154864246
 >                     path: '/dev/da11p1'
 >                     phys_path: '/dev/da11p1'
 >                     whole_disk: 1
 >                     DTL: 2318345
 >                     create_txg: 4
 >             children[2]:
 >                 type: 'raidz'
 >                 id: 2
 >                 guid: 12675549095184402399
 >                 nparity: 2
 >                 metaslab_array: 30
 >                 metaslab_shift: 37
 >                 ashift: 9
 >                 asize: 16003083796480
 >                 is_log: 0
 >                 create_txg: 4
 >                 children[0]:
 >                     type: 'disk'
 >                     id: 0
 >                     guid: 9157198169193782917
 >                     path: '/dev/da16p1'
 >                     phys_path: '/dev/da16p1'
 >                     whole_disk: 1
 >                     DTL: 35652327
 >                     create_txg: 4
 >                 children[1]:
 >                     type: 'disk'
 >                     id: 1
 >                     guid: 14494767130322696735
 >                     path: '/dev/da17p1'
 >                     phys_path: '/dev/da17p1'
 >                     whole_disk: 1
 >                     DTL: 2318365
 >                     create_txg: 4
 >                 children[2]:
 >                     type: 'disk'
 >                     id: 2
 >                     guid: 17589735305722725561
 >                     path: '/dev/da18p1'
 >                     phys_path: '/dev/da18p1'
 >                     whole_disk: 1
 >                     DTL: 2318364
 >                     create_txg: 4
 >                 children[3]:
 >                     type: 'disk'
 >                     id: 3
 >                     guid: 14307956723966398718
 >                     path: '/dev/da19p1'
 >                     phys_path: '/dev/da19p1'
 >                     whole_disk: 1
 >                     DTL: 2318363
 >                     create_txg: 4
 >                 children[4]:
 >                     type: 'disk'
 >                     id: 4
 >                     guid: 15965579855708087029
 >                     path: '/dev/da20p1'
 >                     phys_path: '/dev/da20p1'
 >                     whole_disk: 1
 >                     DTL: 2318362
 >                     create_txg: 4
 >                 children[5]:
 >                     type: 'disk'
 >                     id: 5
 >                     guid: 760237787742339099
 >                     path: '/dev/da21p1'
 >                     phys_path: '/dev/da21p1'
 >                     whole_disk: 1
 >                     DTL: 2318361
 >                     create_txg: 4
 >                 children[6]:
 >                     type: 'disk'
 >                     id: 6
 >                     guid: 2500372020986714069
 >                     path: '/dev/da22p1'
 >                     phys_path: '/dev/da22p1'
 >                     whole_disk: 1
 >                     DTL: 2318360
 >                     create_txg: 4
 >                 children[7]:
 >                     type: 'disk'
 >                     id: 7
 >                     guid: 4544065732274116075
 >                     path: '/dev/label/disk24'
 >                     phys_path: '/dev/label/disk24'
 >                     whole_disk: 1
 >                     DTL: 5572568
 >                     create_txg: 4
 >             children[3]:
 >                 type: 'hole'
 >                 id: 3
 >                 guid: 0
 >                 metaslab_array: 0
 >                 metaslab_shift: 17
 >                 ashift: 0
 >                 asize: 0
 >                 is_log: 0
 >                 is_hole: 1
 >
 > On 07/31/13 14:56, Martin Birgmeier wrote:
 >> No, this is not an LSI HBA, but rather
 >>
 >> atapci0:<JMicron JMB361 UDMA133 controller>  port
 >> 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f
 >> mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3
 >> atapci1:<AHCI SATA controller>  at channel -1 on atapci0
 >> atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported
 >> ata2:<ATA channel>  on atapci1
 >> ata3:<ATA channel>  on atapci1
 >> ata4:<ATA channel>  at channel 0 on atapci0
 >> atapci2:<ATI IXP700/800 SATA300 controller>  port
 >> 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f
 >> mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0
 >> atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported
 >> ata5:<ATA channel>  at channel 0 on atapci2
 >> ata6:<ATA channel>  at channel 1 on atapci2
 >> ata7:<ATA channel>  at channel 2 on atapci2
 >> ata8:<ATA channel>  at channel 3 on atapci2
 >> ata9:<ATA channel>  at channel 4 on atapci2
 >> ata10:<ATA channel>  at channel 5 on atapci2
 >> ada0 at ata5 bus 0 scbus3 target 0 lun 0
 >> ada1 at ata6 bus 0 scbus4 target 0 lun 0
 >> ada2 at ata7 bus 0 scbus5 target 0 lun 0
 >> ada3 at ata8 bus 0 scbus6 target 0 lun 0
 >> ada4 at ata9 bus 0 scbus7 target 0 lun 0
 >> ada5 at ata10 bus 0 scbus8 target 0 lun 0
 >>
 >> (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.)
 >>
 >> Regards,
 >>
 >> Martin
 >>
 >
 >
 >
 

From: Christopher Harrison <harrison@biostat.wisc.edu>
To: Martin Birgmeier <Martin.Birgmeier@aon.at>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Thu, 01 Aug 2013 09:11:30 -0500

 Just last night I got the system to deadlock again.   I grabbed a kernel 
 bt but I realized I do no have full debug enabled.   I think our 
 problems are to different ones.   As my bt points directly to an LSI 
 adapter or kernel driver issue.
      -C
 On 8/1/13 12:20 AM, Martin Birgmeier wrote:
 > It is raidz2, using gpt partitions each covering half of six 2 TB disks.
 > The remainder of these disks are mostly unused, with one containing a
 > small UFS root for booting, and the others some small partitions which
 > are used for various VirtualBox instances.
 >
 > Regards,
 >
 > Martin
 >
 > On 07/31/13 22:32, Christopher D. Harrison wrote:
 >> what raidz level are you using: raidz2, raid0, raid1
 >> also what are the pool config (see below, my pool is called z):
 >>
 >> MOS Configuration:
 >>          version: 28
 >>          name: 'z'
 >>          state: 0
 >>          txg: 214028175
 >>          pool_guid: 5334451366918939808
 >>          hostid: 4266313884
 >>          hostname: 'pisces.biostat.wisc.edu'
 >>          hole_array[0]: 3
 >>          vdev_children: 4
 >>          vdev_tree:
 >>              type: 'root'
 >>              id: 0
 >>              guid: 5334451366918939808
 >>              children[0]:
 >>                  type: 'raidz'
 >>                  id: 0
 >>                  guid: 10192098197416954465
 >>                  nparity: 2
 >>                  metaslab_array: 34
 >>                  metaslab_shift: 37
 >>                  ashift: 9
 >>                  asize: 16003083796480
 >>                  is_log: 0
 >>                  create_txg: 4
 >>                  children[0]:
 >>                      type: 'disk'
 >>                      id: 0
 >>                      guid: 5442684633350386182
 >>                      path: '/dev/da3p1'
 >>                      phys_path: '/dev/da3p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318341
 >>                      create_txg: 4
 >>                  children[1]:
 >>                      type: 'disk'
 >>                      id: 1
 >>                      guid: 4184769488943604229
 >>                      path: '/dev/da2p1'
 >>                      phys_path: '/dev/da2p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318340
 >>                      create_txg: 4
 >>                  children[2]:
 >>                      type: 'disk'
 >>                      id: 2
 >>                      guid: 6673578809124996458
 >>                      path: '/dev/da1p1'
 >>                      phys_path: '/dev/da1p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318358
 >>                      create_txg: 4
 >>                  children[3]:
 >>                      type: 'disk'
 >>                      id: 3
 >>                      guid: 14565784372994613264
 >>                      path: '/dev/da0p1'
 >>                      phys_path: '/dev/da0p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318357
 >>                      create_txg: 4
 >>                  children[4]:
 >>                      type: 'disk'
 >>                      id: 4
 >>                      guid: 17127372890035360647
 >>                      path: '/dev/da6p1'
 >>                      phys_path: '/dev/da6p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318356
 >>                      create_txg: 4
 >>                  children[5]:
 >>                      type: 'disk'
 >>                      id: 5
 >>                      guid: 5667576937780231702
 >>                      path: '/dev/label/disk05'
 >>                      phys_path: '/dev/label/disk05'
 >>                      whole_disk: 1
 >>                      DTL: 26389621
 >>                      create_txg: 4
 >>                  children[6]:
 >>                      type: 'disk'
 >>                      id: 6
 >>                      guid: 16545540667878636920
 >>                      path: '/dev/da5p1'
 >>                      phys_path: '/dev/da5p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318354
 >>                      create_txg: 4
 >>                  children[7]:
 >>                      type: 'disk'
 >>                      id: 7
 >>                      guid: 16180883375304336701
 >>                      path: '/dev/da4p1'
 >>                      phys_path: '/dev/da4p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318353
 >>                      create_txg: 4
 >>              children[1]:
 >>                  type: 'raidz'
 >>                  id: 1
 >>                  guid: 7000405778472969514
 >>                  nparity: 2
 >>                  metaslab_array: 32
 >>                  metaslab_shift: 37
 >>                  ashift: 9
 >>                  asize: 16003083796480
 >>                  is_log: 0
 >>                  create_txg: 4
 >>                  children[0]:
 >>                      type: 'disk'
 >>                      id: 0
 >>                      guid: 11848332905648146086
 >>                      path: '/dev/da10p1'
 >>                      phys_path: '/dev/da10p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318352
 >>                      create_txg: 4
 >>                  children[1]:
 >>                      type: 'disk'
 >>                      id: 1
 >>                      guid: 17180048053682905420
 >>                      path: '/dev/da9p1'
 >>                      phys_path: '/dev/da9p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318351
 >>                      create_txg: 4
 >>                  children[2]:
 >>                      type: 'disk'
 >>                      id: 2
 >>                      guid: 14648362260613207535
 >>                      path: '/dev/da8p1'
 >>                      phys_path: '/dev/da8p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318350
 >>                      create_txg: 4
 >>                  children[3]:
 >>                      type: 'disk'
 >>                      id: 3
 >>                      guid: 2512073543886070959
 >>                      path: '/dev/da7p1'
 >>                      phys_path: '/dev/da7p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318349
 >>                      create_txg: 4
 >>                  children[4]:
 >>                      type: 'disk'
 >>                      id: 4
 >>                      guid: 9213113982598250625
 >>                      path: '/dev/da14p1'
 >>                      phys_path: '/dev/da14p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318348
 >>                      create_txg: 4
 >>                  children[5]:
 >>                      type: 'disk'
 >>                      id: 5
 >>                      guid: 14965308070554000358
 >>                      path: '/dev/da13p1'
 >>                      phys_path: '/dev/da13p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318347
 >>                      create_txg: 4
 >>                  children[6]:
 >>                      type: 'disk'
 >>                      id: 6
 >>                      guid: 8812761598924675500
 >>                      path: '/dev/da12p1'
 >>                      phys_path: '/dev/da12p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318346
 >>                      create_txg: 4
 >>                  children[7]:
 >>                      type: 'disk'
 >>                      id: 7
 >>                      guid: 15539718945154864246
 >>                      path: '/dev/da11p1'
 >>                      phys_path: '/dev/da11p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318345
 >>                      create_txg: 4
 >>              children[2]:
 >>                  type: 'raidz'
 >>                  id: 2
 >>                  guid: 12675549095184402399
 >>                  nparity: 2
 >>                  metaslab_array: 30
 >>                  metaslab_shift: 37
 >>                  ashift: 9
 >>                  asize: 16003083796480
 >>                  is_log: 0
 >>                  create_txg: 4
 >>                  children[0]:
 >>                      type: 'disk'
 >>                      id: 0
 >>                      guid: 9157198169193782917
 >>                      path: '/dev/da16p1'
 >>                      phys_path: '/dev/da16p1'
 >>                      whole_disk: 1
 >>                      DTL: 35652327
 >>                      create_txg: 4
 >>                  children[1]:
 >>                      type: 'disk'
 >>                      id: 1
 >>                      guid: 14494767130322696735
 >>                      path: '/dev/da17p1'
 >>                      phys_path: '/dev/da17p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318365
 >>                      create_txg: 4
 >>                  children[2]:
 >>                      type: 'disk'
 >>                      id: 2
 >>                      guid: 17589735305722725561
 >>                      path: '/dev/da18p1'
 >>                      phys_path: '/dev/da18p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318364
 >>                      create_txg: 4
 >>                  children[3]:
 >>                      type: 'disk'
 >>                      id: 3
 >>                      guid: 14307956723966398718
 >>                      path: '/dev/da19p1'
 >>                      phys_path: '/dev/da19p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318363
 >>                      create_txg: 4
 >>                  children[4]:
 >>                      type: 'disk'
 >>                      id: 4
 >>                      guid: 15965579855708087029
 >>                      path: '/dev/da20p1'
 >>                      phys_path: '/dev/da20p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318362
 >>                      create_txg: 4
 >>                  children[5]:
 >>                      type: 'disk'
 >>                      id: 5
 >>                      guid: 760237787742339099
 >>                      path: '/dev/da21p1'
 >>                      phys_path: '/dev/da21p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318361
 >>                      create_txg: 4
 >>                  children[6]:
 >>                      type: 'disk'
 >>                      id: 6
 >>                      guid: 2500372020986714069
 >>                      path: '/dev/da22p1'
 >>                      phys_path: '/dev/da22p1'
 >>                      whole_disk: 1
 >>                      DTL: 2318360
 >>                      create_txg: 4
 >>                  children[7]:
 >>                      type: 'disk'
 >>                      id: 7
 >>                      guid: 4544065732274116075
 >>                      path: '/dev/label/disk24'
 >>                      phys_path: '/dev/label/disk24'
 >>                      whole_disk: 1
 >>                      DTL: 5572568
 >>                      create_txg: 4
 >>              children[3]:
 >>                  type: 'hole'
 >>                  id: 3
 >>                  guid: 0
 >>                  metaslab_array: 0
 >>                  metaslab_shift: 17
 >>                  ashift: 0
 >>                  asize: 0
 >>                  is_log: 0
 >>                  is_hole: 1
 >>
 >> On 07/31/13 14:56, Martin Birgmeier wrote:
 >>> No, this is not an LSI HBA, but rather
 >>>
 >>> atapci0:<JMicron JMB361 UDMA133 controller>  port
 >>> 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f
 >>> mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3
 >>> atapci1:<AHCI SATA controller>  at channel -1 on atapci0
 >>> atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported
 >>> ata2:<ATA channel>  on atapci1
 >>> ata3:<ATA channel>  on atapci1
 >>> ata4:<ATA channel>  at channel 0 on atapci0
 >>> atapci2:<ATI IXP700/800 SATA300 controller>  port
 >>> 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f
 >>> mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0
 >>> atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported
 >>> ata5:<ATA channel>  at channel 0 on atapci2
 >>> ata6:<ATA channel>  at channel 1 on atapci2
 >>> ata7:<ATA channel>  at channel 2 on atapci2
 >>> ata8:<ATA channel>  at channel 3 on atapci2
 >>> ata9:<ATA channel>  at channel 4 on atapci2
 >>> ata10:<ATA channel>  at channel 5 on atapci2
 >>> ada0 at ata5 bus 0 scbus3 target 0 lun 0
 >>> ada1 at ata6 bus 0 scbus4 target 0 lun 0
 >>> ada2 at ata7 bus 0 scbus5 target 0 lun 0
 >>> ada3 at ata8 bus 0 scbus6 target 0 lun 0
 >>> ada4 at ata9 bus 0 scbus7 target 0 lun 0
 >>> ada5 at ata10 bus 0 scbus8 target 0 lun 0
 >>>
 >>> (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.)
 >>>
 >>> Regards,
 >>>
 >>> Martin
 >>>
 >>
 >>
 

From: Xin Li <delphij@delphij.net>
To: bug-followup@FreeBSD.org, Martin.Birgmeier@aon.at
Cc:  
Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk
 load
Date: Wed, 14 Aug 2013 22:37:22 -0700

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA256
 
 I can't say for sure based on the information you have provided, but
 stable/9 have a lot of fixes that could be related.
 
 One problem we found in field that is not easy to reproduce is that
 there is a lost interrupt issue in FreeBSD core.  This was fixed in
 r253184 (post-9.1-RELEASE and before 9.2, the fix will be part of the
 upcoming FreeBSD 9.2-RELEASE):
 
 http://svnweb.freebsd.org/base/stable/9/sys/kern/kern_intr.c?r1=249402&r2=253184&view=patch
 
 The symptom of this issue is that you basically see a lot of processes
 blocking on zio->zio_cv, while there is no disk activity.  However,
 the information you have provided can neither prove or deny my guess.
  I post the information here so people are aware of this issue if they
 search these terms.
 
 Cheers,
 -----BEGIN PGP SIGNATURE-----
 
 iQEcBAEBCAAGBQJSDGkSAAoJEG80Jeu8UPuzrBwH/01K+aR4WVN6SfK8i9WpjxhW
 nhv0dWP/Q6+XStmNJcthNfT3Ox2lR+yNYhPif3uUtOYGjlEZX/t9QpptXo86Xwji
 kN7cPZwELqE+0FS10uWtlaOYexqD7eg2QK2oj75hTfcKr5XfJWEVgWxlj+Iv6eAI
 6JkNd7mdvOJReAgHVyt6mMdR/p4xhDP+/3lJXg1YoINYVfWzpolLAZcxdPxAxF+O
 4rYM+BA/pcpglbK7CFd59tIUOeUaeOPHPVBeqw1NvMHw2EfJANx8PSyiu2jWdSeQ
 264XgS3ZuphVWFYuapt2nQdkLaBFCQyfCU/3hu1lNQjD+6iwWoKIwtae32kmN+0=
 =cwdX
 -----END PGP SIGNATURE-----
>Unformatted:
