From nobody@FreeBSD.org  Wed Oct 26 17:38:10 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id E88F41065670
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 26 Oct 2011 17:38:10 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id BF2568FC12
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 26 Oct 2011 17:38:10 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p9QHcAff014362
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 26 Oct 2011 17:38:10 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p9QHcAsT014361;
	Wed, 26 Oct 2011 17:38:10 GMT
	(envelope-from nobody)
Message-Id: <201110261738.p9QHcAsT014361@red.freebsd.org>
Date: Wed, 26 Oct 2011 17:38:10 GMT
From: Fabian Keil <fk@fabiankeil.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [geom] Fatal trap 12: page fault while in kernel mode -- Stopped at atomic_subtract_int+0x4
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         162036
>Category:       kern
>Synopsis:       [geom] Fatal trap 12: page fault while in kernel mode -- Stopped at atomic_subtract_int+0x4
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 26 17:40:08 UTC 2011
>Closed-Date:    
>Last-Modified:  Fri Feb 22 17:10:00 UTC 2013
>Originator:     Fabian Keil
>Release:        HEAD
>Organization:
>Environment:
FreeBSD r500.local 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r+3084a02: Mon Oct 24 15:55:58 CEST 2011 fk@r500.local:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
I reproducible get the following (handtranscribed) panic
when sending an zfs snapshot to a geli provider based on a
labeled USB stick that disappears (due to a bug, or because
it's unplugged): 

Fatal trap 12: page fault while in kernel mode
cpuid = 0: apic id = 00
fault virtual address = 0x288
fault code	      = supervisor write data, page not present
instruction pointer   = 0x20:0xffffffff808e2984
stack pointer         = 0x28:0xffffff800023fba0
frame pointer         = 0x28:0xffffff800023fbb0
code segment          = base 0x0, limit 0xfffff, type 0x1b
                      = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags      = interrupt enabled, resume, IOPL = 0
current process       = 13 (g_up)
[ thread pid 13 tid 100010 ]
Stopped at    atomic_subtract_int+0x4: lock subl %esi,(%rdi)
db> where  
Tracing pid 13 tid 100010 td 0xfffffe00027998c0
atomic_subtract_int() at atomic_subtract_int+0x4
g_io_schdule_up() at g_io_schedule_up+0xa6
g_up_procbody() at g_up_procbody+0x5c
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800023fd00, rbp 0 ---

It seems to be important that ZFS is actually writing to the stick.
If the stick is unplugged while the operation is stalled for other
reasons, this particular panic doesn't seem to occur.

In case the zpool layout isn't clear, it's the same as used in:
http://www.freebsd.org/cgi/query-pr.cgi?pr=162010

While I end up in the debugger, dumping core doesn't work
and produces a double fault.

The panic above is from a custom kernel without WITNESS and
INVARIANTS (a bit older than the uname -a output above),
but enabling them doesn't seem to affect the trace
before the double fault.

I wasn't able to reproduce the panic by unplugging the stick
while writing to the pool using dd (but only tried once).
>How-To-Repeat:
Either use a flaky USB stick as geli-encrypted vdev and wait
for it to disappear by itself while a zfs receive is in progress
and actually writing to the device, or use a reliable stick and
unplug it manually while zpool iostat is showing writes.
>Fix:


>Release-Note:
>Audit-Trail:

From: Fabian Keil <fk@fabiankeil.de>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/162036: [geom] Fatal trap 12: page fault while in kernel
 mode -- Stopped at atomic_subtract_int+0x4
Date: Fri, 18 Nov 2011 13:08:43 +0100

 --Sig_/j78gnomfcy4BkUup5X1_Qyd
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 
 With the patch from:
 http://lists.freebsd.org/pipermail/freebsd-current/2011-November/029226.html
 and sources after r227637 dumping core works. I didn't test with r227637
 and without the scheduler-stopping patch.
 
 fk@r500 /usr/crash $kgdb kernel.6/kernel vmcore.6
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain condition=
 s.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 (da0:ugen7.2: <vendor 0x14cd> at usbus7 (disconnected)
 umass-sim0:0:0:0): Request completed with CAM_REQ_CMP_ERR
 umass0: at uhub7, port 2, addr 2 (disconnected)
 (da0:umass-sim0:0:0:0): Retrying command
 (da0:umass-sim0:0:0:0): Selection timeout
 (da0:umass-sim0:0:0:0): Retrying command
 (da0:umass-sim0:0:0:0): Selection timeout
 (da0:umass-sim0:0:0:0): Retrying command
 (da0:umass-sim0:0:0:0): Selection timeout
 (da0:umass-sim0:0:0:0): Retrying command
 (da0:umass-sim0:0:0:0): lost device - 1 outstanding
 GEOM_ELI(pass2:: umass-sim0:0:Crypto WRITE request failed (error=3D6).0: 0)=
 : lost device
 label/extreme.eli[WRITE(offset=3D2791427072, length=3D131072)]
 (pass2:umass-sim0:0:0:0): removing device entry
 GEOM_ELI(da0:umass-sim0:0:0:0): Error 6, Retries exhausted
 (da0:umass-sim0:0:0:0): oustanding 0
 : Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(offset=
 =3D2791558144, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2791689216, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2791886848, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792017920, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792148992, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792280064, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792411136, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792542208, length=3D131072)]
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2791296000, length=3D131072)]
 GEOM_ELI: g_eli_read_done() failed label/extreme.eli[READ(offset=3D270336, =
 length=3D8192)]
 GEOM_ELI: g_eli_read_done() failed label/extreme.eli[READ(offset=3D40226078=
 72, length=3D8192)]
 GEOM_ELI: g_eli_read_done() failed label/extreme.eli[READ(offset=3D40228700=
 16, length=3D8192)]
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792673280, length=3D131072)]
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792804352, length=3D131072)]
 zio_vdev_io_start: Setting zio->io_error to ENXIO for vdev extreme /dev/lab=
 el/extreme.eli
 GEOM_ELI: Crypto WRITE request failed (error=3D6). label/extreme.eli[WRITE(=
 offset=3D2792935424, length=3D131072)]
 
 Fatal trap 12: page fault while in kernel mode
 (da0:cpuid =3D 1; apic id =3D 01
 umass-sim0:0:fault virtual address      =3D 0x288
 0:fault code            =3D supervisor write data, page not present
 0): removing device entry
 instruction pointer     =3D 0x20:0xffffffff808dbef4
 stack pointer           =3D 0x28:0xffffff8000244ba0
 frame pointer           =3D 0x28:0xffffff8000244bb0
 code segment            =3D base 0x0, limit 0xfffff, type 0x1b
                         =3D DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
 current process         =3D 13 (g_up)
 
 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kerne=
 l/zfs.ko.symbols...done.
 done.
 [...]
 Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/k=
 ernel/fdescfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/fdescfs.ko
 #0  doadump (textdump=3D0) at /usr/src/sys/kern/kern_shutdown.c:267
 267             if (textdump && textdump_pending) {
 (kgdb) where
 #0  doadump (textdump=3D0) at /usr/src/sys/kern/kern_shutdown.c:267
 #1  0xffffffff80315990 in db_dump (dummy=3DVariable "dummy" is not availabl=
 e.
 ) at /usr/src/sys/ddb/db_command.c:537
 #2  0xffffffff803152c1 in db_command (last_cmdp=3D0xffffffff80df1800, cmd_t=
 able=3DVariable "cmd_table" is not available.
 ) at /usr/src/sys/ddb/db_command.c:448
 #3  0xffffffff80315510 in db_command_loop () at /usr/src/sys/ddb/db_command=
 .c:501
 #4  0xffffffff80317649 in db_trap (type=3DVariable "type" is not available.
 ) at /usr/src/sys/ddb/db_main.c:229
 #5  0xffffffff80690581 in kdb_trap (type=3D12, code=3D0, tf=3D0xffffff80002=
 44af0) at /usr/src/sys/kern/subr_kdb.c:626
 #6  0xffffffff808f48cd in trap_fatal (frame=3D0xffffff8000244af0, eva=3DVar=
 iable "eva" is not available.
 ) at /usr/src/sys/amd64/amd64/trap.c:814
 #7  0xffffffff808f4c36 in trap_pfault (frame=3D0xffffff8000244af0, usermode=
 =3D0) at /usr/src/sys/amd64/amd64/trap.c:735
 #8  0xffffffff808f50ff in trap (frame=3D0xffffff8000244af0) at /usr/src/sys=
 /amd64/amd64/trap.c:474
 #9  0xffffffff808def33 in calltrap () at /usr/src/sys/amd64/amd64/exception=
 .S:228
 #10 0xffffffff808dbef4 in atomic_subtract_int (p=3D0x288, v=3D1) at atomic.=
 h:289
 #11 0xffffffff805eeb16 in g_io_schedule_up (tp=3DVariable "tp" is not avail=
 able.
 ) at /usr/src/sys/geom/geom_io.c:679
 #12 0xffffffff805ef09c in g_up_procbody (arg=3DVariable "arg" is not availa=
 ble.
 ) at /usr/src/sys/geom/geom_kern.c:97
 #13 0xffffffff80628b6f in fork_exit (callout=3D0xffffffff805ef040 <g_up_pro=
 cbody>, arg=3D0x0, frame=3D0xffffff8000244c50) at /usr/src/sys/kern/kern_fo=
 rk.c:995
 #14 0xffffffff808df45e in fork_trampoline () at /usr/src/sys/amd64/amd64/ex=
 ception.S:602
 #15 0x0000000000000000 in ?? ()
 [...]
 #38 0x0000000000000000 in ?? ()
 #39 0xffffffff80e2cf80 in tdq_cpu ()
 #40 0x0000000000000001 in ?? ()
 #41 0x0000000000000000 in ?? ()
 #42 0xfffffe0002723460 in ?? ()
 #43 0xffffff8000244360 in ?? ()
 #44 0xffffff8000244308 in ?? ()
 #45 0xfffffe0004ca9000 in ?? ()
 ---Type <return> to continue, or q <return> to quit---
 #46 0xffffffff80683250 in sched_switch (td=3D0xffffffff805ef040, newtd=3D0x=
 0, flags=3DVariable "flags" is not available.
 ) at /usr/src/sys/kern/sched_ule.c:1853
 (kgdb) f 11
 #11 0xffffffff805eeb16 in g_io_schedule_up (tp=3DVariable "tp" is not avail=
 able.
 ) at /usr/src/sys/geom/geom_io.c:679
 679                             biodone(bp);
 Current language:  auto; currently c
 (kgdb) l
 674                             g_bioq_unlock(&g_bio_run_up);
 675                             THREAD_NO_SLEEPING();
 676                             CTR4(KTR_GEOM, "g_up biodone bp %p provider=
  %s off "
 677                                 "%jd len %ld", bp, bp->bio_to->name,
 678                                 bp->bio_offset, bp->bio_length);
 679                             biodone(bp);
 680                             THREAD_SLEEPING_OK();
 681                             continue;
 682                     }
 683                     CTR0(KTR_GEOM, "g_up going to sleep");
 (kgdb) p *bp
 $1 =3D {bio_cmd =3D 2 '\002', bio_flags =3D 2 '\002', bio_cflags =3D 0 '\0'=
 , bio_pflags =3D 0 '\0', bio_dev =3D 0x0, bio_disk =3D 0x0, bio_offset =3D =
 2792935424, bio_bcount =3D 131072,
   bio_data =3D 0xffffff800a581000 <Address 0xffffff800a581000 out of bounds=
 >, bio_error =3D 6, bio_resid =3D 131072, bio_done =3D 0xffffffff813597d0 <=
 g_eli_write_done>,
   bio_driver1 =3D 0x0, bio_driver2 =3D 0x0, bio_caller1 =3D 0x0, bio_caller=
 2 =3D 0x0, bio_queue =3D {tqe_next =3D 0x0, tqe_prev =3D 0xffffffff80e10f60=
 }, bio_attribute =3D 0x0,
   bio_from =3D 0xfffffe0012877d00, bio_to =3D 0xfffffe0004c65100, bio_lengt=
 h =3D 131072, bio_completed =3D 0, bio_children =3D 0, bio_inbed =3D 0, bio=
 _parent =3D 0xfffffe0017121bc8,
   bio_t0 =3D {sec =3D 201, frac =3D 2813861954672982723}, bio_task =3D 0, b=
 io_task_arg =3D 0x0, bio_classifier1 =3D 0x0, bio_classifier2 =3D 0x0, bio_=
 pblkno =3D 0}
 
 The "zio_vdev_io_start: ..." messages are the result of:
 
 commit 541719565e4daf8cb6f93896013ae57b867546c4
 Author: Fabian Keil <fk@fabiankeil.de>
 Date:   Mon Mar 14 18:27:13 2011 +0100
 
     Print a message when ENXIOing a device
 
 diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c b/sys/cdd=
 l/contrib/opensolaris/uts/common/fs/zfs/zio.c
 index b38fb0b..7269258 100644
 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c
 +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c
 @@ -2370,6 +2370,8 @@ zio_vdev_io_start(zio_t *zio)
                         return (ZIO_PIPELINE_STOP);
 
                 if (!vdev_accessible(vd, zio)) {
 +                       printf("%s: Setting zio->io_error to ENXIO for vdev=
  %s %s\n",
 +                              __func__, vd->vdev_spa->spa_name, vd->vdev_p=
 ath);
                         zio->io_error =3D ENXIO;
                         zio_interrupt(zio);
                         return (ZIO_PIPELINE_STOP);
 
 Fabian
 
 --Sig_/j78gnomfcy4BkUup5X1_Qyd
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iEYEARECAAYFAk7GStIACgkQSMVSH78upWNTBQCeO6B3YovPxFDqiarhtceE+WHK
 ys4AnjfIblRvmELA3zLdpeLCqgDY+QYZ
 =/0Et
 -----END PGP SIGNATURE-----
 
 --Sig_/j78gnomfcy4BkUup5X1_Qyd--

From: Fabian Keil <fk@fabiankeil.de>
To: bug-followup@FreeBSD.org
Cc:  
Subject: kern/162036: [geom] Fatal trap 12: page fault while in kernel mode
 -- Stopped at atomic_subtract_int+0x4
Date: Fri, 22 Feb 2013 17:59:13 +0100

 --Sig_/ygxLXwLUVd3B0XBB6PqOfsd
 Content-Type: multipart/mixed; boundary="MP_/KEOJ2TQS+uSTGDSq0NXG=p3"
 
 --MP_/KEOJ2TQS+uSTGDSq0NXG=p3
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: inline
 
 The problem still exists with a current kernel but apparently
 the atomic_subtract_int() call has been relocated.
 
 As the result there are now two related panics caused by
 g_eli_read_done() and (sometimes) g_eli_crypto_read_done()
 being called with:
 
 (kgdb) p bp->bio_parent->bio_to->geom->softc
 $1 =3D (void *) 0x0
 
 The attached patch adds the missing pointer checks and seems
 to prevent the panics for me.
 
 So far I haven't seen similar panics in the g_*_write_done()
 functions, although they look rather similar.
 
 Fabian
 
 --MP_/KEOJ2TQS+uSTGDSq0NXG=p3
 Content-Type: text/x-patch
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment;
  filename=kern-162036-Let-g_eli_-read_done-deal-with-lost-devices.patch
 
 =46rom 52a4fac396eaaaadd7a79a0d4311f38bdc3d3141 Mon Sep 17 00:00:00 2001
 From: Fabian Keil <fk@fabiankeil.de>
 Date: Tue, 19 Feb 2013 14:42:00 +0100
 Subject: [PATCH 1/2] Let g_eli_*read_done() deal with lost devices without
  causing panics
 
 Seems to fix kern/162036 for me.
 ---
  sys/geom/eli/g_eli.c         | 3 ++-
  sys/geom/eli/g_eli_privacy.c | 6 ++++--
  2 files changed, 6 insertions(+), 3 deletions(-)
 
 diff --git a/sys/geom/eli/g_eli.c b/sys/geom/eli/g_eli.c
 index 4e35297..24969b0 100644
 --- a/sys/geom/eli/g_eli.c
 +++ b/sys/geom/eli/g_eli.c
 @@ -183,7 +183,8 @@ g_eli_read_done(struct bio *bp)
  			pbp->bio_driver2 =3D NULL;
  		}
  		g_io_deliver(pbp, pbp->bio_error);
 -		atomic_subtract_int(&sc->sc_inflight, 1);
 +		if (sc !=3D NULL)
 +			atomic_subtract_int(&sc->sc_inflight, 1);
  		return;
  	}
  	mtx_lock(&sc->sc_queue_mtx);
 diff --git a/sys/geom/eli/g_eli_privacy.c b/sys/geom/eli/g_eli_privacy.c
 index f353832..938883a 100644
 --- a/sys/geom/eli/g_eli_privacy.c
 +++ b/sys/geom/eli/g_eli_privacy.c
 @@ -88,7 +88,8 @@ g_eli_crypto_read_done(struct cryptop *crp)
  			bp->bio_error =3D crp->crp_etype;
  	}
  	sc =3D bp->bio_to->geom->softc;
 -	g_eli_key_drop(sc, crp->crp_desc->crd_key);
 +	if (sc !=3D NULL)
 +		g_eli_key_drop(sc, crp->crp_desc->crd_key);
  	/*
  	 * Do we have all sectors already?
  	 */
 @@ -105,7 +106,8 @@ g_eli_crypto_read_done(struct cryptop *crp)
  	 * Read is finished, send it up.
  	 */
  	g_io_deliver(bp, bp->bio_error);
 -	atomic_subtract_int(&sc->sc_inflight, 1);
 +	if (sc !=3D NULL)
 +		atomic_subtract_int(&sc->sc_inflight, 1);
  	return (0);
  }
 =20
 --=20
 1.8.1.3
 
 
 --MP_/KEOJ2TQS+uSTGDSq0NXG=p3--
 
 --Sig_/ygxLXwLUVd3B0XBB6PqOfsd
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iQIcBAEBCAAGBQJRJ6PlAAoJEGkYIpGLojccDGMP/064LFc70F8+/Zn6ofTr1sgM
 4s0euYG6ebtB3nOFuiaEoNqfcbga+2+FSJBnxEogc1/MMA9r/CAsqee8HUvT3HGo
 psjeefN6K3k5Emk1GywLqZetgGzMbtqtZEZ5Qppd/Z7m2VzznnKZ+5UPx/tcxo7y
 krPELZjj2qRITBRPX5AN+xTmbYKCqtoOHKxXJIQN8Cf+9zoVhidaWu3Q2GG0AQo5
 lvcWVAWPQcRNoS+lRyyeX4dO+miOPSMUBcNvSmcGxFHjaQxnr61aR4kztu0DNkAz
 rJznXgoEb05f0EZ8bc5sP0nVnv1B+Uo1gEM5JHZ3IR0RugXLJv+nkaSzD0JM9iAd
 X7dmU798cpWiF4wXliIeTxCHeOKxLYeSFnPMams319FQrDyHrGiZl3jWcQh6PHbr
 0SjqB8GGAyb5cV7BkD6szQjJbErmRCcmwFUHEt9Ra13lFl83ZBPHEr2EYf8VfJnt
 2w4xMPx5Z1vu7YHk/cKliVYnl9uoplG9ob6HksDm7VNkxlTyeUqCAGNVNHK8nMi1
 pPy95t3KgM7rgdRlxIldMyJpl5gUX/wWhmieGV8FQ2vh0JiK67ude3KlDATk9pKW
 4BbiTmgfiAcMQvmwDsRI5FhlSK2KhwIFIRQUpW9LIkF0t1LHOHYboqHs4YCx/L3n
 NNw9YRVwLPUQb1G5q4ft
 =L8cH
 -----END PGP SIGNATURE-----
 
 --Sig_/ygxLXwLUVd3B0XBB6PqOfsd--
>Unformatted:
