From eugen@D00015.dialonly.kemerovo.su  Sat Mar 16 00:11:58 2002
Return-Path: <eugen@D00015.dialonly.kemerovo.su>
Received: from D00015.dialonly.kemerovo.su (D00015.dialonly.kemerovo.su [213.184.66.105])
	by hub.freebsd.org (Postfix) with ESMTP id 60B9D37B496
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 16 Mar 2002 00:11:52 -0800 (PST)
Received: (from eugen@localhost)
	by D00015.dialonly.kemerovo.su (8.11.6/8.11.6) id g2G880k17109;
	Sat, 16 Mar 2002 15:08:00 +0700 (KRAT)
	(envelope-from eugen)
Message-Id: <200203160808.g2G880k17109@D00015.dialonly.kemerovo.su>
Date: Sat, 16 Mar 2002 15:08:00 +0700 (KRAT)
From: Eugene Grosbein <eugen@grosbein.pp.ru>
Reply-To: Eugene Grosbein <eugen@grosbein.pp.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         35969
>Category:       kern
>Synopsis:       kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    peter
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Mar 16 00:20:01 PST 2002
>Closed-Date:    Tue Mar 19 20:14:55 PST 2002
>Last-Modified:  Wed Mar 20 10:20:01 PST 2002
>Originator:     Eugene Grosbein
>Release:        FreeBSD 4.5-STABLE i386
>Organization:
Svyaz Service
>Environment:
System: FreeBSD D00015.dialonly.kemerovo.su 4.5-STABLE FreeBSD 4.5-STABLE #4: Sat Mar 9 13:41:04 KRAT 2002 eu@D00015.dialonly.kemerovo.su:/usr/local/obj/usr/local/src/sys/DADV i386
	
	pseudo-device ppp 2
	options	PPP_DEFLATE
	options	PPP_BSDCOMP
	options	PPP_FILTER

>Description:
	
	Using pppd with compression 'deflate' produces kernel panics
	with 4.5-STABLE at least after 1 March 2002.
	Some investigation performed by Kirk McKusick shows
	corription of kernel structures. 
	
	4.5-RELEASE doesn't have this problem. 
	Really the problem appeared much later but I can't say exact date.

>How-To-Repeat:
	
	1. Build kernel with 
	
	pseudo-device ppp 2
	options	PPP_DEFLATE
	options	PPP_BSDCOMP
	options	PPP_FILTER

	For debugging purposes, add 
	
	makeoptions DEBUG=-g
	options		DDB

	and enable creating of crashdumps via /etc/rc.conf (dumpdev parameter)

	2. Create user 'pppuser' with shell '/usr/sbin/pppd', group dialer,
	   create empty ~pppuser/.hushlogin, empty /etc/ppp/options and say:
	
	# cat >~pppuser/.ppprc <<EOF
	local
	-crtscts
	172.20.0.1:172.20.0.2
	EOF

	3. The problem was first discovered with dialin service,
	so you can attach an analog modem with phone line and dial 
	to the system. Use client supporting 'deflate' compression method.
	Make some traffic and you'll get some kind of kernel panic shortly.

	I've found that there is no need to use hardware modem to trigger
	this bug. It's can be shown even with tunnelling 
	PPP over TCP connection via loopback.

	I personally used this small helper to make connection
	pppd <-> pppd over rlogin TCP stream:

#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <limits.h>
#include <stdio.h>
#include <libutil.h>

int main(int argc, char* argv[])
{
    char *me=argv[0];
    char ptyname[PATH_MAX];
    int master;
    
    ++argv; --argc;
    if(argc<1) {
	fprintf(stderr,"usage: %s command [params]\n",me);
	exit(1);
    }
    
    switch(forkpty(&master,ptyname,NULL,NULL)) {
    case -1:	/* failure */
	    fprintf(stderr,"%s: cannot forkpty: %s\n",me,strerror(errno));
	    exit(1);
    case  0:	/* slave, new pty - running process*/
	    execlp("/usr/sbin/pppd","/usr/sbin/pppd",NULL);
	    _exit(1);
	    /* NOT REACHED */
    default:	/* master */
	    dup2(master, 0);
	    dup2(master, 1);
	    dup2(master, 2);
	    if (master > 2) close(master);
	    execvp(*argv,argv);
    }
    return 1;
}

/* END OF FILE */

	Compile this: cc runatpty.c -o runatpty -lutil
	Then enable 'rlogin' in /etc/inetd.conf and say:

	# echo 'localhost +' > ~pppuser/.rhosts
	# chown pppuser ~pppuser/.rhosts
	# chmod 600 ~pppuser/.rhosts

	Test your settings using 'rlogin -8E -l pppuser localhost'.
	You should be able to login without password and see pppd
	trying to talk PPP with you. It will timeout and fail.
	Now say:
	
	# ./runatpty rlogin -8E -l pppuser localhost
	
	It will not go background, leave it run. You should have two
	up and running ppp interfaces - both sides of PPP connection
	over rlogin TCP stream. Try using it, make some traffic
	and you'll get kernel panic.

>Fix:

	Unknown for me.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->peter 
Responsible-Changed-By: cjc 
Responsible-Changed-When: Sat Mar 16 11:48:25 PST 2002 
Responsible-Changed-Why:  
peter is listed as the pppd(8) maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=35969 

From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc: dillon@FreeBSD.org
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
Date: Sun, 17 Mar 2002 12:58:34 +0700

 I've investigated this case little further.
 I've used cvsup to downgrade my home system to
 tag=RELENG_4 date=2002.02.13.00.00.00
 
 Now it does not panic.
 So it's possible that last commit to src/sys/net/if_ppp.c broke something:
 
 revision 1.67.2.2
 date: 2002/02/13 00:43:10;  author: dillon;  state: Exp;  lines: +4 -3
 MFC remove the MFREE() mbuf macro and cleanup twists in related code.
 
 Eugene Grosbein

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Eugene Grosbein <eugen@grosbein.pp.ru>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
Date: Sat, 16 Mar 2002 22:47:26 -0800 (PST)

 :I've investigated this case little further.
 :I've used cvsup to downgrade my home system to
 :tag=RELENG_4 date=2002.02.13.00.00.00
 :
 :Now it does not panic.
 :So it's possible that last commit to src/sys/net/if_ppp.c broke something:
 :
 :revision 1.67.2.2
 :date: 2002/02/13 00:43:10;  author: dillon;  state: Exp;  lines: +4 -3
 :MFC remove the MFREE() mbuf macro and cleanup twists in related code.
 :
 :Eugene Grosbein
 
     Hmm.  That change should not have made any difference, but it's easy
     to test... upgrade your system to 2002.02.13.00.50.00 (just after
     my commit) and see if the bug comes back.  If it does I will run through
     that entire commit.  I perused it just now and did not see anything
     specifically wrong.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
Date: Mon, 18 Mar 2002 22:17:54 +0700

 On Sat, Mar 16, 2002 at 10:47:26PM -0800, Matthew Dillon wrote:
 
 > :Now it does not panic.
 > :So it's possible that last commit to src/sys/net/if_ppp.c broke something:
 > :
 > :revision 1.67.2.2
 > :date: 2002/02/13 00:43:10;  author: dillon;  state: Exp;  lines: +4 -3
 > :MFC remove the MFREE() mbuf macro and cleanup twists in related code.
 > 
 >     Hmm.  That change should not have made any difference, but it's easy
 >     to test... upgrade your system to 2002.02.13.00.50.00 (just after
 >     my commit) and see if the bug comes back.  If it does I will run through
 >     that entire commit.  I perused it just now and did not see anything
 >     specifically wrong.
 
 I was wrong, your commit wasn't the cause of the panics.
 Now I believe I've isolated the moment of breakage.
 The system updated (or downgraded) to tag=RELENG_4 date=2002.02.22.00.00.00
 does not panic using PPP_DEFLATE but suffers from stalled connections.
 One have to send USR2 to pppd to revive PPP link.
 man pppd says:
 
 SIGUSR2
               This signal causes pppd to renegotiate compression.
               This  can  be useful to re-enable compression after
               it has been disabled as a result of a fatal  decom-
               pression error.  (Fatal decompression errors gener-
               ally indicate a bug in  one  or  other  implementa-
               tion.)
 
 The system updated to tag=RELENG_4 date=2002.02.22.03.00.00
 quickly panics if PPP_DEFLATE is used. It seems it panics due to
 corrupted kernel area.
 
 There was the only commit to RELENG_4 during this period,
 it patched src/lib/libz/infblock.c and src/sys/net/zlib.c
 The latter file is used by kernel ppp driver.
 
 I'd like to investigate this further but I'm definitly not a kernel hacker.
 The only workaround I've found is using 'nodeflate' option of pppd.
 This effectivly disables usage of deflate compression and eliminates panics
 even for kernel built from sources after 2002.02.22.03.00.00
 
 Eugene Grosbein

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Eugene Grosbein <eugen@grosbein.pp.ru>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
Date: Mon, 18 Mar 2002 15:58:38 -0800 (PST)

 :I was wrong, your commit wasn't the cause of the panics.
 :Now I believe I've isolated the moment of breakage.
 :The system updated (or downgraded) to tag=RELENG_4 date=2002.02.22.00.00.00
 :does not panic using PPP_DEFLATE but suffers from stalled connections.
 :One have to send USR2 to pppd to revive PPP link.
 :man pppd says:
 :
 :SIGUSR2
 :...
 :              tion.)
 :
 :The system updated to tag=RELENG_4 date=2002.02.22.03.00.00
 :quickly panics if PPP_DEFLATE is used. It seems it panics due to
 :corrupted kernel area.
 :
 :There was the only commit to RELENG_4 during this period,
 :it patched src/lib/libz/infblock.c and src/sys/net/zlib.c
 :The latter file is used by kernel ppp driver.
 :
 :I'd like to investigate this further but I'm definitly not a kernel hacker.
 :The only workaround I've found is using 'nodeflate' option of pppd.
 :This effectivly disables usage of deflate compression and eliminates panics
 :even for kernel built from sources after 2002.02.22.03.00.00
 :
 :Eugene Grosbein
 
     Excellent work Eugene!
 
 cvs diff -j "RELENG_4:2002/02/22 00:00:00 GMT" -j "RELENG_4:2002/02/22 03:00:00 GMT"
 
     Indeed the only change is to the zlib code.  
 
     Ok, I've spent a little while understanding the code. I  believe
     I found the bug!
 
     If you look at around line 3954 you will see this:
 
         s->sub.decode.codes = c;
         s->sub.decode.tl = tl;
         s->sub.decode.td = td;
       }
       ZFREE(z, s->sub.trees.blens);
       s->mode = CODES;
 
     sub.trees is a union with sub.decode, so when s->sub.decode = ...
     is done, it tromps all over the s->sub.trees elements.  Thus
     this ZFREE is in the wrong place.
 
     Plese bring your tree back to the latest -stable and apply the
     patch I include below, then tell me if it fixes your crashes.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>
 
 Index: zlib.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/net/zlib.c,v
 retrieving revision 1.10.2.1
 diff -u -r1.10.2.1 zlib.c
 --- zlib.c	22 Feb 2002 02:49:31 -0000	1.10.2.1
 +++ zlib.c	18 Mar 2002 23:57:59 -0000
 @@ -3951,11 +3951,15 @@
            r = Z_MEM_ERROR;
            LEAVE
          }
 +	/*
 +	 * this ZFREE must occur *BEFORE* we mess with sub.decode, because
 +	 * sub.trees is union'd with sub.decode.
 +	 */
 +        ZFREE(z, s->sub.trees.blens);
          s->sub.decode.codes = c;
          s->sub.decode.tl = tl;
          s->sub.decode.td = td;
        }
 -      ZFREE(z, s->sub.trees.blens);
        s->mode = CODES;
      case CODES:
        UPDATE

From: Giorgos Keramidas <keramida@freebsd.org>
To: Eugene Grosbein <eugen@grosbein.pp.ru>
Cc: bug-followup@freebsd.org
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel
 panics; PPP_BSDCOMP sometimes procudes stalled connections
Date: Tue, 19 Mar 2002 12:13:07 +0200 (EET)

 Adding to audit trail:
 |
 | Date: Sat, 16 Mar 2002 15:46:36 +0700
 | From: Eugene Grosbein <eugen@grosbein.pp.ru>
 | Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes ...
 |
 | Forgot to mention that 'nodeflate' in ppp options file disables
 | this method of compression and kernel panics do not appear.
 | BSDCOMP generally works as expected but sometimes TCP connections hang
 | (even over loopback) and traffic unfreezes only when pppd receives SIGUSR2.
 |
 | Eugene Grosbein
 

From: Eugene Grosbein <eugen@svzserv.kemerovo.su>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: Eugene Grosbein <eugen@grosbein.pp.ru>,
	FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; 
 PPP_BSDCOMP sometimes procudes stalled connections
Date: Wed, 20 Mar 2002 10:05:16 +0700

 Matthew Dillon wrote:
 
 >     Plese bring your tree back to the latest -stable and apply the
 >     patch I include below, then tell me if it fixes your crashes.
 
 Well, my home system with PPP over loopback that I used to locate breakage
 does not panic with this patch. Today I'll test my dialin server.
 
 Eugene Grosbein
State-Changed-From-To: open->closed 
State-Changed-By: dillon 
State-Changed-When: Tue Mar 19 20:12:57 PST 2002 
State-Changed-Why:  
Believed to be fixed w/1.12 & 1.10.2.2 commit to src/net/zlib.z (current 
and stable).  Sent email to security and jedgar in regards to additional 
MFCs.  Awaiting final confirmation from Eugene but closing the ticket now.   
(also note: Ian Dowse came up with the same patch while independantly  
investigating a similar problem). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=35969 

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Eugene Grosbein <eugen@svzserv.kemerovo.su>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; 
 PPP_BSDCOMP sometimes procudes stalled connections
Date: Tue, 19 Mar 2002 20:11:35 -0800 (PST)

 :
 :Matthew Dillon wrote:
 :
 :>     Plese bring your tree back to the latest -stable and apply the
 :>     patch I include below, then tell me if it fixes your crashes.
 :
 :Well, my home system with PPP over loopback that I used to locate breakage
 :does not panic with this patch. Today I'll test my dialin server.
 :
 :Eugene Grosbein
 
     Excellent. I am going to go ahead and commit it.  Ian Dowse also
     came up with the exact same patch independantly so I am confident 
     that it is correct.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>

From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: Matthew Dillon <dillon@apollo.backplane.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG,
	Kirk McKusick <mckusick@mckusick.com>
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections
Date: Wed, 20 Mar 2002 21:35:03 +0700

 >     Plese bring your tree back to the latest -stable and apply the
 >     patch I include below, then tell me if it fixes your crashes.
 
 My dialin server is revived now. It works as expected.
 I thank Matthes Dillon, Kirk McKusic and other people of stable@freebsd.org
 for their help.
 
 It seems, that commit affected src/lib/libz/infblock.c 
 and it's possible it still contatins the bug that is fixed now in the kernel.
 I talk about this:
 
 Index: infblock.c
 ===================================================================
 RCS file: /home/ncvs/src/lib/libz/infblock.c,v
 retrieving revision 1.1.1.4
 retrieving revision 1.1.1.4.6.1
 diff -u -r1.1.1.4 -r1.1.1.4.6.1
 --- infblock.c	10 Jan 1999 09:46:55 -0000	1.1.1.4
 +++ infblock.c	22 Feb 2002 02:48:40 -0000	1.1.1.4.6.1
 @@ -3,6 +3,9 @@
   * For conditions of distribution and use, see copyright notice in zlib.h 
   */
  
 +#include <sys/cdefs.h>
 +__FBSDID("$FreeBSD: src/lib/libz/infblock.c,v 1.1.1.4.6.1 2002/02/22 02:48:40 jedgar Exp $");
 +
  #include "zutil.h"
  #include "infblock.h"
  #include "inftrees.h"
 @@ -249,10 +252,11 @@
                               &s->sub.trees.tb, s->hufts, z);
        if (t != Z_OK)
        {
 -        ZFREE(z, s->sub.trees.blens);
          r = t;
 -        if (r == Z_DATA_ERROR)
 +        if (r == Z_DATA_ERROR) {
 +          ZFREE(z, s->sub.trees.blens);
            s->mode = BAD;
 +	}
          LEAVE
        }
        s->sub.trees.index = 0;
 @@ -313,11 +317,12 @@
          t = inflate_trees_dynamic(257 + (t & 0x1f), 1 + ((t >> 5) & 0x1f),
                                    s->sub.trees.blens, &bl, &bd, &tl, &td,
                                    s->hufts, z);
 -        ZFREE(z, s->sub.trees.blens);
          if (t != Z_OK)
          {
 -          if (t == (uInt)Z_DATA_ERROR)
 +          if (t == (uInt)Z_DATA_ERROR) {
 +            ZFREE(z, s->sub.trees.blens);
              s->mode = BAD;
 +	}
            r = t;
            LEAVE
          }
 @@ -329,6 +334,7 @@
          }
          s->sub.decode.codes = c;
        }
 +      ZFREE(z, s->sub.trees.blens);
        s->mode = CODES;
      case CODES:
        UPDATE
 
 Eugene Grosbein

From: Ian Dowse <iedowse@maths.tcd.ie>
To: Eugene Grosbein <eugen@grosbein.pp.ru>
Cc: Matthew Dillon <dillon@apollo.backplane.com>,
	FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG,
	Kirk McKusick <mckusick@mckusick.com>
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections 
Date: Wed, 20 Mar 2002 16:13:11 +0000

 In message <20020320213503.A344@grosbein.pp.ru>, Eugene Grosbein writes:
 >It seems, that commit affected src/lib/libz/infblock.c 
 >and it's possible it still contatins the bug that is fixed now in the kernel.
 
 Yes, I had seen the libc/libz code too. At first glance it appears
 to have a similar problem:
 
 >@@ -329,6 +334,7 @@
 >         }
 >         s->sub.decode.codes = c;
 >       }
 >+      ZFREE(z, s->sub.trees.blens);
 >       s->mode = CODES;
 >     case CODES:
 >       UPDATE
 
 However the structure of struct inflate_blocks_state is different
 in libz, so it is mostly safe because decode.codes and trees.blens
 are unlikely to overlap on most platforms:
 
   union {
     uInt left;          /* if STORED, bytes left to copy */
     struct {
       uInt table;               /* table lengths (14 bits) */
       uInt index;               /* index into blens (or border) */
       uIntf *blens;             /* bit lengths of codes */
       uInt bb;                  /* bit length tree depth */
       inflate_huft *tb;         /* bit length decoding tree */
     } trees;            /* if DTREE, decoding info for trees */
     struct {
       inflate_codes_statef 
          *codes;
     } decode;           /* if CODES, current state */
   } sub;                /* submode */
 
 The patch applied to libc/libz seems to match exactly what appeared
 in the official zlib 1.1.4 release, but it is still very dodgy
 programming by the zlib people...
 
 Ian

From: Matthew Dillon <dillon@apollo.backplane.com>
To: Ian Dowse <iedowse@maths.tcd.ie>
Cc: Eugene Grosbein <eugen@grosbein.pp.ru>,
	FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG,
	Kirk McKusick <mckusick@mckusick.com>,
	Chris Faulhaber <jedgar@fxp.org>, security-officer@FreeBSD.ORG
Subject: Re: kern/35969: kernel option PPP_DEFLATE often procudes kernel panics; PPP_BSDCOMP sometimes procudes stalled connections 
Date: Wed, 20 Mar 2002 10:17:52 -0800 (PST)

     I would say that libz should probably be fixed too, and given a 
     big whopping comment as well, even though (as Ian noted) the 
     organization of the union is such that the bug will not occur.
 
     I'm adding Chris to this Cc.  Chris, it's your baby :-), I'll leave it
     to you to decide what is best in regards to libz.
 
     p.s. Chris, OpenBSD's infblock.c file in libz is exactly the same as
     ours except for some minor whitespace/bracing changes.
 
 					-Matt
 					Matthew Dillon 
 					<dillon@backplane.com>
 
 :Yes, I had seen the libc/libz code too. At first glance it appears
 :to have a similar problem:
 :
 :>@@ -329,6 +334,7 @@
 :>         }
 :>         s->sub.decode.codes = c;
 :>       }
 :>+      ZFREE(z, s->sub.trees.blens);
 :>       s->mode = CODES;
 :>     case CODES:
 :>       UPDATE
 :
 :However the structure of struct inflate_blocks_state is different
 :in libz, so it is mostly safe because decode.codes and trees.blens
 :are unlikely to overlap on most platforms:
 :
 :  union {
 :    uInt left;          /* if STORED, bytes left to copy */
 :    struct {
 :      uInt table;               /* table lengths (14 bits) */
 :      uInt index;               /* index into blens (or border) */
 :      uIntf *blens;             /* bit lengths of codes */
 :      uInt bb;                  /* bit length tree depth */
 :      inflate_huft *tb;         /* bit length decoding tree */
 :    } trees;            /* if DTREE, decoding info for trees */
 :    struct {
 :      inflate_codes_statef 
 :         *codes;
 :    } decode;           /* if CODES, current state */
 :  } sub;                /* submode */
 :
 :The patch applied to libc/libz seems to match exactly what appeared
 :in the official zlib 1.1.4 release, but it is still very dodgy
 :programming by the zlib people...
 :
 :Ian
>Unformatted:
