From nobody@FreeBSD.org  Mon Apr 12 14:53:55 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 84279106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 12 Apr 2010 14:53:55 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 742508FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 12 Apr 2010 14:53:55 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o3CErsmw034373
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 12 Apr 2010 14:53:54 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o3CErsd7034360;
	Mon, 12 Apr 2010 14:53:54 GMT
	(envelope-from nobody)
Message-Id: <201004121453.o3CErsd7034360@www.freebsd.org>
Date: Mon, 12 Apr 2010 14:53:54 GMT
From: Andrey Smagin <samspeed@mail.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: amd64-curent memory leak in kernel
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         145654
>Category:       amd64
>Synopsis:       amd64-curent memory leak in kernel
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-net
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 12 15:00:04 UTC 2010
>Closed-Date:    Mon Dec 06 21:43:11 UTC 2010
>Last-Modified:  Mon Dec 06 21:43:11 UTC 2010
>Originator:     Andrey Smagin
>Release:        amd64-current 2010 mart xx
>Organization:
DiP Interactive
>Environment:
FreeBSD rizhskaya8.vvt 9.0-CURRENT FreeBSD 9.0-CURRENT #2: Mon Mar 29 23:45:02 UTC 2010     root@rizhskaya8.vvt:/usr/obj/usr/src/sys/SAM  amd64
>Description:
I switched i386->amd64 on my server, with full new ports tree, 
save only configs. Under heavy disk IO, "Wired" memory increased and 
kernel panic in malloc inside IPFW. Uptime back proportional of disk IO load,
with transfer large files from one HDD to another,paniced from 1 to 3-4 hours. With no IO uptime about 2-5 days. It may be ZFS related problem because, 
I switch /usr and /var to ZFS. I try to fix problem by configuring ZFS in 
loader.conf, but nothing help.
loader.conf:
vfs.zfs.arc_max="512M"
vfs.zfs.arc_min="256M"
vfs.zfs.vdev.cache.size="128M"
panic always in:
panic()
kmemmalloc()
uma_large_malloc()
malloc()
ipfw_ctl() at ipfw_ctl+0xb4. 
rip_ctl_output() at rip_ctl_output+0x9f
...
>How-To-Repeat:
start copying large size of data from UFS to ZFS or vice versa. "Wired" memory  increased value and after some hours value above 2GB then hang in malloc always in IPFW.

>Fix:


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: gavin 
State-Changed-When: Mon Apr 12 18:51:29 UTC 2010 
State-Changed-Why:  
Hi, 

Could you please provide the output of "netstat -m" and "vmstat -z" both from 
when the machine has freshly rebooted, and a bit later (if possible, shortly 
before the crash - maybe run them a few times leading up to the crash, and 
provide the latest one you manage to get). 

Thanks! 


Responsible-Changed-From-To: freebsd-amd64->gavin 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Mon Apr 12 18:51:29 UTC 2010 
Responsible-Changed-Why:  
Track 

http://www.freebsd.org/cgi/query-pr.cgi?pr=145654 

From: Andrey Smagin <samspeed@dipinteractive.com>
To: bug-followup@FreeBSD.org, samspeed@mail.ru
Cc:  
Subject: Re: amd64/145654: amd64-curent memory leak in kernel
Date: Thu, 22 Apr 2010 17:19:55 +0400

 After fresh load
  8:12 up 7 mins, 1 user, load averages: 0,34 0,33 0,22
 
 2764/1856/4620 mbufs in use (current/cache/total)
 2175/1217/3392/56800 mbuf clusters in use (current/cache/total/max)
 2161/1167 mbuf+clusters out of packet secondary zone in use (current/cache)
 7/128/135/28400 4k (page size) jumbo clusters in use 
 (current/cache/total/max)
 0/0/0/14200 9k jumbo clusters in use (current/cache/total/max)
 0/0/0/7100 16k jumbo clusters in use (current/cache/total/max)
 5069K/3410K/8479K bytes allocated to network (current/cache/total)
 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
 0/0/0 sfbufs in use (current/peak/max)
 0 requests for sfbufs denied
 0 requests for sfbufs delayed
 0 requests for I/O initiated by sendfile
 0 calls to protocol drain routines
 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
 
 UMA Kegs: 208, 0, 191, 13, 191, 0
 UMA Zones: 896, 0, 191, 1, 191, 0
 UMA Slabs: 568, 0, 14139, 309, 36494, 0
 UMA RCntSlabs: 568, 0, 1831, 3, 1831, 0
 UMA Hash: 256, 0, 54, 36, 81, 0
 16 Bucket: 152, 0, 706, 44, 767, 0
 32 Bucket: 280, 0, 205, 5, 223, 0
 64 Bucket: 536, 0, 196, 7, 219, 19
 128 Bucket: 1048, 0, 555, 27, 615, 38
 VM OBJECT: 216, 0, 15426, 288, 169067, 0
 MAP: 232, 0, 7, 25, 7, 0
 KMAP ENTRY: 120, 150846, 366, 378, 70191, 0
 MAP ENTRY: 120, 0, 4485, 723, 352399, 0
 DP fakepg: 120, 0, 0, 62, 17, 0
 SG fakepg: 120, 0, 0, 0, 0, 0
 mt_zone: 2056, 0, 291, 14, 291, 0
 16: 16, 0, 6169, 383, 725705, 0
 32: 32, 0, 3853, 995, 185798, 0
 64: 64, 0, 57619, 1629, 2144772, 0
 128: 128, 0, 18594, 19367, 417944, 0
 256: 256, 0, 5719, 4526, 1169574, 0
 512: 512, 0, 1522, 242, 80193, 0
 1024: 1024, 0, 151, 181, 17485, 0
 2048: 2048, 0, 1041, 67, 2688, 0
 4096: 4096, 0, 541, 138, 16545, 0
 Files: 80, 0, 614, 6721, 99263, 0
 TURNSTILE: 136, 0, 585, 55, 585, 0
 umtx pi: 96, 0, 0, 0, 0, 0
 MAC labels: 40, 0, 0, 0, 0, 0
 PROC: 1120, 0, 128, 82, 7916, 0
 THREAD: 904, 0, 504, 80, 673, 0
 SLEEPQUEUE: 80, 0, 585, 82, 585, 0
 VMSPACE: 392, 0, 105, 95, 7894, 0
 cpuset: 72, 0, 2, 98, 2, 0
 audit_record: 952, 0, 0, 0, 0, 0
 mbuf_packet: 256, 0, 2166, 1162, 565623, 0
 mbuf: 256, 0, 601, 691, 740290, 0
 mbuf_cluster: 2048, 56800, 3342, 50, 3376, 0
 mbuf_jumbo_page: 4096, 28400, 7, 128, 2183, 0
 mbuf_jumbo_9k: 9216, 14200, 0, 0, 0, 0
 mbuf_jumbo_16k: 16384, 7100, 0, 0, 0, 0
 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0
 g_bio: 232, 0, 4, 828, 415223, 0
 ttyinq: 160, 0, 180, 84, 315, 0
 ttyoutq: 256, 0, 95, 25, 167, 0
 ata_request: 320, 0, 1, 203, 107760, 0
 ata_composite: 336, 0, 0, 0, 0, 0
 VNODE: 480, 0, 25020, 12, 25836, 0
 VNODEPOLL: 112, 0, 29, 103, 29, 0
 S VFS Cache: 108, 0, 19396, 107, 21123, 0
 L VFS Cache: 328, 0, 5836, 32, 5836, 0
 NAMEI: 1024, 0, 0, 96, 384955, 0
 NFSMOUNT: 616, 0, 4, 20, 4, 0
 NFSNODE: 656, 0, 21, 27, 21, 0
 DIRHASH: 1024, 0, 91, 37, 116, 0
 pipe: 728, 0, 29, 76, 4669, 0
 ksiginfo: 112, 0, 204, 852, 1017, 0
 itimer: 344, 0, 0, 0, 0, 0
 KNOTE: 128, 0, 116, 174, 204451, 0
 socket: 680, 56802, 306, 150, 14639, 0
 ipq: 56, 1827, 13, 302, 27186, 0
 udp_inpcb: 336, 56804, 53, 134, 9124, 0
 udpcb: 16, 56952, 53, 787, 9124, 0
 tcp_inpcb: 336, 56804, 290, 73, 2204, 0
 tcpcb: 880, 56800, 170, 102, 2204, 0
 tcptw: 72, 11400, 120, 330, 511, 0
 syncache: 144, 15366, 3, 127, 753, 0
 hostcache: 136, 15372, 76, 92, 76, 0
 tcpreass: 40, 3612, 10, 662, 35376, 0
 sackhole: 32, 0, 0, 505, 256, 0
 sctp_ep: 1272, 56802, 0, 0, 0, 0
 sctp_asoc: 2240, 40000, 0, 0, 0, 0
 sctp_laddr: 48, 80064, 0, 288, 24, 0
 sctp_raddr: 616, 80004, 0, 0, 0, 0
 sctp_chunk: 136, 400008, 0, 0, 0, 0
 sctp_readq: 104, 400032, 0, 0, 0, 0
 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0
 sctp_asconf: 40, 400008, 0, 0, 0, 0
 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0
 ripcb: 336, 56804, 27, 94, 2941, 0
 unpcb: 240, 56800, 33, 111, 319, 0
 rtentry: 200, 0, 196, 51, 248, 0
 IPFW dynamic rule: 120, 0, 0, 0, 0, 0
 divcb: 336, 56804, 14, 41, 14, 0
 selfd: 56, 0, 293, 904, 1804821, 0
 SWAPMETA: 288, 116519, 0, 0, 0, 0
 ip4flow: 56, 99351, 936, 387, 4417, 0
 ip6flow: 80, 99360, 0, 0, 0, 0
 Mountpoints: 760, 0, 18, 12, 18, 0
 FFS inode: 168, 0, 1748, 100, 2111, 0
 FFS1 dinode: 128, 0, 0, 0, 0, 0
 FFS2 dinode: 256, 0, 1748, 127, 2111, 0
 taskq_zone: 48, 0, 0, 1008, 29985, 0
 zio_cache: 720, 0, 0, 1150, 276612, 0
 zio_buf_512: 512, 0, 23347, 1111, 26308, 0
 zio_data_buf_512: 512, 0, 313, 296, 1730, 0
 zio_buf_1024: 1024, 0, 21, 411, 8589, 0
 zio_data_buf_1024: 1024, 0, 4, 52, 53, 0
 zio_buf_1536: 1536, 0, 8, 70, 483, 0
 zio_data_buf_1536: 1536, 0, 5, 59, 72, 0
 zio_buf_2048: 2048, 0, 4, 82, 345, 0
 zio_data_buf_2048: 2048, 0, 0, 42, 27, 0
 zio_buf_2560: 2560, 0, 14, 60, 463, 0
 zio_data_buf_2560: 2560, 0, 5, 22, 31, 0
 zio_buf_3072: 3072, 0, 1, 57, 682, 0
 zio_data_buf_3072: 3072, 0, 0, 14, 8, 0
 zio_buf_3584: 3584, 0, 1, 73, 860, 0
 zio_data_buf_3584: 3584, 0, 3, 11, 6, 0
 zio_buf_4096: 4096, 0, 18, 117, 1452, 0
 zio_data_buf_4096: 4096, 0, 2, 12, 11, 0
 zio_buf_5120: 5120, 0, 0, 74, 841, 0
 zio_data_buf_5120: 5120, 0, 3, 17, 24, 0
 zio_buf_6144: 6144, 0, 1, 70, 1402, 0
 zio_data_buf_6144: 6144, 0, 1, 34, 25, 0
 zio_buf_7168: 7168, 0, 0, 27, 152, 0
 zio_data_buf_7168: 7168, 0, 2, 25, 12, 0
 zio_buf_8192: 8192, 0, 0, 65, 360, 0
 zio_data_buf_8192: 8192, 0, 0, 27, 23, 0
 zio_buf_10240: 10240, 0, 2, 52, 244, 0
 zio_data_buf_10240: 10240, 0, 0, 27, 30, 0
 zio_buf_12288: 12288, 0, 1, 89, 273, 0
 zio_data_buf_12288: 12288, 0, 5, 39, 48, 0
 zio_buf_14336: 14336, 0, 1, 124, 1959, 0
 zio_data_buf_14336: 14336, 0, 4, 40, 51, 0
 zio_buf_16384: 16384, 0, 3913, 30, 4189, 0
 zio_data_buf_16384: 16384, 0, 1, 26, 30, 0
 zio_buf_20480: 20480, 0, 2, 25, 44, 0
 zio_data_buf_20480: 20480, 0, 6, 38, 68, 0
 zio_buf_24576: 24576, 0, 1, 19, 43, 0
 zio_data_buf_24576: 24576, 0, 9, 35, 80, 0
 zio_buf_28672: 28672, 0, 2, 25, 29, 0
 zio_data_buf_28672: 28672, 0, 12, 15, 61, 0
 zio_buf_32768: 32768, 0, 0, 61, 215, 0
 zio_data_buf_32768: 32768, 0, 11, 24, 61, 0
 zio_buf_36864: 36864, 0, 1, 26, 24, 0
 zio_data_buf_36864: 36864, 0, 19, 35, 81, 0
 zio_buf_40960: 40960, 0, 1, 13, 14, 0
 zio_data_buf_40960: 40960, 0, 13, 31, 79, 0
 zio_buf_45056: 45056, 0, 1, 13, 12, 0
 zio_data_buf_45056: 45056, 0, 15, 39, 83, 0
 zio_buf_49152: 49152, 0, 0, 14, 16, 0
 zio_data_buf_49152: 49152, 0, 11, 33, 69, 0
 zio_buf_53248: 53248, 0, 0, 14, 7, 0
 zio_data_buf_53248: 53248, 0, 13, 14, 59, 0
 zio_buf_57344: 57344, 0, 0, 9, 5, 0
 zio_data_buf_57344: 57344, 0, 17, 27, 69, 0
 zio_buf_61440: 61440, 0, 0, 5, 2, 0
 zio_data_buf_61440: 61440, 0, 10, 17, 52, 0
 zio_buf_65536: 65536, 0, 3039, 23, 4180, 0
 zio_data_buf_65536: 65536, 0, 10, 10, 53, 0
 zio_buf_69632: 69632, 0, 0, 2, 1, 0
 zio_data_buf_69632: 69632, 0, 4, 23, 39, 0
 zio_buf_73728: 73728, 0, 0, 0, 0, 0
 zio_data_buf_73728: 73728, 0, 6, 14, 41, 0
 zio_buf_77824: 77824, 0, 0, 0, 0, 0
 zio_data_buf_77824: 77824, 0, 4, 16, 38, 0
 zio_buf_81920: 81920, 0, 0, 0, 0, 0
 zio_data_buf_81920: 81920, 0, 4, 16, 35, 0
 zio_buf_86016: 86016, 0, 0, 2, 1, 0
 zio_data_buf_86016: 86016, 0, 5, 22, 37, 0
 zio_buf_90112: 90112, 0, 0, 0, 0, 0
 zio_data_buf_90112: 90112, 0, 4, 16, 38, 0
 zio_buf_94208: 94208, 0, 0, 0, 0, 0
 zio_data_buf_94208: 94208, 0, 3, 24, 41, 0
 zio_buf_98304: 98304, 0, 0, 0, 0, 0
 zio_data_buf_98304: 98304, 0, 5, 9, 36, 0
 zio_buf_102400: 102400, 0, 0, 2, 1, 0
 zio_data_buf_102400: 102400, 0, 4, 16, 31, 0
 zio_buf_106496: 106496, 0, 0, 0, 0, 0
 zio_data_buf_106496: 106496, 0, 2, 18, 35, 0
 zio_buf_110592: 110592, 0, 0, 0, 0, 0
 zio_data_buf_110592: 110592, 0, 8, 19, 40, 0
 zio_buf_114688: 114688, 0, 0, 20, 13, 0
 zio_data_buf_114688: 114688, 0, 5, 9, 34, 0
 zio_buf_118784: 118784, 0, 0, 5, 4, 0
 zio_data_buf_118784: 118784, 0, 7, 13, 37, 0
 zio_buf_122880: 122880, 0, 0, 0, 0, 0
 zio_data_buf_122880: 122880, 0, 14, 13, 51, 0
 zio_buf_126976: 126976, 0, 0, 14, 11, 0
 zio_data_buf_126976: 126976, 0, 10, 10, 39, 0
 zio_buf_131072: 131072, 0, 2, 133, 595, 0
 zio_data_buf_131072: 131072, 0, 2860, 415, 8547, 0
 dmu_buf_impl_t: 224, 0, 30761, 2032, 47487, 0
 dnode_t: 776, 0, 23628, 352, 24453, 0
 arc_buf_hdr_t: 208, 0, 12283, 2837, 23858, 0
 arc_buf_t: 72, 0, 7459, 1641, 24352, 0
 zil_lwb_cache: 200, 0, 5, 603, 1653, 0
 zfs_znode_cache: 376, 0, 23144, 46, 23596, 0
 md7: 512, 0, 256, 87, 275, 0
 md8: 512, 0, 1868, 50, 1895, 0
 bridge_rtnode: 64, 0, 45, 179, 77, 0
 NetGraph items: 72, 4118, 0, 377, 3371, 0
 NetGraph data items: 72, 522, 0, 232, 215879, 0
 
  procs memory page disks faults cpu
  r b w avm fre flt re pi po fr sr ad0 ad4 in sy cs us sy id
  1 0 0 2051M 2226M 2628 12 20 0 2475 0 0 0 1254 9545 12006 2 3 95
 
 
 Under load, last 5 seconds before panic
  3:07 up 3:54, 6 users, load averages: 0,49 0,43 0,41
 2181/3699/5880 mbufs in use (current/cache/total)
 1203/2405/3608/56800 mbuf clusters in use (current/cache/total/max)
 1121/1414 mbuf+clusters out of packet secondary zone in use (current/cache)
 22/527/549/28400 4k (page size) jumbo clusters in use 
 (current/cache/total/max)
 0/0/0/14200 9k jumbo clusters in use (current/cache/total/max)
 0/0/0/7100 16k jumbo clusters in use (current/cache/total/max)
 3039K/7842K/10882K bytes allocated to network (current/cache/total)
 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
 0/0/0 sfbufs in use (current/peak/max)
 0 requests for sfbufs denied
 0 requests for sfbufs delayed
 246 requests for I/O initiated by sendfile
 0 calls to protocol drain routines
 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
 
 UMA Kegs: 208, 0, 191, 13, 191, 0
 UMA Zones: 896, 0, 191, 1, 191, 0
 UMA Slabs: 568, 0, 29939, 770, 1956135, 0
 UMA RCntSlabs: 568, 0, 2172, 600, 497233, 0
 UMA Hash: 256, 0, 0, 0, 81, 0
 16 Bucket: 152, 0, 285, 440, 1053, 0
 32 Bucket: 280, 0, 131, 79, 637, 0
 64 Bucket: 536, 0, 206, 4, 969, 106
 128 Bucket: 1048, 0, 640, 197, 92248, 10390
 VM OBJECT: 216, 0, 30014, 10792, 7125691, 0
 MAP: 232, 0, 7, 25, 7, 0
 KMAP ENTRY: 120, 150846, 3985, 727, 5109709, 0
 MAP ENTRY: 120, 0, 5344, 1383, 19824395, 0
 DP fakepg: 120, 0, 0, 62, 17, 0
 SG fakepg: 120, 0, 0, 0, 0, 0
 mt_zone: 2056, 0, 291, 41, 291, 0
 16: 16, 0, 46131, 6957, 92224416, 0
 32: 32, 0, 3819, 1332, 52245844, 0
 64: 64, 0, 71347, 14557, 358058147, 0
 128: 128, 0, 23502, 30844, 199185317, 0
 256: 256, 0, 31584, 2106, 218882779, 0
 512: 512, 0, 2677, 711, 16518029, 0
 1024: 1024, 0, 185, 411, 245954, 0
 2048: 2048, 0, 1052, 626, 273564, 0
 4096: 4096, 0, 551, 351, 926798, 0
 Files: 80, 0, 690, 6780, 15685819, 0
 TURNSTILE: 136, 0, 1189, 111, 1189, 0
 umtx pi: 96, 0, 0, 0, 0, 0
 MAC labels: 40, 0, 0, 0, 0, 0
 PROC: 1120, 0, 120, 732, 396120, 0
 THREAD: 904, 0, 1144, 44, 1331, 0
 SLEEPQUEUE: 80, 0, 1189, 116, 1189, 0
 VMSPACE: 392, 0, 97, 793, 396086, 0
 cpuset: 72, 0, 2, 98, 2, 0
 audit_record: 952, 0, 0, 0, 0, 0
 mbuf_packet: 256, 0, 1135, 504, 62608907, 0
 mbuf: 256, 0, 1057, 3184, 86870313, 0
 mbuf_cluster: 2048, 56800, 1721, 1781, 8659376, 0
 mbuf_jumbo_page: 4096, 28400, 22, 399, 718686, 0
 mbuf_jumbo_9k: 9216, 14200, 0, 0, 0, 0
 mbuf_jumbo_16k: 16384, 7100, 0, 0, 0, 0
 mbuf_ext_refcnt: 4, 0, 0, 504, 246, 0
 g_bio: 232, 0, 6, 698, 36176869, 0
 ttyinq: 160, 0, 600, 72, 810, 0
 ttyoutq: 256, 0, 312, 48, 424, 0
 ata_request: 320, 0, 3, 609, 11851875, 0
 ata_composite: 336, 0, 0, 0, 0, 0
 VNODE: 480, 0, 50239, 7281, 1552871, 0
 VNODEPOLL: 112, 0, 29, 103, 29, 0
 S VFS Cache: 108, 0, 52732, 9242, 2574166, 0
 L VFS Cache: 328, 0, 713, 1819, 68917, 0
 NAMEI: 1024, 0, 0, 716, 28500593, 0
 NFSMOUNT: 616, 0, 4, 14, 4, 0
 NFSNODE: 656, 0, 5, 67, 90, 0
 DIRHASH: 1024, 0, 0, 292, 2776, 0
 pipe: 728, 0, 34, 606, 121357, 0
 ksiginfo: 112, 0, 840, 678, 32108, 0
 itimer: 344, 0, 0, 11, 3, 0
 KNOTE: 128, 0, 88, 666, 13030823, 0
 socket: 680, 56802, 353, 895, 316288, 0
 ipq: 56, 1827, 0, 756, 5378376, 0
 udp_inpcb: 336, 56804, 53, 651, 95901, 0
 udpcb: 16, 56952, 53, 787, 95901, 0
 tcp_inpcb: 336, 56804, 378, 2031, 163092, 0
 tcpcb: 880, 56800, 227, 881, 163092, 0
 tcptw: 72, 11400, 151, 1949, 66292, 0
 syncache: 144, 15366, 1, 233, 85663, 0
 hostcache: 136, 15372, 686, 994, 2623, 0
 tcpreass: 40, 3612, 6, 834, 2209238, 0
 sackhole: 32, 0, 4, 602, 15110, 0
 sctp_ep: 1272, 56802, 0, 0, 0, 0
 sctp_asoc: 2240, 40000, 0, 0, 0, 0
 sctp_laddr: 48, 80064, 0, 288, 24, 0
 sctp_raddr: 616, 80004, 0, 0, 0, 0
 sctp_chunk: 136, 400008, 0, 0, 0, 0
 sctp_readq: 104, 400032, 0, 0, 0, 0
 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0
 sctp_asconf: 40, 400008, 0, 0, 0, 0
 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0
 ripcb: 336, 56804, 11, 671, 35006, 0
 unpcb: 240, 56800, 39, 265, 22238, 0
 rtentry: 200, 0, 196, 51, 248, 0
 IPFW dynamic rule: 120, 0, 0, 0, 0, 0
 divcb: 336, 56804, 14, 41, 14, 0
 selfd: 56, 0, 1016, 1000, 285857996, 0
 SWAPMETA: 288, 116519, 98, 383, 11197, 0
 ip4flow: 56, 99351, 417, 1284, 266807, 0
 ip6flow: 80, 99360, 0, 0, 0, 0
 Mountpoints: 760, 0, 18, 22, 18, 0
 FFS inode: 168, 0, 1223, 2385, 170904, 0
 FFS1 dinode: 128, 0, 0, 0, 0, 0
 FFS2 dinode: 256, 0, 1223, 1987, 170903, 0
 taskq_zone: 48, 0, 0, 720, 12587810, 0
 zio_cache: 720, 0, 9, 776, 113062830, 0
 zio_buf_512: 512, 0, 88102, 10668, 2486650, 0
 zio_data_buf_512: 512, 0, 4, 675, 138863, 0
 zio_buf_1024: 1024, 0, 14, 566, 559947, 0
 zio_data_buf_1024: 1024, 0, 0, 800, 18643, 0
 zio_buf_1536: 1536, 0, 0, 330, 79011, 0
 zio_data_buf_1536: 1536, 0, 1, 755, 27243, 0
 zio_buf_2048: 2048, 0, 2, 274, 96658, 0
 zio_data_buf_2048: 2048, 0, 0, 796, 24774, 0
 zio_buf_2560: 2560, 0, 1, 263, 70306, 0
 zio_data_buf_2560: 2560, 0, 0, 431, 19884, 0
 zio_buf_3072: 3072, 0, 0, 239, 64997, 0
 zio_data_buf_3072: 3072, 0, 0, 389, 16507, 0
 zio_buf_3584: 3584, 0, 0, 623, 631285, 0
 zio_data_buf_3584: 3584, 0, 2, 272, 16122, 0
 zio_buf_4096: 4096, 0, 4, 434, 476805, 0
 zio_data_buf_4096: 4096, 0, 0, 365, 13909, 0
 zio_buf_5120: 5120, 0, 0, 401, 410376, 0
 zio_data_buf_5120: 5120, 0, 1, 501, 19994, 0
 zio_buf_6144: 6144, 0, 0, 352, 280457, 0
 zio_data_buf_6144: 6144, 0, 0, 418, 12189, 0
 zio_buf_7168: 7168, 0, 0, 490, 105809, 0
 zio_data_buf_7168: 7168, 0, 0, 300, 10408, 0
 zio_buf_8192: 8192, 0, 0, 361, 118884, 0
 zio_data_buf_8192: 8192, 0, 0, 211, 9536, 0
 zio_buf_10240: 10240, 0, 0, 469, 115875, 0
 zio_data_buf_10240: 10240, 0, 0, 257, 13209, 0
 zio_buf_12288: 12288, 0, 0, 528, 77570, 0
 zio_data_buf_12288: 12288, 0, 0, 189, 10280, 0
 zio_buf_14336: 14336, 0, 0, 492, 2345281, 0
 zio_data_buf_14336: 14336, 0, 0, 203, 8159, 0
 zio_buf_16384: 16384, 0, 3235, 522, 278665, 0
 zio_data_buf_16384: 16384, 0, 0, 184, 7426, 0
 zio_buf_20480: 20480, 0, 0, 466, 46430, 0
 zio_data_buf_20480: 20480, 0, 0, 431, 10925, 0
 zio_buf_24576: 24576, 0, 0, 184, 21103, 0
 zio_data_buf_24576: 24576, 0, 0, 188, 9444, 0
 zio_buf_28672: 28672, 0, 0, 293, 9312, 0
 zio_data_buf_28672: 28672, 0, 2, 92, 7424, 0
 zio_buf_32768: 32768, 0, 0, 214, 13693, 0
 zio_data_buf_32768: 32768, 0, 0, 92, 6210, 0
 zio_buf_36864: 36864, 0, 1, 108, 6194, 0
 zio_data_buf_36864: 36864, 0, 0, 160, 5324, 0
 zio_buf_40960: 40960, 0, 0, 105, 4363, 0
 zio_data_buf_40960: 40960, 0, 0, 84, 4724, 0
 zio_buf_45056: 45056, 0, 0, 83, 3634, 0
 zio_data_buf_45056: 45056, 0, 0, 104, 4132, 0
 zio_buf_49152: 49152, 0, 0, 88, 5886, 0
 zio_data_buf_49152: 49152, 0, 0, 83, 3756, 0
 zio_buf_53248: 53248, 0, 0, 86, 2671, 0
 zio_data_buf_53248: 53248, 0, 0, 54, 3319, 0
 zio_buf_57344: 57344, 0, 0, 56, 2326, 0
 zio_data_buf_57344: 57344, 0, 0, 60, 3128, 0
 zio_buf_61440: 61440, 0, 0, 76, 1944, 0
 zio_data_buf_61440: 61440, 0, 0, 68, 2767, 0
 zio_buf_65536: 65536, 0, 10146, 190, 145362, 0
 zio_data_buf_65536: 65536, 0, 0, 52, 3235, 0
 zio_buf_69632: 69632, 0, 0, 62, 1604, 0
 zio_data_buf_69632: 69632, 0, 0, 68, 2284, 0
 zio_buf_73728: 73728, 0, 0, 62, 1034, 0
 zio_data_buf_73728: 73728, 0, 0, 87, 2513, 0
 zio_buf_77824: 77824, 0, 0, 65, 921, 0
 zio_data_buf_77824: 77824, 0, 0, 70, 2143, 0
 zio_buf_81920: 81920, 0, 0, 176, 4626, 0
 zio_data_buf_81920: 81920, 0, 0, 65, 2398, 0
 zio_buf_86016: 86016, 0, 0, 63, 599, 0
 zio_data_buf_86016: 86016, 0, 1, 64, 1968, 0
 zio_buf_90112: 90112, 0, 0, 37, 842, 0
 zio_data_buf_90112: 90112, 0, 0, 109, 2161, 0
 zio_buf_94208: 94208, 0, 0, 48, 350, 0
 zio_data_buf_94208: 94208, 0, 0, 54, 2064, 0
 zio_buf_98304: 98304, 0, 0, 149, 5939, 0
 zio_data_buf_98304: 98304, 0, 0, 53, 1885, 0
 zio_buf_102400: 102400, 0, 0, 36, 291, 0
 zio_data_buf_102400: 102400, 0, 0, 57, 1859, 0
 zio_buf_106496: 106496, 0, 0, 50, 411, 0
 zio_data_buf_106496: 106496, 0, 0, 82, 1759, 0
 zio_buf_110592: 110592, 0, 0, 42, 265, 0
 zio_data_buf_110592: 110592, 0, 0, 65, 1597, 0
 zio_buf_114688: 114688, 0, 0, 104, 4123, 0
 zio_data_buf_114688: 114688, 0, 0, 44, 1913, 0
 zio_buf_118784: 118784, 0, 0, 45, 190, 0
 zio_data_buf_118784: 118784, 0, 0, 75, 1703, 0
 zio_buf_122880: 122880, 0, 0, 35, 199, 0
 zio_data_buf_122880: 122880, 0, 0, 73, 1952, 0
 zio_buf_126976: 126976, 0, 0, 71, 312, 0
 zio_data_buf_126976: 126976, 0, 0, 59, 1551, 0
 zio_buf_131072: 131072, 0, 3, 461, 27588, 0
 zio_data_buf_131072: 131072, 0, 4, 552, 3830007, 0
 dmu_buf_impl_t: 224, 0, 91367, 16923, 6121798, 0
 dnode_t: 776, 0, 89592, 10163, 1175056, 0
 arc_buf_hdr_t: 208, 0, 30529, 18845, 4722225, 0
 arc_buf_t: 72, 0, 3285, 11265, 5217810, 0
 zil_lwb_cache: 200, 0, 7, 734, 30520, 0
 zfs_znode_cache: 376, 0, 48946, 7624, 1381732, 0
 md7: 512, 0, 2304, 146, 3663, 0
 md8: 512, 0, 1869, 77, 1896, 0
 bridge_rtnode: 64, 0, 37, 299, 211, 0
 NetGraph items: 72, 4118, 1, 840, 1298657, 0
 NetGraph data items: 72, 522, 0, 464, 15454739, 0
 
  procs memory page disks faults cpu
  r b w avm fre flt re pi po fr sr ad0 ad4 in sy cs us sy id
  0 0 0 2084M 211M 9867 3 5 0 12971 3721 0 0 3850 47342 36442 11 11 78
State-Changed-From-To: feedback->open 
State-Changed-By: gavin 
State-Changed-When: Sun Jul 18 12:07:52 UTC 2010 
State-Changed-Why:  
Feedback was received 


Responsible-Changed-From-To: gavin->freebsd-net 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Sun Jul 18 12:07:52 UTC 2010 
Responsible-Changed-Why:  
I can't actually spot anything out of the ordinary in the data received 
but this does sound very much like it is either some sort of network leak 
(or maybe ipfw?) - pass over to the -net guys who may have more of an idea  
or see something that I've missed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=145654 

From: Andriy Gapon <avg@freebsd.org>
To: bug-followup@freebsd.org, samspeed@mail.ru
Cc:  
Subject: Re: amd64/145654: amd64-curent memory leak in kernel
Date: Sun, 05 Dec 2010 15:14:32 +0200

 Is still reproducible?
 How much RAM do you have?  What panic message is printed when this happens?
 This could be a KVA exhaustion or fragmentation issue related to ZFS.
 
 -- 
 Andriy Gapon

From: Andriy Gapon <avg@freebsd.org>
To: Andrey Smagin <samspeed@mail.ru>, bug-followup@freebsd.org
Cc:  
Subject: Re: amd64/145654: amd64-curent memory leak in kernel
Date: Mon, 06 Dec 2010 18:16:30 +0200

 on 05/12/2010 22:47 Andrey Smagin said the following:
 >> Is still reproducible?
 >> How much RAM do you have?  What panic message is printed when this happens?
 >> This could be a KVA exhaustion or fragmentation issue related to ZFS.
 > 
 > I have 4G of RAM, panic messages about malloc memory for network subsystem.
 > After opening PR was some commit helped decrease memory usage under heavy
 > load. I use last night CURRENT right now. Next loader.conf now avoiding this problem:
 > 
 > vfs.zfs.arc_max="300M"  - larger size increase performance but also increase possibility of panic
 
 So you can still reproduce the panic with CURRENT?
 Can you get panic message and stack trace?
 
 > vfs.zfs.vdev.cache.size="16M"
 > vfs.zfs.cache_flush_disable="0"
 > vfs.zfs.prefetch_disable="0"
 >  
 > vfs.zfs.zfetch.array_rd_sz="4M"
 > vfs.zfs.zfetch.block_cap="256"
 > vfs.zfs.zfetch.min_sec_reap="2"
 > vfs.zfs.zfetch.max_streams="16"
 > 
 > vm.kmem_size="4G"
 > 
 > IMHO gigantic problem is -  ZFS use "wired" memory instead "inactive" as UFS.
 
 That's the way ZFS is.
 
 -- 
 Andriy Gapon
State-Changed-From-To: open->closed 
State-Changed-By: avg 
State-Changed-When: Mon Dec 6 21:42:39 UTC 2010 
State-Changed-Why:  
Closing per submitter's request, the issue seems to be resolved. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=145654 
>Unformatted:
