From nobody@FreeBSD.org  Mon Mar  8 21:23:09 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id A1E531065678
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  8 Mar 2010 21:23:09 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 7724A8FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  8 Mar 2010 21:23:09 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o28LN8qs084983
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 8 Mar 2010 21:23:08 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o28LN8ED084982;
	Mon, 8 Mar 2010 21:23:08 GMT
	(envelope-from nobody)
Message-Id: <201003082123.o28LN8ED084982@www.freebsd.org>
Date: Mon, 8 Mar 2010 21:23:08 GMT
From: Evgenii Davidov <dado@korolev-net.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: lltable grows too much
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         144564
>Category:       kern
>Synopsis:       lltable grows too much
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    bz
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 08 21:30:00 UTC 2010
>Closed-Date:    Thu Dec 02 16:08:55 UTC 2010
>Last-Modified:  Thu Dec 02 16:08:55 UTC 2010
>Originator:     Evgenii Davidov
>Release:        8.0
>Organization:
isp kaskad
>Environment:
FreeBSD i1 8.0-STABLE FreeBSD 8.0-STABLE #5: Tue Mar  2 17:29:46 MSK 2010     root@i1:/usr/obj/usr/src/sys/I8  i386

>Description:
vmstat -m | grep llt
      lltable 2055899 257032K       -  2073709  128,256

it's too much, isn't it?

arp -an | wc -l
     625

>How-To-Repeat:
i don't know
>Fix:


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: brucec 
State-Changed-When: Mon Mar 8 21:43:10 UTC 2010 
State-Changed-Why:  
Could you explain how the machine's being used, what firewall if any is loaded, 
what tasks it's running etc.? Without more details it's hard to know where to  
look for the problem. 
The dmesg output could also be useful. 


Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Mon Mar 8 21:43:10 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144564 
State-Changed-From-To: feedback->open  
State-Changed-By: brucec 
State-Changed-When: Tue Mar 9 07:18:55 UTC 2010 
State-Changed-Why:  
Feedback was received. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144564 
State-Changed-From-To: open->analyzed 
State-Changed-By: bz 
State-Changed-When: Fri Mar 19 08:53:31 UTC 2010 
State-Changed-Why:  
The problem has actually crashed cluster machines (ipv6gw) 
already with 
panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated 
as 
db> show malloc 
Type        InUse        MemUse     Requests 
lltable       963374       240540K       963374 
and I have looked at this and have a patch. 


Responsible-Changed-From-To: freebsd-net->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Fri Mar 19 08:53:31 UTC 2010 
Responsible-Changed-Why:  
Take, been working on this and have a patch. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144564 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, dado@korolev-net.ru, 
    "Simon L. Nielsen" <simon@FreeBSD.org>, Qing Li <qingli@FreeBSD.org>
Cc:  
Subject: Re: kern/144564: lltable grows too much
Date: Fri, 19 Mar 2010 22:56:50 +0000 (UTC)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --0-293454317-1269033655=:33454
 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; FORMAT=flowed
 Content-ID: <20100319214317.N33454@maildrop.int.zabbadoz.net>
 
 Dear Evgenii,
 Hi Simon, Qing,
 
 the further down mentioned patch should fix the problem reported in the PR
 (see Subject and net@) and from cluster admins for IPv4.
 
 Please see the notes about FLOWTABLE that would still possibly show the
 leak or worse, cause a panic. I'll send a patch out for this as soon as
 I have updated it for HEAD.
 
 The patch file includes a description of the changes.  It would be great if
 you could test/review them.
 
 You can also fetch the patch temporary from:
 http://people.freebsd.org/~bz/20100319-01-ll-plug-ref-leak-pr144564.diff
 
 While without the patch
  	vmstat -m | awk '/(HighUse|lltable)/ { print; }'
 will show InUse =~ Requests, with the patch you should see
 InUse << Requests.  This will allow you to see if it works at run time.
 
 Regards,
 Bjoern
 
 -- 
 Bjoern A. Zeeb         It will not break if you know what you are doing.
 --0-293454317-1269033655=:33454
 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; NAME=20100319-01-ll-plug-ref-leak-pr144564.diff
 Content-Transfer-Encoding: BASE64
 Content-ID: <20100319212055.G33454@maildrop.int.zabbadoz.net>
 Content-Description: 20100319-01-ll-plug-ref-leak-pr144564.diff
 Content-Disposition: ATTACHMENT; FILENAME=20100319-01-ll-plug-ref-leak-pr144564.diff
 
 IQ0KISBQbHVnIHJlZmVyZW5jZSBsZWFrcyBpbiB0aGUgbGluay1sYXllciBj
 b2RlICgibmV3LWFycCIpIHRoYXQgcHJldmlvdXNseQ0KISBwcmV2ZW50ZWQg
 dGhlIGxpbmstbGF5ZXIgZW50cnkgZnJvbSBiZWluZyBmcmVlZC4gIEFkZCBh
 IEtBU1NFUlQgdG8gY2F0Y2gNCiEgdGhvc2UgY2FzZXMgaW4gdGhlIGZ1dHVy
 ZS4NCiENCiEgTm90ZSB0aGF0IHdpdGggb3B0aW9ucyBGTE9XVEFCTEUgYW4g
 ZXh0cmEgcGF0Y2ggdG8gY2xlYW51cCBpcyBzdGlsbCBuZWVkZWQNCiEgdGhh
 dCB3aWxsIGJlIHByb3ZpZGVkIHNlcGFyYXRlbHkgKGFmdGVyIHVwZGF0aW5n
 IGl0IHRvIHRoZSBsYXN0ZXN0IGNvZGUpLg0KIQ0KISBJbiBib3RoIGluLmMg
 YW5kIGluNi5jICh0aG91Z2ggdGhhdCBjb2RlIHBhdGggc2VlbXMgdG8gYmUg
 YmFzaWNhbGx5IGRlYWQpDQohIHBsdWcgYSByZWZlcmVuY2UgbGVhayBpbiBj
 YXNlIG9mIGEgcGVuZGluZyBjYWxsb3V0IGJlaW5nIGRyYWluZWQuDQohDQoh
 IEluIGlmX2V0aGVyLmMgY29uc2lzdGVudGx5IGFkZCBhIHJlZmVyZW5jZSBi
 ZWZvcmUgcmVzZXR0aW5nIHRoZSBjYWxsb3V0DQohIGFuZCBpbiBjYXNlIHdl
 IGNhbmNlbGVkIGEgcGVuZGluZyBvbmUgcmVtb3ZlIHRoZSByZWZlcmVuY2Ug
 Zm9yIHRoYXQuDQohIEluIHRoZSBmaW5hbCBjYXNlIGluIGFycHRpbWVyLCBi
 ZWZvcmUgZnJlZWluZyB0aGUgZXhwaXJlZCBlbnRyeSwgcmVtb3ZlDQohIHRo
 ZSByZWZlcmVuY2UgYWdhaW4gYW5kIGV4cGxpY2l0bHkgY2FsbCBjYWxsb3V0
 X3N0b3AoKSB0byBjbGVhciB0aGUgYWNjdGl2ZQ0KISBmbGFnLg0KIQ0KISBJ
 biBpZl9sbGF0YmwuYyB3aGVuIGZyZWVpbmcgZW50aXJlIHRhYmxlcyBtYWtl
 IHN1cmUgdGhhdCBpbiBjYXNlIHdlIGNhbmNlbA0KISBhIHBlbmRpbmcgY2Fs
 bG91dCB0byByZW1vdmUgdGhlIHJlZmVyZW5jZSBhcyB3ZWxsLg0KIQ0KIQ0K
 ISBSZXZpZXdlZCBieToJCQ0KISBNRkMgYWZ0ZXI6CQkyIHdlZWtzDQohIFBy
 b2JsZW0gZm91bmQgb246CWlwdjZndyBjbHVzdGVyIG1hY2hpbmUNCiEgUFI6
 CQkJa2Vybi8xNDQ1NjQNCiENCkluZGV4OiBzeXMvbmV0aW5ldC9pbi5jDQo9
 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PT09PT09DQotLS0gc3lzL25ldGluZXQvaW4uYwko
 cmV2aXNpb24gMjA1MzQ1KQ0KKysrIHN5cy9uZXRpbmV0L2luLmMJKHdvcmtp
 bmcgY29weSkNCkBAIC0xMzU3LDggKzEzNTcsMTIgQEAgaW5fbGx0YWJsZV9w
 cmVmaXhfZnJlZShzdHJ1Y3QgbGx0YWJsZSAqbGx0LA0KIA0KIAkJCWlmIChJ
 Tl9BUkVfTUFTS0VEX0FERFJfRVFVQUwoKHN0cnVjdCBzb2NrYWRkcl9pbiAq
 KUwzX0FERFIobGxlKSwgDQogCQkJCQkJICAgICBwZngsIG1zaykpIHsNCi0J
 CQkJY2FsbG91dF9kcmFpbigmbGxlLT5sYV90aW1lcik7DQorCQkJCWludCBj
 YW5jZWxlZDsNCisNCisJCQkJY2FuY2VsZWQgPSBjYWxsb3V0X2RyYWluKCZs
 bGUtPmxhX3RpbWVyKTsNCiAJCQkJTExFX1dMT0NLKGxsZSk7DQorCQkJCWlm
 IChjYW5jZWxlZCkNCisJCQkJCUxMRV9SRU1SRUYobGxlKTsNCiAJCQkJbGxl
 bnRyeV9mcmVlKGxsZSk7DQogCQkJfQ0KIAkJfQ0KSW5kZXg6IHN5cy9uZXRp
 bmV0L2lmX2V0aGVyLmMNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCi0tLSBz
 eXMvbmV0aW5ldC9pZl9ldGhlci5jCShyZXZpc2lvbiAyMDUzNDUpDQorKysg
 c3lzL25ldGluZXQvaWZfZXRoZXIuYwkod29ya2luZyBjb3B5KQ0KQEAgLTE4
 MCw2ICsxODAsOCBAQCBhcnB0aW1lcih2b2lkICphcmcpDQogCWVsc2Ugew0K
 IAkJaWYgKCFjYWxsb3V0X3BlbmRpbmcoJmxsZS0+bGFfdGltZXIpICYmDQog
 CQkgICAgY2FsbG91dF9hY3RpdmUoJmxsZS0+bGFfdGltZXIpKSB7DQorCQkJ
 Y2FsbG91dF9zdG9wKCZsbGUtPmxhX3RpbWVyKTsNCisJCQlMTEVfUkVNUkVG
 KGxsZSk7DQogCQkJKHZvaWQpIGxsZW50cnlfZnJlZShsbGUpOw0KIAkJCUFS
 UFNUQVRfSU5DKHRpbWVvdXRzKTsNCiAJCX0gDQpAQCAtMzgyLDkgKzM4NCwx
 NCBAQCByZXRyeToNCiAJCSAgICBFSE9TVFVOUkVBQ0ggOiBFSE9TVERPV047
 DQogDQogCWlmIChyZW5ldykgew0KKwkJaW50IGNhbmNlbGVkOw0KKw0KIAkJ
 TExFX0FERFJFRihsYSk7DQogCQlsYS0+bGFfZXhwaXJlID0gdGltZV9zZWNv
 bmQgKyBWX2FycHRfZG93bjsNCi0JCWNhbGxvdXRfcmVzZXQoJmxhLT5sYV90
 aW1lciwgaHogKiBWX2FycHRfZG93biwgYXJwdGltZXIsIGxhKTsNCisJCWNh
 bmNlbGVkID0gY2FsbG91dF9yZXNldCgmbGEtPmxhX3RpbWVyLCBoeiAqIFZf
 YXJwdF9kb3duLA0KKwkJICAgIGFycHRpbWVyLCBsYSk7DQorCQlpZiAoY2Fu
 Y2VsZWQpDQorCQkJTExFX1JFTVJFRihsYSk7DQogCQlsYS0+bGFfYXNrZWQr
 KzsNCiAJCUxMRV9XVU5MT0NLKGxhKTsNCiAJCWFycHJlcXVlc3QoaWZwLCBO
 VUxMLCAmU0lOKGRzdCktPnNpbl9hZGRyLA0KQEAgLTY5Niw5ICs3MDMsMTQg
 QEAgbWF0Y2g6DQogCQlFVkVOVEhBTkRMRVJfSU5WT0tFKGFycF91cGRhdGVf
 ZXZlbnQsIGxhKTsNCiANCiAJCWlmICghKGxhLT5sYV9mbGFncyAmIExMRV9T
 VEFUSUMpKSB7DQorCQkJaW50IGNhbmNlbGVkOw0KKw0KKwkJCUxMRV9BRERS
 RUYobGEpOw0KIAkJCWxhLT5sYV9leHBpcmUgPSB0aW1lX3NlY29uZCArIFZf
 YXJwdF9rZWVwOw0KLQkJCWNhbGxvdXRfcmVzZXQoJmxhLT5sYV90aW1lciwg
 aHogKiBWX2FycHRfa2VlcCwNCi0JCQkgICAgYXJwdGltZXIsIGxhKTsNCisJ
 CQljYW5jZWxlZCA9IGNhbGxvdXRfcmVzZXQoJmxhLT5sYV90aW1lciwNCisJ
 CQkgICAgaHogKiBWX2FycHRfa2VlcCwgYXJwdGltZXIsIGxhKTsNCisJCQlp
 ZiAoY2FuY2VsZWQpDQorCQkJCUxMRV9SRU1SRUYobGEpOw0KIAkJfQ0KIAkJ
 bGEtPmxhX2Fza2VkID0gMDsNCiAJCWxhLT5sYV9wcmVlbXB0ID0gVl9hcnBf
 bWF4dHJpZXM7DQpJbmRleDogc3lzL25ldC9pZl9sbGF0YmwuYw0KPT09PT09
 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PQ0KLS0tIHN5cy9uZXQvaWZfbGxhdGJsLmMJKHJl
 dmlzaW9uIDIwNTM0NSkNCisrKyBzeXMvbmV0L2lmX2xsYXRibC5jCSh3b3Jr
 aW5nIGNvcHkpDQpAQCAtMzAsNiArMzAsNyBAQCBfX0ZCU0RJRCgiJEZyZWVC
 U0QkIik7DQogI2luY2x1ZGUgIm9wdF9kZGIuaCINCiAjaW5jbHVkZSAib3B0
 X2luZXQuaCINCiAjaW5jbHVkZSAib3B0X2luZXQ2LmgiDQorI2luY2x1ZGUg
 Im9wdF9yb3V0ZS5oIg0KIA0KICNpbmNsdWRlIDxzeXMvcGFyYW0uaD4NCiAj
 aW5jbHVkZSA8c3lzL3N5c3RtLmg+DQpAQCAtMTExLDYgKzExMiwxNSBAQCBs
 bGVudHJ5X2ZyZWUoc3RydWN0IGxsZW50cnkgKmxsZSkNCiAJaWYgKGxsZS0+
 bGFfaG9sZCAhPSBOVUxMKQ0KIAkJbV9mcmVlbShsbGUtPmxhX2hvbGQpOw0K
 IA0KKyNpZm5kZWYgRkxPV1RBQkxFDQorCS8qIFhYWC1CWiBmbG93dGFibGUg
 Y2xlYW51cCBuZWVkcyB0byBoYXBwZW4gaGVyZS4gT3RoZXIgcGF0Y2guICov
 DQorCS8qDQorCSAqIFdpdGggdGhlIGxsZW50cnkgcmVtb3ZlZCBmcm9tIHRo
 ZSBsaXN0IHdlIG11c3QgYmUgdGhlIGxhc3QNCisJICogb25lIHRvIGhvbGQg
 YSByZWZlcmVuY2Ugb3Igd2Ugd2lsbCBsZWFrIHRoZSBlbnRyeSBuZXZlciBm
 cmVlaW5nIGl0Lg0KKwkgKi8NCisJS0FTU0VSVChsbGUtPmxsZV9yZWZjbnQg
 PT0gMSwgKCIlczogbGxlPSVwIGxsZV9yZWZjbnQ9JWQgIT0gMSIsDQorCSAg
 ICBfX2Z1bmNfXywgbGxlLCBsbGUtPmxsZV9yZWZjbnQpKTsNCisjZW5kaWYN
 CiAJTExFX0ZSRUVfTE9DS0VEKGxsZSk7DQogfQ0KIA0KQEAgLTE3MCw5ICsx
 ODAsMTIgQEAgbGx0YWJsZV9mcmVlKHN0cnVjdCBsbHRhYmxlICpsbHQpDQog
 DQogCWZvciAoaT0wOyBpIDwgTExUQkxfSEFTSFRCTF9TSVpFOyBpKyspIHsN
 CiAJCUxJU1RfRk9SRUFDSF9TQUZFKGxsZSwgJmxsdC0+bGxlX2hlYWRbaV0s
 IGxsZV9uZXh0LCBuZXh0KSB7DQorCQkJaW50IGNhbmNlbGVkOw0KIA0KLQkJ
 CWNhbGxvdXRfZHJhaW4oJmxsZS0+bGFfdGltZXIpOw0KKwkJCWNhbmNlbGVk
 ID0gY2FsbG91dF9kcmFpbigmbGxlLT5sYV90aW1lcik7DQogCQkJTExFX1dM
 T0NLKGxsZSk7DQorCQkJaWYgKGNhbmNlbGVkKQ0KKwkJCQlMTEVfUkVNUkVG
 KGxsZSk7DQogCQkJbGxlbnRyeV9mcmVlKGxsZSk7DQogCQl9DQogCX0NCklu
 ZGV4OiBzeXMvbmV0aW5ldDYvaW42LmMNCj09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
 PT0NCi0tLSBzeXMvbmV0aW5ldDYvaW42LmMJKHJldmlzaW9uIDIwNTM0NSkN
 CisrKyBzeXMvbmV0aW5ldDYvaW42LmMJKHdvcmtpbmcgY29weSkNCkBAIC0y
 MzQ0LDggKzIzNDQsMTIgQEAgaW42X2xsdGFibGVfcHJlZml4X2ZyZWUoc3Ry
 dWN0IGxsdGFibGUgKmxsdCwNCiAJCQkJICAgICYoKHN0cnVjdCBzb2NrYWRk
 cl9pbjYgKilMM19BRERSKGxsZSkpLT5zaW42X2FkZHIsIA0KIAkJCQkgICAg
 JnBmeC0+c2luNl9hZGRyLCANCiAJCQkJICAgICZtc2stPnNpbjZfYWRkcikp
 IHsNCi0JCQkJY2FsbG91dF9kcmFpbigmbGxlLT5sYV90aW1lcik7DQorCQkJ
 CWludCBjYW5jZWxlZDsNCisNCisJCQkJY2FuY2VsZWQgPSBjYWxsb3V0X2Ry
 YWluKCZsbGUtPmxhX3RpbWVyKTsNCiAJCQkJTExFX1dMT0NLKGxsZSk7DQor
 CQkJCWlmIChjYW5jZWxlZCkNCisJCQkJCUxMRV9SRU1SRUYobGxlKTsNCiAJ
 CQkJbGxlbnRyeV9mcmVlKGxsZSk7DQogCQkJfQ0KIAkJfQ0K
 
 --0-293454317-1269033655=:33454--

From: Evgenii Davidov <dado@korolev-net.ru>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>
Cc: bug-followup@FreeBSD.org, "Simon L. Nielsen" <simon@FreeBSD.org>,
	Qing Li <qingli@FreeBSD.org>
Subject: Re: kern/144564: lltable grows too much
Date: Sat, 20 Mar 2010 08:31:17 +0300

 ,
 
 thank you, Bjoern!
 
 as i understand to use this patch i have to cvsup my source to HEAD ?
 is it safe..?
 it's a serious production machine, now i just reboot it at nigth once a week to free memory
 
 On Fri, Mar 19, 2010 at 10:56:50PM +0000, Bjoern A. Zeeb :
 
 > Dear Evgenii,
 > Hi Simon, Qing,
 > 
 > the further down mentioned patch should fix the problem reported in the PR
 > (see Subject and net@) and from cluster admins for IPv4.
 > 
 > Please see the notes about FLOWTABLE that would still possibly show the
 > leak or worse, cause a panic. I'll send a patch out for this as soon as
 > I have updated it for HEAD.
 > 
 > The patch file includes a description of the changes.  It would be great if
 > you could test/review them.
 > 
 > You can also fetch the patch temporary from:
 > http://people.freebsd.org/~bz/20100319-01-ll-plug-ref-leak-pr144564.diff
 > 
 > While without the patch
 > 	vmstat -m | awk '/(HighUse|lltable)/ { print; }'
 > will show InUse =~ Requests, with the patch you should see
 > InUse << Requests.  This will allow you to see if it works at run time.
 > 
 > Regards,
 > Bjoern
 > 
 > -- 
 > Bjoern A. Zeeb         It will not break if you know what you are doing.
 
 Content-Description: 20100319-01-ll-plug-ref-leak-pr144564.diff
 > !
 > ! Plug reference leaks in the link-layer code ("new-arp") that previously
 > ! prevented the link-layer entry from being freed.  Add a KASSERT to catch
 > ! those cases in the future.
 > !
 > ! Note that with options FLOWTABLE an extra patch to cleanup is still needed
 > ! that will be provided separately (after updating it to the lastest code).
 > !
 > ! In both in.c and in6.c (though that code path seems to be basically dead)
 > ! plug a reference leak in case of a pending callout being drained.
 > !
 > ! In if_ether.c consistently add a reference before resetting the callout
 > ! and in case we canceled a pending one remove the reference for that.
 > ! In the final case in arptimer, before freeing the expired entry, remove
 > ! the reference again and explicitly call callout_stop() to clear the acctive
 > ! flag.
 > !
 > ! In if_llatbl.c when freeing entire tables make sure that in case we cancel
 > ! a pending callout to remove the reference as well.
 > !
 > !
 > ! Reviewed by:		
 > ! MFC after:		2 weeks
 > ! Problem found on:	ipv6gw cluster machine
 > ! PR:			kern/144564
 > !
 > Index: sys/netinet/in.c
 > ===================================================================
 > --- sys/netinet/in.c	(revision 205345)
 > +++ sys/netinet/in.c	(working copy)
 > @@ -1357,8 +1357,12 @@ in_lltable_prefix_free(struct lltable *llt,
 >  
 >  			if (IN_ARE_MASKED_ADDR_EQUAL((struct sockaddr_in *)L3_ADDR(lle), 
 >  						     pfx, msk)) {
 > -				callout_drain(&lle->la_timer);
 > +				int canceled;
 > +
 > +				canceled = callout_drain(&lle->la_timer);
 >  				LLE_WLOCK(lle);
 > +				if (canceled)
 > +					LLE_REMREF(lle);
 >  				llentry_free(lle);
 >  			}
 >  		}
 > Index: sys/netinet/if_ether.c
 > ===================================================================
 > --- sys/netinet/if_ether.c	(revision 205345)
 > +++ sys/netinet/if_ether.c	(working copy)
 > @@ -180,6 +180,8 @@ arptimer(void *arg)
 >  	else {
 >  		if (!callout_pending(&lle->la_timer) &&
 >  		    callout_active(&lle->la_timer)) {
 > +			callout_stop(&lle->la_timer);
 > +			LLE_REMREF(lle);
 >  			(void) llentry_free(lle);
 >  			ARPSTAT_INC(timeouts);
 >  		} 
 > @@ -382,9 +384,14 @@ retry:
 >  		    EHOSTUNREACH : EHOSTDOWN;
 >  
 >  	if (renew) {
 > +		int canceled;
 > +
 >  		LLE_ADDREF(la);
 >  		la->la_expire = time_second + V_arpt_down;
 > -		callout_reset(&la->la_timer, hz * V_arpt_down, arptimer, la);
 > +		canceled = callout_reset(&la->la_timer, hz * V_arpt_down,
 > +		    arptimer, la);
 > +		if (canceled)
 > +			LLE_REMREF(la);
 >  		la->la_asked++;
 >  		LLE_WUNLOCK(la);
 >  		arprequest(ifp, NULL, &SIN(dst)->sin_addr,
 > @@ -696,9 +703,14 @@ match:
 >  		EVENTHANDLER_INVOKE(arp_update_event, la);
 >  
 >  		if (!(la->la_flags & LLE_STATIC)) {
 > +			int canceled;
 > +
 > +			LLE_ADDREF(la);
 >  			la->la_expire = time_second + V_arpt_keep;
 > -			callout_reset(&la->la_timer, hz * V_arpt_keep,
 > -			    arptimer, la);
 > +			canceled = callout_reset(&la->la_timer,
 > +			    hz * V_arpt_keep, arptimer, la);
 > +			if (canceled)
 > +				LLE_REMREF(la);
 >  		}
 >  		la->la_asked = 0;
 >  		la->la_preempt = V_arp_maxtries;
 > Index: sys/net/if_llatbl.c
 > ===================================================================
 > --- sys/net/if_llatbl.c	(revision 205345)
 > +++ sys/net/if_llatbl.c	(working copy)
 > @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD$");
 >  #include "opt_ddb.h"
 >  #include "opt_inet.h"
 >  #include "opt_inet6.h"
 > +#include "opt_route.h"
 >  
 >  #include <sys/param.h>
 >  #include <sys/systm.h>
 > @@ -111,6 +112,15 @@ llentry_free(struct llentry *lle)
 >  	if (lle->la_hold != NULL)
 >  		m_freem(lle->la_hold);
 >  
 > +#ifndef FLOWTABLE
 > +	/* XXX-BZ flowtable cleanup needs to happen here. Other patch. */
 > +	/*
 > +	 * With the llentry removed from the list we must be the last
 > +	 * one to hold a reference or we will leak the entry never freeing it.
 > +	 */
 > +	KASSERT(lle->lle_refcnt == 1, ("%s: lle=%p lle_refcnt=%d != 1",
 > +	    __func__, lle, lle->lle_refcnt));
 > +#endif
 >  	LLE_FREE_LOCKED(lle);
 >  }
 >  
 > @@ -170,9 +180,12 @@ lltable_free(struct lltable *llt)
 >  
 >  	for (i=0; i < LLTBL_HASHTBL_SIZE; i++) {
 >  		LIST_FOREACH_SAFE(lle, &llt->lle_head[i], lle_next, next) {
 > +			int canceled;
 >  
 > -			callout_drain(&lle->la_timer);
 > +			canceled = callout_drain(&lle->la_timer);
 >  			LLE_WLOCK(lle);
 > +			if (canceled)
 > +				LLE_REMREF(lle);
 >  			llentry_free(lle);
 >  		}
 >  	}
 > Index: sys/netinet6/in6.c
 > ===================================================================
 > --- sys/netinet6/in6.c	(revision 205345)
 > +++ sys/netinet6/in6.c	(working copy)
 > @@ -2344,8 +2344,12 @@ in6_lltable_prefix_free(struct lltable *llt,
 >  				    &((struct sockaddr_in6 *)L3_ADDR(lle))->sin6_addr, 
 >  				    &pfx->sin6_addr, 
 >  				    &msk->sin6_addr)) {
 > -				callout_drain(&lle->la_timer);
 > +				int canceled;
 > +
 > +				canceled = callout_drain(&lle->la_timer);
 >  				LLE_WLOCK(lle);
 > +				if (canceled)
 > +					LLE_REMREF(lle);
 >  				llentry_free(lle);
 >  			}
 >  		}
 
 
 -- 
 Evgenii V Davidov

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: Evgenii Davidov <dado@korolev-net.ru>
Cc: bug-followup@FreeBSD.org, "Simon L. Nielsen" <simon@FreeBSD.org>, 
    Qing Li <qingli@FreeBSD.org>
Subject: Re: kern/144564: lltable grows too much
Date: Sat, 20 Mar 2010 20:38:33 +0000 (UTC)

 On Sat, 20 Mar 2010, Evgenii Davidov wrote:
 
 > ????????????,
 >
 > thank you, Bjoern!
 >
 > as i understand to use this patch i have to cvsup my source to HEAD ?
 
 No, the patch should equally apply to stable/8.
 
 Flowtable is different in stable/8 and HEAD though that's why this
 change wasn't included here, as we can simply rule that problem out
 removing FLOWTABLE support form the kernel config for the moment.
 
 > is it safe..?
 
 Well who knows;-)  I assume you are not running with INVARIANTS in
 production so it should be.  You can remove the entire change to
 llentry_free() in sys/net/if_llatbl.c (the #ifndef FLOWATBLE part with
 the KASSERT to be one the safe side.)
 I know there exist more problems wrt. IPv6 but you hand't mentioned
 that you are using that.  It won't harm without the KASSERT either.
 
 > it's a serious production machine, now i just reboot it at nigth once a week to free memory
 
 Maybe you can try it with the next reboot over night and see if it
 keeps up for an hour and if it does continue to test?
 
 I'll meanwhile try to see if I can find more testers (apart from
 having done my own tests) and if Qing Li can review it.
 
 /bz
 
 -- 
 Bjoern A. Zeeb         It will not break if you know what you are doing.

From: Evgenii Davidov <dado@korolev-net.ru>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>
Cc: bug-followup@FreeBSD.org, "Simon L. Nielsen" <simon@FreeBSD.org>,
	Qing Li <qingli@FreeBSD.org>
Subject: Re: kern/144564: lltable grows too much
Date: Sun, 21 Mar 2010 00:23:50 +0300

 ,
 
 On Sat, Mar 20, 2010 at 08:38:33PM +0000, Bjoern A. Zeeb :
 
 > No, the patch should equally apply to stable/8.
 
 thank you! maybe i'll can try it next week
 i have no FLOWTABLE in kernel because i noticed flowcleaner eating a lot of cpu once
 
 -- 
 Evgenii V Davidov

From: Qing Li <qingli@freebsd.org>
To: "Bjoern A. Zeeb" <bz@freebsd.org>
Cc: bug-followup@freebsd.org, dado@korolev-net.ru, 
	"Simon L. Nielsen" <simon@freebsd.org>
Subject: Re: kern/144564: lltable grows too much
Date: Mon, 22 Mar 2010 11:04:03 -0700

 The patch looks good. Thank you for fixing it.
 
 At the time I was wondering if I needed to do something if
 callout_reset() returned
 non-zero value, I guess now I know.
 
 -- Qing
 
 On Fri, Mar 19, 2010 at 3:56 PM, Bjoern A. Zeeb <bz@freebsd.org> wrote:
 > Dear Evgenii,
 > Hi Simon, Qing,
 >
 > the further down mentioned patch should fix the problem reported in the P=
 R
 > (see Subject and net@) and from cluster admins for IPv4.
 >
 > Please see the notes about FLOWTABLE that would still possibly show the
 > leak or worse, cause a panic. I'll send a patch out for this as soon as
 > I have updated it for HEAD.
 >
 > The patch file includes a description of the changes. =A0It would be grea=
 t if
 > you could test/review them.
 >
 > You can also fetch the patch temporary from:
 > http://people.freebsd.org/~bz/20100319-01-ll-plug-ref-leak-pr144564.diff
 >
 > While without the patch
 > =A0 =A0 =A0 =A0vmstat -m | awk '/(HighUse|lltable)/ { print; }'
 > will show InUse =3D~ Requests, with the patch you should see
 > InUse << Requests. =A0This will allow you to see if it works at run time.
 >
 > Regards,
 > Bjoern
 >
 > --
 > Bjoern A. Zeeb =A0 =A0 =A0 =A0 It will not break if you know what you are=
  doing.

From: Kevin Day <toasty@dragondata.com>
To: bug-followup@FreeBSD.org,
 dado@korolev-net.ru
Cc:  
Subject: Re: kern/144564: lltable grows too much
Date: Sun, 28 Mar 2010 12:34:42 -0500

 This patch also corrected a long-standing bug we were running into on =
 our public 6to4 relay that would cause llentry to grow over time until =
 we hit the limit. I've been testing it for 5 days now with moderate load =
 (>50mbps, 3kpps) including all sorts of junk traffic to invalid =
 destinations, and there doesn't seem to be a leak now.
 =20=

From: Evgenii Davidov <dado@korolev-net.ru>
To: Kevin Day <toasty@dragondata.com>
Cc: bug-followup@FreeBSD.org, freebsd-net@freebsd.org
Subject: Re: kern/144564: lltable grows too much
Date: Wed, 31 Mar 2010 15:34:52 +0400

 ,
 
 On Sun, Mar 28, 2010 at 12:34:42PM -0500, Kevin Day :
 
 > This patch also corrected a long-standing bug we were running into on our public 6to4 relay that would cause llentry to grow over time until we hit the limit. I've been testing it for 5 days now with moderate load (>50mbps, 3kpps) including all sorts of junk traffic to invalid destinations, and there doesn't seem to be a leak now.
 
 yesterday i cvsuped to RELENG_8
 and tried the patch
 as i could see the problem with lltable was fixed by the patch
 but some of other stuff was broken, i saw some new messages:
 
 Mar 31 00:33:04 i1 kernel: ip_dn_ctl  dummynet: compat option 60
 Mar 31 00:33:04 i1 kernel: ip_dummynet_compat setting compatibility with FreeBSD 8
 Mar 31 01:49:10 i1 kernel: in_cksum_skip: out of data by 38505
 Mar 31 09:02:46 i1 kernel: ipfw: ouch!, skip past end of rules, denying packet
 
 and some ip-addresses in vlanes was not working
 
 so i rebooted to the old kernel
 
 it may be somthing with ipfw in RELENG_8
 
 maybe i could test the patch with RELENG_8_0 but list time i tried it there was a mpd error like "ifa_add_loopback_route: insertion failed" i tried it on march 12 and don't kown is it fixed..
 
 -- 
 Evgenii V Davidov

From: Evgenii Davidov <dado@korolev-net.ru>
To: "Bjoern A. Zeeb" <bz@FreeBSD.org>
Cc: bug-followup@FreeBSD.org, "Simon L. Nielsen" <simon@FreeBSD.org>,
	Qing Li <qingli@FreeBSD.org>
Subject: Re: kern/144564: lltable grows too much
Date: Wed, 7 Apr 2010 23:56:30 +0400

 ,
 
 On Sat, Mar 20, 2010 at 08:38:33PM +0000, Bjoern A. Zeeb :
 
 > >as i understand to use this patch i have to cvsup my source to HEAD ?
 > 
 > No, the patch should equally apply to stable/8.
 
 i have moved to "tag=RELENG_8 date=2010.03.13.01.01.01" with that patch 
 and now it looks much better:
 
       lltable  1010   173K       -   678051  128,256
 
 thank you!
 
 -- 
 Evgenii V Davidov

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/144564: commit references a PR
Date: Sun, 11 Apr 2010 16:04:18 +0000 (UTC)

 Author: bz
 Date: Sun Apr 11 16:04:08 2010
 New Revision: 206481
 URL: http://svn.freebsd.org/changeset/base/206481
 
 Log:
   Plug reference leaks in the link-layer code ("new-arp") that previously
   prevented the link-layer entry from being freed.
   
   In both in.c and in6.c (though that code path seems to be basically dead)
   plug a reference leak in case of a pending callout being drained.
   
   In if_ether.c consistently add a reference before resetting the callout
   and in case we canceled a pending one remove the reference for that.
   In the final case in arptimer, before freeing the expired entry, remove
   the reference again and explicitly call callout_stop() to clear the active
   flag.
   
   In nd6.c:nd6_free() we are only ever called from the callout function and
   thus need to remove the reference there as well before calling into
   llentry_free().
   
   In if_llatbl.c when freeing entire tables make sure that in case we cancel
   a pending callout to remove the reference as well.
   
   Reviewed by:		qingli (earlier version)
   MFC after:		10 days
   Problem observed, patch tested by: simon on ipv6gw.f.o,
   			Christian Kratzer (ck cksoft.de),
   			Evgenii Davidov (dado korolev-net.ru)
   PR:			kern/144564
   Configurations still affected:	with options FLOWTABLE
 
 Modified:
   head/sys/net/if_llatbl.c
   head/sys/netinet/if_ether.c
   head/sys/netinet/in.c
   head/sys/netinet6/in6.c
   head/sys/netinet6/nd6.c
 
 Modified: head/sys/net/if_llatbl.c
 ==============================================================================
 --- head/sys/net/if_llatbl.c	Sun Apr 11 15:35:17 2010	(r206480)
 +++ head/sys/net/if_llatbl.c	Sun Apr 11 16:04:08 2010	(r206481)
 @@ -170,9 +170,12 @@ lltable_free(struct lltable *llt)
  
  	for (i=0; i < LLTBL_HASHTBL_SIZE; i++) {
  		LIST_FOREACH_SAFE(lle, &llt->lle_head[i], lle_next, next) {
 +			int canceled;
  
 -			callout_drain(&lle->la_timer);
 +			canceled = callout_drain(&lle->la_timer);
  			LLE_WLOCK(lle);
 +			if (canceled)
 +				LLE_REMREF(lle);
  			llentry_free(lle);
  		}
  	}
 
 Modified: head/sys/netinet/if_ether.c
 ==============================================================================
 --- head/sys/netinet/if_ether.c	Sun Apr 11 15:35:17 2010	(r206480)
 +++ head/sys/netinet/if_ether.c	Sun Apr 11 16:04:08 2010	(r206481)
 @@ -180,6 +180,8 @@ arptimer(void *arg)
  	else {
  		if (!callout_pending(&lle->la_timer) &&
  		    callout_active(&lle->la_timer)) {
 +			callout_stop(&lle->la_timer);
 +			LLE_REMREF(lle);
  			(void) llentry_free(lle);
  			ARPSTAT_INC(timeouts);
  		} 
 @@ -382,9 +384,14 @@ retry:
  		    EHOSTUNREACH : EHOSTDOWN;
  
  	if (renew) {
 +		int canceled;
 +
  		LLE_ADDREF(la);
  		la->la_expire = time_second + V_arpt_down;
 -		callout_reset(&la->la_timer, hz * V_arpt_down, arptimer, la);
 +		canceled = callout_reset(&la->la_timer, hz * V_arpt_down,
 +		    arptimer, la);
 +		if (canceled)
 +			LLE_REMREF(la);
  		la->la_asked++;
  		LLE_WUNLOCK(la);
  		arprequest(ifp, NULL, &SIN(dst)->sin_addr,
 @@ -696,9 +703,14 @@ match:
  		EVENTHANDLER_INVOKE(arp_update_event, la);
  
  		if (!(la->la_flags & LLE_STATIC)) {
 +			int canceled;
 +
 +			LLE_ADDREF(la);
  			la->la_expire = time_second + V_arpt_keep;
 -			callout_reset(&la->la_timer, hz * V_arpt_keep,
 -			    arptimer, la);
 +			canceled = callout_reset(&la->la_timer,
 +			    hz * V_arpt_keep, arptimer, la);
 +			if (canceled)
 +				LLE_REMREF(la);
  		}
  		la->la_asked = 0;
  		la->la_preempt = V_arp_maxtries;
 
 Modified: head/sys/netinet/in.c
 ==============================================================================
 --- head/sys/netinet/in.c	Sun Apr 11 15:35:17 2010	(r206480)
 +++ head/sys/netinet/in.c	Sun Apr 11 16:04:08 2010	(r206481)
 @@ -1357,8 +1357,12 @@ in_lltable_prefix_free(struct lltable *l
  
  			if (IN_ARE_MASKED_ADDR_EQUAL((struct sockaddr_in *)L3_ADDR(lle), 
  						     pfx, msk)) {
 -				callout_drain(&lle->la_timer);
 +				int canceled;
 +
 +				canceled = callout_drain(&lle->la_timer);
  				LLE_WLOCK(lle);
 +				if (canceled)
 +					LLE_REMREF(lle);
  				llentry_free(lle);
  			}
  		}
 
 Modified: head/sys/netinet6/in6.c
 ==============================================================================
 --- head/sys/netinet6/in6.c	Sun Apr 11 15:35:17 2010	(r206480)
 +++ head/sys/netinet6/in6.c	Sun Apr 11 16:04:08 2010	(r206481)
 @@ -2344,8 +2344,12 @@ in6_lltable_prefix_free(struct lltable *
  				    &((struct sockaddr_in6 *)L3_ADDR(lle))->sin6_addr, 
  				    &pfx->sin6_addr, 
  				    &msk->sin6_addr)) {
 -				callout_drain(&lle->la_timer);
 +				int canceled;
 +
 +				canceled = callout_drain(&lle->la_timer);
  				LLE_WLOCK(lle);
 +				if (canceled)
 +					LLE_REMREF(lle);
  				llentry_free(lle);
  			}
  		}
 
 Modified: head/sys/netinet6/nd6.c
 ==============================================================================
 --- head/sys/netinet6/nd6.c	Sun Apr 11 15:35:17 2010	(r206480)
 +++ head/sys/netinet6/nd6.c	Sun Apr 11 16:04:08 2010	(r206481)
 @@ -1125,6 +1125,7 @@ nd6_free(struct llentry *ln, int gc)
  	ifp = ln->lle_tbl->llt_ifp;
  	IF_AFDATA_LOCK(ifp);
  	LLE_WLOCK(ln);
 +	LLE_REMREF(ln);
  	llentry_free(ln);
  	IF_AFDATA_UNLOCK(ifp);
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: analyzed->patched 
State-Changed-By: bz 
State-Changed-When: Mon Apr 12 13:12:43 UTC 2010 
State-Changed-Why:  
Changes have been comitted to HEAD and will be merged to stable/8 
next week. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144564 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/144564: commit references a PR
Date: Wed, 21 Apr 2010 19:59:04 +0000 (UTC)

 Author: bz
 Date: Wed Apr 21 19:51:22 2010
 New Revision: 207013
 URL: http://svn.freebsd.org/changeset/base/207013
 
 Log:
   MFC r206481:
   
     Plug reference leaks in the link-layer code ("new-arp") that previously
     prevented the link-layer entry from being freed.
   
     In both in.c and in6.c (though that code path seems to be basically dead)
     plug a reference leak in case of a pending callout being drained.
   
     In if_ether.c consistently add a reference before resetting the callout
     and in case we canceled a pending one remove the reference for that.
     In the final case in arptimer, before freeing the expired entry, remove
     the reference again and explicitly call callout_stop() to clear the active
     flag.
   
     In nd6.c:nd6_free() we are only ever called from the callout function and
     thus need to remove the reference there as well before calling into
     llentry_free().
   
     In if_llatbl.c when freeing the entire tables make sure that in case we
     cancel a pending callout to remove the reference as well.
   
     Reviewed by:          qingli (earlier version)
     MFC after:            10 days
     Problem observed, patch tested by: simon on ipv6gw.f.o,
                           Christian Kratzer (ck cksoft.de),
                           Evgenii Davidov (dado korolev-net.ru)
   PR:			kern/144564
   Configurations still affected:	with options FLOWTABLE
 
 Modified:
   stable/8/sys/net/if_llatbl.c
   stable/8/sys/netinet/if_ether.c
   stable/8/sys/netinet/in.c
   stable/8/sys/netinet6/in6.c
   stable/8/sys/netinet6/nd6.c
 Directory Properties:
   stable/8/sys/   (props changed)
   stable/8/sys/amd64/include/xen/   (props changed)
   stable/8/sys/cddl/contrib/opensolaris/   (props changed)
   stable/8/sys/contrib/dev/acpica/   (props changed)
   stable/8/sys/contrib/pf/   (props changed)
   stable/8/sys/dev/xen/xenpci/   (props changed)
   stable/8/sys/geom/sched/   (props changed)
 
 Modified: stable/8/sys/net/if_llatbl.c
 ==============================================================================
 --- stable/8/sys/net/if_llatbl.c	Wed Apr 21 19:48:40 2010	(r207012)
 +++ stable/8/sys/net/if_llatbl.c	Wed Apr 21 19:51:22 2010	(r207013)
 @@ -170,9 +170,12 @@ lltable_free(struct lltable *llt)
  
  	for (i=0; i < LLTBL_HASHTBL_SIZE; i++) {
  		LIST_FOREACH_SAFE(lle, &llt->lle_head[i], lle_next, next) {
 +			int canceled;
  
 -			callout_drain(&lle->la_timer);
 +			canceled = callout_drain(&lle->la_timer);
  			LLE_WLOCK(lle);
 +			if (canceled)
 +				LLE_REMREF(lle);
  			llentry_free(lle);
  		}
  	}
 
 Modified: stable/8/sys/netinet/if_ether.c
 ==============================================================================
 --- stable/8/sys/netinet/if_ether.c	Wed Apr 21 19:48:40 2010	(r207012)
 +++ stable/8/sys/netinet/if_ether.c	Wed Apr 21 19:51:22 2010	(r207013)
 @@ -180,6 +180,8 @@ arptimer(void *arg)
  	else {
  		if (!callout_pending(&lle->la_timer) &&
  		    callout_active(&lle->la_timer)) {
 +			callout_stop(&lle->la_timer);
 +			LLE_REMREF(lle);
  			(void) llentry_free(lle);
  			ARPSTAT_INC(timeouts);
  		} 
 @@ -382,9 +384,14 @@ retry:
  		    EHOSTUNREACH : EHOSTDOWN;
  
  	if (renew) {
 +		int canceled;
 +
  		LLE_ADDREF(la);
  		la->la_expire = time_second + V_arpt_down;
 -		callout_reset(&la->la_timer, hz * V_arpt_down, arptimer, la);
 +		canceled = callout_reset(&la->la_timer, hz * V_arpt_down,
 +		    arptimer, la);
 +		if (canceled)
 +			LLE_REMREF(la);
  		la->la_asked++;
  		LLE_WUNLOCK(la);
  		arprequest(ifp, NULL, &SIN(dst)->sin_addr,
 @@ -694,9 +701,14 @@ match:
  		la->la_flags |= LLE_VALID;
  
  		if (!(la->la_flags & LLE_STATIC)) {
 +			int canceled;
 +
 +			LLE_ADDREF(la);
  			la->la_expire = time_second + V_arpt_keep;
 -			callout_reset(&la->la_timer, hz * V_arpt_keep,
 -			    arptimer, la);
 +			canceled = callout_reset(&la->la_timer,
 +			    hz * V_arpt_keep, arptimer, la);
 +			if (canceled)
 +				LLE_REMREF(la);
  		}
  		la->la_asked = 0;
  		la->la_preempt = V_arp_maxtries;
 
 Modified: stable/8/sys/netinet/in.c
 ==============================================================================
 --- stable/8/sys/netinet/in.c	Wed Apr 21 19:48:40 2010	(r207012)
 +++ stable/8/sys/netinet/in.c	Wed Apr 21 19:51:22 2010	(r207013)
 @@ -1357,8 +1357,12 @@ in_lltable_prefix_free(struct lltable *l
  
  			if (IN_ARE_MASKED_ADDR_EQUAL((struct sockaddr_in *)L3_ADDR(lle), 
  						     pfx, msk)) {
 -				callout_drain(&lle->la_timer);
 +				int canceled;
 +
 +				canceled = callout_drain(&lle->la_timer);
  				LLE_WLOCK(lle);
 +				if (canceled)
 +					LLE_REMREF(lle);
  				llentry_free(lle);
  			}
  		}
 
 Modified: stable/8/sys/netinet6/in6.c
 ==============================================================================
 --- stable/8/sys/netinet6/in6.c	Wed Apr 21 19:48:40 2010	(r207012)
 +++ stable/8/sys/netinet6/in6.c	Wed Apr 21 19:51:22 2010	(r207013)
 @@ -2337,8 +2337,12 @@ in6_lltable_prefix_free(struct lltable *
  				    &((struct sockaddr_in6 *)L3_ADDR(lle))->sin6_addr, 
  				    &pfx->sin6_addr, 
  				    &msk->sin6_addr)) {
 -				callout_drain(&lle->la_timer);
 +				int canceled;
 +
 +				canceled = callout_drain(&lle->la_timer);
  				LLE_WLOCK(lle);
 +				if (canceled)
 +					LLE_REMREF(lle);
  				llentry_free(lle);
  			}
  		}
 
 Modified: stable/8/sys/netinet6/nd6.c
 ==============================================================================
 --- stable/8/sys/netinet6/nd6.c	Wed Apr 21 19:48:40 2010	(r207012)
 +++ stable/8/sys/netinet6/nd6.c	Wed Apr 21 19:51:22 2010	(r207013)
 @@ -1124,6 +1124,7 @@ nd6_free(struct llentry *ln, int gc)
  	ifp = ln->lle_tbl->llt_ifp;
  	IF_AFDATA_LOCK(ifp);
  	LLE_WLOCK(ln);
 +	LLE_REMREF(ln);
  	llentry_free(ln);
  	IF_AFDATA_UNLOCK(ifp);
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: bz 
State-Changed-When: Thu Dec 2 16:08:05 UTC 2010 
State-Changed-Why:  
Even though a problem with FLOWTABLE still exists it's a FT 
and not a llatbl problem in first place and this has been fixed 
for long enough so close it. 
Thanks a lot for reporting and testing! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=144564 
>Unformatted:
