From nobody@FreeBSD.org  Thu Dec 27 20:00:35 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 70D5AF8C
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 27 Dec 2012 20:00:35 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 57D1F8FC12
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 27 Dec 2012 20:00:35 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id qBRK0Yu6000985
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 27 Dec 2012 20:00:34 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id qBRK0Y8r000984;
	Thu, 27 Dec 2012 20:00:34 GMT
	(envelope-from nobody)
Message-Id: <201212272000.qBRK0Y8r000984@red.freebsd.org>
Date: Thu, 27 Dec 2012 20:00:34 GMT
From: Radek Krejca <radek.krejca@starnet.cz>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Unexpected change of default route
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         174749
>Category:       kern
>Synopsis:       Unexpected change of default route
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    rrs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Dec 27 20:10:00 UTC 2012
>Closed-Date:    Mon Jun 24 09:50:27 UTC 2013
>Last-Modified:  Mon Jun 24 09:50:27 UTC 2013
>Originator:     Radek Krejca
>Release:        9.0
>Organization:
STARNET, s.r.o.
>Environment:
FreeBSD nat-62.starnet.cz 9.0-RELEASE-p4 FreeBSD 9.0-RELEASE-p4 #1: Mon Nov  5 14:24:05 CET 2012     root@storage.starnet.cz:/usr/obj/usr/src/sys/NAT-9.0  amd64
>Description:
Hello,

I have a lot of diskless machines (about 60) booted over network from one readonly image. This machines are NATs realized over PF. Image is relative clean system, there is only PF support compiled in kernel, snmpd (readonly, but if I shut it down, problem is still here), fprobe for reports and that is all. I dont block any port at this moment, except smtp.

No route software or dhclient is running.

Only on one machine is problem (probably client with virus), but this problem is very often and serious. Currently I have script for watching default route and after change turn it back and send me e-mail. Change come on about 10 times (randomly) pred day, but in batch, I got 15 e-mail per 5 second (15 changes) 10 minutes ago.

Default route (bad default route) is random and last change was to 192.168.1.5 - correct is 178.255.168.254. Bad default route need not be from ip range on this machine.

Ipv6 routing is untouched so I can analyze traffic, but I dont know which. All traffic is impossible, because bandwidth of this machine is about 300-450 mbit and I cannot predict time of problem and ip range of new bad default route.

I thought that could be a problem in icmp redirect (if there is huge of theese packet, change of default route comes) but I switched on log.icmp.redirect and I dont have positive match.
>How-To-Repeat:
Its hard to say - if there is a lot of icmp redirect packets sometimes change comes, but not always. Instalation is relative clean, so problem could be in default instalation. The same problem comes also on 8.2 FBSD, the same machine (physical machine was changed, but the same set of clients).
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu Dec 27 20:12:29 UTC 2012 
Responsible-Changed-Why:  
reclassify. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=174749 
Responsible-Changed-From-To: freebsd-net->freebsd-ipfw 
Responsible-Changed-By: qingli 
Responsible-Changed-When: Thu Dec 27 23:36:00 UTC 2012 
Responsible-Changed-Why:  
similar to kern/157796 


http://www.freebsd.org/cgi/query-pr.cgi?pr=174749 

From: =?iso-8859-2?Q?Radek_Krej=E8a?= <radek.krejca@starnet.cz>
To: "'bug-followup@FreeBSD.org'" <bug-followup@FreeBSD.org>,
	=?iso-8859-2?Q?Radek_Krej=E8a?= <radek.krejca@starnet.cz>
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Thu, 3 Jan 2013 06:31:33 +0100

 Hello,
 
 I see, that category was changed to From-To: 	freebsd-net->freebsd-ipfw, bu=
 t I dont use ipfw, but PF.
 
 And problem is going to be critical, because there are a lot of incidents, =
 currently already on 2 machines. I am trying catch pakets, but still withou=
 t any match.
 
 Radek
 

From: Krzysztof Barcikowski <krzysiek@airnet.opole.pl>
To: bug-followup@FreeBSD.org, radek.krejca@starnet.cz
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Thu, 10 Jan 2013 11:26:28 +0100

 Hello,
 Kindly please take a look at the following threads, similar problem appears:
 http://lists.freebsd.org/pipermail/freebsd-net/2012-March/031879.html
 http://lists.freebsd.org/pipermail/freebsd-net/2012-September/033209.html
 http://lists.freebsd.org/pipermail/freebsd-net/2012-September/033394.html
 
 I've also received email from other user reporting this problem:
 
 "Hello fellow.
 I found a thread in FreeBSD-net mailing list, where you was told 
 about unexpectedly changed (on some kind of junk address) static routes 
 http://lists.freebsd.org/pipermail/freebsd-net/2012-March/031879.html.
 I have a similar  problem, but with default gateway.
 I think I found one more likeness in our systems, I am using renamed 
 vlan interface. It was made in rc.conf 
   by ifconfig_vlan3400_name="comstar_w".
 Do you have something like that in your rc.conf?
 Or maybe you already found solution for these trouble?
 
 Thanks."
 
 Best regards!
 Chris
 
 
 

From: =?iso-8859-2?Q?Radek_Krej=E8a?= <radek.krejca@starnet.cz>
To: 'Krzysztof Barcikowski' <krzysiek@airnet.opole.pl>,
	"bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>
Cc:  
Subject: RE: kern/174749: Unexpected change of default route
Date: Thu, 10 Jan 2013 12:29:15 +0100

 Hi, thank you for response, because problem is very bad for us, because our=
  customers leave us. I have script which checks default route and switch it=
  back and send e-mail to me so situation is better.
 
 To problem - in your text:
 
 > From: Krzysztof Barcikowski [mailto:krzysiek@airnet.opole.pl]
 > Sent: Thursday, January 10, 2013 11:26 AM
 > To: bug-followup@FreeBSD.org; Radek Krej=E8a
 > Subject: Re: kern/174749: Unexpected change of default route
 >=20
 > Hello,
 > Kindly please take a look at the following threads, similar problem appea=
 rs:
 > http://lists.freebsd.org/pipermail/freebsd-net/2012-March/031879.html
 > http://lists.freebsd.org/pipermail/freebsd-net/2012-September/033209.html
 > http://lists.freebsd.org/pipermail/freebsd-net/2012-September/033394.html
 >=20
 > I've also received email from other user reporting this problem:
 >=20
 > "Hello fellow.
 > I found a thread in FreeBSD-net mailing list, where you was told
 > about unexpectedly changed (on some kind of junk address) static routes
 > http://lists.freebsd.org/pipermail/freebsd-net/2012-March/031879.html.
 > I have a similar  problem, but with default gateway.
 > I think I found one more likeness in our systems, I am using renamed
 > vlan interface. It was made in rc.conf
 >   by ifconfig_vlan3400_name=3D"comstar_w".
 > Do you have something like that in your rc.conf?
 > Or maybe you already found solution for these trouble?
 
 
 I have some points to above:
 
 - route monitor is useless - it only tells, that default route is changed a=
 nd pid of process - but process doesnt exists at watching time....
 - i have clean system, only with PF nat (it could be interesting)
 - situation is the same on 8.2 and 9.0 (9.1 not tested)
 - change is in reaction on traffic - in time of change, threre is a lot of =
 garbage on network
 
 I find out that ip of bad default route is used for traffic long time ago b=
 efore change - udp traffic, I think, that it is torrent (or something simil=
 ar) traffic. There could be 10 changes per minute (like yesterday).=20
 
 I wrote script, which store all traffic (collected over tcpdump) in 10 seco=
 nd files and in case of change stops collecting and deleting old logs - but=
  I havent time to analyze it yet (i have about 200 vlans and 500 Mbit traff=
 ic on this router). My konwledge of internet protocols is on bad level also=
 ....
 
 Here are a little of commands on machine (mpd is new and wasnt installed du=
 ring monitoring, snmpd too):
 
 root@nat-62 /root# cat /etc/rc.conf
 nat_number=3D"62"
 ipv6_defaultrouter=3D"2a02:768:0:4000::4000"
 ifconfig_em0_ipv6=3D"inet6 2a02:768:0:4000::${nat_number}"
 keymap=3D"us.iso"
 
 # enable routing
 gateway_enable=3D"YES"
 
 # enable ssh
 sshd_enable=3D"YES"
 
 # enable packet filter
 pf_enable=3D"YES"                         # Enable PF (load module if requi=
 red)
 pf_rules=3D"/etc/pf.conf"                 # rules definition file for pf
 pf_flags=3D""                             # additional flags for pfctl star=
 tup
 pflog_enable=3D"NO"                       # start pflogd(8)
 pflog_logfile=3D"/var/log/pflog"          # where pflogd should store the l=
 ogfile
 pflog_flags=3D""                          # additional flags for pflogd sta=
 rtup
 pfsync_enable=3D"NO"                      # Expose pf state to other hosts =
 for syncing
 
 # enable snmp
 snmpd_enable=3D"YES"
 snmpd_flags=3D"-a"
 snmpd_pidfile=3D"/var/run/snmpd.pid"
 
 fprobe_enable=3D"YES"
 fprobe_server=3D"some_server"
 
 ifconfig_em1=3D"up"
 
 ipv6_activate_all_interfaces=3D"YES"      # Set to YES to set up for IPv6.
 ipv6_gateway_enable=3D"YES"               # Set to YES if this host will be=
  a gateway.
 
 radvd_enable=3D"YES"
 
 ntpdate_enable=3D"YES"                    # Run ntpdate to sync time on boo=
 t (or NO).
 ntpd_enable=3D"YES"
 
 mpd_enable=3D"YES"
 
 init_nat_enable=3D"YES"
 
 
 root@nat-62 /root# ifconfig -l
 em0 em1 lo0 vlan1208 vlan1210 vlan1212 vlan1214 vlan1216 vlan1218 vlan1220 =
 vlan1222 vlan1224 vlan1226 vlan1228 vlan1230 vlan1232 vlan1234 vlan1236 vla=
 n1238 vlan1240 vlan1248 vlan1246 vlan1244 vlan1242 vlan1207 vlan100 vlan106=
  vlan107 vlan1001 vlan1003 vlan1005 vlan1007 vlan1009 vlan1011 vlan1013 vla=
 n1015 vlan1017 vlan1019 vlan1021 vlan453 vlan1206 vlan1023 vlan1025 vlan102=
 7 vlan1029 vlan1031 vlan1033 vlan1035 vlan1037 vlan332 vlan345 vlan341 vlan=
 327 vlan333 vlan335 vlan336 vlan334 vlan337 vlan338 vlan339 vlan340 vlan342=
  vlan343 vlan449 vlan329 vlan448 vlan401 vlan402 vlan403 vlan1051 vlan801 v=
 lan297 vlan299
 
 Important point - I have this machine diskless, readonly, dhclient isnt run=
 ning:
 
 root@nat-62 /root# ps -uax
 USER       PID  %CPU %MEM    VSZ    RSS  TT  STAT STARTED         TIME COMM=
 AND
 root        11 371.5  0.0      0     64  ??  RL   19Dec12 111079:00.52 [idl=
 e]
 root         0  11.1  0.0      0    192  ??  DLs  19Dec12   4491:00.35 [ker=
 nel]
 root        12  10.4  0.0      0    288  ??  WL   19Dec12   3404:19.05 [int=
 r]
 root      1159   1.3  0.1  22332   3428  ??  Ss   19Dec12    615:51.38 /usr=
 /sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift
 root     70422   0.4  0.0  14636   1604   1- S     9:07PM      5:59.16 sh .=
 /reset_gw
 root         1   0.0  0.0   6280    424  ??  ILs  19Dec12      0:01.22 /sbi=
 n/init --
 root         2   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [sct=
 p_iterator]
 root         3   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [xpt=
 _thrd]
 root         4   0.0  0.0      0     16  ??  DL   19Dec12      0:01.22 [pag=
 edaemon]
 root         5   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [vmd=
 aemon]
 root         6   0.0  0.0      0     16  ??  DL   19Dec12      0:00.02 [pag=
 ezero]
 root         7   0.0  0.0      0     16  ??  DL   19Dec12      0:30.66 [buf=
 daemon]
 root         8   0.0  0.0      0     16  ??  DL   19Dec12      0:09.11 [vnl=
 ru]
 root         9   0.0  0.0      0     16  ??  DL   19Dec12      3:37.36 [syn=
 cer]
 root        10   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [aud=
 it]
 root        13   0.0  0.0      0     48  ??  DL   19Dec12      0:02.22 [geo=
 m]
 root        14   0.0  0.0      0     16  ??  DL   19Dec12     58:39.99 [yar=
 row]
 root        15   0.0  0.0      0    128  ??  DL   19Dec12      1:19.63 [usb=
 ]
 root        16   0.0  0.0      0     16  ??  DL   19Dec12      0:20.35 [acp=
 i_thermal]
 root        17   0.0  0.0      0     16  ??  DL   19Dec12      0:04.53 [acp=
 i_cooling1]
 root        18   0.0  0.0      0     16  ??  DL   19Dec12      0:11.27 [sof=
 tdepflush]
 root        33   0.0  0.0      0     16  ??  DL   19Dec12      0:01.36 [md0=
 ]
 root       107   0.0  0.0      0     16  ??  DL   19Dec12      0:00.15 [md1=
 ]
 root       112   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [md2=
 ]
 root       117   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [md3=
 ]
 root       122   0.0  0.0      0     16  ??  DL   19Dec12      0:00.32 [md4=
 ]
 root       127   0.0  0.0      0     16  ??  DL   19Dec12      0:00.00 [md5=
 ]
 root       139   0.0  0.0      0     16  ??  DL   19Dec12      0:01.77 [md6=
 ]
 root       712   0.0  0.1  10372   3280  ??  Is   19Dec12      0:00.02 /sbi=
 n/devd
 root       731   0.0  0.0      0     16  ??  DL   19Dec12      5:55.99 [pfp=
 urge]
 root       927   0.0  0.0  12184   1448  ??  Ss   19Dec12      0:15.95 /usr=
 /sbin/syslogd -s
 root      1052   0.0  0.0      0     64  ??  DL   19Dec12      0:00.00 [ng_=
 queue]
 root      1062   0.0  0.1  33532   6128  ??  S    19Dec12     29:38.98 /usr=
 /local/sbin/snmpd -p /var/run/snmpd.pid -a
 root      1075   0.0  0.4  35504  16400  ??  Ss   19Dec12    178:17.51 /usr=
 /local/sbin/fprobe -iem1 -fvlan&&ip -B4096 -r2 -q10000 -t10000:10000000 -K1=
 8 something
 root      1197   0.0  0.1  46876   3808  ??  Is   19Dec12      0:02.02 /usr=
 /sbin/sshd
 root      1204   0.0  0.1  20384   3432  ??  Ss   19Dec12      0:20.92 send=
 mail: accepting connections (sendmail)
 smmsp     1208   0.0  0.1  20384   3224  ??  Is   19Dec12      0:00.22 send=
 mail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
 root      1214   0.0  0.0  14260   1440  ??  Is   19Dec12      0:04.18 /usr=
 /sbin/cron -s
 root     57633   0.0  0.1  68016   4728  ??  Is   12:21PM      0:00.02 sshd=
 : darius [priv] (sshd)
 darius   58105   0.0  0.1  68016   4740  ??  S    12:21PM      0:00.01 sshd=
 : darius@pts/0 (sshd)
 root     86691   0.0  0.0  14636   1604  ??  S    12:24PM      0:00.00 sh .=
 /reset_gw
 root     86692   0.0  0.0  10052   1136  ??  S    12:24PM      0:00.00 /sbi=
 n/route get default
 root     86693   0.0  0.0  16424   1272  ??  S    12:24PM      0:00.00 grep=
  gateway
 root     86694   0.0  0.0  10056    920  ??  S    12:24PM      0:00.00 cut =
 -d: -f2
 root     86695   0.0  0.0  10056    968  ??  S    12:24PM      0:00.00 tr -=
 d
 root      1281   0.0  0.0  41300   1904  v0  Is   19Dec12      0:00.01 logi=
 n [pam] (login)
 jvelisek  8423   0.0  0.1  17668   2468  v0  I    19Dec12      0:00.01 -csh=
  (csh)
 root      8426   0.0  0.1  44572   2652  v0  I    19Dec12      0:00.01 sudo=
  su -l
 root      8427   0.0  0.0  41296   1796  v0  I    19Dec12      0:00.00 su -=
 l
 root      8428   0.0  0.1  17668   2464  v0  I+   19Dec12      0:00.01 -su =
 (csh)
 root      1282   0.0  0.0  12184   1100  v1  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv1
 root      1283   0.0  0.0  12184   1100  v2  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv2
 root      1284   0.0  0.0  12184   1100  v3  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv3
 root      1285   0.0  0.0  12184   1100  v4  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv4
 root      1286   0.0  0.0  12184   1100  v5  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv5
 root      1287   0.0  0.0  12184   1100  v6  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv6
 root      1288   0.0  0.0  12184   1100  v7  Is+  19Dec12      0:00.00 /usr=
 /libexec/getty Pc ttyv7
 darius   58106   0.0  0.1  17668   2540   0  Is   12:21PM      0:00.01 -csh=
  (csh)
 root     58889   0.0  0.0  41304   1888   0  I    12:21PM      0:00.00 su -=
 l
 root     59480   0.0  0.1  17668   2856   0  S    12:21PM      0:00.02 -su =
 (csh)
 root     86696   0.0  0.0  14328   1272   0  R+   12:24PM      0:00.00 ps -=
 uax
 
 If you need any more informations please let me know.=20
 
 Radek

From: Mark Linimon <linimon@lonesome.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Sat, 12 Jan 2013 02:01:07 -0600

 ----- Forwarded message from Vadim Urazaev <demiurg@tica.com.ua> -----
 
 Date: Fri, 11 Jan 2013 07:46:05 +0200
 From: Vadim Urazaev <demiurg@tica.com.ua>
 To: Radek Krejča <radek.krejca@starnet.cz>
 Cc: freebsd-ipfw@freebsd.org, freebsd-net@freebsd.org,
 	freebsd-hackers@freebsd.org
 Subject: RE: kern/174749: Unexpected change of default route
 
 Do some body know how can we debug kernel memory corruption on live system?
 We need to find out which function/subsystem is cause of this mess.
 Or maybe is there some way to lock particular memory area, where default
 gateway lies and watch which subsystem will cause system crash?
 
 ----- End forwarded message -----

From: Ian Smith <smithi@nimnet.asn.au>
To: bug-followup@FreeBSD.org, radek.krejca@starnet.cz
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Mon, 11 Feb 2013 23:50:56 +1100

 It seems clear that this PR is misclassified, perhaps denying it the
 attention it deserves. Radek is using pf nat, not ipfw nat. The only
 way it seems 'similar to kern/157796' is the changed default route
 symptom, but I have no clue for either PR.  If not pf@ then where?
 
 cheers, Ian

From: Nick Rogers <ncrogers@gmail.com>
To: bug-followup@FreeBSD.org, radek.krejca@starnet.cz
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Mon, 4 Mar 2013 08:01:10 -0800

 FWIW I am experiencing the same problem under FreeBSD 9.1-RELEASE
 (GENERIC kernel + PF/ALTQ). The default route changes about once per
 day on a system pushing around 50Mb/s of traffic. The route changes to
 an IPv4 address (I do not use IPv6) that is not on my network. I am
 using PF for NAT, RDR, and filtering and ALTQ. I do not have dhclient
 running. The problem is seemingly random and I am unable to reproduce
 it reliably. This is a critical problem and I hope someone capable can
 give it the attention it needs.

From: =?koi8-r?B?+8HT1MnOIOHMxcvTwc7E0iDzxdLHxcXXyd4=?=
	<ShastinAS@sf.sibserv.com>
To: "'bug-followup@FreeBSD.org'" <bug-followup@FreeBSD.org>,
	"'radek.krejca@starnet.cz'" <radek.krejca@starnet.cz>
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Sun, 7 Apr 2013 06:40:05 +0000

 --_000_DFF2BF98BA64E94F82B01B663522984C798340MX01SFsibservcom_
 Content-Type: text/plain; charset="koi8-r"
 Content-Transfer-Encoding: quoted-printable
 
 I have same problem.
 Freebsd 9.1R
 Kernel ipfw dummynet, pf nat
 Same config ipfw, pf work fine on freebsd 8.3 stable (December), if I start=
  freebsd 9.1
 Routes changes automaticly and not only default, any other changed too
 Routes changed every 5 minets and message log contain this:
 ----------
 Apr  5 20:39:04 gw kernel: pf: state key linking mismatch! dir=3DOUT, if=3D=
 igb0, stored af=3D2, a0: 46.219.13.140:38308, a1: 46.x.x.x:52242, proto=3D6=
 , found af=3D2, a0: 172.24.37.19:59953, a1: 85.26.164.174:6881, proto=3D6.
 Apr  5 20:39:14 gw kernel: pf: state key linking mismatch! dir=3DOUT, if=3D=
 igb0, stored af=3D2, a0: 88.201.177.111:12749, a1: 172.24.22.192:57573, pro=
 to=3D17, found af=3D2, a0: 172.24.37.21:4704, a1: 36.234.221.79:7288, proto=
 =3D6.
 Apr  5 20:39:16 gw kernel: pf: state key linking mismatch! dir=3DOUT, if=3D=
 igb0, stored af=3D2, a0: 178.208.241.243:10262, a1: 46.x.x.x:52727, proto=
 =3D6, found af=3D2, a0: 172.24.70.68:53987, a1: 37.220.178.247:35412, proto=
 =3D6.
 Apr  5 20:39:16 gw kernel: pf: state key linking mismatch! dir=3DOUT, if=3D=
 igb0, stored af=3D2, a0: 178.125.65.157:22887, a1: 46.x.x.x:55395, proto=3D=
 17, found af=3D2, a0: 172.24.70.68:53988, a1: 85.26.234.152:46651, proto=3D=
 6.
 Apr  5 20:39:17 gw kernel: pf: state key linking mismatch! dir=3DOUT, if=3D=
 igb0, stored af=3D2, a0: 109.252.190.162:16837, a1: 172.24.9.125:50591, pro=
 to=3D17, found af=3D2, a0: 172.24.99.113:1734, a1: 176.121.217.249:35692, p=
 roto=3D6.
 Apr  5 20:39:18 gw kernel: arpresolve: can't allocate llinfo for 81.109.86.=
 123
 Apr  5 20:39:18 gw last message repeated 359 times
 Apr  5 20:39:18 gw kernel: 81.109.86.123
 Apr  5 20:39:18 gw kernel: arpresolve: can't allocate llinfo for 81.109.86.=
 123
 ------------ igb0 is internal interface,    81.109.86.123 is fake route, an=
 d not my network
 Route delete default, route add my router isp and small time later again
 Apr  5 20:37:15 gw kernel: pf: state key linking mismatch! dir=3DOUT, if=3D=
 igb0, stored af=3D2, a0: 172.24.49.75:26167, a1: 93.77.135.82:46401, proto=
 =3D17, found af=3D2, a0: 172.24.70.30:50699, a1: 94.20.68.43:11789, proto=
 =3D6.
 Apr  5 20:37:17 gw kernel: arpresolve: can't allocate llinfo for 5.9.99.11
 Apr  5 20:37:17 gw last message repeated 158 times
 Apr  5 20:37:17 gw kernel: allocate llinfo for 5.9.99.11
 Apr  5 20:37:17 gw kernel: arpresolve: can't allocate llinfo for 5.9.99.11
 ------- 5.9.99.11 fake gateway
 
 
 
 --_000_DFF2BF98BA64E94F82B01B663522984C798340MX01SFsibservcom_
 Content-Type: text/html; charset="koi8-r"
 Content-Transfer-Encoding: quoted-printable
 
 <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
 osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
 xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
 //www.w3.org/TR/REC-html40">
 <head>
 <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dkoi8-r">
 <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
 <style><!--
 /* Font Definitions */
 @font-face
 	{font-family:Calibri;
 	panose-1:2 15 5 2 2 2 4 3 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
 	{margin:0cm;
 	margin-bottom:.0001pt;
 	font-size:11.0pt;
 	font-family:"Calibri","sans-serif";
 	mso-fareast-language:EN-US;}
 a:link, span.MsoHyperlink
 	{mso-style-priority:99;
 	color:blue;
 	text-decoration:underline;}
 a:visited, span.MsoHyperlinkFollowed
 	{mso-style-priority:99;
 	color:purple;
 	text-decoration:underline;}
 span.EmailStyle17
 	{mso-style-type:personal-compose;
 	font-family:"Calibri","sans-serif";
 	color:windowtext;}
 .MsoChpDefault
 	{mso-style-type:export-only;
 	font-family:"Calibri","sans-serif";
 	mso-fareast-language:EN-US;}
 @page WordSection1
 	{size:612.0pt 792.0pt;
 	margin:2.0cm 42.5pt 2.0cm 3.0cm;}
 div.WordSection1
 	{page:WordSection1;}
 --></style><!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
 </xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
 <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
 </head>
 <body lang=3D"RU" link=3D"blue" vlink=3D"purple">
 <div class=3D"WordSection1">
 <p class=3D"MsoNormal"><span lang=3D"EN-US">I have same problem.<o:p></o:p>=
 </span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Freebsd 9.1R<o:p></o:p></span><=
 /p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Kernel ipfw dummynet, pf nat<o:=
 p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Same config ipfw, pf work fine =
 on freebsd 8.3 stable (December), if I start freebsd 9.1<o:p></o:p></span><=
 /p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Routes changes automaticly and =
 not only default, any other changed too<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Routes changed every 5 minets a=
 nd message log contain this:<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">----------<o:p></o:p></span></p=
 >
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:04 gw kernel:=
  pf: state key linking mismatch! dir=3DOUT, if=3Digb0, stored af=3D2, a0: 4=
 6.219.13.140:38308, a1: 46.x.x.x:52242, proto=3D6, found af=3D2, a0: 172.24=
 .37.19:59953, a1: 85.26.164.174:6881, proto=3D6.<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:14 gw kernel:=
  pf: state key linking mismatch! dir=3DOUT, if=3Digb0, stored af=3D2, a0: 8=
 8.201.177.111:12749, a1: 172.24.22.192:57573, proto=3D17, found af=3D2, a0:=
  172.24.37.21:4704, a1: 36.234.221.79:7288, proto=3D6.<o:p></o:p></span></p=
 >
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:16 gw kernel:=
  pf: state key linking mismatch! dir=3DOUT, if=3Digb0, stored af=3D2, a0: 1=
 78.208.241.243:10262, a1: 46.x.x.x:52727, proto=3D6, found af=3D2, a0: 172.=
 24.70.68:53987, a1: 37.220.178.247:35412, proto=3D6.<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:16 gw kernel:=
  pf: state key linking mismatch! dir=3DOUT, if=3Digb0, stored af=3D2, a0: 1=
 78.125.65.157:22887, a1: 46.x.x.x:55395, proto=3D17, found af=3D2, a0: 172.=
 24.70.68:53988, a1: 85.26.234.152:46651, proto=3D6.<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:17 gw kernel:=
  pf: state key linking mismatch! dir=3DOUT, if=3Digb0, stored af=3D2, a0: 1=
 09.252.190.162:16837, a1: 172.24.9.125:50591, proto=3D17, found af=3D2, a0:=
  172.24.99.113:1734, a1: 176.121.217.249:35692, proto=3D6.<o:p></o:p></span=
 ></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:18 gw kernel:=
  arpresolve: can't allocate llinfo for 81.109.86.123<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:18 gw last me=
 ssage repeated 359 times<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:18 gw kernel:=
  81.109.86.123<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:39:18 gw kernel:=
  arpresolve: can't allocate llinfo for 81.109.86.123<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">------------ igb0 is internal i=
 nterface,&nbsp;&nbsp;&nbsp; 81.109.86.123 is fake route, and not my network=
 <o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Route delete default, route add=
  my router isp and small time later again<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:37:15 gw kernel:=
  pf: state key linking mismatch! dir=3DOUT, if=3Digb0, stored af=3D2, a0: 1=
 72.24.49.75:26167, a1: 93.77.135.82:46401, proto=3D17, found af=3D2, a0: 17=
 2.24.70.30:50699, a1: 94.20.68.43:11789, proto=3D6.<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:37:17 gw kernel:=
  arpresolve: can't allocate llinfo for 5.9.99.11<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:37:17 gw last me=
 ssage repeated 158 times<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:37:17 gw kernel:=
  allocate llinfo for 5.9.99.11<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">Apr&nbsp; 5 20:37:17 gw kernel:=
  arpresolve: can't allocate llinfo for 5.9.99.11<o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US">------- 5.9.99.11 fake gateway<=
 o:p></o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p>&nbsp;</o:p></span></p>
 <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p>&nbsp;</o:p></span></p>
 </div>
 </body>
 </html>
 
 --_000_DFF2BF98BA64E94F82B01B663522984C798340MX01SFsibservcom_--

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/174749: commit references a PR
Date: Wed, 24 Apr 2013 18:30:39 +0000 (UTC)

 Author: rrs
 Date: Wed Apr 24 18:30:32 2013
 New Revision: 249848
 URL: http://svnweb.freebsd.org/changeset/base/249848
 
 Log:
   This fixes the issue with the "randomly changing" default
   route. What it was is there are two places in ip_output.c
   where we do a goto again. One place was fine, it
   copies out the new address and then resets dst = ro->rt_dst;
   But the other place does *not* do that, which means earlier
   when we found the gateway, we have dst pointing there
   aka dst = ro->rt_gateway is done.. then we do a
   goto again.. bam now we clobber the default route.
   
   The fix is just to move the again so we are always
   doing dst = &ro->rt_dst; in the again loop.
   
   PR:	 174749,157796
   MFC after:	1 week
 
 Modified:
   head/sys/netinet/ip_output.c
 
 Modified: head/sys/netinet/ip_output.c
 ==============================================================================
 --- head/sys/netinet/ip_output.c	Wed Apr 24 18:00:28 2013	(r249847)
 +++ head/sys/netinet/ip_output.c	Wed Apr 24 18:30:32 2013	(r249848)
 @@ -196,8 +196,8 @@ ip_output(struct mbuf *m, struct mbuf *o
  		hlen = ip->ip_hl << 2;
  	}
  
 -	dst = (struct sockaddr_in *)&ro->ro_dst;
  again:
 +	dst = (struct sockaddr_in *)&ro->ro_dst;
  	ia = NULL;
  	/*
  	 * If there is a cached route,
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/174749: commit references a PR
Date: Thu, 25 Apr 2013 11:24:53 +0000 (UTC)

 Author: rrs
 Date: Thu Apr 25 11:24:40 2013
 New Revision: 249891
 URL: http://svnweb.freebsd.org/changeset/base/249891
 
 Log:
   MFC of PR r249848.
   
   PR:		174749, 157796
 
 Modified:
   stable/8/sys/netinet/ip_output.c
 Directory Properties:
   stable/8/sys/   (props changed)
   stable/8/sys/netinet/   (props changed)
 
 Modified: stable/8/sys/netinet/ip_output.c
 ==============================================================================
 --- stable/8/sys/netinet/ip_output.c	Thu Apr 25 08:57:15 2013	(r249890)
 +++ stable/8/sys/netinet/ip_output.c	Thu Apr 25 11:24:40 2013	(r249891)
 @@ -197,8 +197,8 @@ ip_output(struct mbuf *m, struct mbuf *o
  		hlen = ip->ip_hl << 2;
  	}
  
 -	dst = (struct sockaddr_in *)&ro->ro_dst;
  again:
 +	dst = (struct sockaddr_in *)&ro->ro_dst;
  	/*
  	 * If there is a cached route,
  	 * check that it is to the same destination
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/174749: commit references a PR
Date: Thu, 25 Apr 2013 11:25:42 +0000 (UTC)

 Author: rrs
 Date: Thu Apr 25 11:25:24 2013
 New Revision: 249892
 URL: http://svnweb.freebsd.org/changeset/base/249892
 
 Log:
   MFC of r249848
   
   PR:	174749, 157796
 
 Modified:
   stable/9/sys/netinet/ip_output.c
 Directory Properties:
   stable/9/sys/   (props changed)
 
 Modified: stable/9/sys/netinet/ip_output.c
 ==============================================================================
 --- stable/9/sys/netinet/ip_output.c	Thu Apr 25 11:24:40 2013	(r249891)
 +++ stable/9/sys/netinet/ip_output.c	Thu Apr 25 11:25:24 2013	(r249892)
 @@ -194,8 +194,8 @@ ip_output(struct mbuf *m, struct mbuf *o
  		hlen = ip->ip_hl << 2;
  	}
  
 -	dst = (struct sockaddr_in *)&ro->ro_dst;
  again:
 +	dst = (struct sockaddr_in *)&ro->ro_dst;
  	ia = NULL;
  	/*
  	 * If there is a cached route,
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: =?iso-8859-2?Q?Radek_Krej=E8a?= <radek.krejca@starnet.cz>
To: "'bug-followup@FreeBSD.org'" <bug-followup@FreeBSD.org>
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Thu, 25 Apr 2013 20:32:08 +0200

 Hello,
 
 thank you very much, I will try it. The same problem I noticed also in 9.1.
 
 Radek
 
Responsible-Changed-From-To: freebsd-ipfw->rrs 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Fri Apr 26 08:58:18 UTC 2013 
Responsible-Changed-Why:  
Randall fixed this recently. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=174749 

From: =?iso-8859-2?Q?Radek_Krej=E8a?= <radek.krejca@starnet.cz>
To: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>
Cc:  
Subject: Re: kern/174749: Unexpected change of default route
Date: Tue, 30 Apr 2013 20:25:13 +0200

 Hello again,
 
 after three days of testing I can tell, that It looks like problem is solve=
 d.
 
 My question is, when it will be on source codes on svn (for updating over s=
 vn)?
 
 http://svnweb.freebsd.org/base/releng/9.1/sys/netinet/ip_output.c?view=3Dlo=
 g
 
 Radek
 
State-Changed-From-To: open->closed 
State-Changed-By: glebius 
State-Changed-When: Mon Jun 24 09:50:04 UTC 2013 
State-Changed-Why:  
Merged to stable/9 on April 24th as r249848. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=174749 
>Unformatted:
