Upgraded to 13.2-RELEASE: Unbound DNS broken - ideas? reddit / freebsd ---------------------------------------------------------------------- squirtcow · 2023-04-17 18:32 Hi guys, I started having problems with Unbound DNS after the upgrade to 13.2 yesterday, and I've not been able to determine any sensible cause for it. I only use Unbound as local-resolver and forwarder to Quad9 over IPv6 w. TLS. LAN clients communicate with unbound on port 53/udp, as per normal. There is no NAT involved on either client- or server side. The disruption seems to come and go, where I suddenly can get a few minutes of things working like it should, before something breaks again. I have done the following; * Rebuilt all ports post-upgrade (portmaster -af). * Temporarily disabled PF to see if it could have been a firewall issue. * Tcpdump and logging using unbound-control. * Synced system-clock (ntp) ​ **Client -> Server** 20:15:30.049478 IP 85.19.n.n.48091 > 213.236.n.n.domain: 43060+ A? www.google.com. (32) 20:15:30.049525 IP 213.236.n.n.domain > 85.19.n.n.48091: 43060 ServFail 0/0/0 (32) 20:15:30.058660 IP 85.19.n.n.48091 > 213.236.n.n.domain: 43060+ A? www.google.com. (32) 20:15:30.058683 IP 213.236.n.n.domain > 85.19.n.n.48091: 43060 ServFail 0/0/0 (32) 20:15:30.079285 IP 85.19.n.n.1664 > 213.236.n.n.domain: 35506+ A? www.google.com. (32) 20:15:30.079307 IP 213.236.n.n.domain > 85.19.n.n.1664: 35506 ServFail 0/0/0 (32) **Server -> Quad9 (From unbound.log)** [1681755233] unbound[2723:0] info: validation failure : no DNSSEC records from 2620:fe::fe for DS google.com. while building chain of trust [1681755309] unbound[2723:0] info: validation failure : no DNSSEC records from 2620:fe::9 for DS google.com. while building chain of trust [1681755371] unbound[2723:0] info: validation failure : no DNSSEC records from 2620:fe::fe for DS google.com. while building chain of trust However, manual query with dig works just fine. # dig @dns.quad9.net google.com A +short 142.250.74.142 # dig @dns.quad9.net google.com AAAA +short 2a00:1450:400f:805::200e **.. and when it works, it works great!** 20:30:12.076492 IP6 2a02:nn:nn:200::c01e.47612 > dns.quad9.net.domain-s: Flags [.], ack 4254, win 1035, options [nop,nop,TS val 3598800785 ecr 3299683544], length 0 20:30:12.076509 IP6 2a02:nn:nn:200::c01e.47612 > dns.quad9.net.domain-s: Flags [P.], seq 548:572, ack 4254, win 1035, options [nop,nop,TS val 3598800785 ecr 3299683544], length 24 20:30:12.076562 IP6 2a02:nn:nn:200::c01e.47612 > dns.quad9.net.domain-s: Flags [F.], seq 572, ack 4254, win 1035, options [nop,nop,TS val 3598800785 ecr 3299683544], length 0 20:30:12.084340 IP6 dns.quad9.net.domain-s > 2a02:nn:nn:200::c01e.47612: Flags [R], seq 3095997412, win 0, length 0 20:30:12.084350 IP6 dns.quad9.net.domain-s > 2a02:nn:nn:200::c01e.47612: Flags [R], seq 3095997412, win 0, length 0 20:30:12.144344 IP6 dns.quad9.net.domain-s > 2a02:nn:nn:200::c01e.38263: Flags [P.], seq 4001:4025, ack 548, win 120, options [nop,nop,TS val 3299683612 ecr 1440472467], length 24 20:30:12.144352 IP6 dns.quad9.net.domain-s > 2a02:nn:nn:200::c01e.38263: Flags [F.], seq 4025, ack 548, win 120, options [nop,nop,TS val 3299683612 ecr 1440472467], length 0 Ideas, anyone? ---------------------------------------------------------------------- EinalButtocks · 2023-04-17 19:43 Could be fallback to tcp fails, since it’s dnssec and the respnse could be bigger than 512. Try “dig … +tcp” and see if that works ---------------------------------------------------------------------- squirtcow · 2023-04-17 19:58 That seems to work fine, but worth looking into.. Could it be TCP MSS related, perhaps? I am running jumboframes locally. ​ $ dig @2620:fe::fe dagbladet.no AAAA +tcp +short 2606:4700::6812:13de 2606:4700::6812:12de ​ 21:53:59.532793 IP6 2a02:nn:nn:200::c01e.19477 > dns.quad9.net.domain: Flags [.], ack 1, win 1030, options [nop,nop,TS val 1282857055 ecr 3304711042], length 0 21:53:59.532855 IP6 2a02:nn:nn:200::c01e.19477 > dns.quad9.net.domain: Flags [P.], seq 1:56, ack 1, win 1030, options [nop,nop,TS val 1282857055 ecr 3304711042], length 55 8999+ [1au] AAAA? dagbladet.no. (53) 21:53:59.540771 IP6 dns.quad9.net.domain > 2a02:nn:nn:200::c01e.19477: Flags [.], ack 56, win 112, options [nop,nop,TS val 3304711050 ecr 1282857055], length 0 21:53:59.542464 IP6 dns.quad9.net.domain > 2a02:nn:nn:200::c01e.19477: Flags [P.], seq 1:100, ack 56, win 112, options [nop,nop,TS val 3304711052 ecr 1282857055], length 99 8999 2/0/1 AAAA 2606:4700::6812:13de, AAAA 2606:4700::6812:12de (97) ---------------------------------------------------------------------- Kumba42 · 2023-04-17 21:35 A Quad9 technician on a support ticket I opened with them last year (2022), upon learning I ran my own local Unbound instances, stated:  >Is DNSSEC enabled in your DNS forwarder settings? If yes, it should be disabled, since Quad9 already performs DNSSEC validation, and because DNSSEC validation at the forwarder level can cause issues. He pointed me to [this article](https://www.nlnetlabs.nl/documentation/unbound/howto-turnoff- dnssec/) (nlnetlabs.nl) and said to follow item #2 to disable trust validation. ---------------------------------------------------------------------- [deleted] · 2023-04-17 22:03 Interesting! I just turned off DNSSEC in the forwarder settings and now my performance has improved. Good to know on this front. ---------------------------------------------------------------------- Quad9DNS · 2023-04-18 06:55 This. DNSSEC at the forwarder level significantly reduces performance due to duplicating the DNSSEC validation process that the upstream recursive service already performs, causes excess DNS queries to be sent to the recursive resolver, and can cause occasional failures. Also, Cloud9??? :) ---------------------------------------------------------------------- squirtcow · 2023-04-18 12:48 Many thanks for sharing this! I have just made the changes, and will report back on the results. I was not aware of this previously! ---------------------------------------------------------------------- Kumba42 · 2023-04-19 23:12 Also see FreeBSD [PR#270912](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270912). It looks like there is an issue w/ Unbound and ASLR in 13.2-RELEASE that causes Unbound to randomly return SERVFAIL because its DoT connection to upstream forwarders randomly get reset at the syscall level. Quick fix is either disable ASLR globally (I wouldn't recommend this): `sysctl kern.elf64.aslr.enable=0` ^( ) Or disable it for just the Unbound binary (this is what I did on my setup): `elfctl -e +noaslr /usr/local/sbin/unbound` ^( ) Edit: s/ALSR/ASLR/g ---------------------------------------------------------------------- grahamperrin · 2023-05-08 14:24 [Timeouts to forward servers on BSD based system with ASLR · Issue #887 · NLnetLabs/unbound](https://github.com/NLnetLabs/unbound/issues/887)