[HN Gopher] DNS resolution issue in Alpine Linux
___________________________________________________________________
DNS resolution issue in Alpine Linux
Author : nelsonmarcos
Score : 56 points
Date : 2022-03-11 17:08 UTC (5 hours ago)
(HTM) web link (christoph.luppri.ch)
(TXT) w3m dump (christoph.luppri.ch)
| tptacek wrote:
| We're a somewhat popular hosting provider that runs Docker
| containers (as VMs) for our customers and does private networking
| over IPv6, which expands the size of our DNS requests, and we run
| into this all the time with Alpine. It's kind of baffling.
|
| TCP DNS is not hard. It's part of the spec. Normally, that
| argument doesn't mean much to me --- lots of things are parts of
| specs that I think are silly and not worth doing --- but TCP DNS
| seems like a basic necessity for DNS to work at all.
|
| What's holding this up? TCP DNS is just UDP DNS, but over a TCP
| connection, with the packet length sent before the packet itself.
| It's the simplest thing you could possibly come up with to make
| TCP DNS work. It's been there since the 1980s. They should add
| it.
| toast0 wrote:
| I mean sure, musl should add TCP DNS. But real world networking
| says if your DNS response is over the limit for basic UDP DNS,
| a meaningful proportion of clients won't get that response.
| Which I guess is okish if it's a TXT for mail servers, cause
| mail servers probably have a reasonable DNS setup; but if you
| wanted to return a lot of A records or AAAA records (or A
| records that a provider DNS64s into AAAA records), you need to
| be careful about how many you return, before the response gets
| too large and doesn't arrive.
|
| This is something networks and DNS client libraries should fix,
| but we live in a world where PathMTU only mostly works, so
| realism gets your service working.
| skuhn wrote:
| It's non-trivial to determine if your response will fit into
| the UDP packet size, other than erring heavily on the side of
| caution.
|
| Part of the request query is included in the response, IPv6
| AAAA records are considerably larger than IPv4 A records,
| DNSSEC adds size, RFC 1035 message compression comes into
| play, EDNS0 comes into play, etc.
|
| Which is all to say that you're right that for maximum
| compatibility you should endeavor to fit all responses into a
| UDP packet, but most places lack the sophistication to ensure
| that is always the case.
|
| Supporting TCP DNS on the client side should not be
| considered optional if you want to successfully resolve
| records that you don't control.
| tptacek wrote:
| That's acceptable in the real world because virtually nobody
| uses IPv6, and people who run into IPv6 problems will just
| avoid musl.
| cmeacham98 wrote:
| This is literally the opposite of true. The OS-supplied
| resolver supports TCP DNS on all of the most popular desktop
| and mobile OSes and has for ages. I'd estimate a minimum of
| 90% of users have setups that support it, and I'm only
| hedging the 10% because I'm not sure how prevalent horribly
| misconfigured corporate firewalls are.
|
| By not supporting TCP DNS you are the odd one out.
| toast0 wrote:
| The OS supplied resolver may support it, but many networks
| block TCP DNS.
| tptacek wrote:
| Many networks also block AAAA DNS, because middleboxes
| are stupid. That doesn't mean it's OK for libc not to
| implement AAAA.
| nwmcsween wrote:
| Possibly ask on #musl or submit a patch?
| tptacek wrote:
| They know about it! I could write the patch, but they're not
| going to accept it; there's something weird going on about
| this.
| nwmcsween wrote:
| There isn't anything weird, I went on #musl asked the
| question, the functionality is desired but it should be
| worked with the community to ensure correctness.
| rascul wrote:
| The functionality was explicitly not added by the
| developer.
|
| > My choice not to do TCP in musl's stub resolver was
| based on an interpretation that truncated results are not
| just acceptable but better ux - not only do you save
| major round-trip delays to DNS but you also get a
| reasonable upper bound on # of addrs in result.
|
| https://twitter.com/RichFelker/status/994629795551031296
|
| Maybe he is willing to consider it now, I don't know.
| infogulch wrote:
| It's not a bug, it's a feature! You're welcome. (tm)
| tptacek wrote:
| Yes, there is. I'm not just making this up. I've gotten
| takes from "musl people", including Rich Felker, that
| this was a deliberate choice, and that not only is musl's
| behavior of pretending that truncated UDP responses are
| valid acceptable, but that it's somehow better (maybe for
| performance?) --- that you can always serve "getaddrinfo"
| off of just the truncated UDP result. Which is, of
| course, false.
|
| There is a weird thing going on; maybe you can relay this
| back to the #musl channel you're talking on, and they can
| keep discussing.
| bogantech wrote:
| > There is a weird thing going on
|
| It's not weird they're just incompetent / a bozo.
| jve wrote:
| Few days ago, I spent quite a few hours trying to make `apk
| update` work for alpine on WSL2 on Windows. It didn't want to
| resolve dl-cdn.alpinelinux.org within alpine. Did resolve on host
| ubuntu.
|
| 1. WFH from VPN, firstly I had to lower mtu from 1500 to 1392 (My
| VPN specific issue) https://github.com/microsoft/WSL/issues/4698
|
| 2. Next, I had to run some powershell script that updates
| /etc/resolv.conf to use my VPN DNS (WSL specific stuff)
| https://github.com/microsoft/WSL/issues/1350
|
| 3. And I still don't know if apk works properly. Kind of works in
| Docker build, but I have a feeling something not quite right.
|
| See this example. Why does it "hang"? Docker command not exiting
| docker run -it alpine:3.15 apk update fetch https://dl-
| cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
|
| Now, doing it within container itself, works:
| docker run -it alpine:3.15 sh / # apk update fetch
| https://dl-
| cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
| fetch https://dl-
| cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
| v3.15.0-342-g4fee739486 [https://dl-
| cdn.alpinelinux.org/alpine/v3.15/main]
| v3.15.0-340-g4ed6115e99 [https://dl-
| cdn.alpinelinux.org/alpine/v3.15/community] OK: 15859
| distinct packages available / # exit
|
| Can someone shed some light?
| newman314 wrote:
| Is DNS set in /etc/docker/daemon.json?
| yakubin wrote:
| _> [...] the standard was extended by two options:_
|
| _> - Increasing the size of the UPD packet above 512 bytes via
| the Extension Mechanism for DNS (EDNS)_
|
| _> - Switching the protocol from UDP to TCP_
|
| _> Alpine Linux, or rather musl libc, doesn't support either of
| those options._
|
| It still seems weird to me that such details are decided by libc.
| My reflex idea when designing a system would be to put DNS
| functionality in a system service, while libraries would only
| query the service, without troubling themselves with system
| caches, TCP vs UDP etc. Then possibly the service could be even
| swapped for another with a compatible interface, but making
| different decisions, without perturbing the applications. It
| sounds like systemd-resolved is a move in that direction, but I
| still don't understand why putting all that in libc, essentially
| making all applications perform their own independent DNS work,
| was the original choice.
| richardfey wrote:
| Alpine has had DNS issues since the very beginning, but I am
| surprised to read that it still has some.
| rascul wrote:
| TLDR:
|
| The solution to an intentionally broken resolver is to use a
| third party library.
| blueflow wrote:
| Its only broken if you dont consider robustness a feature.
| tptacek wrote:
| It's not just "robustness". Not supporting TCP DNS breaks DNS
| if your responses are "large", for values of "large" that
| include numbers that are in fact very small.
| 34tlkjlaegrlk wrote:
| largest safe size to use is ~548 bytes - anything more than
| that and you need tcp
| zamadatix wrote:
| Or, as the article says, EDNS.
| blueflow wrote:
| Its okay to ignore "modern" features if they create bad
| edge cases.
| tptacek wrote:
| This feature is from 1986.
| blueflow wrote:
| So are absolute domain names, but everyone is using
| relative domain specifications now, omitting the final
| dot.
|
| HTTP Transfer-Encoding also got specified, and then
| collectively mis-implemented.
|
| That its in the standard for decades doesnt mean it will
| be good when used.
| charcircuit wrote:
| I can use absolute domain names just fine in the software
| I use.
| tptacek wrote:
| This doesn't make sense as an argument. Without TCP DNS,
| you're stuck with an untenably low limit for how much
| data can fit in a DNS response. Not having TCP DNS breaks
| DNS. It's not an aesthetic argument.
| [deleted]
| zamadatix wrote:
| Things can be broken in more than one way not just solely
| when the thing broken is robustness (for some value of
| robust).
| traceroute66 wrote:
| The moment I saw Alpine Linux in the title, my first guess was "I
| bet this is something to do with musl libc". Briefly looking
| through the blog, it looks like my gut feeling was correct.
|
| A while ago I evaluated Alpine Linux. I wanted to like it, I
| really did, it ticked so many boxes.
|
| But time and time again, I kept on running into issues with their
| adoption of musl libc.
|
| The last straw for me was when I discovered packages in their
| package repo (some of which were well-known names) that were
| compiled against musl when the upstream developers quite clearly
| wrote in their docs that "if you compile X against anything other
| than glibc, you're on your own". For me, the fact that Alpine
| ignored this and compiled against musl anyway, was a big red
| flag. (And yes I raised some of these as bug reports, but the
| cases got closed and nothing done about it).
| Arnavion wrote:
| >The last straw for me was when I discovered packages in their
| package repo (some of which were well-known names) that were
| compiled against musl when the upstream developers quite
| clearly wrote in their docs that "if you compile X against
| anything other than libc, you're on your own".
|
| (I assume you meant "glibc", not "libc".)
|
| That's how every software works. The software developer cares
| about A, B, C distros at most, so other distros are on their
| own. The maintainer of distro D takes responsibility themselves
| to make the package work on D. The maintainer needs to
| understand the software well enough to be able to assert that
| it will work on their distro, patch it as necessary to make
| that happen, and maintain those patches in the light of bug
| reports from the distro users.
|
| >(And yes I raised some of these as bug reports, but the cases
| got closed and nothing done about it).
|
| Well, yeah. Unless you find something that is irrecoverably
| broken against musl such that it can only be fixed by compiling
| against glibc, your bug report is pointless.
| traceroute66 wrote:
| > (I assume you meant "glibc", not "libc".)
|
| Yes, corrected.
|
| > The maintainer of distro D takes responsibility themselves
| to make the package work on D.
|
| Yeah, but Alpine are not doing that, or at least not at the
| time I evaluated. I got tired of finding distro supplied
| packages that just didn't work as expected because things
| broke unexpectedly because of musl.
| blueflow wrote:
| "you're on your own" is like, the expected thing when you are
| doing something different.
|
| Still, Alpine is on top of everyone else when it comes to
| Docker Images sizes. Thats why it will stick.
| qbasic_forever wrote:
| If you care about image size you need to switch to a
| distroless image base. It's incredibly easy nowadays with
| multi-stage builds. IMHO people use alpine not because they
| seek it out but because they're reading old guides and advice
| that hasn't been updated with the times.
|
| Blindly switching to alpine to save space is opening yourself
| up to exactly these sorts of musl-libc quirks and issues--you
| better have solid test coverage and a plan to make sure there
| aren't unexpected edge cases.
| encryptluks2 wrote:
| There are a lot of tools that simply can't run distroless
| and adding in all the requirements is often tedious with
| little reward.
| bogomipz wrote:
| What are some of the tools that can't run on distroless?
| sciurus wrote:
| Are they still on to of everyone else in a meaningful way,
| though?
|
| There was discussion yesterday [0] that pointed out that
| there are 29MB Debian [1] and Ubuntu [2] images.
|
| [0] https://news.ycombinator.com/item?id=30633670
|
| [1] https://hub.docker.com/layers/debian/library/debian/stabl
| e-s...
|
| [2] https://ubuntu.com/blog/minimal-ubuntu-released
| blueflow wrote:
| I run Alpine Linux from a 512 MB CF card, so yes.
| dividedbyzero wrote:
| What do you use this for?
| blueflow wrote:
| On a Siemens Futro S-300. I use it as a Home Server. A
| 2-TB disk is mounted there, also a printer.
| traceroute66 wrote:
| > "you're on your own" is like, the expected thing when you
| are doing something different.
|
| Indeed. And that's fine. As long as you're willing to support
| that difference.
|
| But the "Alpine compiling XYZ against musl" thing is/was just
| being done blindly by Alpine (i.e. load X into auto-build and
| let it rip). Sure it compiled without errors. But it never
| ran properly.
| encryptluks2 wrote:
| I've rarely had issues and if I do there is a process to
| report them to get them fixed. Not yet mad and act like the
| entire distro is worthless because I can't be bothered with
| reporting an issue for something that is already free. I've
| had less issues with Alpine than I've had with software on
| Windows.
| johnklos wrote:
| There are two problems here:
|
| * musl should support EDNS and DNS over TCP/IP without issues
|
| * People should be smart enough to use DNS services that don't
| have stupid edge cases
|
| For the latter, if you use Google for resolving DNS, you get what
| you deserve. Run your own resolver if DNS resolution matters.
| h1fra wrote:
| DNS in Alpine is notoriously buggy but it can get months until
| you realise that. One easy and effective solution is to force dns
| resolution like so
|
| dnsConfig: options: - name: ndots value: '1'
|
| cc: https://support.cloudbees.com/hc/en-
| us/articles/360040999471...
|
| There are also plenty of dormant issue, enough so that I won't be
| using Alpine ever again imo :'(
___________________________________________________________________
(page generated 2022-03-11 23:01 UTC)