[HN Gopher] Show HN: Subtrace - Wireshark for Docker Containers
___________________________________________________________________
Show HN: Subtrace - Wireshark for Docker Containers
Hey HN, we built Subtrace (https://subtrace.dev) to let you see all
incoming and outgoing requests in your backend server--like
Wireshark, but for Docker containers. It comes with a Chrome
DevTools-like interface. Check out this video:
https://www.youtube.com/watch?v=OsGa6ZwVxdA, and see our docs for
examples: https://docs.subtrace.dev. Subtrace lets you see every
request with full payload, headers, status code, and latency
details. Tools like Sentry and OpenTelemetry often leave out these
crucial details, making prod debugging slow and annoying. Most of
the time, all I want to see are the headers and JSON payload of
real backend requests, but it's impossible to do that in today's
tools without excessive logging, which just makes everything slower
and more annoying. Subtrace shows you every backend request
flowing through your system. You can use simple filters to search
for the requests you care about and inspect their details.
Internally, Subtrace intercepts all network-related Linux syscalls
using Seccomp BPF so that it can act as a proxy for all incoming
and outgoing TCP connections. It then parses HTTP requests out of
the proxied TCP stream and sends them to the browser over
WebSocket. The Chrome DevTools Network tab is already ubiquitous
for viewing HTTP requests in the frontend, so we repurposed it to
work in the browser like any other app (we were surprised that it's
just a bunch of TypeScript). Setup is just one command for any
Linux program written in any language. You can use Subtrace by
adding a `subtrace run` prefix to your backend server startup
command. No signup required. Try for yourself:
https://docs.subtrace.dev
Author : adtac
Score : 156 points
Date : 2025-02-18 23:29 UTC (23 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| Onkar-Hanchate wrote:
| Interesting! How does this handle latency? Does it introduce any
| noticeable delay?
| adtac wrote:
| More than 0.01 ms, less than 0.10 ms. Noticeable when you're a
| high frequency trading firm, probably not otherwise :)
|
| In the hot path, Subtrace is just a dumb proxy that copies
| bytes. All of the processing + indexing happens offline in a
| different machine (in Clickhouse).
| smw wrote:
| Can it decrypt tls? Perhaps by hooking the calls to common
| libraries?
| adtac wrote:
| Yes, but we've managed to do it automatically without any
| library/language specific hooks! It's probably one of my
| favourite things in Subtrace :)
|
| We generate an ephemeral TLS root CA certificate and inject it
| into the system store. The generated certificate is entirely
| in-memory and never leaves the machine. To make this work
| without root privileges, we intercept the open(2) syscall to
| see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent).
| If so, we append the ephemeral root CA to the list of actual CA
| certificates; if not, we let the kernel handle the file open
| like usual. This way, none of the other programs are affected,
| so only the program you start with `subtrace run` sees and
| trusts the ephemeral root CA.
|
| After we get the program to trust the ephemeral root CA, we can
| proxy outgoing TLS connections through Subtrace transparently
| but also read the cleartext bytes.
|
| All of this is fully automatic, of course.
| hassleblad23 wrote:
| Wow. I think more network inspection tools should adopt an
| approach like this. Great job.
| cyberax wrote:
| I can't decide if I'm horrified or amazed by this :)
| memhole wrote:
| Up front, this is not my area of expertise. You would still
| not want to run this on the same server as your containers
| since someone could inject their own cert? I think it allows
| someone to decrypt traffic that isn't just proxied to
| Subtrace?
|
| Edit: I've reviewed the docs and it looks like you do run it
| on the same server. For clarity, I've used Sentry before.
| adtac wrote:
| Subtrace proxies the program's connection using a regular
| TLS connection to the upstream server. For example, if you
| do `subtrace run -- curl https://example.com`, curl thinks
| it's talking to example.com over TLS, but it's really
| talking to Subtrace locally. Since we injected the
| ephemeral root CA into the system store, curl will trust
| the valid TLS certificate that Subtrace presents for
| example.com. From within the same server, Subtrace will
| handle the actual TLS connection to upstream example.com.
| That upstream connection is undecipherable to outsiders.
|
| Everything is exactly as secure as before Subtrace. In
| other words, using Subtrace doesn't make the NSA's job any
| easier ;)
| memhole wrote:
| That's helpful! Thanks for clarifying. I'll have to check
| it out.
| Arnavion wrote:
| This will not work with HPKP but hopefully nothing is using
| that any more. (
| https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )
|
| It won't work with programs that defensively validate the
| cert chain but those are rare.
|
| It won't work with programs that embed their own root cert
| store, which is also rare but I would guess less rare than
| the previous one. The usual reason to do this is to minimize
| OS deps, and in the case of Docker containers to save on
| container image size by only including the roots you care
| about.
|
| But yes for the vast majority of programs it should work
| fine.
| adtac wrote:
| Yep, certificate pinning is the one scenario Subtrace can't
| handle in my experience, but thankfully, it's fairly rare
| like you said. And IMO there is no general solution to the
| problem [1], but it's one of those very interesting
| problems to daydream thinking about when you're stuck in
| traffic or whatever :)
|
| We still try our best by handling as much of the long tail
| of environments with some library/framework specific
| workaround (e.g. Deno bundles all TLS certs in its binary
| so we set the DENO_CERT env var when applicable).
|
| [1] https://news.ycombinator.com/item?id=42923998
| TachyonicBytes wrote:
| Is this a different method from the httptap [1] that was on
| hackernews a few weeks ago? Somebody in that post seemed to
| say that it also generates CA certificates on the fly.
|
| [1] https://news.ycombinator.com/item?id=42919909
| adtac wrote:
| httptap is really cool! Their technique is different (they
| do a filesystem mount instead of intercepting syscalls like
| Subtrace does) but both tools effectively reach the same
| goal using different routes.
| 29athrowaway wrote:
| You can use mitmproxy and mitmweb to achieve the same. It is in
| Docker hub and you can pass environment variables to your other
| containers to make it work.
|
| The TLS certificate setup is more tricky but that is always going
| to be a pain.
|
| Burp Proxy is another great tool that is even more powerful but
| harder to set up.
| vednig wrote:
| please add YC to the title
| choilive wrote:
| Always wanted a tool like this. Will try it out next time I need
| to inspect traffic of a docker container.
| kristopolous wrote:
| stratoshark, the docker container part of wireshark, may be a
| better match for that description.
|
| I'd probably use a postman related pitch instead. This is much
| closer to that and looks like a nice complement to that workflow
| westurner wrote:
| Stratoshark: https://wiki.wireshark.org/Stratoshark :
|
| > _Stratoshark captures and analyzes system calls and logs
| using libsinsp and libscap, and can share capture files with
| the Sysdig command line tool and Falco_
| arguflow wrote:
| Very happy subtrace user here. Especially useful to possess the
| Server-Timing headers.
| adtac wrote:
| standards ftw!
| ayewo wrote:
| Can you share more about how you are doing this?
|
| IIRC, Server-Timing headers are Firefox-only and they are not
| available via fetch or XHR.
| jgauth wrote:
| Looks like it is for http requests only? If so, wireshark is not
| an apt comparison.
| adtac wrote:
| For now, yes :)
|
| Since we operate at the TCP level, we can actually handle
| pretty much any protocol. I have an implementation of a
| postgres handler in my git stash that intercepts and shows the
| SQL queries executed + the resulting rows alongside the HTTP
| request that triggered it (I still need to do some robustness
| and correctness testing before it's ready to merge). With a
| handful of other protocols like MySQL, Mongo, Redis, Kafka, or
| even FTP lol, I think Subtrace can cover most practical dev
| workloads.
|
| Btw Subtrace can already record .pcap files today since it's
| just a simple TCP stream proxy, but raw network packet captures
| are mostly only useful when you're implementing new protocols,
| which 99% of the people using Docker containers today aren't
| doing. It's also a solved problem because you can just run
| `apt-get install tcpdump` inside the container.
|
| Automatic tracing for _app-level protocols_ that is easy to
| setup, works everywhere, lightweight for prod, fast to search,
| and can show the data in a clean interface is still insanely
| difficult today. That 's the problem Subtrace is trying to
| solve.
| sophacles wrote:
| It's a pretty cool looking product. It's not wireshark, it's
| not close to wireshark just because it can capture some tcp
| pcaps, and there are more protocols relevant to container
| networking than a handful of TCP app-level protocols.
|
| When I came in I was hoping to see a product that actually
| was for container networking, not just app data flows. Again
| - this is a neat tool, and probably incredibly useful for
| people developing way up the stack like that, but a lot of us
| live below the bottom of a "full-stack developer's" stack.
| Some features I would expect in a "wireshark for Docker
| containers":
|
| * Ability to inspect DNS traffic
|
| * Ability to trace packets the enter the conainter network
| stack (e.g. the packet(s) generated when the server calls
| send() ) into the virtual interface for the namespace and
| through the host machine. Ideally with multiple observation
| points and correlation of packets in the overlay/underlay
| contexts (decrypting from local keys whenever possible).
|
| * Ability to see where a packet dies between the app and exit
| NIC on the host. Including things like "packets delivered to
| this container even though the dest IP/subnet isn't in this
| container"
|
| * Similar to the previous point: ability to track packets
| through all the NAT steps containers introduce.
|
| * See arp traffic on virtual interfaces.
|
| * Ability to observe the TLS handshake and gather all the
| parameters of the connection.
|
| * Packet dissection and protocol session tracing for all the
| tunnels.
|
| * Bonus points if you can capture weird teleports caused by
| ebpf programs in the network path.
|
| I expect this because that's how I use wireshark_+ bpftrace
| in container environments for the most part. I've also used
| it to debug while implementing protocols, but that's a less
| common use case of packet dissection and tracing in
| wireshark.
|
| What you've built is cool, and I can see it expanding in a
| lot of directions very, very, usefully. I just really dislike
| something calling itself a wireshark while not really helping
| with networking (and in fact - the networking has to work
| reasonably well for this tool to be effective).
| adtac wrote:
| That's totally fair, I can see why Wireshark wouldn't be
| the most accurate description to someone working on those
| kinds of problems. And fwiw, I wish my problems (before
| Subtrace) were cool enough to need to whip out Wireshark
| for packet-level inspection :)
| polithrow22 wrote:
| anything similar for k8s?
| adtac wrote:
| Subtrace already works great on Kubernetes
| (https://docs.subtrace.dev/kubernetes)! Add a single line to
| your image's Dockerfile and that's it.
|
| I'm working on an even simpler way where you can just `kubectl
| apply` a DaemonSet or a Helm chart to get automatic tracing for
| _all pods_ in your cluster instantly without any code-level
| changes. If anyone is interested in beta testing this, email me
| at adtac@subtrace.dev, I 'd love to understand your usecase!
| robinhoodexe wrote:
| It'd be neat to use subtrace in an ephemeral pod for
| debugging purposes, that just runs alongside the regular pod.
|
| For monitoring the network traffic for the whole cluster, the
| CNI and/or whatever ebpf-based runtime security stuff you're
| using (falco, tetragon, tracee) is usually enough, but I can
| definitely see the usefulness of subtract for more specific
| debugging purposes. If run as a DaemonSet make sure to add
| some pod filtering such as namespace and label selectors (but
| I'm sure you've already thought about that).
| adtac wrote:
| > use subtrace in an ephemeral pod for debugging purposes
|
| That's a great suggestion. It'd be like kubectl exec-ing
| into a shell inside the pod, but for network activity. I
| think I'm going to prototype this tonight :)
|
| > pod filtering such as namespace and label selectors
|
| Yep, Subtrace already tags each request with a bunch of
| metadata about the place where it originated so that you
| can filter on those in the dashboard :) Things like the
| hostname, pod, cluster, AWS/GCP location are automatically
| populated, but you can also set custom tags in the config
| [1].
|
| [1] https://docs.subtrace.dev/tags
| ddelnano wrote:
| Disclaimer: I'm a maintainer of the project
|
| Pixie (https://px.dev) can be installed in under 5 mins and
| gives this level of visibility across all applications. No need
| to change your application (wrap in `subtrace run`) to get
| instant visibility.
|
| We also support 11 application protocols
| (https://docs.px.dev/reference/datatables/) with TLS handshake
| tracing and MQTT support coming soon (encrypted traffic tracing
| has been supported for a long time).
| qwertox wrote:
| Wireshark seems a bit misleading. More like a "network inspector"
| if one leans towards the browser's network tab in the inspector?
|
| But it really looks useful and I'll definitely play with it to
| see if I put it into my toolbox.
| adtac wrote:
| Thanks!
|
| re the Wireshark analogy: the reason I used that was because:
| (1) Subtrace operates at roughly the same level in the
| operating system stack, (2) has the similar capabilities, (3)
| has a large overlap in use-cases even if not entirely the same,
| and (4) has been the most effective at communicating what
| Subtrace is in my experience so far. I can see why the analogy
| is not a perfect 1:1 mapping (obligatory xkcd:
| https://xkcd.com/624), but naming things is hard and taglines
| are just names in idea space :)
| ksdme9 wrote:
| Have not played around with it, but, curious, how does debugging
| on production work for a specific request/session? Can I filter
| by some sort if request trace id or something?
| adtac wrote:
| You can tag each request with arbitrary key-value maps and
| search over these later. For example, if you add a `x-subtrace-
| tags: user=foo, project=bar` header on the response, you can
| apply a `tags.user == "foo"` filter in the dashboard to see all
| requests across your entire infra from that user and _only_
| that user. Each request is pre-populated by default with tags
| like hostname, pod name, AWS /GCP location, etc.
|
| It's like Honeycomb's wide events but even better because: (1)
| you can see whole request including the payload alongside the
| event fields, and (2) it's fully automatic and requires no code
| changes out of the box (you can incrementally add these tags
| when you find a need for each one instead of the huge upfront
| cost from instrumenting the hell out of your codebase).
___________________________________________________________________
(page generated 2025-02-19 23:00 UTC)