[HN Gopher] Show HN: Subtrace - Wireshark for Docker Containers
___________________________________________________________________
Show HN: Subtrace - Wireshark for Docker Containers
Hey HN, we built Subtrace (https://subtrace.dev) to let you see all
incoming and outgoing requests in your backend server--like
Wireshark, but for Docker containers. It comes with a Chrome
DevTools-like interface. Check out this video:
https://www.youtube.com/watch?v=OsGa6ZwVxdA, and see our docs for
examples: https://docs.subtrace.dev. Subtrace lets you see every
request with full payload, headers, status code, and latency
details. Tools like Sentry and OpenTelemetry often leave out these
crucial details, making prod debugging slow and annoying. Most of
the time, all I want to see are the headers and JSON payload of
real backend requests, but it's impossible to do that in today's
tools without excessive logging, which just makes everything slower
and more annoying. Subtrace shows you every backend request
flowing through your system. You can use simple filters to search
for the requests you care about and inspect their details.
Internally, Subtrace intercepts all network-related Linux syscalls
using Seccomp BPF so that it can act as a proxy for all incoming
and outgoing TCP connections. It then parses HTTP requests out of
the proxied TCP stream and sends them to the browser over
WebSocket. The Chrome DevTools Network tab is already ubiquitous
for viewing HTTP requests in the frontend, so we repurposed it to
work in the browser like any other app (we were surprised that it's
just a bunch of TypeScript). Setup is just one command for any
Linux program written in any language. You can use Subtrace by
adding a `subtrace run` prefix to your backend server startup
command. No signup required. Try for yourself:
https://docs.subtrace.dev
Author : adtac
Score : 351 points
Date : 2025-02-18 23:29 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| Onkar-Hanchate wrote:
| Interesting! How does this handle latency? Does it introduce any
| noticeable delay?
| adtac wrote:
| More than 0.01 ms, less than 0.10 ms. Noticeable when you're a
| high frequency trading firm, probably not otherwise :)
|
| In the hot path, Subtrace is just a dumb proxy that copies
| bytes. All of the processing + indexing happens offline in a
| different machine (in Clickhouse).
| smw wrote:
| Can it decrypt tls? Perhaps by hooking the calls to common
| libraries?
| adtac wrote:
| Yes, but we've managed to do it automatically without any
| library/language specific hooks! It's probably one of my
| favourite things in Subtrace :)
|
| We generate an ephemeral TLS root CA certificate and inject it
| into the system store. The generated certificate is entirely
| in-memory and never leaves the machine. To make this work
| without root privileges, we intercept the open(2) syscall to
| see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent).
| If so, we append the ephemeral root CA to the list of actual CA
| certificates; if not, we let the kernel handle the file open
| like usual. This way, none of the other programs are affected,
| so only the program you start with `subtrace run` sees and
| trusts the ephemeral root CA.
|
| After we get the program to trust the ephemeral root CA, we can
| proxy outgoing TLS connections through Subtrace transparently
| but also read the cleartext bytes.
|
| All of this is fully automatic, of course.
| hassleblad23 wrote:
| Wow. I think more network inspection tools should adopt an
| approach like this. Great job.
| cyberax wrote:
| I can't decide if I'm horrified or amazed by this :)
| 1oooqooq wrote:
| as with anything too clever and undocumented, you will be
| amazed until you waste a few weeks debugging an issue
| caused by it.
| memhole wrote:
| Up front, this is not my area of expertise. You would still
| not want to run this on the same server as your containers
| since someone could inject their own cert? I think it allows
| someone to decrypt traffic that isn't just proxied to
| Subtrace?
|
| Edit: I've reviewed the docs and it looks like you do run it
| on the same server. For clarity, I've used Sentry before.
| adtac wrote:
| Subtrace proxies the program's connection using a regular
| TLS connection to the upstream server. For example, if you
| do `subtrace run -- curl https://example.com`, curl thinks
| it's talking to example.com over TLS, but it's really
| talking to Subtrace locally. Since we injected the
| ephemeral root CA into the system store, curl will trust
| the valid TLS certificate that Subtrace presents for
| example.com. From within the same server, Subtrace will
| handle the actual TLS connection to upstream example.com.
| That upstream connection is undecipherable to outsiders.
|
| Everything is exactly as secure as before Subtrace. In
| other words, using Subtrace doesn't make the NSA's job any
| easier ;)
| memhole wrote:
| That's helpful! Thanks for clarifying. I'll have to check
| it out.
| Arnavion wrote:
| This will not work with HPKP but hopefully nothing is using
| that any more. (
| https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )
|
| It won't work with programs that defensively validate the
| cert chain but those are rare.
|
| It won't work with programs that embed their own root cert
| store, which is also rare but I would guess less rare than
| the previous one. The usual reason to do this is to minimize
| OS deps, and in the case of Docker containers to save on
| container image size by only including the roots you care
| about.
|
| But yes for the vast majority of programs it should work
| fine.
| adtac wrote:
| Yep, certificate pinning is the one scenario Subtrace can't
| handle in my experience, but thankfully, it's fairly rare
| like you said. And IMO there is no general solution to the
| problem [1], but it's one of those very interesting
| problems to daydream thinking about when you're stuck in
| traffic or whatever :)
|
| We still try our best by handling as much of the long tail
| of environments with some library/framework specific
| workaround (e.g. Deno bundles all TLS certs in its binary
| so we set the DENO_CERT env var when applicable).
|
| [1] https://news.ycombinator.com/item?id=42923998
| chatmasta wrote:
| No general solution? :) If you came up with this hack I'm
| sure you can extend it...
|
| Cert pinning has to read a public cert from memory,
| right? And a public cert has a well-known shape... and
| you have bpf and access to the memory...
| adtac wrote:
| in a different context, maybe:
| https://news.ycombinator.com/item?id=38044786 :)
| smw wrote:
| kubeshark [0] is using ebpf to catch calls to
| openssl/go's tls lib and thus no need to juggle certs.
| Has pros and cons compared to your method, but an
| interesting comparison.
|
| [0] https://www.kubeshark.co/
| ddelnano wrote:
| The approach you describe above is common for similar
| projects:
|
| - Pixie (https://px.dev) -- which I contribute to
|
| - Beyla (https://github.com/grafana/beyla)
|
| - Coroot (https://github.com/coroot/coroot)
|
| If you are interested in the details and how the strategy
| for this tracing has evolved, you can learn more in this
| blog (https://blog.px.dev/ebpf-tls-tracing-past-present-
| future/).
| TachyonicBytes wrote:
| Is this a different method from the httptap [1] that was on
| hackernews a few weeks ago? Somebody in that post seemed to
| say that it also generates CA certificates on the fly.
|
| [1] https://news.ycombinator.com/item?id=42919909
| adtac wrote:
| httptap is really cool! Their technique is different (they
| do a filesystem mount instead of intercepting syscalls like
| Subtrace does) but both tools effectively reach the same
| goal using different routes.
| chatmasta wrote:
| This is a great hack. It won't work 100% of the time but
| it'll be close, and damn it's just so clean. Proxies and
| intercepts all the way down... I love this.
| 29athrowaway wrote:
| You can use mitmproxy and mitmweb to achieve the same. It is in
| Docker hub and you can pass environment variables to your other
| containers to make it work.
|
| The TLS certificate setup is more tricky but that is always going
| to be a pain.
|
| Burp Proxy is another great tool that is even more powerful but
| harder to set up.
| vednig wrote:
| please add YC to the title
| choilive wrote:
| Always wanted a tool like this. Will try it out next time I need
| to inspect traffic of a docker container.
| kristopolous wrote:
| stratoshark, the docker container part of wireshark, may be a
| better match for that description.
|
| I'd probably use a postman related pitch instead. This is much
| closer to that and looks like a nice complement to that workflow
| westurner wrote:
| Stratoshark: https://wiki.wireshark.org/Stratoshark :
|
| > _Stratoshark captures and analyzes system calls and logs
| using libsinsp and libscap, and can share capture files with
| the Sysdig command line tool and Falco_
| mdaniel wrote:
| Stratoshark was discussed last month
| https://news.ycombinator.com/item?id=42793777
| dang wrote:
| Thanks! Macroexpanded:
|
| _Show HN: Stratoshark, a sibling application to Wireshark_ -
| https://news.ycombinator.com/item?id=42793777 - Jan 2025 (50
| comments)
| arguflow wrote:
| Very happy subtrace user here. Especially useful to possess the
| Server-Timing headers.
| adtac wrote:
| standards ftw!
| ayewo wrote:
| Can you share more about how you are doing this?
|
| IIRC, Server-Timing headers are Firefox-only and they are not
| available via fetch or XHR.
| jgauth wrote:
| Looks like it is for http requests only? If so, wireshark is not
| an apt comparison.
| adtac wrote:
| For now, yes :)
|
| Since we operate at the TCP level, we can actually handle
| pretty much any protocol. I have an implementation of a
| postgres handler in my git stash that intercepts and shows the
| SQL queries executed + the resulting rows alongside the HTTP
| request that triggered it (I still need to do some robustness
| and correctness testing before it's ready to merge). With a
| handful of other protocols like MySQL, Mongo, Redis, Kafka, or
| even FTP lol, I think Subtrace can cover most practical dev
| workloads.
|
| Btw Subtrace can already record .pcap files today since it's
| just a simple TCP stream proxy, but raw network packet captures
| are mostly only useful when you're implementing new protocols,
| which 99% of the people using Docker containers today aren't
| doing. It's also a solved problem because you can just run
| `apt-get install tcpdump` inside the container.
|
| Automatic tracing for _app-level protocols_ that is easy to
| setup, works everywhere, lightweight for prod, fast to search,
| and can show the data in a clean interface is still insanely
| difficult today. That 's the problem Subtrace is trying to
| solve.
| sophacles wrote:
| It's a pretty cool looking product. It's not wireshark, it's
| not close to wireshark just because it can capture some tcp
| pcaps, and there are more protocols relevant to container
| networking than a handful of TCP app-level protocols.
|
| When I came in I was hoping to see a product that actually
| was for container networking, not just app data flows. Again
| - this is a neat tool, and probably incredibly useful for
| people developing way up the stack like that, but a lot of us
| live below the bottom of a "full-stack developer's" stack.
| Some features I would expect in a "wireshark for Docker
| containers":
|
| * Ability to inspect DNS traffic
|
| * Ability to trace packets the enter the conainter network
| stack (e.g. the packet(s) generated when the server calls
| send() ) into the virtual interface for the namespace and
| through the host machine. Ideally with multiple observation
| points and correlation of packets in the overlay/underlay
| contexts (decrypting from local keys whenever possible).
|
| * Ability to see where a packet dies between the app and exit
| NIC on the host. Including things like "packets delivered to
| this container even though the dest IP/subnet isn't in this
| container"
|
| * Similar to the previous point: ability to track packets
| through all the NAT steps containers introduce.
|
| * See arp traffic on virtual interfaces.
|
| * Ability to observe the TLS handshake and gather all the
| parameters of the connection.
|
| * Packet dissection and protocol session tracing for all the
| tunnels.
|
| * Bonus points if you can capture weird teleports caused by
| ebpf programs in the network path.
|
| I expect this because that's how I use wireshark_+ bpftrace
| in container environments for the most part. I've also used
| it to debug while implementing protocols, but that's a less
| common use case of packet dissection and tracing in
| wireshark.
|
| What you've built is cool, and I can see it expanding in a
| lot of directions very, very, usefully. I just really dislike
| something calling itself a wireshark while not really helping
| with networking (and in fact - the networking has to work
| reasonably well for this tool to be effective).
| adtac wrote:
| That's totally fair, I can see why Wireshark wouldn't be
| the most accurate description to someone working on those
| kinds of problems. And fwiw, I wish my problems (before
| Subtrace) were cool enough to need to whip out Wireshark
| for packet-level inspection :)
| polithrow22 wrote:
| anything similar for k8s?
| adtac wrote:
| Subtrace already works great on Kubernetes
| (https://docs.subtrace.dev/kubernetes)! Add a single line to
| your image's Dockerfile and that's it.
|
| I'm working on an even simpler way where you can just `kubectl
| apply` a DaemonSet or a Helm chart to get automatic tracing for
| _all pods_ in your cluster instantly without any code-level
| changes. If anyone is interested in beta testing this, email me
| at adtac@subtrace.dev, I 'd love to understand your usecase!
| robinhoodexe wrote:
| It'd be neat to use subtrace in an ephemeral pod for
| debugging purposes, that just runs alongside the regular pod.
|
| For monitoring the network traffic for the whole cluster, the
| CNI and/or whatever ebpf-based runtime security stuff you're
| using (falco, tetragon, tracee) is usually enough, but I can
| definitely see the usefulness of subtract for more specific
| debugging purposes. If run as a DaemonSet make sure to add
| some pod filtering such as namespace and label selectors (but
| I'm sure you've already thought about that).
| adtac wrote:
| > use subtrace in an ephemeral pod for debugging purposes
|
| That's a great suggestion. It'd be like kubectl exec-ing
| into a shell inside the pod, but for network activity. I
| think I'm going to prototype this tonight :)
|
| > pod filtering such as namespace and label selectors
|
| Yep, Subtrace already tags each request with a bunch of
| metadata about the place where it originated so that you
| can filter on those in the dashboard :) Things like the
| hostname, pod, cluster, AWS/GCP location are automatically
| populated, but you can also set custom tags in the config
| [1].
|
| [1] https://docs.subtrace.dev/tags
| ddelnano wrote:
| Disclaimer: I'm a maintainer of the project
|
| Pixie (https://px.dev) can be installed in under 5 mins and
| gives this level of visibility across all applications. No need
| to change your application (wrap in `subtrace run`) to get
| instant visibility.
|
| We also support 11 application protocols
| (https://docs.px.dev/reference/datatables/) with TLS handshake
| tracing and MQTT support coming soon (encrypted traffic tracing
| has been supported for a long time).
| ZloyKrys wrote:
| Might be an overkill to use it in this capacity but SUSE
| NeuVector (https://github.com/neuvector/neuvector) can build a
| connectivity graph between pods and allows to perform packet
| capture to PCAP. It works at the virtual switch layer and can
| capture all traffic.
| qwertox wrote:
| Wireshark seems a bit misleading. More like a "network inspector"
| if one leans towards the browser's network tab in the inspector?
|
| But it really looks useful and I'll definitely play with it to
| see if I put it into my toolbox.
| adtac wrote:
| Thanks!
|
| re the Wireshark analogy: the reason I used that was because:
| (1) Subtrace operates at roughly the same level in the
| operating system stack, (2) has similar capabilities, (3) has
| an overlap in use-cases, and (4) has been the most effective at
| communicating what Subtrace is in my experience so far. I can
| see why the analogy is not a perfect 1:1 mapping (obligatory
| xkcd: https://xkcd.com/624), but naming things is hard and
| taglines are just names in idea space :)
| conradev wrote:
| Can I download a pcap file? Being able to view and debug the
| application protocol is one thing, but I can use Wireshark to
| view and debug TCP itself.
|
| Reading the documentation makes it sound like this sits one
| or two levels above where Wireshark usually operates, which
| is why I think the analogy is tough.
| jolmg wrote:
| My first impression with "Wireshark for Docker containers" is
| "... Well, Wireshark already works with Docker containers, so
| if your goal is to be Wireshark for Docker containers, you'll
| just fail by definition. Can't beat Wireshark at being
| Wireshark."
|
| I'm saying this just FYI. I haven't actually looked at what
| your product does, but if it were to matter to me, it'd be
| based on what it can offer that Wireshark can't, rather than
| how similar it is to Wireshark.
| beardedwizard wrote:
| I'm struggling to understand why I wouldn't just use
| wireshark and specify the interface as well. Reading docs
| shows you need to invoke your container with this as a
| wrapper which to me is quite a bit scarier than attaching
| to an interface.
| Mekoloto wrote:
| Wireshark is a weird choice in my opinion because you know,
| why would i use Subtrace if i can use Wireshark?
|
| and People who don't know how to use Wireshark want probably
| a better motivation to use Subtrace than 'its like wireshark'
| because there is a reason why they don't use it.
| chatmasta wrote:
| I can already use Wireshark itself with docker containers.
|
| IMO the real UX problem is that there's nothing in between
| "Wireshark" and "DevTools."
|
| What would be cool is a program that renders the real DevTools
| UI from a .pcap file (or stdin).
|
| There's a lot of use cases where I can capture all the traffic
| with Wireshark but I just want a basic DevTools interface.
| ksdme9 wrote:
| Have not played around with it, but, curious, how does debugging
| on production work for a specific request/session? Can I filter
| by some sort if request trace id or something?
| adtac wrote:
| You can tag each request with arbitrary key-value maps and
| search over these later. For example, if you add a `x-subtrace-
| tags: user=foo, project=bar` header on the response, you can
| apply a `tags.user == "foo"` filter in the dashboard to see all
| requests across your entire infra from that user and _only_
| that user. Each request is pre-populated by default with tags
| like hostname, pod name, AWS /GCP location, etc.
|
| It's like Honeycomb's wide events but even better because: (1)
| you can see whole request including the payload alongside the
| event fields, and (2) it's fully automatic and requires no code
| changes out of the box (you can incrementally add these tags
| when you find a need for each one instead of the huge upfront
| cost from instrumenting the hell out of your codebase).
| johannes1234321 wrote:
| From the video it seems the dashboard is hosted on sibtrace.dev.
| Is my data being sent there? That's an absolute no-go for me.
| adtac wrote:
| You can use Subtrace entirely locally without talking to
| subtrace.dev using the -devtools flag. For example:
| subtrace run -devtools=/subtrace -- python3 -m http.server
|
| Then go to http://localhost:8000/subtrace and you'll see the
| Chrome DevTools interface running in your browser like a
| regular app. Any request sent to http://localhost:8000 should
| appear there in realtime. Note that the -devtools flag is kinda
| best effort because Subtrace is intended to be used in
| production.
|
| We have fully on-prem installations for enterprises, of course
| (e.g. if you're in a highly regulated industry like
| healthcare).
| zekenie wrote:
| This is such a cool product! I was looking through the docs
| and I think this could be a lot more prominent. Thanks for
| building in public!
| rob_c wrote:
| or, there's you know... wireshark...
| codegeek wrote:
| Side question. Why not do a Launch HN instead of Show HN since
| you are backed by YC ? I thought YC companies can do a launch HN?
| dang wrote:
| They can, but for open-source startups I recommend they do a
| Show HN first, for the following reasons:
|
| (1) If the Show HN succeeds, then they can save their Launch HN
| card to play later; if it doesn't succeed, they can go ahead
| and do a Launch HN soon afterwards.
|
| (2) Show HN is more about the project/tech, especially when
| it's open-source. Launch HN is more about the startup/business.
|
| (3) Nowadays many startups offer an open-source (or open core)
| version for free, then make money with a hosted version. For
| these, it's a good fit to first do a Show HN about the open-
| source offering, and then later do a Launch HN about the cloud
| product.
| 0xbadcafebee wrote:
| There needs to be a new term for this grassroots astroturf...
| Grasstroturf? Get the nerds' attention with free tech, down-
| playing the fact that it's a business, so later they can pump
| them for money easier. The pig doesn't want to see the knife,
| they just want the free trough.
| dang wrote:
| FWIW, I've spent time at YC open-source events where later-
| stage founders advise earlier-stage ones, and this isn't
| the kind of advice they give. In fact they would describe
| that as a terrible strategy and tell startups to avoid it.
|
| If I were to give an example of actual advice I've heard,
| using the terms of your comment, it might be: keep the
| nerds happy with free open-source software that actually
| works, then charge money for things that companies
| (especially enterprise companies) actually prefer to pay
| for.
|
| The "paid hosted offering" pattern is the most common of
| these, since nerds might want to run their own instances
| but many IT departments do not.
| kylegalbraith wrote:
| Do you envision going beyond just network calls from bpf?
| adtac wrote:
| What would you like to see?
|
| One of things we're thinking about is automatic method/function
| call tracing. Something like attaching the entire stack trace
| of calls done to handle the API request. Ideally using the same
| UI so that you can see the headers/payload that was sent and
| the function-level stack trace right next to each other. None
| of the OpenTelemetry verbosity, all of the observability!
| thebabayaga29 wrote:
| Cool product, I can imagine my engineers using it. Out of
| curiosity, how would you monitize that?
| adtac wrote:
| Flat pricing with unlimited requests hosted in the cloud
| (https://subtrace.dev/pricing).
|
| We also have an on-prem version of Subtrace for enterprises. It
| runs in their own AWS account without ever talking to
| subtrace.dev so that companies in regulated industries like
| healthcare can use Subtrace.
| withinboredom wrote:
| We use https://treblle.com/ at work for this in production. Very
| handy to see what requests are being made and by whom.
| gerwim wrote:
| Looks great! Reading through the docs it seems the subtrace
| process sends all data to your server. I'm reluctant to do that
| on a production environment, where API keys and personal data are
| being handled.
|
| Is there any way to run it completely self hosted? If not, are
| there plans? And how will you monitize self hosted options (if
| it's possible)?
| ddelnano wrote:
| Disclaimer: I'm a maintainer of the project
|
| Pixie [1] is a similar project and offers the self hosted model
| you are looking for.
|
| We also support 11 application protocols [2] with TLS handshake
| tracing and MQTT support coming soon (encrypted traffic tracing
| has been supported for a long time).
|
| [1] https://px.dev
|
| [2] https://docs.px.dev/reference/datatables/
| edoceo wrote:
| From a dictionary: The meaning of DISCLAIMER is a denial or
| disavowal of legal claim : relinquishment of or formal
| refusal to accept an interest or estate.
|
| Perhaps you meant DISCLOSURE
| adtac wrote:
| re self-hosting: yep! Use the -devtools flag to get a stripped
| down version of Subtrace running entirely locally:
| subtrace run -devtools=/subtrace -- python3 -m http.server
|
| This starts a Python server on localhost:8000 but with
| Subtrace. Everything except /subtrace is forwarded to the
| Python server like usual, but if you go to
| http://localhost:8000/subtrace you should see the Chrome
| DevTools network tab running in the browser like a regular app.
| Any request you send to localhost:8000 + all outgoing requests
| made from inside the Python app will automatically appear in
| that dashboard!
| Vordimous wrote:
| This will help those developers who don't want to navigate
| Wireshark's setup and just want the familiar Chrom DevTools. Very
| cool project!
| mhils wrote:
| Congrats on the seccomp-based interception, that's a really neat
| way to solve this problem! We did some BPF_PROG_TYPE_CGROUP_SOCK
| eBPF shenanigans in mitmproxy for redirection, but that doesn't
| work with containers at all. Cool to see that intercepting all
| relevant syscalls works that well.
| IggleSniggle wrote:
| My most painful debugging scenarios with Docker networking (for
| me) has always been dealing with non-TCP traffic. But still, this
| seems useful. One thing I don't understand is why this requires
| an account token? Does this require a network connection to
| subtrace? It seems like this should all be running locally, and
| these kinds of connection details are _exactly_ the kind I would
| not want to leave the host, let alone go to a third party.
| adtac wrote:
| One of my litmus tests for software is "can I use it on a
| flight?", so you'll be pleased to know Subtrace works great
| locally without ever talking to subtrace.dev!
| subtrace run -devtools=/subtrace -- python3 -m http.server
|
| Run the above command in a Linux machine, go to
| http://localhost:8000/subtrace and send some requests to
| localhost:8000 to see a stripped down version of the Subtrace
| dashboard working fully locally.
| sirjaz wrote:
| Any Windows Server support coming?
| adtac wrote:
| Not in the very short term, but we've talked about it
| internally and we might do it in the future!
|
| If you'd like to use Subtrace on Windows, it would be super
| helpful to understand your use-case deeply so that we build the
| right things in the right order. Please reach out to me at
| adtac@subtrace.dev, I'd love to chat!
___________________________________________________________________
(page generated 2025-02-20 23:02 UTC)