[HN Gopher] Show HN: Httpdbg - A tool to trace the HTTP requests...
___________________________________________________________________
Show HN: Httpdbg - A tool to trace the HTTP requests sent by your
Python code
Hi, I created httpdbg, a tool for Python developers to easily
debug HTTP(S) client requests in Python programs. I developed it
because I needed a tool that could help me trace the HTTP requests
sent by my tests back to the corresponding methods in our API
client. The goal of this tool is to simplify the debugging
process, so I designed it to be as simple as possible. It requires
no external dependencies, no setup, no superuser privileges, and no
code modifications. I'm sharing it with you today because I use it
regularly, and it seems like others have found it useful too--so it
might be helpful for you as well. Hope you will like it. cle
Source: https://github.com/cle-b/httpdbg Documentation:
https://httpdbg.readthedocs.io/ A blog post on a use case:
https://medium.com/@cle-b/trace-all-your-http-requests-in-py...
Author : cle-b
Score : 81 points
Date : 2024-09-25 19:18 UTC (3 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| billconan wrote:
| this is very useful, but why can it only work with python code?
| At which level does it intercept the http traffic?
|
| do I have to use specific http library?
| tredre3 wrote:
| It seems to intercept calls for some popular http client libs:
|
| https://github.com/cle-b/httpdbg/tree/main/httpdbg/hooks
| cle-b wrote:
| It works only with Python code because it intercepts HTTP
| requests by hooking into certain Python functions.
|
| It supports any HTTP library based on Python's standard socket
| library. Specifically, it works with libraries like requests,
| httpx, aiohttp, and urllib3, as well as pytest, providing more
| detailed information about the initiator of the requests.
| cdfuller wrote:
| Is there a way to use it with jupyter notebooks? `pyhttpdbg -m
| jupyter notebook` didn't work for me
| cle-b wrote:
| If you want to trace the HTTP requests in a notebook, you must
| install the package notebook-httpdbg.
|
| This is documented here:
| https://httpdbg.readthedocs.io/en/latest/notebook/
| nfgrars wrote:
| Alternatively use man (1) ngrep for http or man (1) openssl for
| https.
| stuaxo wrote:
| This is great -
|
| It would be good to be be able to have django debug toolbar
| integration, that way I could see which requests were made to
| backend APIs without leaving Django.
|
| Having tried MITMProxy something like httpdbg is definitely
| needed.
| diegoallen wrote:
| You can do that with django debug toolbar. If you have an
| endpoint that doesn't return HTML, and hence wouldn't render
| debug toolbar, you can go to django admin (or any other
| endpoint that would render ddt) and go to the history pane,
| check other requests and switch to them.
| ricardo81 wrote:
| I could be lost here (C/PHP/Node coder mainly in code I've used)
|
| Why is it a special case to track HTTP/s requests, that otherwise
| couldn't be logged like any other process/function? I'd guess
| most people use libcurl and you can wrap something around that.
|
| I guess I'm lost on why this is HTTP or Python specific, or if it
| is, fine.
| robertlagrant wrote:
| I think the nice thing about HTTP for this is different parts
| of the stack can introduce default headers etc and it's helpful
| to be able to see the actual request after all that
| processing's been done.
| ricardo81 wrote:
| With curl there's always CURLOPT_VERBOSE as per the library.
| seanc wrote:
| In the old days we'd use tcpdump and wireshark for this, but
| nowadays everything is encrypted up in the application layer so
| you need this kind of thing. Or tricky key dumping hacks.
| toomuchtodo wrote:
| https://www.charlesproxy.com/ ?
| cle-b wrote:
| Unlike other tools such as proxies that allow you to trace HTTP
| requests, httpdbg makes it possible to link the HTTP request to
| the Python code that initiated it. This is why it is specific
| to Python and does not work with other languages.
| ricardo81 wrote:
| I'm still not understanding.
|
| If you're coding something up, why wouldn't you know that
| piece of code does a HTTP/s request? Based on what you said,
| it sounds like a scenario where a programmer doesn't know how
| a request was made. Are there examples of scenarios where
| that's the case?
|
| Sounds like a bit of a security nightmare where there's code
| doing arbitrary requests.
| bityard wrote:
| Maybe you are working with an application or library that
| you didn't write, and want to see the raw requests and
| responses it generates without reading the entirety of the
| source code.
|
| Maybe you are generating HTTP requests through an API and
| need to see which headers it sets by default, or which
| headers are or are not getting set due to a
| misconfiguration or bug.
|
| There are probably loads more use cases, and if I actually
| did programming for a living, I could probably list a lot
| more.
| ricardo81 wrote:
| The 3rd party library stuff makes sense, to an extent.
| But then you're debugging a 3rd party library.
| diegoallen wrote:
| If a 3rd party library you depend on has bugs, you have
| bugs. And you need to either submit a patch to the
| library or find a workaround.
| ricardo81 wrote:
| Or just not use arbitrary 3rd party stuff hoping it works
| :)
|
| libcurl is used on billions of devices across the world
| and has plenty of debugging capabilities.
|
| MITM proxy works across all languages.
| fragmede wrote:
| The NIH is strong in this once.
| golergka wrote:
| > If you're coding something up, why wouldn't you know that
| piece of code does a HTTP/s request?
|
| Because tracing all side-effects in a huge codebase with a
| lot of libraries and layers can be a daunting task.
|
| Update: if you haven't worked with 20 year old >1m LOC
| codebase which went through many different teams and
| doesn't have any documentation whatsoever, you may lack
| necessary perspective to see value tools like this.
| ricardo81 wrote:
| Sounds like people dealing with code they have no idea
| what it does. No amount of tools are going to help with
| that.
| actionfromafar wrote:
| I think you attract downvotes because tools are helpful.
| If you have a huge unknown codebase, it can be nice to
| attack it from different angles. Reading code is useful,
| but observing what it does in runtime can be useful, too.
| Also, with hairier code, it can be more useful to first
| observe and prod it like a black box.
| ricardo81 wrote:
| Just the 1 downvote.
|
| Yes, "tools are helpful", but whether there's a
| python/http specific tool that doesn't do what more
| generic tools do remains to be seen.
| whirlwin wrote:
| Here's a concrete scenario for you: Say you are in a team
| of 10 developers with a huge codebase that has accumulated
| over 5+ years. If you're new in the team, and you need to
| understand when a specific HTTP header is sent, or just
| snoop the value in the payload you otherwise wouldn't be
| able to see.
| ricardo81 wrote:
| Snooping traffic isn't new though, so what's specific
| about this tool and Python.
| judofyr wrote:
| Looks neat!
|
| A similar tool for this would be VCR (originally built in Ruby,
| but ported to other languages since):
| https://vcrpy.readthedocs.io/en/latest/. This injects itself into
| the request pipeline, records the result in a local file which
| can then also be replayed later in tests. It's a quite nice
| approach when you want to write tests (or just explore) a highly
| complicated HTTP API without actually hitting it all the time.
| seanc wrote:
| The inspection and debugging features this offers are great
| additions though. I've stared at VCR yaml enough times to not
| want to ever do it again.
| cle-b wrote:
| I really like vrcpy. I used it a lot with pytest in my previous
| job. httpdbg isn't exactly the same; the idea is more about
| seeing HTTP requests in real-time and being able to easily
| study them.
| zug_zug wrote:
| I've always used a proxy, like charles proxy, for this exact
| purpose.
|
| A neutral middle-man that gives exact timing/response data.
| actionfromafar wrote:
| That's fine if you can, but say you want to trace deployed on
| stage, or _shudders_ even production. Or you code tests against
| some CI and can only add Python.
| hartator wrote:
| I wonder if the same exists for Ruby?
| Jugurtha wrote:
| That's pretty cool! I was playing last night and implemented
| resumable downloads[0] for pip so that it could pick up where it
| stopped upon a network disconnect or a user interruption. It
| sucks when large packages, especially ML related, fail at the
| last second and pip has to download from scratch. This tool would
| have been nice to have. Thanks a bunch,
|
| - [0]: https://asciinema.org/a/1r8HmOLCfHm40nSvEZBqwm89k
___________________________________________________________________
(page generated 2024-09-25 23:00 UTC)