[HN Gopher] Mitmproxy 7.0
___________________________________________________________________
Mitmproxy 7.0
Author : mhils
Score : 216 points
Date : 2021-07-16 12:15 UTC (10 hours ago)
(HTM) web link (mitmproxy.org)
(TXT) w3m dump (mitmproxy.org)
| beermonster wrote:
| I've used this tool to identify and fix security issues and have
| also taught software testers how to use it just to spread the
| love and knowledge. Kudos!
| ericb wrote:
| Such an amazing and well-maintained tool!!!
|
| Anyone interested in sharing use-cases where it has helped you?
| mhils wrote:
| We have a super diverse userbase, so this is a fun question!
| Some pointers:
|
| Development:
|
| - Debug your own apps/devices (which do not come with Chrome
| Devtools).
|
| - Simple Python scripts to replace/inject website content.
| (https://docs.mitmproxy.org/stable/addons-overview/)
|
| - Capture a web page/API and then replay the responses offline.
| (https://docs.mitmproxy.org/stable/overview-
| features/#server-...).
|
| - Replay requests for testing.
| (https://docs.mitmproxy.org/stable/overview-
| features/#client-...)
|
| Privacy:
|
| - Figure out what your Covid app is sending out
| (https://seancoates.com/blogs/how-i-helped-fix-canadas-
| covid-...)
|
| - Look at how evil IoT devices for pets are
| (https://www.nytimes.com/2018/12/11/realestate/spying-on-
| the-...)
|
| - Generally find more evil stuff
| (https://mitmproxy.org/publications/)
|
| Security:
|
| - Reverse-engineering custom binary protocols is never fun, but
| with mitmproxy you can at least write custom contentviews to
| dissect stuff. (https://github.com/mitmproxy/mitmproxy/blob/mai
| n/examples/ad...)
|
| - Fiddle with generic TLS-protected TCP streams. (https://githu
| b.com/mitmproxy/mitmproxy/blob/main/examples/ad...)
|
| - Dump TLS master secrets for WireShark.
| (https://docs.mitmproxy.org/stable/howto-wireshark-tls/)
| mpicker0 wrote:
| We have a product that depends on data received from external
| web services. We often want to test certain conditions in the
| data (specific results, error scenarios, etc), and it's
| difficult or impossible to guarantee these conditions when we
| call out to our external services. We use mitmproxy to
| intercept and manipulate the responses we get.
|
| Even if we don't need to manipulate the data, it's a very
| useful tool for watching requests go back and forth.
| thinkmassive wrote:
| I've used it to diagnose (and work around) webhook problems
| between third-party applications that are supposedly
| compatible.
|
| One that sticks out in my mind is trying to receive Slack
| webhooks in Mattermost. I was unaware of Slack's new BlockKit
| message format until I saw what was being sent. Then I wrote a
| simple python script to transform the payload, all within
| mitmproxy. It's not the most efficient long term solution, but
| the fact I could diagnose and resolve the issue with a single
| tool, all within a few hours of first discovering it, was very
| satisfying.
|
| Here's the result for anyone interested (warning: it'sa quick
| hack, not a polished/maintained project)
|
| https://github.com/thinkmassive/mattermost-webhook-slack-pro...
|
| Edit: this doesn't use TLS at all. I understand that's probably
| the primary use case for the tool, but I thought this was a
| cool example of other ways it can be useful too.
| Proven wrote:
| Tried it last week to see what APIs are used to communicate
| with home appliance. Couldn't get it to work.
| KORraN wrote:
| I've introduced mitmproxy in my team and we use it quite often
| for flows' inspection and manipulation while testing a desktop
| app. Can't count cases where a quick look into a proxy made
| debugging a lot easier, because we were able to point at
| unexpected requests / responses.
|
| Thank you for your work, @mhils & the team. Enjoy your moments
| of glory on HN :)
| rbdixon wrote:
| Scripting flow manipulations in mitmproxy is easy. There are a
| lot of great examples to learn from which help. The filter
| syntax is expressive and simplifies writing code to mangle
| flows. I've done body conversion [0], weird types of
| authorization testing, manipulated system update packages on
| the fly, all sorts of bizarre things. Auto-reloading scripts on
| change is such a time saver. If you want a python-based
| programmable HTTPS proxy for testing I have not found anything
| better.
|
| [0]: https://carvesystems.com/news/proxying-wbxml-services/
| gumby wrote:
| Is anyone using this for page rewriting? Not just for the all-
| important ad blocking but removing the whitespace where the
| removed ad was, rewriting autoplay videos and gifs into click-to-
| play, etc. So many apps have web views that the doing it in the
| browser is a step backwards.
|
| I used to do all my rewrite via a proxy but it stopped being
| particularly useful when https became almost universal.
| zo1 wrote:
| It's getting harder and harder to do even with HTTPS! Look at
| the other commenter on here that described the absolute
| wasteland with apps and OS-vendors (Android,IOS) being actively
| hostile to device-owners.
|
| Edit. Typo.
| mhils wrote:
| Mitmproxy dev here, happy to answer any questions! :)
| alias_neo wrote:
| How practical is it (or not) to intercept TLS traffic for IoT
| devices on my network where I don't have the option of setting
| a proxy?
|
| I've been on a crusade the past few years to try and get my
| self-signed CA onto "smart" devices in my home, but with some
| it's difficult or I risk damaging expensive hardware.
|
| I've tried various ARP poisoning techniques and such in the
| past with limited success.
| egberts1 wrote:
| Many IoTs have local search and scan, so block the DNS server
| at your firewall then provide your own DNS server via your
| own DHCP server.
| mhils wrote:
| I think that's a classic "it depends". I see two main
| challenges for IoT monitoring:
|
| 1. Getting the IoT traffic to your proxy machine. As you say
| this is fairly trivial when you can set a proxy, but a bit
| tricky otherwise as you need to run mitmproxy in transparent
| mode (https://docs.mitmproxy.org/stable/concepts-modes/).
| With some luck you can set a custom gateway on your device
| which simplifies the setup, otherwise you need to have the
| capability to set iptables rules on the router. I personally
| have a small $20 OpenWRT router dedicated for that.
| Alternatively you can run your own DNS server and point
| everything to mitmproxy. v7 now also supports SNI/host-header
| based proxying! The good news is that this setup is the same
| for all your devices and once it works, it works.
|
| 2. Getting the device to trust your (personal) mitmproxy CA
| cert. This one really depends on the particular IoT device
| and may involve some serious reverse-engineering if you are
| unlucky. I wish there would be a more of a "I should be able
| to inspect my own traffic" movement, but it's a tricky
| problem to solve.
| atatatat wrote:
| ...because of enforcement on cheap Chinese etc goods.
| alias_neo wrote:
| The first part is the easy part, I have routers and run
| DNS, and have experience with IPTables, I can usually find
| ways to get hold of the traffic.
|
| The second part is the challenge, and for some devices I've
| been unable to embed my own CA. I was hoping there was a
| proxy trick or some magic sauce I was missing. Perhaps
| where DHCP could automatically configure clients with a
| (insecure) proxy or something along those lines?
| oauea wrote:
| > I was hoping there was a proxy trick or some magic
| sauce I was missing. Perhaps where DHCP could
| automatically configure clients with a (insecure) proxy
| or something along those lines?
|
| Nope, you're not missing anything.
| alias_neo wrote:
| Thanks for confirming, will save me some time messing
| around! Ill stick to getting my soldering iron out and
| EPPROM dumping/flashing.
| rsgrn wrote:
| If I wanted to use mitmproxy to mitm and observe traffic during
| automated testing, what mode/settings should I be using on
| Windows?
|
| The setup is: A client (dev machine or pipeline) running tests
| that communicate to a Windows VM that hosts the app under test.
| The client RPCs to call automation APIs on the host/server.
|
| However I couldn't get both communicating and using mitmproxy
| to observe traffic to work at the same time.
| mhils wrote:
| If your server is really just one service on one port, I
| would just run mitmproxy as a reverse proxy and point your
| client to that (https://docs.mitmproxy.org/stable/concepts-
| modes/). If there are multiple services, having it running as
| a regular proxy sounds best to me.
| chmaynard wrote:
| I'm interested in how you generate your docs. I'd like to do
| something similar using Jekyll.
| mhils wrote:
| Our main docs are built with Hugo
| (https://github.com/mitmproxy/mitmproxy/tree/main/docs). For
| our API docs we use pdoc (https://pdoc.dev), which integrates
| well with most static site generators. pdoc is also
| maintained by us. :)
| chmaynard wrote:
| Thanks. When you began working on the main docs, did you
| start with a pre-built theme or did you start from scratch?
| mhils wrote:
| If I'm not mistaken they are built from scratch, but that
| part wasn't built by me. IMO the by far hardest part is
| actually writing the docs. :)
| punnerud wrote:
| I am still using version 2.0 after the big changes in
| shortcuts. I used mitmproxy weekly, now I have switched to
| mostly using Charles, but would love to switch back.
|
| Any plans for a recommended shortcut setup again?
|
| Not writing 30 characters for copy body of a POST request, but
| fast key bindings.
| mhils wrote:
| I don't recall what changed with v3, but sorry for letting
| you down. Shortcuts are fully configurable since a few
| releases: https://docs.mitmproxy.org/stable/concepts-
| commands/#custom-....
|
| If you have specific feedback about where the keyboard UX
| sucks I'd be happy if you could open an issue on GitHub so
| that we can take a closer look!
| aritmo wrote:
| Thanks for mitmproxy!
|
| Suppose I have set an interception filter and I am viewing a
| live packet. I am about to edit it, etc.
|
| But when a new packet arrives, the text UI moves to the new
| packet! I have lost the packet that I was looking a few seconds
| ago.
|
| Is this an issue in mitmproxy or am doing things the wrong way?
| mhils wrote:
| Could you clarify what you mean by packet? Are you talking
| about HTTP requests or about the new non-HTTP TCP streams?
| The UI/UX for the latter is very much work-in-progress.
| aritmo wrote:
| I am talking about the HTTP requests.
| KORraN wrote:
| Do you have by any chance enabled focus to follow new
| requests? Keyboard shortcut Shift + F. Although it's
| disabled by default (IIRC), you may have enabled it by
| accident.
| mhils wrote:
| The currently focused flow should never change when a new
| one comes in. If that's happening to you, please file a
| bug over on GitHub or ping me on our dev Slack and we'll
| get this fixed ASAP. :)
| monai wrote:
| mitmproxy is a great tool, but it lacks upstream certificate
| lookup and attributes transfer to a generated host certificate.
| For example, Charles does that. Some applications perform
| attribute inspection to detect MITM proxies, namely Spotify.
| Are you planning to implement this feature? Or maybe you
| already did it in version 7?
| mhils wrote:
| mitmproxy does lookup the upstream certificate by default,
| but at the moment we only transfer common name, subject
| alternative names and (new) organization name. The good news
| is that with version 7 it's very easy to add more here, I'd
| be more than happy to accept contributions. :)
| monai wrote:
| Good news indeed! I tried to contribute 1-2 years ago and
| quickly realized that such a feature would require a
| substantial rewrite after a quick code inspection.
| mhils wrote:
| If you (or anyone else reading this) is interested,
| please do reach out on GitHub or our dev Slack. Happy to
| help you get started. :)
| prinzhorn wrote:
| Is this related to
| https://github.com/mitmproxy/mitmproxy/issues/4575 ?
| mhils wrote:
| Yes, exactly!
| cjbprime wrote:
| Thanks for Mitmproxy! It would be amazing to be able to see
| non-HTTPS (but still TLS) traffic in the Flows view, and even
| to intercept it live instead of having to drop to raw TCP mode
| and write Python. Any idea whether that's on the roadmap?
| mhils wrote:
| Starting with this release you do see TCP-over-TLS in the
| flows view, we have a small screenshot in the blog post.
| Interception/modification is still something where we need to
| iron out the UX for, with version 7 you can at least inject
| messages with the `inject.tcp`/`inject.websocket` commands.
| This is not very well documented yet, but the technical
| foundations are there now. :)
| filleokus wrote:
| Mitmproxy is so awesome, I use it all the time! Can recommend it
| for anyone who needs to do some fiddling with HTTP(S) traffic for
| debugging / reverse engineering purposes.
|
| I'm slightly too much of a GUI person to use it in a
| "explorative" phase, then I'm more a fan of Charles/Proxyman, but
| whenever I wanna modify the traffic or do some scripting to i.e
| log certain traffic, I reach for Mitmproxy.
| res0nat0r wrote:
| I just used this app the other week do dig into what I think is
| a bug with rclone. Seeing the exact http traffic to / from the
| AWS s3 endpoint made it much simpler to understand what was
| going on. The CLI is super intuitive, it's a great application
| all around.
| MostKaleido wrote:
| I'd highly recommend https://httptoolkit.tech/ for that
| explorative GUI phase. I found it recently and the rule
| configuration, UI and interception setup is significantly
| better than Charles/Fiddler/Proxyman.
| enumjorge wrote:
| It's such a bummer that a tool like this has a subscription
| model pricing. I don't know what it's like in other
| companies, but where I work getting a recurring subscription
| approved is a hassle compared to a one-time thing. I've been
| looking for something like this for work but a recurring cost
| is almost an instant dealbreaker.
| pimterry wrote:
| Hi! I'm the dev behind HTTP Toolkit.
|
| There is a subscription for Pro features, but most users
| don't need it - you can do most things common use cases
| need (all interception & inspection, manual traffic
| rewriting, almost everything that doesn't need advanced
| configuration) with just the free version. It's also 100%
| open-source, even the paid bits.
|
| That said, imo subscriptions provide lots of benefits as a
| model for both sides. Basically by aligning how much value
| users get with how much they pay - cheap for quick use,
| expensive for extensive heavy use, strong incentive to fix
| bugs & support existing users instead of chasing the next
| shiny launch.
|
| I hear your point about approvals anyway. If subscriptions
| are impossible, I can do one-off indefinite licenses - send
| me an email at tim@httptoolkit.tech
| enumjorge wrote:
| Thank you for clarifying. I initially misread the pricing
| page and thought the free could only be used for personal
| and not commercial use. That really helps with adoption.
| If I can get multiple people to try it and if they like
| it, it's an easier conversation with the people who
| approve expensed if I'm not the only one vouching for a
| piece of software.
|
| I get what you mean about aligning incentives. I do want
| to support devs in a way that these types of projects are
| sustainable, but the way other products shut off access
| to your stuff once you stop paying make subscription
| models an unnecessary risk. Having a free tier like yours
| really helps.
| InvaderFizz wrote:
| I like the look of the product, but there is no mention
| whatsoever of websockets in the docs.
|
| That is the thing missing from Charles for our QA folks.
|
| It is highly annoying to have to hook the iOS app up to
| Xcode just to be able to see the wss traffic when
| tracking down a bug.
| wildbook wrote:
| I've successfully used Fiddler Classic as alternative to
| Charles to look at live websocket traffic when debugging
| things before. Websocket support in general is a bit of a
| hidden feature though since you need to go to the request
| that established the connection and double-click the
| websocket logo to the left on it, which opens a new panel
| showing its traffic.
|
| The newer "Fiddler Everywhere" might also be able to do
| it, possibly even better, but I dislike the UI and I'm
| used to Classic so I've stuck to that.
| pimterry wrote:
| Yep, websockets are coming later this year, but it's not
| done yet I'm afraid.
|
| Tracked here:
| https://github.com/httptoolkit/httptoolkit/issues/36. You
| can subscribe to the GH issue for updates if you're
| interested.
| Cu3PO42 wrote:
| While I don't currently need such a tool at work, I just
| want to chime in to agree with the approval point. It's
| significantly less hassle to get an expensive one time
| purchase approved than a EUR5 recurring payment. I very
| much appreciate your openness in that regard and I'll
| keep it in mind for the future :-)
| [deleted]
| mhils wrote:
| Thanks for the kind words! We have a Google Summer of Code
| student working with us on mitmweb, our web interface, at the
| moment, so we'll hopefully have a better GUI story soon as well
| :)
| GekkePrutser wrote:
| Yeah it's really hard though these days to sniff traffic from
| Android because most apps choose not to trust user-added root
| CAs anymore :(
| InvaderFizz wrote:
| It's not quite that. It's usually certificate pinning. Our
| iOS app won't talk to anything not issued by the list of CAs
| we include at build time.
|
| This is specifically to prevent mitm attacks on the app.
| GekkePrutser wrote:
| Yeah certificate pinning is a thing too. But since Android
| 7.0 apps also have to specify whether they allow user-added
| root CAs and most don't!
|
| You have to get the APK and mess with the manifest to
| bypass that. And bypassing pinning is even more difficult.
|
| Edit: pimterry below me explains it better:
| https://news.ycombinator.com/item?id=27857429
|
| I find this a problem because MITM isn't always bad. It can
| be used for inspecting apps for tracking and other
| malicious practices by app makers.
| InvaderFizz wrote:
| > But since Android 7.0 apps also have to specify whether
| they allow user-added root CAs and most don't!
|
| I was unaware of this as I do little interaction with
| Android. TIL.
|
| And yes, I do appreciate being able to MITM all kinds of
| apps, especially ones that talk to a device I own to be
| able to add functionality/integrations not supported by
| the manufacturer.
|
| I also see the benefit of preventing MITM, in our case,
| it is because we intentionally offload logic onto the
| client to simplify GDPR/CCPA compliance(can't store what
| you don't collect) among other design decisions which
| mean a malicious client could circumvent rulesets we have
| in place.
| oarsinsync wrote:
| > _which mean a malicious client could circumvent
| rulesets we have in place_
|
| Note that your HTTP certificate pinning is easily
| bypassed on both Android and iOS by a determined actor. A
| malicious client _will_ circumvent rulesets you have in
| place. If you do not validate client inputs, you will be
| exploited.
|
| Indeed, it doesn't really differ that much from DRM. It's
| an attempt by the author of the software to impose
| control over the user, preventing them from using the
| software how they see fit.
|
| If I've trusted my own root, and you've decided you wont
| trust anything on my root, I now need to take steps to
| disable the certificate validation routines completely on
| my device, in order to bypass your restrictions.
|
| I get that there are cases where it's extremely valuable
| (and in some cases, required by regulators) to enforce
| certificate pinning, e.g. mobile banking apps, but there
| are plenty of cases where it's just bullshit DRM (e.g.
| gym app that performs unauthenticated requests to an
| https endpoint to retrieve occupancy stats on each of the
| chain's gyms).
|
| In the latter case, all I wanted was to track how busy
| the gym was over time, so I could adjust my schedule to
| enable me to go when the gym was quiet. I couldn't be
| bothered to jailbreak my iphone and install
| nabla-c0d3/ssl-kill-switch2, so instead I quit my gym and
| switched to one that was less stupid.
| InvaderFizz wrote:
| > A malicious client will circumvent rulesets you have in
| place. If you do not validate client inputs, you will be
| exploited.
|
| I agree. That is why there is still serverside
| enforcement, but performance tuning is heavily cache
| based and many client request strategies are specifically
| designed to not bust the cache and cause a high latency
| call.
|
| Also, in our case, the app is the product. I'm not very
| positive on it's market fit, so you may never hear of it
| if it launches to a fizzle, as I suspect is a large
| probability.
| eklavya wrote:
| That sans io core sure sounds like a free monad interpreter.
| EvanAnderson wrote:
| Mitmproxy was an interesting solution to a problem I had a few
| weeks ago. One of my IT support gigs was trying to use
| "Mevo"-branded cameras to stream some live events to Facebook
| using an iPad. We had to use the venue's Wi-Fi network. We found
| the Mevo app would not detect that we were "connected to the
| Internet" and wouldn't allow us to stream.
|
| The venue's network admin said they weren't filtering anything
| outbound to the Internet. We could access websites from the iPad
| just fine. Same w/ Facebook, Youtube, etc.
|
| I put up Mitmproxy, an adhoc Wi-Fi network on a second Wi-Fi NIC,
| a DHCP server, and iptables NAT on my laptop. I set the iPad to
| use my adhoc SSID and my machine as an explicit HTTP proxy. My
| intention was to snoop on the traffic to determine what Mevo was
| using for criteria to consider being "connected".
|
| For whatever reason the Mevo app just started working in this
| configuration. I ended up sticking a spare Wi-Fi router and a PC
| w/ Mitmproxy installed in the venue for the duration of the
| event. I never did figure out what the Mevo app didn't like about
| the venue's network.
| derimagia wrote:
| If I had to guess why you had issues, the app probably had
| certificate pinning. Apple makes it pretty easy to pin domains,
| but it's also not too difficult to circumvent it.
|
| This looks to cover it pretty well:
| https://www.guardsquare.com/blog/leveraging-infoplist-based-...
| Forbo wrote:
| If I had to guess, there may have been some sort of captive
| portal in place. I've seen even the "invisible" kind that don't
| require user intervention cause interference with connectivity
| checks for some devices.
| pedrovhb wrote:
| Does anyone know how the story currently is for MITM'ing Android?
|
| I had fun reverse engineering app APIs a few years back, but I
| looked into it again more recently and found that Android has
| started pinning certificates by default even on apps that didn't
| pin anything themselves. I also had trouble getting my custom
| certificate to be used at all, when it used to be pretty easy.
|
| Can Frida still bypass pinning, even this new default one, or is
| it done at OS level?
| pimterry wrote:
| In short:
|
| * By default, all modern Android apps only trust CA system
| certificates, and on a normal device you cannot change those.
|
| * You can change system certificates on rooted devices and most
| (but not all) emulators.
|
| * Apps can opt-in to trusting user-installed CA certificates
| within their manifest config, but almost all don't.
|
| If you're on a non-rooted device, the only way to MITM traffic
| is by modifying the application itself, so that it opts into
| trusting your cert. You can either inject Frida-Gadget to do
| that, do it manually, or use
| https://github.com/shroudedcode/apk-mitm which tries to do it
| automatically (but it's a bit hit & miss whether it works).
|
| If you're on an emulator/rooted device, it's totally possible,
| see https://httptoolkit.tech/blog/intercepting-android-
| https/#in... for how that works.
|
| That's separate to explicit certificate pinning. That's also
| possible on Android (with some standard config settings, or
| manually in code) although it's got much less popular I think
| since the defaults were tightened up. In that case you do
| usually need Frida, and it's generally easiest to just use a
| rooted device. I wrote a blog post about exactly how to do that
| last week: https://httptoolkit.tech/blog/frida-certificate-
| pinning/#how...
| mirashii wrote:
| > By default, all modern Android apps only trust CA system
| certificates, and on a normal device you cannot change those.
|
| Sure you can. It's in a settings pane, Settings -> Security
| -> Encryption and Credentials -> Install a certificate.
|
| MDM profiles can also install certificates.
|
| There's pretty much zero possibility of this being removed as
| a feature as it is essential to a wide variety of corporate
| environments.
| mumphster wrote:
| This doesn't bypass cert pinning in apps which was the
| original question
| pimterry wrote:
| Those certificates are user-added certificates, not system
| certificates. Android has two separate certificate stores,
| the UI only exposes the user certificate store, and by
| default apps only trust the system certificate store.
|
| This was changed back in 2016, the original announcement
| post has some good details: https://android-
| developers.googleblog.com/2016/07/changes-to...
|
| Enterprises can add certificates to the system store I
| believe, but only as part of the initial provisioning of
| the device, and only for enterprise-managed devices.
| GekkePrutser wrote:
| Indeed, and such enterprise-management devices must be
| wiped for enrolment.
|
| There's some management modes which don't require that
| but they don't offer the ability to install system certs.
___________________________________________________________________
(page generated 2021-07-16 23:02 UTC)