[HN Gopher] Decrypting your own HTTPS traffic with Wireshark
___________________________________________________________________
Decrypting your own HTTPS traffic with Wireshark
Author : rl1987
Score : 322 points
Date : 2022-03-20 14:58 UTC (8 hours ago)
(HTM) web link (www.trickster.dev)
(TXT) w3m dump (www.trickster.dev)
| candiddevmike wrote:
| I was trying to hack/reverse engineer some IoT API calls, and I
| really wish there was some middle ground between "Don't trust any
| network, encrypt everything end to end" and "I can MitM anything
| I want". Everything on my network is a (sometimes literally)
| black box sending and receiving who knows what. As a network
| operator, I should be able to MitM everything on my network. But
| I don't want people to MitM me when I'm on a public network, so
| I'm at a loss on how to fix this.
| dec0dedab0de wrote:
| Most b2b gear will play well with a proxy server, and that will
| allow you to explicitly mitm everything going to the internet.
| Just don't allow anything else out.
|
| One annoying problem is certificate pinning, if you have deices
| that join your network and others, but it's doable.
|
| The real problem is devices sold to consumers that don't allow
| you to install custom certs or allow you to configure a proxy.
| There should probably be a law that says every network device
| should have a means for the owner to decrypt the traffic on the
| fly.
| candiddevmike wrote:
| Maybe with some locally installed b2b software, but any kind
| of hardware device or appliance has trusted root CAs. Unless
| you have access to one of those, nothing will trust your MitM
| setup (at least, I couldn't get Nest equipment to connect via
| a MitM proxy).
|
| Maybe there could be some kind standard DHCP option for a
| trusted root CA on the network. For user controlled devices
| (i.e. phones), they wouldn't trust it, but any kind of vendor
| managed device would. Probably too naive of a solution.
| NavinF wrote:
| That totally defeats the purpose of TLS. On most home
| networks, any device can act as a DHCP server. Indeed, a
| lot of those "limit internet time for kids" devices depend
| on ARP and/or DHCP spoofing to work. I'm glad the security
| model of the last couple of decades (trust every device on
| the LAN) is slowly dying.
|
| Oh and adding a nerdy feature to home routers is never
| gonna happen. Consider how long it took for IPv6 to be
| enabled by default.
| wildmanx wrote:
| A problem is that the easier you make it to sneak in other
| trusted root CAs, the more you open attack surfaces which
| malicious parties can exploit.
|
| As a device manufacturer that uses certificate pinning, do
| you really want to trust whatever cheap crap router people
| have at home to provide a trusted root CA? Let's say that
| one gets hacked (which happens too often..), you are now
| also wide open. Sure, it's the owners responsibility, and
| with certificate pinning that's not configurable, you are
| depriving responsible owners of the possibility to decrypt
| their own traffic, but we at least need to acknowledge that
| there is a balance to strike here, and it's not obvious
| what the perfect solution would be.
|
| Maybe an opt-out of certificate pinning that is so "hard"
| to reach that it's really only for power users that know
| what they are doing. (And with "hard" I mean more than some
| window popping up that says "Are you sure you want to do
| this? You'll void all warranty blah blah" -- people have
| already been conditioned to just click away certificate
| warnings in their browser; the same people would just click
| anything to just go on with their lives..)
| userbinator wrote:
| There is no "balance", this is purely a question of
| ownership. If I can't inspect or modify the traffic, it's
| not mine, I'm just "renting" perpetually.
| EvanAnderson wrote:
| A physical switch on the device would be a good solution.
| Physical access is mostly "game over" anyway.
| wildmanx wrote:
| It needs to be something that requires certain skill.
| Maybe a dip switch inside the enclosure. Average Joe
| won't dare and it's too much work anyway. At least if you
| just try random things to get your IoT device to work if
| something is off.
|
| I'd love to have that.
| unnouinceput wrote:
| On public network you use tunnels aka end-to-end-encryption. On
| your private network you control (theoretically) everything.
| teraflop wrote:
| "Private" and "public" are relative terms. From the
| perspective of an IoT smart thermostat talking to its
| mothership, your home network is a public connection, and
| it's successfully using E2E encryption to protect its private
| data from you -- even though you theoretically own it.
| getcrunk wrote:
| Nice! It's a subtle shift but I think your framing is key.
| vinhboy wrote:
| Are you me? This perfectly describe something I have been
| wanting to do recently. I really do wish that everything that
| connect to "MY" network I can inspect freely.
|
| This stupid wireless lightbulb I have want to ping home to its
| manufacturer and I have no way to stop it. I had to go through
| the trouble of putting it on its own network and blocking it
| from everything. Very annoying. In the end I returned it
| because I couldn't trust it.
| cube2222 wrote:
| From an SSL/TLS perspective you could have an HTTPS proxy on
| your network proxying all your traffic and impersonating all
| websites, by using a certificate authority trusted by your
| devices.
|
| You'd be the sole owner of that root cert / its private key, so
| you wouldn't have to worry about people in public networks
| mitm'ing you, but you'd mitm everything on your own network
| (that is, the traffic coming from devices where you trusted
| that custom CA).
|
| Unless you want to do that with devices that don't explicitly
| trust your custom CA, but in that case, it's a feature that
| this isn't possible.
| renw0rp wrote:
| This idea is used in some "corporate security" tools. For
| example companies which use ZScaler install zscaler
| certificates on employees machines and use ZScaler servers as
| MitM kind of proxy which allows them to scan all traffic,
| including SSL/TLS.
| candiddevmike wrote:
| I know how to do it, you just can't do it unless you control
| the device. Which for the IoT devices I'm trying to
| "hack"/reverse engineer the API server like Nest, I cant
| change anything.
| EvanAnderson wrote:
| Sysadmins, private network operators, and device owners are
| just a subset of malicious attackers and "evil" nation
| states.
|
| The average user doesn't know to care. They keep buying
| this garbage and, in turn, financing the lobby against
| consumer protection laws for embedded devices.
|
| It makes me want to give up on technology and go live in a
| shack in the woods.
| boppo1 wrote:
| Is it any comfort that we might be staring down financial
| collapse that crushes consumer electronics?
| tialaramex wrote:
| But this is a decision you in effect made, those devices
| needn't even use TLS, they could use Jim Bob's secret MITM-
| defeating crypto thing and you can't "hack" that either,
| you bought devices that are resistant to being reverse
| engineered.
|
| Governments could _legislate_ of course, there could be a
| rule that means you can 't sell Nest because it doesn't
| have some "hack" feature. After all the EU mandated people
| provide a sane charge port on phones and it more or less
| worked, even if you don't live in the EU your phone likely
| no longer has some arbitrary custom charge port on it, and
| you can buy replacement chargers or keep the one from a
| previous phone, reducing waste.
|
| You also could legislate from the far end, for public
| services, you could say if I own a device using my
| credentials to do stuff, I get to have the agreed key
| material so that I can decrypt messages after the fact.
| That's a huge _pain_ to implement, most of the ways to do
| it make everybody 's security a bit worse, but it isn't
| technically impossible.
|
| But technologically we can't change the mathematics of the
| encryption, Diffie-Hellman works, two parties can agree a
| secret (which in effect becomes a symmetric key) without
| either of them uttering the secret.
| frazbin wrote:
| "decision you in effect made" == "you were basically
| coerced"
| israrkhan wrote:
| You don't own the communication as a network operator,
| unless you can also prove that you have control over the
| device. IoT devices sometime rely on sending sensitive data
| to their backends. This data may include API keys, client
| authentication secrets and such. By having access to that
| communication, you may be able to spoof identity of the IoT
| device from a PC or a hacked-device. Not very desirable
| from IoT vendor point of view.
| inetknght wrote:
| > _You don 't own the communication as a network
| operator, unless you can also prove that you have control
| over the device._
|
| Sure, but devices that you don't have control over
| shouldn't be permitted to communicate anywhere at all
| ever.
|
| > _IoT devices sometime rely on sending sensitive data to
| their backends._
|
| Such as literally anything.
|
| > _This data may include API keys, client authentication
| secrets and such._
|
| Yeah that sounds like sensitive data alright. It's all
| data that the owner of the device should have but often
| do not because of misplaced corporate greed.
|
| > _By having access to that communication, you may be
| able to spoof identity of the IoT device from a PC or a
| hacked-device._
|
| So what? If I own the device and it's on my network then
| I have every right to everything on the device.
|
| > _Not very desirable from IoT vendor point of view._
|
| And that right there is exactly the problem. IoT vendors
| don't want to really give up ownership of something
| they've sold.
| freedomben wrote:
| Exactly. Even Android now with Android 12 is difficult. I
| spent a bunch of time I didn't really have trying to get my
| Android to proxy through my Burp suite, which is something
| I used to be able to do easily. But now it's extremely
| difficult, and the use of HSTS makes it basically
| impossible for all big tech sites, especially Google owned,
| but even HN. All I want is to be able to see who my device
| is using my network to call, and what they're saying. It
| used to be that the tech was there to serve me, but now it
| seems the other way around.
| userbinator wrote:
| I've been running a MITM proxy for many years to do filtering
| (including adblocking). Fortunately I don't have any IoT to
| worry about.
| bmm6o wrote:
| I can't tell if you know this or not, but TFA is not about
| MitM. Wireshark isn't acting as a proxy, it's using keys saved
| by the client application to decrypt the traffic after the
| fact.
| [deleted]
| candiddevmike wrote:
| I know, it's just related to something I was trying to do
| recently (MitM some IoT devices to inspect the traffic with
| Wireshark). I wish there was some way to get the PMS on
| remote devices via a proxy or configuration option.
| bragr wrote:
| Arguably this is still MitM since Wireshark needs to set up a
| network tap. MitM does not necessarily require modifying the
| data. This would work just as well captured at the switch or
| ISP level if you still had access to the keys.
| alar44 wrote:
| No, that's not what MitM is. The machine needs to rely
| traffic whether it modifies it or not. There's nothing _in
| the middle_ here, clearly. It 's saving packets and
| decrypting later, this is just eavesdropping, it's like a
| wire tap.
| [deleted]
| GekkePrutser wrote:
| Easy. Set up your own PKI and trust it. Keep the private key
| safe and nobody else can use it.
|
| Only problem is that on Android 7 and above apps can choose to
| ignore any added certificates in the user store and the system
| store is off limits. Only way you can do it is by managing the
| phone in COBO or COPE mode with Android Enterprise. Which
| requires an expensive MDM system and a factory reset to enroll
| it.
|
| Also the app can do certificate pinning which is even worse to
| get around. You'll have to modify the actual app.
|
| I wouldn't be surprised if this rigmarole was the result of app
| development companies trying to keep secret how much data they
| exfiltrate.
| caf wrote:
| You don't need a factory reset to enrol in MDM, you can
| create a managed "work" profile and leave the unmanaged user
| profile alone.
| linsomniac wrote:
| (Sorry folks, was replying to the (at the time) only other
| comment about wireshark automatically doing decryption - edit)
|
| This doesn't work if you use DH, and you should be using DH.
|
| To work with DH, you need to run your browser with a SSL key
| logfile, as discussed here:
|
| https://security.stackexchange.com/questions/35639/decryptin...
| Retr0id wrote:
| That's exactly what this article covers.
| kingforaday wrote:
| Pop Quiz: Who remembers Wireshark's original name?
|
| What a fun tool for over two decades!
| ziml77 wrote:
| Ethereal. But I thought it was pronounced Ether Real. I'm not
| sure I'd encountered the word ethereal in any other context at
| that point (pre-teen/young teen).
| vocram wrote:
| But real "hackers" used Ettercap instead!
| nonrandomstring wrote:
| Real hackers use tshark :) [1] I try to get my students doing
| everything from commandline, creating pcap files. When they
| finally see the GUI of Wireshark they think they died and
| went to heaven.
|
| [1] It's actually the capture part of Wireshark
| geraldcombs wrote:
| Apologies for "well, actually"-ing here, but the capture
| part of Wireshark is dumpcap and the various extcap
| utilities (sshdump, ciscodump, etc). TShark is the command
| line interface to Wireshark's dissection engine, which, as
| you point out can be a really useful thing to have. It
| ships with a bunch of other command line utilities as well,
| such as editcap, mergecap, and capinfos:
| https://www.wireshark.org/docs/man-pages/
| maze-le wrote:
| I still remember how flashed I was seeing the byte-by-byte
| analysis of a network package for the first time -- it was a
| real eye opening moment for me.
| qiskit wrote:
| Hate to be that guy, but network packet. But yeah, it's quite
| the revelation especially when you realize other people can
| also sniff the network and capture the unencrypted traffic.
| Remember thinking nobody would waste so much time and energy.
| How young and naive I was back then.
| tssva wrote:
| Hate to be that guy, but Wireshark captures and decodes
| protocols starting at the data layer. So in the most common
| case what you see is the decode of the ethernet frame which
| in the majority of cases but not all includes a network
| packet inside it which Wireshark further decodes. If you
| were capturing from an ATM network you would see ATM cells
| being decoded. For some protocols the data layer unit is
| also referred to as a packet in which case you would see
| the data layer packet which includes the network packet.
| But yeah, it is pretty cool no matter what you want to call
| things.
| gelstudios wrote:
| The original domain name is crypto bait now.
|
| Wireshark and its predecessors have made it easy to visualize
| how protocols actually work, and peel back the magic that makes
| the internet work.
|
| Between the ability to "follow TCP stream" or the
| "conversation" views between two hosts, wireshark has helped
| connect the abstract and the observed / real behavior of
| networks. Which seems to make the difference for people trying
| to get beyond the basics of networking in workshops / training
| sessions.
|
| The built-in protocol dissectors are also a big help for
| teaching.
|
| Some common situations won't work well out of the box (multi
| interface PCAPs), or I need to read the manual again... but
| with a bit of scripting / light processing of the PCAP it works
| very well.
| khimaros wrote:
| also very useful for reverse engineering devices using a
| custom protocol over USB bulk data (in conjunction with a
| Windows/OSX VM and the usbmon kernel module).
| mistrial9 wrote:
| dude- wireshark was the name from a really, really long time
| ago.. that is real trivia
| cortesoft wrote:
| Whether it was a "really, really" long time ago or not is
| dependent on how old you are.
| tssva wrote:
| I remember replacing Novell LANalyzer for Windows with
| Ethereal, so it doesn't seem that long ago to me. Of course
| I also remember replacing The Sniffer from Network General
| which ran under DOS with LANalyzer for Windows.
| seba_dos1 wrote:
| Initially I dismissed this comment as I remember it changing
| name and I'm still in my twenties, but it turns out that it
| got renamed the same year I actually switched to GNU/Linux,
| so I was very close to missing the original name :)
| Existenceblinks wrote:
| Right after TCP/IP class around 2009, I haven't touched it
| since. Zero gas Ethereal!?
| astura wrote:
| Ethereal. I was working a job where I had to use it daily when
| they changed the name to Wireshark. Around 2006-2007 IIRC. I
| never got used to the new name and left that job shortly
| afterwards.
|
| Wireshark is an objectively better name though, it's for the
| best.
| bsedlm wrote:
| ethersomething??
| cookiengineer wrote:
| > Pop Quiz: Who remembers Wireshark's original name?
|
| That's a _real_ question for the _ethernet_ people ;)
| pieter_mj wrote:
| Hehe, I've never used it when it was called ethereal.
| rileymat2 wrote:
| How does this work with perfect forward secrecy, does curl save
| all the keys? Or is it disabled?
| racingmars wrote:
| First step the article gives: `export
| SSLKEYLOGFILE=~/.sslkeyfile`
|
| This will tell curl, among many other applications, to save the
| necessary keys.
| rileymat2 wrote:
| Yes, I saw this and tried that, but when running curl only
| saw one key. I just expected more.
| dorianmariefr wrote:
| I prefer mitmproxy
| mhils wrote:
| You can have both! mitmproxy will dump TLS master secrets for
| any app: https://docs.mitmproxy.org/stable/howto-wireshark-tls/
| geraldcombs wrote:
| As the blog post points out, this requires coordinating
| Wireshark's "(Pre-)Master Secret Log filename" / tls.keylog_file
| preference with you web browser's environment, so a while back I
| wrote a Wireshark Lua plugin that automates as much of this as
| possible. It pops up a window that lets you launch Chrome or
| Firefox with the SSLKEYLOGFILE environment variable set to the
| correct value: https://gitlab.com/-/snippets/2156054
| vxNsr wrote:
| Thank you for this
| funshed wrote:
| I can't figure out how to do virtualized android devices? any
| tips.
| raro11 wrote:
| Today I got everything working using BlueStacks.
|
| You still have to root the emulated device as you would with a
| regular device. But it works
| salmaanp wrote:
| how would this work for custom applications? Say I have a C
| program which does openssl handshakes inside. How can we
| implement the same SSLKEYLOGFILE functionality in it
| throwra620 wrote:
| This blog is awful to read. Wireshark will decrypt your ssl
| traffic automatic if you tell it to.
| noselasd wrote:
| Wireshark doesn't come with a built in way to extract the
| dynamically exhanged TLS keys from applications and use that to
| decrypt the traffic, so no it does not do what you claim.
| collsni wrote:
| confidently incorrect
| Sophira wrote:
| How would one go about doing this? I've never been able to get
| it to do that.
| Retr0id wrote:
| With which keys?
| ravishi wrote:
| How would one go about doing that for Android apps? I guess
| android apps don't respect that flag?
| freedomben wrote:
| You can add a self-signed trusted root ca, and set a system
| proxy in Android and apps will (usually) use that, although the
| Google and other big-tech apps will certificate pin and/or use
| HSTS, so they won't trust your cert. Also newer versions of
| Android make it a huge PITA.
| punnerud wrote:
| Can use frida-sslkeylog to extract the keys on Android, and put
| it into Wireshark: https://github.com/saleemrashid/frida-
| sslkeylog
|
| This require a rooted phone. Or you can patch the app with
| objection, so you don't need the root:
| https://github.com/sensepost/objection/wiki/Patching-Android...
|
| Can download the APK from places like https://apkpure.com/
| almog wrote:
| apkpure.com has a track record of serving malicious APKs.
|
| If you run it in a sandbox environment it's probably okay-
| ish, but if you already have the app installed from a trusted
| source on your phone, you can grab the APK file with `adb
| pull`.
| punnerud wrote:
| Did not know that. Thanks
| dylanowen wrote:
| Yeah I've had good success with patching apps. If the app
| complains the phone is an emulator, just patch the byte code
| that is checking for emulators too :)
| rl1987 wrote:
| As long as there's no certificate pinning implemented in the
| apps (which can be defeated with some effort) I would go with a
| different approach and use mitmproxy. See:
| https://www.trickster.dev/post/setting-up-mitmproxy-with-and...
| ThePhysicist wrote:
| Neat, I didn't know about SSLKEYLOGFILE, seems like a great way
| to debug apps that support it without installing custom root
| certificates.
___________________________________________________________________
(page generated 2022-03-20 23:00 UTC)