https://ericdraken.com/pfsense-decrypt-ad-traffic/ Menu * Latest * Accomplishments * StackOverflow * Instagram * GitHub * About Block YouTube Ads on AppleTV by Decrypting and Stripping Ads from Profobuf EricJanuary 6, 2022June 9, 2022 So many adsSo many ads In a Nutshell I discovered that putting a man-in-the-middle proxy between my Apple TV and the world lets me decrypt HTTPS traffic. From there, I can read the Protocol Buffer data Google uses to populate YouTube with ads. It is too CPU-intensive to decode Protobuf on the fly, so instead, I found a flaw in the Protobuf format which allows me to reliably change one byte to obliterate ads. What follows is a reference guide for setting up a bare-metal network router to block malicious ads, obnoxious ads, tracking, clickbait, crypto-jackers, scam popups, Windows spying on you, etc. using blocklists to protect all networked devices. --------------------------------------------------------------------- Goal: Let's build a cryptographically-strong router with FreeBSD and pfSense to completely block YouTube ads using a flaw in the Google Protocol Buffer format to completely block pre-roll, mid-roll, and end-roll YouTube ads on Apple TV and iPhones, network-wide. Disclaimer: I want to support content creators, so to be fair, after a few months of blocking YouTube ads, I am now paying for YouTube Premium; Just because I can break something, doesn't mean I need to. Sections Part 1 - Setup pfSense on Bare-Metal 1. Why block Ads and Behaviour Tracking? 2. Required Router Hardware 3. Unboxing the Hardware 4. Install pfSense on Bare Metal 5. First pfSense Boot 6. Enable the AES-NI Cryptographic Instruction 7. Enable RAM Disk 8. Dashboard Widgets 9. Adblocking with pfBlockerNG 10. Isolate LANs for Security 11. Class B IPv4 172.31.1.0/24 Network for Untrusted Devices 12. Add Firewall Rules Part 2 - Isolate Network LANs 1. Setup the Untrusted Wi-Fi AP 2. Automatic pfSense Configuration Backups 3. Unable to Reach 172.31.1.x from 192.168.10.x 4. Replace Stock Firmware on the AC1200 Wi-Fi Access Point 5. Archer C5 v2 into the Refuse Bin, R7000 as the New Wi-Fi AP 6. Set up the Trusted Wireless Network 7. Network Devices Interconnectivity Check 8. Windows File Sharing Gotchas 9. Public Service Announcement: Edge Browser Part 3 - Setup DNS Adblocking 1. Block Clickbait, Incessant Ads, and Dangerous Sites 2. Intercept all DNS Requests, Even to Hardcoded DNS Servers Part 4 - Trick the YouTube Ad Algorithm 1. How to Restrict Apple TV YouTube Ads? 2. Trick the YouTube Ad Algorithm Instead 3. Research into YouTube Advertizing Spend 4. New Goal: Convince YouTube I'm 70 and in Italy 5. Selectively Route Apple TV Over the VPN 6. Selectively Route Apple TV YouTube Traffic Over the VPN 7. Gotcha: DNS Race Condition 8. Gotcha: Authentication Trouble, Forbidden 403 Error 9. Gotcha: YouTube is Now Showing UK Ads, Not Italian Ads 10. Find a VPN Exit Node with no ASN Leak 11. Hijack Google Video DNS Queries 12. New Goal: Programmatically add IPs to the Firewall Policy Rule 13. Research Python Methods to Hijack DNS Queries i. Rsync Disk Backup ii. Install pfSense REST API iii. Explore the Unbound Python Module 14. Smoke Test: A Python DNS Hijacking Script Part 5 - Decrypt HTTPS Traffic 1. New Goal: Research and install a Squid-like proxy i. Fun fact: Jailbreaking iPhones in Japan 2. Install a Fake-but-Trusted CA Cert on Apple TV and iPhone? 3. Experiment with Squid and SquidGuard 4. Self-Host the CA Certificate 5. Abandoning Squid: Too Slow, Too Heavy i. Rsync Diff of Changes 6. Install MITMProxy in a FreeBSD Jail 7. Exploring MITMProxy 8. Patch MITMProxy Source Code for Server SNI Interrogation Part 6 - Intercept Apple TV and iOS YouTube Ads 1. Smoke Test: Intercept YouTube Ads with MITMProxy 2. Examine uBlock Origin Regex Patterns for Inspiration 3. Surgically Alter the JSON Response to Remove Ads 4. The iOS YouTube App Uses Protobuf, not JSON 5. Timing Analysis to Detect Ad Videos? 6. Decode the YouTube Protobuf Responses 7. Ad URL Polymorphism 8. Smoke Test: Intercept and Decode Protobuf in Python i. Pure Python Benchmarks ii. Pure C++ Benchmarks 9. Fuzzing the YouTube Video Ad Responses 10. Enter Burp Suite Tools for Penetration Testing 11. Exfil the Proto Schemas from the App, Cleanly? Part 7 - Reverse-Engineer Protobuf Messages 1. Hardcore Deep-Dive into Protobuf and Wire Format 2. Exploit a Protobuf Flaw to Easily Remove All Ads by Changing One Byte 3. Smoke Test: Remove Ads from Protobuf in O(n)-Time 4. Analysis of this Successful Adblocking Technique i. Summary ii. Timing Analysis iii. Knock-On Benefits iv. Future-Proof v. Should Google be Worried? 5. The MITMProxy YouTube Adblocking Script Part 8 - Summary 1. YouTube Premium i. Experiment in Ad Viewing ii. $0.15 as a Ballpark CPV iii. CPV from US Advertising Spend Divided by Total Views iv. Is YouTube Premium Worth It? 2. DMCA, Sony, Viacom 3. Summary of Accomplishments --------------------------------------------------------------------- Why block Malicious Ads and Behaviour Tracking? You are a valuable commodity that is bought and sold without your knowledge or consent. You will be tricked with clickbait, distracted with large ads, and enticed to leave the site you are on at every opportunity. Plus, everything you do online is being monitored so your habits and searches can be remarketed and sold over and over again for years. clickbaitclickbait Privacy - Knowing what you like to watch and read, what phone you have, what you watch on Netflix, what you shop for, what you ask Alexa about, yout taste in music, etc. is unbelievably valuable to advertisers. Spying on people is such a big problem that Europe passed the GDPR law so every site you visit asks if you are okay with cookies (and we blindly click "ok" to hide the banner). We must wrestle back privacy ourselves. Bandwidth - If privacy doesn't concern you, how about this: it is well-known that between 25% and 40% of network traffic is ads, tracking, JavaScript to load trackers (fingerprint.js, googletagmanager.js), websocket traffic to collect how you scroll and what you type (Hotjar), and the like. Do you have a 100 Mbps internet connection? Consider it 60 Mbps! Clickbait - Then there is clickbait. "You won't believe what Tom Cruise did. He..." and you may want to click. Then you are in the spider's web. How about fake news? Or articles that don't say "sponsored" in size-8 font, but now say "underscored" to be clever. What is even real anymore? As soon as you click on clickbait, you may end up on a page with a dozen more ads that aren't approved by Google but lead to a dark world of maliciousness. Clickbait is so incredibly profitable to scammers. Cryptojacking - Some websites will load crypto-mining JavaScript (e.g. CoinHive.js) so while you read, they overheat and abuse your computer to try to make a few pennies. Some sites will load JavaScript that tries to steal from your crypto wallet or trick you into transferring cryptocurrency. Takeaway: It is highly lucrative yet detrimental to you to track and trick you, and only you can do something about it. Top - --------------------------------------------------------------------- Required Router Hardware A virtual machine, Docker image, or Raspberry Pi are not performant enough to protect a whole SMB network; We need dedicated hardware with a cryptographic instruction set so that its only function is to route, decrypt, and monitor packets in and out. Here is what I used. * A mini PC with the AES-NI instruction set (e.g. J4125) * Several gigabytes of DDR4 RAM (e.g. 32 GiB) * A decent mSATA SSD drive (e.g. 128 GiB) * A USB drive to transfer pfSense Top - --------------------------------------------------------------------- Unboxing the Hardware I've ordered a mini J4125 PC from AliExpress, ordered 32 GB of DDR4 RAM and a 128 GB mSATA from Amazon, and will assemble them for the first time now. Warning: Out of caution, I searched diligently for a barebones mini PC that did not include RAM or an SSD; there is nothing stopping an overseas seller from including some generic RAM and SSD but charging Samsung prices. Tip: 128 GB of disk space on a router? Yup. That should be plenty of space to hold logs and not wear down the SSD too quickly, and to allow beautiful packet capture (and maybe an edge cache for NPM and Docker?). A beautiful box, isn't it? It only has 3 LAN ports, but it can be extended with network switches. The J4125 AES-NI quad-core fanless mini PCThe J4125 AES-NI quad-core fanless mini PCThe J4125 AES-NI quad-core fanless mini PCThe J4125 pfSense router from a fanless mini PCThe J4125 pfSense router from a fanless mini PCThe pfSense router from a J4125 fanless mini PC Top - --------------------------------------------------------------------- Install pfSense on Bare Metal I've never used pfSense before, so we will explore this together. The compressed image is about 360 MB and can be flashed to a USB drive with an AppImage binary of Etcher (very cool). Decisions, decisions: VGA install or serial? Let's serial into the new router. Why not? Let's not serial into the routerLet's not serial into the routerLet's not serial into the router Well, that looks painful. It would also be a whole production to serial into the box in case of an emergency because the serial port is inside, and there isn't even an RS232 or JTAG connector - just some narrow header pins. Yikes. Let's go with VGA and plug a keyboard into the USB port - get ready to navigate with arrows and tabs. J4125 mini PC BIOS over a VGA cableJ4125 mini PC BIOS over a VGA cableJ4125 mini PC BIOS over a VGA cable I'll follow this guide on YouTube. I'll pass on encrypting the disk since I would like to avoid entering a passphrase each time the mini PC reboots. A stripe disk is fine since there is only one disk. I have no idea what to expect yet, so I will pass on dropping to a shell for a more advanced configuration. Top - --------------------------------------------------------------------- First pfSense Boot I ejected the USB containing the boot image (important) and rebooted the little box. It played a melody on the internal speaker (there is an internal buzzer and thankfully it isn't very loud). Do I need to have a LAN connection already, or can I just start the thing? I'll just start pfSense and let it complain to me if it wants... and according to the YouTube tutorial, I should guess which port is LAN 1. I'll do that now. I figured out that I should set the LAN 1 to a static IP address that is not in my existing router's DHCP range, so I went with 192.168.1.3. Now I can access an admin web portal (admin/pfsense). Hooray. Yikes, the mini PC beeped at me and informed me that 'admin' has logged in. That startled me a bit, but hey that is pretty neat. First time logging into pFsense admin UIFirst time logging into pFsense admin UIFirst time logging into pFsense admin UI Top - --------------------------------------------------------------------- Enable the AES-NI Cryptographic Instruction I played around with the wizard, used defaults, and got to the web configurator. The first thing that caught my eye was AES-NI CPU Crypto: Yes (inactive). I went out of my way to get a mini PC with AES-NI. What gives? Ah, this needs to be enabled in System > Advanced > Miscellaneous. Why not auto-detect this and use the best option? I'm glad I spotted that, or else this mini PC might as well be a Celeron J1900 of yesteryear. Top - --------------------------------------------------------------------- Enable RAM Disk Having 32 GiB of RAM, let's take advantage of that and use a generous amount of RAM for /var and /tmp, and since hopefully this 128 GiB SSD has wear levelling, let's take a RAM Disk backup every hour. Let's take advantage of RAM diskLet's take advantage of RAM diskLet's take advantage of RAM disk Reboot! AES-NI is now active. Top - --------------------------------------------------------------------- Dashboard Widgets This dashboard is pretty slick. I'm just discovering that there are widgets that can be added to the Dashboard, including S.M.A.R.T to alert us if the SSD is going bad. Nice. Final pfSense dashboard after all setupFinal pfSense dashboard after all setupFinal pfSense dashboard after all setup Hang on, when I added the Services Status widget, something called PC /SC Smart Card Daemon shows up. What is that? Research shows it's a daemon for hardware smart keys that we can probably do without(?). It can be disabled in the /etc/rc.bootup file like so: [/* pcscd daemon must] 1 /* pcscd daemon must be started before IPsec */ 2 echo "SKIPPING PC/SC Smart Card Services..."; 3 # echo "Starting PC/SC Smart Card Services..."; 4 # mwexec_bg("/usr/local/sbin/pcscd"); 5 # echo "done.\n"; Wait. After some time went by, I noticed the router slowed down, fatally. IPsec without the SD Card Service will cripple the routerIPsec without the SD Card Service will cripple the routerIPsec without the SD Card Service will cripple the router Warning: Do NOT try to disable the Smart Card Service as it is needed by IPsec; if you start experimenting with an IPsec VPN tunnel and the pcscd daemon is disabled, then your hard disk will fill up with logs and your CPU will run hot. Top - --------------------------------------------------------------------- Adblocking with pfBlockerNG This unboxing and setup has been fun, but I'd like to block all the bad traffic on my network. I've been using a workhorse of a DNS-level adblocker called Pi-Hole on a... yes, Pi, but it would be nice if I can reclaim that wee bit of hardware for something else and use a comparable add-on module in pfSense. Let's explore that now. pfBlockerNG is a very powerful package for pfSense(r) which provides advertisement and malicious content blocking along with geo-blocking capabilities. Question: Do I install the first pfBlockerNG or the pfBlockerNG-devel which feels like a developer version? I'm a software developer, so this is for me, but am I a pfSense developer? No. Maybe it will show me advanced logs or I can mess about with LUA? Let's Google this. From here, random people are saying to install the development version. Another blogger advocates using the dev version as well. Meh, I guess we can install jq, rsync, and Python 3.8. This doesn't feel like a development version since it has exciting dependencies. Install pfBlockerNG-devel not the other oneInstall pfBlockerNG-devel not the other oneInstall pfBlockerNG-devel not the other one That was painless and only added an extra 20 MiB. It seems a lot of the dependencies are part of pfSense already. The knight at the end of Raiders would say that I have chosen wisely (hey, why did Indy age like a normal person up to Indy 4 if he drank the immortality water that the thousand-year-old knight also drank?). [New packages to be I] 1 New packages to be INSTALLED: 2 gmp: 6.2.1 [pfSense] 3 grepcidr: 2.0 [pfSense] 4 iprange: 1.0.4 [pfSense] 5 jq: 1.6 [pfSense] 6 libmaxminddb: 1.6.0 [pfSense] 7 lighttpd: 1.4.59 [pfSense] 8 lua52: 5.2.4 [pfSense] 9 nettle: 3.7.2_2 [pfSense] 10 pfSense-pkg-pfBlockerNG-devel: 3.1.0 [pfSense] 11 py38-maxminddb: 2.0.3 [pfSense] 12 py38-sqlite3: 3.8.10_7 [pfSense] 13 rsync: 3.2.3_1 [pfSense] 14 whois: 5.5.7 [pfSense] 15 xxhash: 0.8.0 [pfSense] 16 zstd: 1.5.0 [pfSense] Wizard time. The pfBlockerNG wizard had four steps but step three is like 50 steps in oneThe pfBlockerNG wizard had four steps but step three is like 50 steps in oneThe pfBlockerNG wizard had four steps but step three is like 50 steps in one There are a lot of options in step three. This is not like Pi-hole at all. I'm going to come back to this and set up my network instead so I accomplish retiring my Nighthawk R700 or giving it new life as a Wi-Fi AP. Fix: If the pfb_dnsbl service won't start or the status tab states [ Missing CRON task ], try deleting the empty file /var/run/booting ( ref). Top - --------------------------------------------------------------------- Isolate LANs for Security An opportunity has presented itself: I can create real networks on each of the three router Gigabit ports (not VLANs), and should I do so? Yes, yes I should. I would like a dedicated hardware network for all my home-phoning spy devices (Alexas and Apple TV) so they don't flood my network with metrics info and "sure I'm muted and not listening to you" audio payloads back to their HQs. I can see it now: A Wi-Fi AP on a hardware LAN that is isolated from everything else and dedicated to these gadgets, and runs through the adblocker and traps hard-coded DNS queries to 1.1.1.1 and 9.9.9.9 and others (I'll have to explore this) so YouTube on my TV doesn't sneakily bypass [DEL:Pi-Hole:DEL] any DNS-level blocker. It's so Utopian an outcome I may not be able to sleep. I've decided that my bottom-shelf TP-Link wireless router that is so old that AC1200 might as well be "A.D. 1200" is going to be my Wi-Fi AP for those IoT spy devices. In sum, there will be a dedicated hardware LAN * with a wireless AP (AC1200) for Amazon/Apple gadgets and the TV. * with a wired switch for all the beefy computers and clusters in my lab. * with another wireless AP (R7000) just for iPhones and watches. As an aside, since doing an Offensive Security hacking course in my spare time, and I rare-earth-magnet-strongly suggest isolating Wi-Fi devices from any critical LAN segments connected to devices that touch daily banking or stock trading (or crypto wallets). Top - --------------------------------------------------------------------- Class B IPv4 172.31.1.0/24 Network for Untrusted Devices The class B IPv4 range 172.16/16 is a valid range of private IP addresses. I'm not uncomfortable with Alexa and Apple TV being even on the same class network as my main LAN segment, so I will banish them to the class B private network at the hardware level, and my more trusted LANs will be on the traditional class C network (192.168 /16). This helps mitigate any misconfigured iptables rules by naturally having no routes between the two networks. Set up a physical network for the untrusted smart devicesSet up a physical network for the untrusted smart devicesSet up a physical network for the untrusted smart devices Be sure to enable the DHCP resolver on the physical NIC that will connect smart devices (which mainly just tell me the weather and creepily listen to me sleep). From this point, DHCP works on this new network, but by default, it assigns IP addresses but does no routing. All traffic is blocked by default. Top - --------------------------------------------------------------------- Add Firewall Rules We need to manually add rules so traffic on the physical NICs goe somewhere. Our first rule to allow eth3 to access the InternetOur first rule to allow eth3 to access the InternetOur first rule to allow eth3 to access the Internet There is a logging message. Let me reproduce it below. Hint: the firewall has limited local log space. Don't turn on logging for everything. I read that to mean, "Congratulations on not cheaping out on your SSD. Now go forth and log everything, my son." I'm not a new-age, fancy-jazz, coloured-light- or smart-plug-controlling guy who forgot how to turn on a light without his phone, so I do not need to have smart devices on the same network as my phone (why create dozens of wireless attack vectors into your home?). I'm classicly trained to actuate an electromechanical current interrupter on the wall and light let there be. Top - --------------------------------------------------------------------- Setup the Untrusted Wi-Fi AP How do I reach the admin UI of AC1200 Wi-Fi AP now? I factory reset it, plugged the WAN NIC into the ETH3 NIC of the pfSense router, but both devices are just blinking at me. I suppose I can just Wi-Fi into the factory-reset AC1200. Yikes, 2016 was a bad year for responsive web UIs I take it. This is horrible; I'll pull out a netbook for this. One sec. It seems the Archer C5 has no AP Mode. This is my problem, not yours, but I'm still going to vent. Oh, and the "refresh" icon on the top of the DCHP Leases page in pfSense is not "refresh", but "reload service". Whoops. Well, I bricked the AC1200 router. I will have to run an Ethernet cable manually... but, wait, my thin notebook has no Ethernet ports and needs a USB-NIC adapter. Happy Friday. Tip: Connect LAN to LAN, not the AP's WAN to pfSense's LAN unless you want to do double NATing. There were shenanigans, but I set the LAN IP of the AC1200 to 172.31.1.100, the ETH3 NIC IP of the pfSense router to 172.31.1.1/24, and set the pfSense DHCP service on ETH3 to assign addresses 172.31.1.101~150. What failed was setting the AC1200 to 172.31.1.2 as it was unreachable (reason unknown). Oh yes, I had to turn off firewally things and NAT Boost, and basically drop the horsepower of this TP-Link router down to that of a potato battery. The above settings allow me to access the AC1200 remotely now. The other video ran its course, so I started following this YouTube video (set the speed to 1.5x). There are some good tutorials on advanced pfSense... if you can sit through the adsThere are some good tutorials on advanced pfSense... if you can sit through the adsThere are some good tutorials on advanced pfSense... if you can sit through the ads One more thing: I installed the nmap package for pfSense and scanned the AC1200 router, and found some sneaky ports open. [Running: /usr/local/] 1 Running: /usr/local/bin/nmap -sT -P0 -e igb1 '172.31.1.100' 2 Starting Nmap 7.91 ( https://nmap.org ) at 2021-11-15 17:40 PST 3 Nmap scan report for 172.31.1.100 4 Host is up (0.0017s latency). 5 Not shown: 969 closed ports, 28 filtered ports 6 PORT STATE SERVICE 7 22/tcp open ssh 8 80/tcp open http 9 20005/tcp open btx Port 20005/tcp is a print server port that I've now closed. However, the Archer C5 AC1200 is vulnerable to all kinds of Kali mischief so it was wise to put it on its own network. I'm not sure how to close port 22 and the SSHd service on it the AC1200 because the stock firmware is ancient and crippled, so I'll just have to block port 22 on the whole LAN segment. I've also taken care of disallowing private networks to ingress on the WAN (see the next section to set up DMZ). Top - --------------------------------------------------------------------- Unable to Reach 172.31.1.x from 192.168.10.x Ping and Traceroute are aiding me in my efforts to connect to the AC1200 Wi-Fi AP from my Trusted LAN. I went ahead and added the subnet to the Symantec Firewall rules just in case (Symantec has its place now and then, but yes, definitely have available PC CPU horsepower to spare). Configure Symantec to allow the Untrusted subnetConfigure Symantec to allow the Untrusted subnetConfigure Symantec to allow the Untrusted subnet Now, it seems ICMP packets are no longer blocked between networks, but I still cannot ping the AP web management UI even though I can see the pings in the traffic logs. I've even added an "any to any" firewall rule on the Untrusted network. No change. Warning: If you run nmap as I did, software firewalls may detect a port scan and kick your network connection for an hour by default. Disable Symantec port scan detectionDisable Symantec port scan detection Let's try a stealth scan instead: sudo nmap -sS -v 172.31.1.*. I think a port scan has been detectedI think a port scan has been detectedI think a port scan has still been detected Nope, pfSense doesn't like that at all. And, the whole network stopped working. Nice security! Also, dang. The good news is that I've isolated the packet malaise to the TP-Link AC1200 box itself. I suspect that I need to add net.ipv4.ip_forward=1 to forward packets with no addresses in them, but I'd need root access to the AC1200. Let's burn it to the ground and rebuild from its sprinkler-soaked ashes. Top - --------------------------------------------------------------------- Replace Stock Firmware on the AC1200 Wi-Fi Access Point Of course, I cannot actually stop Untrusted LAN devices from reaching the AC1200 as they all exist downstream from the pfSense box. DD-WRT open-source router firmware, meet my ancient Archer C5 and do your thing. DD-WRT supports the Archer C5DD-WRT supports the Archer C5DD-WRT supports the Archer C5 The Archer C5 did not accept the DD-WRT firmware. Hmm... how about OpenWRT? OpenWRT supports the Archer C5OpenWRT supports the Archer C5OpenWRT supports the Archer C5 The Archer C5 did not accept the OpenWRT firmware either. What the actual facepalm (WTAF)? Wait. My hardware is revision 2 using the Broadcom chipsets which are notoriously difficult networking chips. Careful: Devices with Broadcom Wi-Fi chipsets have limited OpenWrt supportability (due to limited FLOSS driver availability for Broadcom chips). (REF: OpenWRT.org) Alright, so OpenWRT, DD-WRT, and Tomato projects have no firmware for this AC1200 with unpopular Broadcom chipsets. Into the refuse bin it goes. Top - --------------------------------------------------------------------- Archer C5 v2 into the Refuse Bin, R7000 as the New Wi-Fi AP I've dismantled the AC1200 so I do not forget why I threw it out. It's too bad because it's so pretty on the inside, and they always say, "It is what is inside that counts... except if you are a router with Broadcom chips." Inside the Archer C5 v2 with Broadcom chipsInside the Archer C5 v2 with Broadcom chipsInside the Archer C5 v2 with Broadcom chips The R7000 is factory reset, and here is the first problem: Tip: On factory reset, the Nighthawk R7000 is pretty uptight about the format of the password. One rule is that no more than two identical consecutive characters are allowed. Well, thanks Netgear for pretty much providing a Regex to password crackers. Let's disable all those rules with a few keystrokes to delete the JavaScript "blocking" the form submission. Now my admin password does not conform to the Regex and is super long. Muhahaha to Netgear password crackers. The R7000 is in AP mode, but I can still access the pfSense web management page from the Untrusted network. Let's lock down the web UI in pfSense under Firewall Rules. Untrusted network firewall rulesUntrusted network firewall rules Untrusted network firewall rules Top - --------------------------------------------------------------------- Set up the Trusted Wireless Network The Untrusted network is now looking good. It's time to make the other R7000 Nighthawk I have into a Wi-Fi AP as well so my phone and watch have a safe place to connect to, as well as a laptop when I want to RDP into my wired machines from the kitchen. I was saving that for a honeypot AP, but I can come back to that later. Let's see if I can Wi-Fi into the Wireless LAN's R7000... Tip: Remember to physically unplug the pfSense upstream router from the R7000 because the R7000 is too helpful and will enter into AP mode by sensing any upstream routers, then you cannot get into the web UI anymore. Since only my trusted devices should be on the Wireless LAN, I'll turn off 2.4 GHz wi-fi because anything recent and wireless should support 5 GHz. That means those pesky AliExpress Pineapple wi-fi password stealers on the cheap side only use 2.4 GHz, so a neighbour is going to have to put in some effort to snoop on my network. Plus, 5 GHz is blocked more easily by walls and concrete, so I prefer it for averting medium-range snooping. But, I so am going to set up a honeypot and to brake check my faith in humanity. It is normally straightforward to put a Wi-Fi router into AP mode by disabling WAN and DHCP. Top - --------------------------------------------------------------------- Network Devices Interconnectivity Check Do all my dozens of computers, laptops, Pis, clusters, NAS drives, and the like still connect as before? Most important is my web-scraping bot in a hardened, RAIDed, dedicated machine with its own UPS. But alas, I cannot SSH into it even though the SSH handshake packets make it to the hefty box. Could this be our old frienemy IPv4 forwarding being disabled? Possibly. I'm able to SSH into the machine from my iPhone (seriously) when on the same network. Nope. Adding net.ipv4.ip_forward = 1 in the right place with a restart did not yield joy. According to dmesg -w (to tail dmesg logs), UFW (Uncomplicated Firewall) is not blocking ICMP requests or TCP requests on port 22. When I do something nutty like try to SSH on, say, port 23, then I can see the UFW block logs in dmesg. Confirmed: Packets can reach that machine. Running tcpdump src 192.168.10.100 where the IP is from the Trusted network on the target machine shows it is responding to pings. I'm even getting replies to SSH handshake requests. So now we know that return packets are being dropped. Interesting! Aside: tcpdump is awesome. Let's follow the trail. Digging a little deeper I see replies to ICMP and SSH handshakes are being sent to some IP over HTTPS that I do not recognize. Bizzare. When I run the usual ipinfo tools I see that replies are going over a VPN that I completely forgot about. Ha. Replies to a different subnet are egressing over the VPN, but cannot return properly. Neat. VPN causes ACK packets to return over the wrong adapterVPN causes ACK packets to return over the wrong adapterVPN causes ACK packets to return over the wrong adapter Now that I remember what I did in 2019, I re-added NAT alias rules, and it's showtime again. Top - --------------------------------------------------------------------- Windows File Sharing Gotchas Your path may be smoother, but I've always seem to make the Trench Run instead of remote-piloting a handful of lead-filled X-Wings at light speed right through the Death Star's reactor to make it go boom: the easy way. I've added some rules to allow Static DHCP devices to talk to each other - Windows devices - but by default, the Private Network in the Windows Defender uses the local subnet as the rule scope. That means different subnets are isolated. We can't just relax the pfSense DHCP subnet mask to say 192.168.20.0/16 because it conflicts with another subnet. Instead, just to get file sharing working, I relax the scope in Advanced Settings like below. Be sure to modify In and Out for SMB and ICMP. Windows file sharing across subnetsWindows file sharing across subnetsWindows file sharing across subnets Again, please add whatever subnets you desire instead of any. Top - --------------------------------------------------------------------- Public Service Announcement: Edge Browser Why does the Microsoft Edge browser start automatically and run in the background, and why can't I kill it when I ctrl+alt+del? If you've asked yourself this, you're not alone. It turns out Edge starts up when you log in and it keeps running in the background. Here is the fix: Prevent Microsoft Edge from starting or running in the background. Sneaky browser.Prevent Microsoft Edge from starting or running in the background. Sneaky browser.Prevent Microsoft Edge from starting or running in the background. Sneaky browser. I suggest downloading Winaero Tweaker and applying registry tweaks to cut down on the Redmond Spy Machine. Stop Microsoft from spying on youStop Microsoft from spying on you Stop Microsoft from spying on you Top - --------------------------------------------------------------------- Block Clickbait, Endless Ads, and Dangerous Sites Thanks to web-browser and DNS-level adblockers (i.e. Pi-hole), it's commonplace to block bad sites, crypto-miners, fingerprinters, trackers, remarketers, banners, pop-ups, fake tech-support scam alerts, and all manner of unscrupulousness designed to take advantage of you. Let's take pfBlockerNG on pfSense for spin. pfBlockerNG blocking ad domains with graphspfBlockerNG blocking ad domains with graphspfBlockerNG blocking ad domains with graphs The pie chart looks great. I followed this pfBlockerNG tutorial. Tekgru.com pfBlockerNG tutorial blogTekgru.com pfBlockerNG tutorial blogTekgru.com pfBlockerNG tutorial blog This is mportant: If you have multiple network interfaces (the mini PC has four), then you need to enable the Permit Firewall Rules for multiple interfaces and select them. DNSBL Perfmit Firewall Rules for multiple interfacesDNSBL Perfmit Firewall Rules for multiple interfacesDNSBL Perfmit Firewall Rules for multiple interfaces Would you like to have discretion over blocklists? Let's add a DNS blocklist related to gambling and reload pfBlockerNG to see if a poker site is blocked on the Trusted LAN. Some sketchy poker sites are now blockedSome sketchy poker sites are now blockedSome sketchy poker sites are now blocked If you would prefer the connection to just close instead of rendering a PHP page, create a new PHP script with the following code and select it in the pfBlockerNG settings page: [ DNS Resolver page, click "Display Custom Options", and add the lines: [server: log-queries:] 1 server: 2 log-queries: yes Well, hello there, Microsoft Windows. What are you up to trying to reach Google Tag Manager? Naughty OS. That request is now black-holed to a non-existent IP at 10.10.10.1. Windows is trying to reach Google Tag ManagerWindows is trying to reach Google Tag ManagerWindows is trying to reach Google Tag Manager Let's turn our attention to the TV and see how it fares under DNS interception. Top - --------------------------------------------------------------------- How to Restrict Apple TV and iPhone YouTube Ads? YouTube: Regarding YouTube, YouTube has been showing 7-second and 15-second ads, twice, back-to-back, nearly every few minutes. Why are the ads incessant and so long? I do not mind the occasional ad, similar to live TV, but these frequent ads would warrant FTC complaints it they were on live TV. YouTube is tricky because ads are also videos that come from the same domain, so domain-name blockers like pfBlockerNG cannot act on them. The best pfBlockerNG and Pi-hole can do is block googleadservices.com only after you watch an ad video and click on the ad. Many people opt to use a web browser like Firefox or Chrome with uBlock Origin that acts on JavaScript as a workaround. It might be enough to watch YouTube on a web browser and stream that to a smart TV. However, we cannot restrict ads on the iPhone (without jailbreaking and compromising it). What are our options? How can we safely restrict YouTube ads on all network devices? Top - --------------------------------------------------------------------- Trick the YouTube Ad Algorithm Instead Thought Experiment: Among friends, let's say that English-speaking countries get ads for the most ridiculous things because their residents are assumed to have disposable income. Can we instead make YouTube think we are an undesirable advertising target? What do ads in other parts of the world look like? Are those living in Antarctica or Low Earth Orbit getting a lot of ads too? Xkcd.com: Mess with advertisersXkcd.com: Mess with advertisers Xkcd.com: Mess with advertisers What would happen if we leverage the capabilities of this pfSense router to route YouTube Location Tracking information through a VPN that terminates in some remote part of the world with fewer YouTube viewers per capita? In other words, let's make ourselves undesirable to advertisers and see if we get fewer ads. Scotty from TNG episode 'Relics' understands the planScotty from TNG episode 'Relics' understands the planScotty from TNG episode 'Relics' understands the plan Top - --------------------------------------------------------------------- Research into YouTube Advertizing Spend Let's do some YouTube demographics research to find a part of the world avoided by advertisers. Mobile advertiser spend by country in 2020 (REF: statista.com)Mobile advertiser spend by country in 2020 (REF: statista.com)Mobile advertiser spend by country in 2020 Let's also check some YouTube statistics about viewers by country for insights. Thinking about following some Reddit advice and VPN'ing into India? Think again. Total YouTube views by country in 2019 (REF: ChannelMeter)Total YouTube views by country in 2019 (REF: ChannelMeter)Total YouTube views by country in 2019 (REF: ChannelMeter) That was 2019. This is 2020: Top ten YouTube countries with population (REF: backlinko.com)Top ten YouTube countries with population (REF: backlinko.com)Top ten YouTube countries with population (REF: backlinko.com) I'm not a digital advertiser, but I can see that people in the UK and Canada watch a large number of videos per sitting. If I were an advertiser though, I'd pump those two countries with video ad after video ad because, statistically, those residents will take the eyeball kicking. All things being equal, I definitely need a VPN to terminate outside of Canada, the UK, and the United States (English-speaking countries) to enjoy YouTube more. Does age play a factor? Who don't advertisers want? I want to be that guy on paper. YouTube age demographics as of 2020 (REF: backlinko.com)YouTube age demographics as of 2020 (REF: backlinko.com)YouTube age demographics as of 2020 (REF: backlinko.com) Top - --------------------------------------------------------------------- New Goal: Let's trick YouTube into believing I am a 70-year-old male living in Italy. Yes, that should definitely cut down on the Nespresso and Starbucks ads, at least. How then to convince YouTube that I am a retired Sicilian living on a small chain island? I embellished that last part. Seventy and in Italy is sufficient. Let's do this. In the YouTube account... I am 71 years oldI am 71 years oldI am [DEL:Iron Man:DEL] 71 years oldI am in ItalyI am in ItalyI am in Italy It is doubtful that this is all it takes for our goal. Let's find a VPN exit point in Italy. NordVPN has 60+ servers in ItalyNordVPN has 60+ servers in Italy NordVPN has 60+ servers in Italy Nice. NordVPN has about 60 servers in Italy (that's an affiliate link by the way). Top - --------------------------------------------------------------------- Selectively Route Apple TV Over the VPN Let's go through some tutorials to set up OpenVPN in pfSense. Just kidding! We're going to use WireGuard - we have the Intel AES-NI crypto instruction set because we didn't go cheap and get a yesteryear J1900 mini PC that sellers are trying to offload. I'll now install the FreeBSD WireGuard package. Install the WireGuard package in pfSenseInstall the WireGuard package in pfSenseInstall the WireGuard package in pfSense Next, add a tunnel and enable it. According to this thread and this thread on Reddit, we need to get some information for WireGuard and NordLynx from a sacrificial Linux VM to transpose the settings (i.e. private key) to the pfSense router. No problem. Connect to Italy over WireGuardConnect to Italy over WireGuardConnect to Italy over WireGuardWireGuard config information via wg show WireGuard config information via wg showWireGuard config information via wg show Run sudo wg showconf nordlynx to see your private key needed by the pfSense tunnel config. Here are various screenshots that show the steps in more detail. VPN > WireGuard > TunnelsVPN > WireGuard > TunnelsVPN > WireGuard > TunnelsVPN > WireGuard > PeersVPN > WireGuard > PeersVPN > WireGuard > Peers Tip: Enter 1.0.0.0 and then 0 as the subnet mask. Do not go for 0.0.0.0 as there is a glitch or bug in the UI or whathaveyou. The result will still be 0.0.0.0/0. VPN > WireGuard > SettingsVPN > WireGuard > SettingsVPN > WireGuard > SettingsInterfaces > Interface AssignmentsInterfaces > Interface AssignmentsInterfaces > Interface Assignments That should be enough to allow Diagnostics to curl Italy. Successfully connected to NordVPN through WireGuard on pfSense Successfully connected to NordVPN through WireGuard on pfSense Successfully connected to NordVPN through WireGuard on pfSense Successfully connect to Italy and verifiedSuccessfully connect to Italy and verifiedSuccessfully connect to Italy and verified Now that the easy part is out of the way, let's set some Policy rules to send the Apple TV traffic over the VPN to Italy as a baseline test. From Netgate, on the order of Firewall/NAT processing: Traffic from LAN to WAN is processed as described in the following more detailed example. + Port forwards or 1:1 NAT on the LAN interface (e.g. proxy or DNS redirects) + Firewall rules for the LAN interface: o Floating rules inbound on LAN o Rules for interface groups including the LAN interface o LAN tab rules + 1:1 NAT or Outbound NAT rules on WAN + Floating rules that match outbound on WAN I'll make an alias, for now, to hold some clients that have static DHCP entries and hostnames I gave them in pfSense. VPN clients in the Firewall > Aliases > IP pageVPN clients in the Firewall > Aliases > IP pageVPN clients in the Firewall > Aliases > IP page Floating rules in have high precedence, so I'll add some rules below the automatic pfBlockerNG rules that were created, and I'll add a nice little blue separator while I'm here. Floating rule to route select clients over the VPN to ItalyFloating rule to route select clients over the VPN to ItalyFloating rule to route select clients over the VPN to Italy And here is that rule as a very long screenshot: Firewall > Rules > Floating rule to route select clients over the VPN Firewall > Rules > Floating rule to route select clients over the VPN Firewall > Rules > Floating rule to route select clients over the VPN Apply. Wait. Let's try it out using one of my notebooks connected to the Untrusted network. Google is entirely in Italian nowGoogle is entirely in Italian now Google is entirely in Italian now Google is in Italian. Very cool. Now for the Apple TV. Apple TV's YouTube reports I am in ItalyApple TV's YouTube reports I am in ItalyApple TV's YouTube reports I am in Italy Winner winner, chicken diner. All my YouTube is in Italian. I get some ads, not as many, but because Italians speak slowly and with a kind of sexy accent I do not mind the ads for Nutella at all. With this technique, I no longer feel manipulated by non-English ads. I have personalized ads off, but given my new status as a retired gentleman I should turn that back on to scare away advertising dollars, er, euros. I wonder if Netflix and Amazon Prime behave any differently... Some Netflix assets are being blockedSome Netflix assets are being blockedSome Netflix assets are being blocked Dang. Netflix is having problems. Amazon Prime is even worse. It looks like some CSS or font files are blocked as well, and the thumbnails aren't loading. It's time to move to Phase Two: Tunnel only YouTube traffic over the VPN. Warning: Do not try to send all the Apple TV traffic over a VPN because Netflix, Prime, and others are wise to VPN providers and have gotten great at geofencing. Top - --------------------------------------------------------------------- Selectively Route Apple TV YouTube Traffic Over the VPN Let's start by adding Firewall Policy rules to send the most common YouTube domains over the VPN. As I'm about to add the rules, my hands hover over the keyboard not knowing what domains to tunnel. They need to be FQDN (fully-qualified domain names, no wildcards). Let's open up a Chromium-based browser and see what traffic it generates in DevTools. Add the domains column to DevTools to see where YouTube callsAdd the domains column to DevTools to see where YouTube callsAdd the domains column to DevTools to see where YouTube calls Here are some candidate FQDNs to add: [www.youtube.com yout] 1 www.youtube.com 2 youtube.com 3 googlevideo.com 4 accounts.google.com 5 googleapis.com 6 gstatic.com But wait, I hear you ask, why accounts.google.com and gstatic.com? This is a preventative measure just in case one of those domains is geo-jacked (Geo-IP LowJacking). I wouldn't put it past Google engineers to geo-jack the fonts domains like fonts.googleapis.com, but I'll take a chance they don't in the interest of scaling to billions of page views efficiently. Here are my new rules where I chain two of them using a tag so I can limit YouTube tunnelling to only the same untrusted machines (including Apple TV). YouTube domains to tunnelYouTube domains to tunnelYouTube domains to tunnelUse a match rule before the tunnel ruleUse a match rule before the tunnel ruleUse a match rule before the tunnel ruleThe first rule matches VPN clients and tags themThe first rule matches VPN clients and tags themThe first rule matches VPN clients and tags themThe second rule tunnels tagged requests through the VPNThe second rule tunnels tagged requests through the VPNThe second rule tunnels tagged requests through the VPN And with that, YouTube thinks I'm in Milan, Netflix and Prime Video think I am still in Canada, and the ads... oh the ads... they are few and far between, and when they do come on, they are just a treat to listen to in that slow, lack-of-harsh-aspirants-or-yelling of a beautiful language Italian is. YouTube, ItalyYouTube, ItalyYouTube, Italy Top - --------------------------------------------------------------------- Time goes by... --------------------------------------------------------------------- Gotcha: DNS Race Condition A day has gone by and I've noticed that I only get Nutella and Ferrero Roche ads in the middle of videos, not at the start. Odd. I did some research and this is what I found: Pertinent information about pfSense and hostname aliasesPertinent information about pfSense and hostname aliasesPertinent information about pfSense and hostname aliases This means that the hostnames are resolved to IP addresses once and those IPs are used in my VPN tunnelling policy rules. A hostname entry in a host or network type alias is periodically resolved and updated by the firewall every few minutes. The default interval is 300 seconds (5 minutes), and can be changed by adjusting the value of Aliases Hostnames Resolve Interval on System > Advanced, Firewall & NAT tab. - pfSense Ah-ha, so I suspect there is a DNS race condition. Let me explain: This happens if, say, the Alias Daemon updates the IPs of the FQDNs. Then, I turn on the Apple TV for the first time all day. Since the usual TTL (time-to-live) of DNS queries is 1440 seconds (30 minutes), all the YouTube DNS entries will be cache misses and will need to be updated. At this point, the IPs from the second DNS queries may be from a pool and are not guaranteed to be the same that the Alias Daemon has. When the Alias Daemon checks again in five minutes, it may resolve the FQDNs to yet different IPs! Let's solve this by overwriting whatever TTL (time-to-live) YouTube has in its DNS entries: Do not abide by the minimum TTL of the target DNS entryDo not abide by the minimum TTL of the target DNS entryDo not abide by the minimum TTL of the target DNS entry And with that, no more DNS lookup race condition. Top - --------------------------------------------------------------------- Gotcha: Authentication Trouble, Forbidden 403 Error Sometimes videos will not play. For security, YouTube embeds your IP in the googlevideo.com request. I've known about this since my post about Download YouTube 4K Videos with PHP back in 2016. The new problem is that various JavaScript and "are you human?" assets are tunnelled over VPN, but those darn domains like r5---sn-hpa7kn76.googlevideo.com are not tunnelled and thus come from the wrong IP. Queue the 403 Forbidden error. YouTube authentication failureYouTube authentication failureYouTube authentication failure Let's fail fast with a quick experiment: I've gotten the IP of the above second-level domain name (SLD), added it manually to the list of domains/IPs to VPN tunnel, applied the change, and refreshed YouTube: Success. We need to route the mangled domains over the VPN as well. Success. We need to route the mangled domains over the VPN as well. Success. We need to route the mangled domains over the VPN as well. Excellent. Now, we just need a way to tunnel that wildcard *.googlevideo.com domain. Unfortunately, the NAT and Firewall rules work with IPs, not wildcard domain names. Can we predict or enumerate these domains? Here is a Wireshark capture of DNS requests to *.googlevideo.com to show that the SLDs (second-level domains) are not eyeballably predictable: Random SLDs from googlevideo.comRandom SLDs from googlevideo.com Random SLDs from googlevideo.com Let's drop into a web browser with adblocking disabled and walk the HAR waterfall of my interaction with YouTube that led to ads showing up. Waterfall showing ad interactions coming from www.youtube.com Waterfall showing ad interactions coming from www.youtube.com Waterfall showing ad interactions coming from www.youtube.com What are GET requests like GET https://r7---sn-uxa0n-t8ge.googlevideo.com/generate_204 doing, exactly? I'll give this problem some thought offline. Top - --------------------------------------------------------------------- Gotcha: YouTube is Now Showing UK Ads, Not Italian Ads Before I could even solve the previous gotcha, British ads started showing up with the same frequency as if we did nothing. Ads from the UK are even more incessant than those from Canada, trailing behind the USA and India according to my earlier stats. It would be a complete failure if we get UK ads. Why does this happen suddenly? I've opened a fresh browser in a VM and tunnelled all traffic through Italy. The only leak I can find is when I query ipinfo.io on my Italian tunnel and see a UK address in the ASN. Could this small leak be our undoing? It is possible the VPN is leaking unintended informationIt is possible the VPN is leaking unintended informationIt is possible the VPN is leaking unintended information Even with my browser's language set to en_US and location data off, this is the only leak I can spot. Then, in addition to a VPN exiting in Italy, it has to be one that doesn't leak ASN (Autonomous System Numbers - used for automated routing) that gives up a different country. Dang, Google, you're good. I'm going to have to bring my A+ game to this one. Top - --------------------------------------------------------------------- Find a VPN Exit Node with no ASN Leak By visiting https://nordvpn.com/servers/tools/, I can see the VPN endpoint nodes in Italy. There are many Wireguard endpoints with NordVPN. Just to move things forward for this exercise, I'll add an OpenVPN tunnel in pfSense and connect to several VPN nodes and examine the ASNs. It's better than nothing, and more importantly, I'd like to eliminate the ASN as the leak of GeoIP information. Here is the guide I used. Through trial and error, I found a VPN node that is registered to an ISP in Italy as found in the Abuse and ASN info. Found an exit node with no ASN leaksFound an exit node with no ASN leaksFound an exit node with no ASN leaks Beautiful. Bellissimo. Italian content with Italian ads againItalian content with Italian ads againItalian content with Italian ads again Top - --------------------------------------------------------------------- Hijack Google Video DNS Queries To make any of this work, I need a technique to route the wildcard *.googlevideos.com domain through the VPN. Thought Experiment: Suppose I write a plugin for pfSense that periodically greps the DNS query log, keeps track of the *.googlevideo.com queries, and adds them to a unique list of aliases for Google Video domains; if backed by an LRU eviction policy, this could keep working indefinitely. However, if each video uses a unique, mangled domain, then this does not work unless I hit refresh on every single video. On the other hand, if I "hold up" the DNS query for the *.googlevideo.com domains, add the IPs to some alias list, then allow the DNS response to finish the round trip, we may be in business! pfSense DNS resolver has user Python supportpfSense DNS resolver has user Python supportpfSense DNS resolver has user Python support Where to even start? Here are some Python example scripts just to get some inspiration. A quick, mental reverse-engineering of a handful of scripts reveals that there are some event hooks available. Nice. Among friends, let's say that I can build up the pool of Google video IPs in real-time. How then to add these IPs programmatically to the firewall alias list for YouTube without restarting the firewall? One person actually hacked the PHP scripts in pfSense. Tempting, but I'll do more research. Another person created a REST API for pfSense. Jackpot! Top - --------------------------------------------------------------------- New Goal: We need to add IPs to the firewall policy rule to route YouTube videos over a VPN to avoid incessant and obnoxious North-American ads, but the IPs keep changing due to changing, mangled second-level domain names (SLDs). Using Python 3 and a REST API, we will monitor the appropriate DNS queries, note the IP(s) of the response, hold the response, add the IP(s) to the VPN tunnelling policy rule, then release the DNS query response. Research Python Methods to Hijack DNS Requests Why this approach? It's future-proof, modular, elegant, maintainable, automated, and it lends itself to a future decision tree that could truly restrict YouTube ads outright. First, I will enable SSHd in pfSense and take a peek around. Enable SSHd in pfSenseEnable SSHd in pfSenseEnable SSHd in pfSenseSSH into pfSense using the GUI credentialsSSH into pfSense using the GUI credentialsSSH into pfSense using the GUI credentials Rsync Disk Backup Let's take this opportunity to make a disk backup. du -h or "duh" shows that only 800 MiB is in use on the SSD. Let's rsync the whole box from our local machine in about four minutes. [# Rsync the pfSense ] 1 # Rsync the pfSense router locally, then compress to an archive. 2 # Tell the remote rsync to preserve ownership information. 3 # Fix brace expansion and execute (easy to read with tr and sed). 4 cat << EOF | tr -s ' ' | sed 's/, "/,"/g' | bash 5 time \ 6 rsync \ 7 --archive \ 8 --acls \ 9 --xattrs \ 10 --hard-links \ 11 --fake-super \ 12 --numeric-ids \ 13 --checksum \ 14 --info=progress2 \ 15 --no-compress \ 16 --whole-file \ 17 --inplace \ 18 --rsync-path="/usr/local/bin/rsync --fake-super --numeric-ids" \ 19 --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*",\ 20 "/var/*","/mnt/*","/media/*","/lost+found"} \ 21 --rsh="ssh -p 2222" \ 22 admin@pfsense:/ \ 23 ~/.pfsense-backup && \ 24 tar \ 25 --gzip \ 26 --create \ 27 --file ~/pfsense-backup-`date +"%Y-%m-%d"`.tar.gz \ 28 ~/.pfsense-backup 29 EOF Tip: To verify the owners and permissions are set in the extended attributes locally, run getfattr -d -m ^ -R -- ~/.pfsense-backup Install pfSense REST API Now that we have a pfSense backup (I'm told just backing up config.xml works too), let's install the REST API. This part had me confused. You see, I was looking at the bottom of the screen wondering how the heck I could copy a truncated hash as a token. After a few tries, I noticed the green message at the top that I had been trained to ignore. It has the token. Tricky UI screen to get the API tokenTricky UI screen to get the API tokenTricky UI screen to get the API token Next, with the API credentials set up, let's try out the API: [curl -k -s \ -H "C] 1 curl -k -s \ 2 -H "Content-Type: application/json" \ 3 -H "Authorization: 61646d696e 978c197c37a882f6da23553c152c1203" \ 4 -X GET https://pfsense/api/v1/firewall/alias \ 5 | jq '.data[] | select(.name == "VPN_domains")' Successful API testSuccessful API testSuccessful API test Explore the Unbound Python Module Running find / -name "py*" shows that the current version of Python is 3.8. As for the Unbound DNS Resolver, I had some luck tinkering in nano and writing simple Python 3.8 code to log DNS query messages. We now have both parts needed to dynamically update the firewall aliases and tunnel all YouTube traffic once and for all. If you are looking for Python module docs for Unbound, here they are: There are no readily available Python module docs for UnboundThere are no readily available Python module docs for UnboundThere are no readily available Python module docs for Unbound Run these commands to quickly get the documentation. [# Do this in a PyCha] 1 # Do this in a PyCharm venv terminal 2 git clone --depth 1 -b master --single-branch https://github.com/ 3 NLnetLabs/unbound.git unbound 4 pip3 install sphinx 5 cd unbound sphinx-build -b html pythonmod/doc/ doc/html/pythonmod/ Warning: The example code is from Python 2.4, so be prepared to run Black and PyCharm code formatting, or run 2to3. Also, the most important part of this whole exercise (getting the IPs from the DNS reply) is missing, so here is the hint: import ipaddress. Don't forget to manually hack the byte strings to pull out the proper IP addresses in binary form, first. Now we have Python docs and access to all the capabilities. Excellent. Successful generation of Unbound Python docs with SphinxSuccessful generation of Unbound Python docs with SphinxSuccessful generation of Unbound Python docs with Sphinx Next, take a backup of your OS or VM and install libtools and swig wherever, ./configure --with-pythonmodule, make, fix some errors in the Unbound code, make again, then you'll have the generated python module (unboundmodule.py) in order to remove all the missing-method red error lines in PyCharm. PyCharm can now find the missing methods we don't actually need to worry aboutPyCharm can now find the missing methods we don't actually need to worry aboutPyCharm can now find the missing methods we don't actually need to worry aboutFirst successful DNS response logging scriptFirst successful DNS response logging scriptFirst successful DNS response logging script Top - --------------------------------------------------------------------- Smoke Test: A Python DNS-Hijacking Script Here is a smoke test of the ability to hijack *.google.com DNS requests with reply IPs that the script has caught in just a few minutes (the timestamps are just to maintain a crude LRU cache): Smoke test for collecting IP addresses of *.google.comSmoke test for collecting IP addresses of *.google.comSmoke test for collecting IP addresses of *.google.com Duplicate IP addresses are possible, and that is fine. I let the smoke test run overnight. Here is the PoC (proof of concept) script I ran as the Unbound Python module script. [# -*- coding: utf-8 ] # -*- coding: utf-8 -*- # Copyright (c) 2021. Eric Draken (ericdraken.com) import ipaddress import json import os import ssl import sys import time import urllib.request from typing import Final, Union FILENAME: Final = os.path.splitext(os.path.basename(__file__))[0] .upper() ALIAS_VPN_WILDCARDS: Final = "VPN_wildcards" ALIAS_VPN_DOMAINS: Final = "VPN_domains" ALIAS_VPN_WILDCARDS_TTL: Final = 60 * 60 # 1 Hour ALIAS_VPN_WILDCARDS_CAPACITY: Final = 500 AUTH_CODE: Final = "61646d696e 978c197c37a882f6da23553c1xxxxxxx" TEST_MODE = True 1 2 if TEST_MODE: 3 API_ALIAS_URL: Final = "https://pfsense/api/v1/firewall/ 4 alias" 5 API_ALIAS_ENTRY_URL: Final = "https://pfsense/api/v1/firewall 6 /alias/entry" 7 else: 8 API_ALIAS_URL: Final = "https://127.0.0.1/api/v1/firewall/ 9 alias" 10 API_ALIAS_ENTRY_URL: Final = "https://127.0.0.1/api/v1/ 11 firewall/alias/entry" 12 13 # *********************************** 14 15 __wildcard_patterns = set() 16 17 if TEST_MODE: 18 19 def log_info(msg=""): 20 print(f"{FILENAME}: {msg}") 21 22 def log_err(msg=""): 23 print(f"{FILENAME}: {msg}") 24 25 26 else: 27 try: 28 # noinspection 29 PyUnresolvedReferences,PyUnboundLocalVariable 30 log_info 31 except NameError: 32 # Added to suppress IDE errors about missing functions 33 and constants 34 from unbound.pythonmod.unboundmodule import ( 35 log_info, 36 register_inplace_cb_reply, 37 register_inplace_cb_reply_cache, 38 register_inplace_cb_reply_local, 39 MODULE_EVENT_NEW, 40 MODULE_EVENT_PASS, 41 MODULE_WAIT_MODULE, 42 MODULE_EVENT_MODDONE, 43 MODULE_FINISHED, 44 log_err, 45 MODULE_ERROR, 46 ) 47 48 # Clarity of log messages 49 __old_log_info = log_info 50 __old_log_err = log_err 51 52 def log_info(msg=""): 53 __old_log_info(f"{FILENAME}: {msg}") 54 55 def log_err(msg=""): 56 __old_log_err(f"{FILENAME}: {msg}") 57 58 def log_response(qstate): 59 if not qstate: 60 return 61 62 r = None 63 if qstate.return_msg and qstate.return_msg.rep: 64 r = qstate.return_msg.rep 65 66 q = None 67 if qstate.return_msg and qstate.return_msg.qinfo: 68 q = qstate.return_msg.qinfo 69 70 if q: 71 test = str(q.qname_str) 72 if any(x in test for x in __wildcard_patterns): 73 log_info("HIT Query: %s, type: %s (%d), class: %s 74 (%d) " % (q.qname_str, q.qtype_str, q.qtype, q.qclass_str, q. 75 qclass)) 76 77 if r: 78 # Do not crash the whole Unbound service 79 try: 80 for i in range(0, r.rrset_count): 81 rr = r.rrsets[i] # ReplyInfo_RRSet 82 rk = rr.rk 83 if rk.rrset_class_str == "IN": 84 d = rr.entry.data # 85 RRSetData_RRData 86 for j in range(0, d.count + d. 87 rrsig_count): 88 if rk.type_str == "A": 89 ip = ipaddress. 90 IPv4Address(d.rr_data[j][2:]).exploded 91 elif rk.type_str == "AAAA": 92 ip = ipaddress. 93 IPv6Address(d.rr_data[j][2:]).exploded 94 else: 95 # Not an A or AAAA record 96 continue 97 98 log_info(f"{j}: IP: {ip!s}, 99 TTL={d.rr_ttl[j]!s}") 100 add_wildcard_ips(str(ip)) 101 except Exception as e: 102 exc_type, exc_obj, exc_tb = sys.exc_info( 103 ) 104 log_err(f"{exc_type}, {exc_tb.tb_lineno}, 105 {e}") 106 107 def inplace_reply_callback(qinfo, qstate, rep, rcode, edns, 108 opt_list_out, region, **kwargs): 109 log_response(qstate) 110 return True 111 112 def inplace_cache_callback(qinfo, qstate, rep, rcode, edns, 113 opt_list_out, region, **kwargs): 114 # log_response(qstate) 115 return True 116 117 def inplace_local_callback(qinfo, qstate, rep, rcode, edns, 118 opt_list_out, region, **kwargs): 119 # log_response(qstate) 120 return True 121 122 def init_standard(id_, env): 123 log_info("Init start") 124 125 # Register the inplace_reply_callback function as an 126 inplace callback 127 # function when answering a resolved query. 128 if not register_inplace_cb_reply(inplace_reply_callback, 129 env, id_): 130 return False 131 132 # Register the inplace_cache_callback function as an 133 inplace callback 134 # function when answering from cache. 135 if not register_inplace_cb_reply_cache( 136 inplace_cache_callback, env, id_): 137 return False 138 139 # Register the inplace_local_callback function as an 140 inplace callback 141 # function when answering from local data. 142 if not register_inplace_cb_reply_local( 143 inplace_local_callback, env, id_): 144 return False 145 146 # Prepare the aliases 147 recreate_vpn_wildcards() 148 global __wildcard_patterns 149 __wildcard_patterns = get_wildcard_patterns() 150 151 log_info("Init finished") 152 return True 153 154 def deinit(id_): 155 return True 156 157 def inform_super(id_, qstate, superqstate, qdata): 158 return True 159 160 def operate(id_, event, qstate, qdata): 161 # Wait for the Python module 162 if (event == MODULE_EVENT_NEW) or (event == 163 MODULE_EVENT_PASS): 164 qstate.ext_state[id_] = MODULE_WAIT_MODULE 165 return True 166 167 # Release when the Python module is finished 168 elif event == MODULE_EVENT_MODDONE: 169 qstate.ext_state[id_] = MODULE_FINISHED 170 return True 171 172 qstate.ext_state[id_] = MODULE_ERROR 173 return True 174 175 176 def request(url: str, method: str = "GET", body: object = None): 177 # Must be HTTPS 178 req = urllib.request.Request(url=url, method=method) 179 req.add_header("Content-Type", "application/json") 180 req.add_header("Accept", "application/json") 181 req.add_header("Authorization", AUTH_CODE) 182 183 ctx = ssl.create_default_context() 184 ctx.check_hostname = False 185 ctx.verify_mode = ssl.CERT_NONE 186 187 data = None 188 if body: 189 data = json.dumps(body).encode() 190 req.add_header("Content-Length", str(len(data))) 191 192 try: 193 res = urllib.request.urlopen(req, data, context=ctx, 194 timeout=1) # Short timeout! 195 json_ = json.load(res) 196 if "data" not in json_: 197 log_err(f"data attribute is missing: {json_}") 198 return False 199 # log_info(json_) 200 return json_ 201 except Exception as e: 202 log_err(e) 203 return False 204 205 206 def recreate_vpn_wildcards(): 207 # Check if the VPN_wildcard_ips alias exists 208 aliases = request(API_ALIAS_URL, "GET") 209 for alias in aliases["data"]: 210 if "name" in alias and alias["name"] == 211 ALIAS_VPN_WILDCARDS: 212 log_info(f"Deleting existing {ALIAS_VPN_WILDCARDS}") 213 # FIXME: If tied to a rule... it 400s 214 request(API_ALIAS_URL, "DELETE", {"id": 215 ALIAS_VPN_WILDCARDS, "apply": True}) 216 break 217 218 # Create 219 log_info(f"Creating {ALIAS_VPN_WILDCARDS}") 220 request( 221 API_ALIAS_URL, 222 "POST", 223 { 224 "name": ALIAS_VPN_WILDCARDS, 225 "type": "host", 226 "descr": f"Automatic {ALIAS_VPN_DOMAINS} wildcard 227 expansions", 228 "address": [], 229 "detail": [], 230 "apply": True, 231 }, 232 ) 233 234 # Check 235 aliases = request(API_ALIAS_URL, "GET") 236 for alias in aliases["data"]: 237 if "name" in alias and alias["name"] == 238 ALIAS_VPN_WILDCARDS: 239 log_info(f"Successfully created {ALIAS_VPN_WILDCARDS} 240 ") 241 return 242 log_info(f"Unable to create {ALIAS_VPN_WILDCARDS}") 243 244 245 def evict_wildcard_ips(): 246 cutoff = int(time.time()) - ALIAS_VPN_WILDCARDS_TTL 247 res = request(API_ALIAS_URL, "GET", {"name": 248 ALIAS_VPN_WILDCARDS}) 249 data: dict = res["data"] 250 if data: 251 alias = data.popitem()[1] 252 addresses = str(alias["address"]).split(" ") 253 timestamps = str(alias["detail"]).split("||") 254 assert len(addresses) == len(timestamps) 255 evictable = [] 256 for timestamp, address in zip(timestamps, addresses): 257 if int(timestamp) < cutoff: 258 evictable.append(address) 259 260 if len(evictable): 261 log_info(f"Evicting {evictable}") 262 request(url=API_ALIAS_ENTRY_URL, method="DELETE", 263 body={"name": ALIAS_VPN_WILDCARDS, "address": evictable, "apply": 264 True}) 265 266 267 def add_wildcard_ips(ips: Union[str, list]): 268 if isinstance(ips, str): 269 ips = [ips] 270 ips_repr = ", ".join(ips) 271 log_info(f"Adding [{ips_repr}]") 272 res = request( 273 API_ALIAS_ENTRY_URL, 274 "POST", 275 { 276 "name": ALIAS_VPN_WILDCARDS, 277 "type": "host", 278 "descr": ips_repr, 279 "address": ips, 280 "detail": [str(int(time.time()))] * len(ips), # Must 281 be a string 282 }, 283 ) 284 details = res["data"]["detail"] 285 # len("1638002792||") == 12 286 if len(details) >= (12 * ALIAS_VPN_WILDCARDS_CAPACITY) - 2: 287 log_info("Capacity reached. Starting eviction...") 288 evict_wildcard_ips() 289 290 291 def get_wildcard_patterns(): 292 patterns = set() 293 res = request(API_ALIAS_URL, "GET", {"name": 294 ALIAS_VPN_DOMAINS}) 295 data: dict = res["data"] 296 if data: 297 alias = data.popitem()[1] 298 details = str(alias["detail"]).split("||") 299 for detail in details: 300 if "*." in detail or ".*" in detail: 301 patterns.add(detail.replace("*.", ".").replace( ".*", ".")) # TODO: Make robust addresses = str(alias["address"]).split(" ") for address in addresses: patterns.add(address) log_info(f"Found wildcard patterns: {patterns}") return patterns if TEST_MODE: if __name__ == "__main__": log_info("Init start") recreate_vpn_wildcards() add_wildcard_ips("1.2.3.4") add_wildcard_ips(["1.2.3.4", "1.2.3.5"]) evict_wildcard_ips() get_wildcard_patterns() When I woke up, the Unbound DNS resolver service segfaulted. Here are the logs: We can see a full FQDN alias re-process on each firewall config updateWe can see a full FQDN alias re-process on each firewall config updateWe can see a full FQDN alias re-process on each firewall config update Failure: Capturing all the IPs from the DNS queries to *.googlevideo.com and *.google.com puts pfSense into a crawl as all the rules need to be reloaded on each addition. Top - --------------------------------------------------------------------- New Goal: Research and install a Squid-like proxy, create a fake-but-trusted CA certificate, host it, install it in a browser as a PoC, decode TLS traffic, and victory dance. Actually, it is not illegal to jailbreak most Apple TV boxes, so we could break in, add a root certificate valid for the pfSense box, MITM traffic from the Apple TV, and then Microsoft Bob is your uncle. That works because the pfSense box as the gateway can decrypt Apple TV traffic, inspect the request headers for the offending ad hostname, block the request, and re-encrypt other valid requests to Mountainview, California. But, then my iPhone would still show ads because it is harder to jailbreak, plus banking apps may detect this and not work anymore. Jailbreaking is too extreme, anyway. Fun fact: I used a jailbroken iPhone all the time in Japan because of a quirky cellphone law. You see, because of icky perverts who like to take photos inappropriately on elevators and escalators, Japan passed a law that made the camera shutter sound mandatory on all photos. Super unfortunate was that taking a screenshot of a web page also made the same loud, unmuteable shutter sound. Imagine you are on a train and you screenshot a Google map, it makes that loud shutter noise, and then you get dirty looks from the train riders. Yeah, I had to jailbreak and zero out the camera sound file. Let's see what it takes to spy on the HTTPS traffic from the Apple TV and iPhone to see if we can block ad URLs that way. Top - --------------------------------------------------------------------- Install a Fake-but-Trusted CA Cert on Apple TV and iPhone? Not wanting to jailbreak and add self-signed certs to Apple TV and iPhone, how hard would it be instead to add fake-but-trusted Certificate Authority (CA) certificates to each device? The 'A' in CA means there is no one higher to vet such a certificate. The 'A' is so powerful, that back in 2001 only a Windows patch was able to revoke some dangerous Verisign certificates. As a thought experiment, new CAs must come into existence from time to time. Let's Encrypt is relatively new, for example. There should then be an in-warranty way to get a fake, trusted CA cert into an Apple TV and iPhone. If that is possible, then an entire world of MITM spycraft is available to decrypt TLS packets in the clear and use good 'ol URL blocking on requests like [https://www.youtube.] 1 https://www.youtube.com/pagead/viewthroughconversion/... 2 https://www.youtube.com/pagead/conversion/... Let's see how easy this would be. We can add fake, trusted CA certs to iPhone tooWe can add fake, trusted CA certs to iPhone tooWe can add fake, trusted CA certs to iPhone too In fact, there are many, many CAs. Here is a quick find / -name "*.pem" in pfSense: Many CAs exist alreadyMany CAs exist alreadyMany CAs exist already Top - --------------------------------------------------------------------- Experiment with Squid and SquidGuard I'm aware of mitmproxy, but it needs to be side-channel installed onto the pfSense router. Let's see if the squid3 proxy that is available as a pfSense package can do what we need. First, I will take a bare-metal backup again so I can roll back in case mitmproxy is better. Install squid3 and ancillary packagesInstall squid3 and ancillary packagesInstall squid3 and ancillary packages I've installed those packages, and naturally, there are more buttons and options than in a space shuttle. I'll find a guide. I've followed the steps in the guide, however, since I have a large SSD and generous RAM, I've made a dedicated folder /squid_cache (and chown squid:proxy) with 8 GiB of cache and a juicy allowance on the per-item cache size which should also help with Docker and NPM speed-up. Two birds, one stone. With Transparent HTTPS support, this should be pretty rad. Tip: If web traffic slows down while using Squid, here are some System Tunables that can make Squid faster (ref): vfs.read_max 128 kern.ipc.nmbclusters 32768 Also, for local disk cache, aufs is asynchronous ufs (great for Docker too) and uses POSIX-threads to avoid blocking the main Squid process on disk-I/O. We can actually generate a CA cert in pfSense itself. Generate a CA in pfSenseGenerate a CA in pfSenseGenerate a CA in pfSense Now, how to get it into the Apple TV and iPhone? It should be hosted somewhere, right? How about on the router? Top - --------------------------------------------------------------------- Self-Host the MITM CA Certificate Self-hosting with a single command is ridiculously easy. From the SSH shell into pfSense, I can create a web folder and server like so: [mkdir /www chown -R ] 1 mkdir /www 2 chown -R squid:proxy /www 3 chmod -R 644 /www 4 echo "Hello" > /www/index.php 5 php -S 0.0.0.0:8000 -t /www When I visit //pfsense:8000 I should get a blank page with "Hello". From here, clients behind the pfSense router can temporarily access static documents. To make like easier, here is a PHP script to cause the MITM cert to download. [ not found (0) 9 libz.so.1 => not found (0) 10 libpthread.so.0 => not found (0) 11 libc.so.6 => not found (0) root@mitmproxy:~/mitm-7.0.4 # Apparently, there is a pkg install compat6x that can solve this for us (unavailable on pfSense), however, this is getting ridiculous! Let's try a new tactic. Since we are in a jail, we are not bound to the crippled (read: secured) pfSense environment. Maybe we can install the mitmproxy package normally in a jail? pkg install mitmproxy [... py38-urwid: ] 1 ... 2 py38-urwid: 2.1.2 3 py38-werkzeug: 2.0.1 4 py38-wsproto: 1.0.0 5 py38-zstandard: 0.15.2 6 python38: 3.8.12 7 readline: 8.1.1 8 sqlite3: 3.35.5_3,1 9 zstd: 1.5.0 10 11 Number of packages to be installed: 50 12 13 The process will require 206 MiB more space. 14 33 MiB to be downloaded. 15 16 Proceed with this action? [y/N]: And, Bingo was his name-o. After this, simply running mitmproxy in the jailed console opens the MITMProxy UI. Nice. Note, this version may be one or two minor versions behind the master branch. Let's clean up with rm -rf ~/mitm* /lib64 and do another bare-metal backup. Top - --------------------------------------------------------------------- Exploring MITMProxy This is getting exciting. First, in pfSense, add a virtual IP for 127.0.1.1 attached to localhost. Then, add a NAT rule to temporarily forward port [Private IPs]:8080 to 127.0.1.1:8080 to access the proxy from the LANs. If not in the jail console, I'll run [ezjail-admin console] 1 ezjail-admin console mitmproxy 2 mitmproxy --listen-port 8080 --set console_focus_follow=true and add the proxy setting 192.168.20.1:8080 to my sacrificial notebook (that is auto-wiped daily). When the browser opens, we can already see colourful log entries in the MITMProxy UI. First logs of MITMProxyFirst logs of MITMProxyFirst logs of MITMProxy The next step is to get the auto-generated CA PEM file used by MITMProxy (~/.mitmproxy/mitmproxy-ca-cert.pem). Since any CA cert here is snake oil, I'll use the provided one. TLS traffic from my devices is safe as long as I use my own proxies. Let's put our experience from our previous attempt at self-hosting a CA into action. However, there is no PHP in the jail, so we can use a Python 3 web server instead. [set PYTHON='/usr/loc] 1 set PYTHON='/usr/local/bin/python3.8' 2 mkdir ~/www 3 # Both Python and mitmproxy run as 'root' 4 chmod 444 ~/.mitmproxy/mitmproxy-ca-cert.pem 5 ln -s ~/.mitmproxy/mitmproxy-ca-cert.pem ~/www/cert.pem 6 $PYTHON -m http.server --bind 127.0.1.1 --directory ~/www 8001 Tip: MITMProxy conveniently has onboarding settings to serve the same CA cert, as we did manually, just by visiting mitm.it. After installing the CA in the Trusted Root Store on my clean notebook (and rebooting), I am treated to this display: MITM TLS interception is working wellMITM TLS interception is working wellMITM TLS interception is working well Let's see if we can get this cert on my iPhone. Successfully added a root CA to the iPhoneSuccessfully added a root CA to the iPhoneSuccessfully added a root CA to the iPhone This is incredibly exciting. Can we LoJack the Apple TV box next? Successfully installed a root CA on the Apple TVSuccessfully installed a root CA on the Apple TVSuccessfully installed a root CA on the Apple TV Excellent. But wait, the router is slowing down. mitmproxy is burning up the CPU... on idle. MITMProxy is burning up the CPU while on idleMITMProxy is burning up the CPU while on idleMITMProxy is burning up the CPU while on idle Of course: Python is a single-threaded paradigm with the GIL (Global Interpreter Lock) ensuring threads do not actually run concurrently - unless they are blocking on I/O, which is the case here(?). Except, most of the CPU work is to generate TLS certs on the fly for each request. Yikes. Running mitmdump forgoes the UI and extreme logging. The extreme logging of all the headers and full responses heavily slows down mitmproxy, but mitmdump by default only logs entries like classic Apache logs - much kinder on the CPU. Certificate Pinning Some advanced, high-security web servers have trouble with the MITMProxy certificates due to Certificate Pinning - this is a technique where the server or the client know the fingerprint of the expected certificate in advance so it cannot be forged. A workaround is to use the --ignore-hosts option to let them bypass the proxy. For my fun, I'll go with this CLI command: [# Try to avoid compr] 1 # Try to avoid compression to save CPU usage 2 # Ignore some difficult sites 3 mitmproxy \ 4 --listen-port 8080 \ 5 --listen-host 127.0.1.1 \ 6 --anticomp \ 7 --mode regular \ 8 --ignore-hosts '^(?:.+\.)?apple\.com:443$' \ 9 --ignore-hosts '^(?:.+\.)?icloud\.com:443$' \ 10 --set console_focus_follow=true While on YouTube, we can see the page ads clear as day with their unencrypted headers; can a simple regex now block them? They are exposed, and afraid, and their days have run out. MITMProxy can see the YouTube ad URLsMITMProxy can see the YouTube ad URLsMITMProxy can see the YouTube ad URLs We can even see details about each request. For example, all the SAN info is laid out for this wide-reaching certificate. There are curiously a lot of *-cn.com domains covered by this cert. We can see rich request and response detailsWe can see rich request and response detailsWe can see rich request and response details [# Try to avoid compr] 1 # Try to avoid compression to save CPU usage 2 # Use a script to block YouTube ads 3 mitmdump \ 4 --listen-port 8080 \ 5 --listen-host 127.0.1.1 \ 6 --anticomp \ 7 --mode regular \ 8 --ignore-hosts '^(.+\.)?apple\.com(:443)?$' \ 9 --ignore-hosts '^(.+\.)?icloud\.com(:443)?$' \ 10 --scripts "youtube.py" # <-- This is new Shortly, I'll write a Python script to block YouTube /pagead/ URLs. Top - --------------------------------------------------------------------- Patch MITMProxy Source Code for Server SNI Interrogation This step may be optional for most, but as a reminder to myself, to make --allowed-hosts work better in Transparent Proxy Mode, the SNI of the server request needs to be checked against the list of regular expressions or else only the server's IP is used for matching in many cases. Here is a quick patch I made that can be applied directly in the jail shell (or just type a few lines manually) for mitmproxy version 7.0.4: [Index: venv/lib/pyth] Index: venv/lib/python3.8/site-packages/mitmproxy/addons/ next_layer.py IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP 1 <+>UTF-8 2 ================================================================== 3 = 4 diff --git a/usr/local/lib/python3.8/site-packages/mitmproxy/ 5 addons/next_layer.py b/usr/local/lib/python3.8/site-packages/ 6 mitmproxy/addons/next_layer.py 7 --- a/usr/local/lib/python3.8/site-packages/mitmproxy/addons/ 8 next_layer.py (date 1641187083049) 9 +++ b/usr/local/lib/python3.8/site-packages/mitmproxy/addons/ 10 next_layer.py (date 1641187083049) 11 @@ -59,7 +59,7 @@ 12 re.compile(x, re.IGNORECASE) for x in 13 ctx.options.allow_hosts 14 ] 15 - def ignore_connection(self, server_address: Optional 16 [connection.Address], data_client: bytes) -> Optional[bool]: 17 + def ignore_connection(self, server: Optional 18 [connection.Server], data_client: bytes) -> Optional[bool]: 19 """ 20 Returns: 21 True, if the connection should be ignored. 22 @@ -70,8 +70,11 @@ 23 return False 24 hostnames: List[str] = [] 25 - if server_address is not None: 26 - hostnames.append(server_address[0]) 27 + if server is not None: 28 + if server.address is not None: 29 + hostnames.append(server.address[0]) 30 + if server.sni is not None: 31 + hostnames.append(server.sni) 32 if is_tls_record_magic(data_client): 33 try: 34 ch = parse_client_hello(data_client) 35 @@ -122,7 +125,7 @@ 36 return stack_match(context, layers) 37 # 1. check for --ignore/--allow 38 - ignore = self.ignore_connection(context.server.address, 39 data_client) 40 + ignore = self.ignore_connection(context.server, data_client) if ignore is True: return layers.TCPLayer(context, ignore=True) if ignore is None: With the above patch, I can now reliably intercept a few hosts and let all others pass through. Reliable server host interception in MITMProxy transparent proxy mode Reliable server host interception in MITMProxy transparent proxy mode Reliable server host interception in MITMProxy transparent proxy mode Top - --------------------------------------------------------------------- Smoke Test: Intercept YouTube Ads with MITMProxy After reading the docs and navigating the mitmproxy source code in the PyCharm IDE, I've written a little script to block ads and tracking URLs coming from YouTube from my clean notebook. I won't reproduce the code just yet because it didn't succeed in blocking ads as hoped, so instead, I'll spend the time investigating why. Here are the smoke test filters I used where for a given top-level domain, URLs with the following partial strings are blocked: [blocked_partials: di] blocked_partials: dict = { 1 "youtube.com": ["/pagead/", "/log_event?", "/stats/ads", "/ 2 stats/qoe?", "/ptracking?", "/generate_204", "el=adunit", "adformat 3 =", "/activeview?"], 4 "google.com": ["/pagead/"], 5 "google.ca": ["/pagead/"], 6 "ggpht.com": ["."], } My initial results on blocking are positive. Everything I wanted to be blocked is faithfully blocked. Note, the (failed) entries are due to my script, and the 502 failures are due to pfBlockerNG black-holing the request. MITMProxy blocking script is workingMITMProxy blocking script is workingMITMProxy blocking script is working Even in the DevTools network panel, the requests are truly blocked. YouTube requests are truly blocked in DevTools network panelYouTube requests are truly blocked in DevTools network panelYouTube requests are truly blocked in DevTools network panel Then how come I am still seeing ads? I've disabled HTTP/2 so that subsequent requests on the same channel don't slide by. Mind you, sometimes the ads skip on their own, or fail to play, but they still show up. Interesting. Could YouTube be using WebSockets? I need some inspiration, so I'll look at uBlock Origin's regex filters for some ideas. Tip: If you see the error OpenSSL Error([('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert internal error')]), then the DNS blocker (i.e. pfBlockerNG) is breaking the upstream TLS handshake for a given domain. Either whitelist it in pfBlockerNG (so the request goes through), or intercept it and block the connection in mitmproxy. This error happens to black-holed domains when the upstream TLS cert cannot be sniffed. The cleanest strategy is to use transparent MITM mode. Top - --------------------------------------------------------------------- Examine uBlock Origin Regex Patterns for Inspiration Here are some of the regex/filters that uBlock Origin uses on YouTube. uBlock Origin YouTube regex/filters from a web browseruBlock Origin YouTube regex/filters from a web browseruBlock Origin YouTube regex/ filters from a web browser At first blush, it seems that a community of like-minded individuals is playing whack-a-mole with YouTube's HTML and JavaScript. This has got me thinking: How does a video know to play an ad with JavaScript? How does YouTube know if the ad converts? They must target ads for individuals, so a given video must receive some unique information about an ad, such as the click link and alt text. WebSockets would be a pain to maintain, especially with all the mobile clients. They must be using stateless JSON to relay that pertinent information in an innocuous URL request that has no telltale signs of ad-ness. Let's hunt for this info in the JSON replies captured by mitmproxy. Key advertizement information contained in a JSON responseKey advertizement information contained in a JSON responseKey advertizement information contained in a JSON response Snap, Crackle, and Pop. We have a new plan: surgically alter the JSON response body to eliminate or Byzantine-up the ad information. Top - --------------------------------------------------------------------- Surgically Alter the JSON Response to Remove Ads After a bit more playful exploration, a trove of blocklorne URLs is right there in the JSON payload. In fact, most of what I am trying to block shows up right here: [ ... "playerAds": [] ... "playerAds": [ { "playerLegacyDesktopWatchAdsRenderer": { "playerAdParams": { "showContentThumbnail": true, "enabledEngageTypes": "3,6,4,5,17,1" }, "gutParams": { 1 "tag": "\\4061\\ytpwmpu" 2 }, 3 "showCompanion": true, 4 "showInstream": true, 5 "useGut": true 6 } 7 } 8 ], 9 "playbackTracking": { 10 "videostatsPlaybackUrl": { 11 "baseUrl": "https://s.youtube.com/api/stats/playback?cl= 12 417308503&docid=IgF3..." 13 }, 14 "videostatsDelayplayUrl": { 15 "baseUrl": "https://s.youtube.com/api/stats/delayplay?cl= 16 417308503&docid=IgF..." 17 }, 18 "videostatsWatchtimeUrl": { 19 "baseUrl": "https://s.youtube.com/api/stats/watchtime?cl= 20 417308503&docid=IgF..." 21 }, 22 "ptrackingUrl": { 23 "baseUrl": "https://www.youtube.com/ptracking?ei= 24 KnzDYZv1B86ikwa0no7AAg&oid=MjD-gn49GocgAFypi8EDnQ&plid= 25 AAXTwR1aNKG2iTgr&pltype=content&ptchn=HnyfMqiRRG1u-2MsSQLbXA&ptk= 26 youtube_single&video_id=IgF3OX8nT0w" 27 }, 28 "qoeUrl": { 29 "baseUrl": "https://s.youtube.com/api/stats/qoe?cl=417308503& 30 docid=IgF3OX8nT..." 31 }, 32 "atrUrl": { 33 "baseUrl": "https://s.youtube.com/api/stats/atr?docid= 34 IgF3OX8nT0w&ei=KnzDYZv1B86ikwa0no7AAg&feature=g-high-trv&len=1213&ns= 35 yt&plid=AAXTwR1aNKG2iTgr&ver=2", 36 "elapsedMediaTimeSeconds": 5 37 }, 38 "videostatsScheduledFlushWalltimeSeconds": [ 39 10, 40 20, 41 30 42 ], 43 "videostatsDefaultFlushIntervalSeconds": 40, 44 "youtubeRemarketingUrl": { 45 "baseUrl": "https://www.youtube.com/pagead/ 46 viewthroughconversion/962985656/?backend=innertube&cname=1&cver= 47 2_20211221&data= 48 backend%3Dinnertube%3Bcname%3D1%3Bcver%3D2_20211221%3Bptype%3Df_view%3Btype%3Dview%3Butuid%3DHnyfMqiRRG1u-2MsSQLbXA%3Butvid%3DIgF3OX8nT0w 49 &foc_id=HnyfMqiRRG1u-2MsSQLbXA&label=followon_view&ptype=f_view& 50 random=37068419&utuid=HnyfMqiRRG1u-2MsSQLbXA", 51 "elapsedMediaTimeSeconds": 0 52 }, "googleRemarketingUrl": { "baseUrl": "https://www.google.com/pagead/1p-user-list/ 962985656/?backend=innertube&cname=1&cver=2_20211221&data= backend%3Dinnertube%3Bcname%3D1%3Bcver%3D2_20211221%3Bptype%3Df_view%3Btype%3Dview%3Butuid%3DHnyfMqiRRG1u-2MsSQLbXA%3Butvid%3DIgF3OX8nT0w &is_vtc=0&ptype=f_view&random=838827488&utuid=HnyfMqiRRG1u-2MsSQLbXA" , "elapsedMediaTimeSeconds": 0 } }, However, YouTube has bobby-trapped their UI and there is more than one way their obfuscated JavaScript code can pull down the ad details. Let's blow it all away right now. After a lot of fun taking apart the YouTube UI and HTTP workflow, taking into account cookies and naughty service workers, I am successfully able to strip away all the pre-roll, post-roll, mid-video, and, well, all the video ads. Here is a screenshot from mitmdump showing how select REST queries are intercepted, decrypted, modified, put back into the response, and the headers updated (content length, etc.). Success in removing YouTube ads via decrypted JSON responsesSuccess in removing YouTube ads via decrypted JSON responsesSuccess in removing YouTube ads via decrypted JSON responses With this new ability, we could even inject JavaScript into the main YouTube web page and subvert their JavaScript in a sort of ECMAScript arms race, possibly even leveraging some of the filters from uBlock Origin. However, we can hang our hats on this accomplishment for today. Success: We can strip out ads from the JSON payload for YouTube web ads using a router. Top - --------------------------------------------------------------------- The iOS YouTube App Uses Protobuf, not JSON I can see very similar data in the Protocol Buffer (Protobuf) version of the same API calls as the web version to that of the YouTube iOS app. That complicates things, somewhat: We cannot lean on JSONPath to hunt down advertisement sections of JSON because with Protobuf the keys are just numbers that can even change. The iOS version of the YouTube app uses ProtobufThe iOS version of the YouTube app uses ProtobufThe iOS version of the YouTube app uses Protobuf Fun fact: YouTube compiles a large list of all the ads you are going to see and sends that to you in a sneaky payload. In fact, it is easier to visualize this when reading Protobuf. If you manage to exhaust that list, then another large list will be coming your way. I can see strings like "Telus" and "Samsung TV" and "Boxing Week" and "Buy now". Remember when YouTube was a fun place? A fable about a Golden Goose comes to mind, Alphabet. What is a Protocol Buffer? Here is an infographic from Data Science Blog. Protobuf introductionProtobuf introductionProtobuf introduction (Credit: Data Science Blog) As a consequence of being able to see unencrypted traffic from my iPhone, I'm taken aback by the sheer amount of tracking information laid bare; It's like I have electrodes on my head and chest while I'm running on a treadmill and a bunch of scientists in white lab coats with clipboards are standing shoulder-to-shoulder recording everything about my internals. Privacy concern: Your apps are tracking you like crazy: what you do, how long you dwell, when you leave a given app, and so much more. The URL https://play.googleapis.com/log/batch shows up a lot in my logs. The next question is: Does the iOS app protocol behave like the web app? Top - --------------------------------------------------------------------- Timing Analysis to Detect Ad Videos? The iOS network traffic is not like the web traffic; Google has teams and teams of engineers dedicated to making sure blocking their ads isn't computationally feasible. Daunted but undeterred, I was staring at network requests to let my mind zone out and wander when I noticed a pattern I had not noticed before. For the web version of YouTube, I can eyeball which URLs are ads and which are the videos I want to watch. Take a look: Which are ad videos and which are content videos?Which are ad videos and which are content videos?Which are ad videos and which are content videos? How am I able to eyeball which video URLs are ads in this chaos? Two ad videos between content videosTwo ad videos between content videosTwo ad videos between content videos Take a look at the query parameter range. For the web version, a chunk of the video I want is fetched from the 0th byte, then immediately another video is fetched with a range starting again at the 0th byte. Both happen near-simultaneously - faster than a human can click on a new video. It turns out this, as well as examining the clen parameter for the length of the full video (short videos are likely ads), can reasonably allow us to detect and doctor ad videos. However, the iOS YouTube protocol does not use the range query parameter or even the Range header; video chunks use a counter like & nr=2 and &nr=3 etc. We must reverse engineer the Protobuf responses. Top - --------------------------------------------------------------------- Decode the YouTube Protobuf Responses Here are some decoded Protobuf log files I created then opened in the PyCharm IDE. Let's examine some Protobuf logs in the IDELet's examine some Protobuf logs in the IDELet's examine some Protobuf logs in the IDE After logging decoded Protobuf messages to disk for offline analysis, I did notice something that piqued my interest. [ 2 { 1: has] 1 2 { 2 1: has_unlimited_entitlement 3 2: False 4 } 5 2 { 6 1: has_premium_lite_entitlement 7 2: False 8 } I wonder what would happen if I were to, say, toggle those? This is tantalizing, but it is cheating, and hence no fun. Back to heuristics. Thought Experiment: As with JSON, can I blow away the Protobuf sections that serve up ads? Could I instead detect the ad videos in the payload, then dynamically modify their responses to be, say, a cached 0.01s video file? The 30s ~ 300s of unskippable ads could be over in the blink of an eye without blocking all those URLs. Intercepted ad URLs from the Protobuf payloadIntercepted ad URLs from the Protobuf payloadIntercepted ad URLs from the Protobuf payload Let's start by blocking the ads as intended. Top - --------------------------------------------------------------------- Ad URL Polymorphism The Protobuf responses are a hot mess of bytes, but there are human-readable URLs that can be grepped. You'd think a simple LRU cache that blocks soon-encountered ad URLs could be the way to go, but, alas, the ad URLs do not quite match the URLs sent over the wire. Also, who is to say that YouTube won't randomize the position of query-string parameters one day? We need an O(1) lookup of flagged ad URLs that are polymorphic (and group homomorphic) to live ad URLs. Detected ad URLs vs intercepted ad URLsDetected ad URLs vs intercepted ad URLsDetected ad URLs vs intercepted ad URLs It might be tempting to split a query string into a sorted dictionary and reassemble it, but we have no way of knowing what the query string boundary is. Plus, a live ad URL could add a key and disrupt the sorting. Addionally, I've encountered URLs like this that purposely try to obfuscate the query params: https://r4--sn-vgqsrns6.googlevideo.com/videoplayback /expire/1640607416 /ei/WFrJYdWnFfyTsfIP4s2BsAk /ip/121.35.98.26 /id/o-AE7swWOPOwXu3GyRght /source/youtube /requiressl/yes /mh/wU/ mm/31,26/... Notice how /ip/121.35.98.26/ is just &ip=121.35.98.26? I propose heuristically scanning for query and path parameters of ad URLs with high entropy and using those as keys (fingerprints). For example, in https://rr6--sn-uxa0n-t8gz.googlevideo.com/initplayback?source= youtube &orc=1&oeis=1&c=IOS&oss=1&oda=1&oad=5500&ovd=5500&oaad=11000&oavd =11000 &ocs=700&oputc=1&oses=1&ofpcc=1&osbr=1&osnz=1&msp=1&odeak=1&odepv =1 &osfc=1&id=58cc678216d6aaca&ip=121.35.98.26&initcwndbps=2125000 &mt=1640373902 One could note the following candidates in descending order of length: * rr6--sn-uxa0n-t8gz * 58cc678216d6aaca * 121.35.98.26 * 1640373902 * 2125000 Any or all of them could be lookup keys each pointing to the same dictionary of deconstructed query parameters. A lookup of a live URL would involve the same process of finding the highest entropy parameters and checking the URL dictionary for a match. The cache data structure can even be multi-level with the root keys being just the length of the high-entropy strings. Failure: Even with the ability to block polymorphic URLs, the video ads are still indistinguishable from content video without context from the Protobuf structure. Top - --------------------------------------------------------------------- Smoke Test: Intercept and Decode Protobuf in Python Python is Slow: Decoding ~500 kiB of raw Protobuf in pure Python is painfully slow. Decoding ~500 kiB of Protobuf in pure Python, especially the decoding step of converting it to over 1 MiB of human-readable text to parse the ad URLs, takes more time than the connection timeout most of the time. I'll run some benchmarks using pure Python vs. the native C++ library. Pure Python Benchmarks [from timeit import r] from timeit import repeat 1 from mitmproxy.contentviews.protobuf import format_pbuf 2 with open("proto.raw", "rb") as f: 3 data: bytes = f.read() 4 print(repeat(lambda: format_pbuf(data), number=5)) 5 6 # On a i7-6700 CPU @ 3.40GHz desktop 7 # [2.10792827908881, 2.0718665630556643, 2.0739889848046005, 8 2.065321908099577, 2.070936748990789] 9 10 # On the pfSense router: 11 # [24.182968072011136, 22.833560551982373, 23.53838806191925, 22.842924927012064, 22.81738876597956] Pure C++ Benchmarks [# On a i7-6700 CPU @] # On a i7-6700 CPU @ 3.40GHz desktop 1 TIMEFORMAT=%R 2 for i in {1..5}; do time protoc --decode_raw < proto.raw > /dev/ 3 null; done 4 # 0.018 5 # 0.017 6 # 0.022 7 # 0.018 8 # 0.018 9 10 # On the pfSense router: 11 printf 'foreach f ( 1 2 3 4 5 )\n time protoc --decode_raw < 12 proto.raw > /dev/null \n end \n' | tcsh 13 # 0.030u 0.114s 0:00.14 100.0% 30+153k 2+0io 0pf+0w 14 # 0.024u 0.104s 0:00.12 100.0% 8+132k 2+0io 0pf+0w 15 # 0.022u 0.106s 0:00.12 100.0% 34+156k 2+0io 0pf+0w 16 # 0.016u 0.114s 0:00.13 92.3% 8+143k 2+0io 0pf+0w # 0.023u 0.102s 0:00.12 100.0% 8+132k 2+0io 0pf+0w If you caught that, it takes about 23s in Python, and 100ms in C++! In this Never Ending Story, we have to find a way to parse the raw Protobuf payloads in Python using the C++ library libprotobuf.so. In the interest of time, I'll use subprocess.Popen and communicate with the C++ protoc binary directly (since raw decoding is not supported in Python anyway). Top - --------------------------------------------------------------------- Fuzzing the YouTube Video Ad Responses How about fuzzing the ad video responses? Now being able to isolate ad videos, as a smoke test, I sent back 200 responses with empty bodies and the iOS app went bananas; it was as if there is an infinite loop with no delay just hammering YouTube's own servers trying to get the next part of the video in panic mode. I felt bad for their servers, so I stopped. Then, what would a happy-path response payload look like? Infinite spin-lock loop of YouTube trying to get the next bytes of the ad videoInfinite spin-lock loop of YouTube trying to get the next bytes of the ad videoInfinite spin-lock loop of YouTube trying to get the next bytes of the ad video Try as I might, when I send back empty 200s, 404s, 503s, truncate response bodies, or just null-out part of the ad video, the iOS app crawls then crashes spectacularly with a dying breath of a messed up iOS UI. I now block some error reporting endpoint at /error_204/ that indicates a "dev assertion failed" so I don't make some overworked QA pull out their hair. Failure: We've learned that blocking ad URLs causes the app to deploy countermeasures and even when defeated, the app hangs forever on the ad screen. We've also learned that fuzzing ad videos often causes the app to crash - there is even session meta data in the video response chunks. Let's go back to what worked with JSON and obliterate the section of the Protobuf responses that contain the array of ad details. Top - --------------------------------------------------------------------- Enter Burp Suite Tools for Penetration Testing There is a library for Burp Suite called blackboxprotobuf (get the original Burp Suite version, not the PyPi fork, unless you like infinite recursion bugs) that is designed to decode raw Protobuf wire messages, inject something naughty, then re-encode them again to see how a Protobuf endpoint behaves. We are going to have so much fun together in this next section. [# Install blackboxpr] 1 # Install blackboxprotobuf from source 2 mkdir blackboxprotobuf_src && cd blackboxprotobuf_src 3 git clone https://github.com/nccgroup/blackboxprotobuf.git . 4 pip3 install poetry 5 cd lib 6 poetry install 7 # pwd -> blackboxprotobuf_src/lib/ 8 cp -r blackboxprotobuf your/project/folder 9 # We only need this folder tree for the Py3 API You may encounter a small world of pain because some forks of blackboxprotobuf will cause a stack overflow due to deep recursion. You can see this by adding sys.setrecursionlimit(200). Compiling the original library source code for Burp Suite and using the C++ bindings will allow us to transcode ~500 kiB of raw Protobuf bytes in just a few seconds. Tip: At the top of your import chain before you import protobuf, add 1 import os 2 os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "cpp" to use the C++ libprotobuf.so implementation whenever possible. It is now possible to generate a best-guess .proto schema with a single function: [import os os.environ] 1 import os 2 os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "cpp" 3 4 from mitmdump.blackboxprotobuf.lib import protobuf_to_json 5 6 data: bytes = ... 7 message, typedef = protobuf_to_json(data) 8 # print(message) 9 print(typedef) The schema isn't perfect, and it is huge and deeply nested, and takes forever to pretty-print, and is probably wrong, but is just good enough to pull out the ad details like so (Protobuf to JSON in this sample): Sample Protobuf to JSON showing a section of adsSample Protobuf to JSON showing a section of adsSample Protobuf to JSON showing a section of ads The Python schema is huge and looks like this for about 250,000 more charcters: [OrderedDict([ ('] 1 OrderedDict([ 2 ('1', OrderedDict([ 3 ('type', 'message'), ('message_typedef', OrderedDict([ 4 ('6', OrderedDict([ 5 ('type', 'message'), ('message_typedef', 6 OrderedDict([ ('1', OrderedDict([... Reverse engineering the Protobuf schema sounds good on paper, but our target is spectacularly complex and a moving target. Top - --------------------------------------------------------------------- Exfil the Proto Schemas from the App, Cleanly? As fun as it to reverse the Protobuf and generate a best-guess schema, wouldn't it be more ninja-like to exfil the actual, working .proto or schema files from the smartphone app? Let's pull out the Protobuf schemas from the Android version of the YouTube app and see if the schemas are the same or compatible. This is what I tried at first, but it went nowhere with the Protobuf Toolkit (PBTK). I reproduce it here so I remember what I tried: [sudo apt update sudo] 1 sudo apt update 2 sudo apt install libqt5x11extras5 python3-pyqt5.qtwebengine python3 3 -pyqt5 4 pip3 install pyqt5 pyqtwebengine requests websocket-client 5 mkdir pbtk && cd pbtk 6 git clone https://github.com/marin-m/pbtk . ./gui.py After installing Qt dependencies (pronounced "cute"), I was treated to a GUI. PBTK - The Protobuf ToolkitPBTK - The Protobuf ToolkitPBTK - The Protobuf Toolkit Next, I got the most recent release of a 100 MiB Android APK file from apkpure.com. Excited in vain, the most PBTK could get was a 59-byte proto file. Another tool called Apktool also looked promising, but the best it can do is disassemble bytecode, not decompile it - this may be good enough for Pen Testers, however. What ended up working for APK decompilation is a combination of a dedicated person's dex2jar tool and a Java Decompiler. A helpful guide can be found here. [# Follow the install] # Follow the install steps at https://stackoverflow.com/a/4177581/ 1 1938889 2 cd dex2jar 3 chmod -R +x *.sh 4 sh d2j-dex2jar.sh -f -o ../output.jar ../YouTube_v16. 5 49.37_apkpure.com.apk 6 cd ../jd-gui java -jar jd-gui-1.6.6.jar You can see that Google went out of its way to complicate reverse engineering. YouTube APK reversed into obfuscated Java classesYouTube APK reversed into obfuscated Java classesYouTube APK reversed into obfuscated Java classes Google thoughtfully did leave some hints. All the Protobuf schemas laid bare and human-readableAll the Protobuf schemas laid bare and human-readableAll the Protobuf classes laid bare and human-readable Upon deeper inspection, the Protobuf classes are right here, in Java, decorated with getters and setters. Since we are using Python, and we cannot get the true schema files, I will leave this approach for now. Top - --------------------------------------------------------------------- Hardcore Deep-Dive into Protobuf and Wire Format After gazing into a sea of decrypted network traffic again, then triggering errors and assertion fails on my iPhone with Protobuf fuzzing, and taking a peek at the error logs being phoned home, I've noticed that ads register for "slots" in a given video. They can register for pre-roll, mid-roll, end-roll, full-page, and ad pods (back-to-back ads). Blocking an ad URL causes an error along the lines of "some ad that doesn't exist booked a slot" and UI panic sets in. I'm going to Sun Tzu the Protobuf Wire Format and come back in a bit... I'm back. The Wire Format is surprisingly elegant, except for ZigZag encoding. Through trial and error, editing out chunks of Protobuf with a hex editor is just a no-go. While computationally expensive, decoding, editing, and re-encoding without the original schema leads to a modified encoding. This is likely because we cannot detect if ZigZag encoding is being used, or if a number is an int32, int64, sint32/64, varint, etc., plus the order of object fields is normally non-deterministic. Here is some Protobuf trivia on the matter: Protobuf serialization gotachasProtobuf serialization gotachas Protobuf serialization gotachas Top - --------------------------------------------------------------------- Exploit a Protobuf Flaw to Easily Remove All Ads by Changing One Byte Casually poring over the C++ source code, an interesting comment in the Protobuf code caught my eye: UnknownFieldSet is used to keep track of fields that were seen when parsing a protocol message but whose field numbers or types are unrecognized. This most frequently occurs when new fields are added to a message type and then messages containing those fields are read by old software that was compiled before the new types were added. (ref) Yes, what to do with unknown fields? What to do indeed. And, how easy would it be to say, change a 49399797 field key to, say, 49399796 thus making an entire substructure of advertisement and tracking information suddenly unavailable? Tantalizing. And, if we can calculate the field tags in bytes with bit-twiddling, then can we use a simple regex to AMF^1 the section of ads in O(n) time? As a motivating example, I'd like to find the field key 49399797 which is not as simple as searching for 2F1C7F5. Here is an implementation of a tag-scanning algorithm so you can see the bit-twiddling: [def DecodeVarint(buf] 1 def DecodeVarint(buffer, pos): 2 mask = (1 << 64) - 1 3 result = 0 4 shift = 0 5 while 1: 6 b = buffer[pos] 7 result |= ((b & 0x7f) << shift) 8 pos += 1 9 if not (b & 0x80): 10 result &= mask 11 result = int(result) 12 return (result, pos) 13 shift += 7 We know the wire type is 2 (length-delimited nested string/message), and one target field key is 49399797. When bit-twiddled, we get the target tag AA FF B8 BC 01 where the final 01 happens to mean 2 (the wire type) in hex. In binary, this is: 10101010 11111111 10111000 10111100 00000001 Let's lose the MSB from each byte as per the var-length wire format: .0101010 .1111111 .0111000 .0111100 .0000001 Then we shift and add only the first four bytes since the LSB is first: [42 + (127 * 2^7) + (] 1 42 + (127 * 2^7) + (56 * 2^14) + (188 * 2^21) = 2 42 + (127 * 128) + (56 * 16384) + (188 * 2097152) = 3 42 + 16256 + 917504 + 394264576 = 4 395198378 Finally, we shift out the number of wire type bits (3) to get back the field key: 395198378 >> 3 = 49399797 And that, folks, is a taste of how Wire Format works. Fantastic. Now, all we have to do is scan the Protobuf bytes for classic ad URL signatures like /pagead/ to bound our field search, then move backward from there until we find the target(s) field tags and thus field keys we would like to denature (e.g. 49399797 -> 49399796). [>> Request(POST yout] >> Request(POST youtubei.googleapis.com:443/youtubei/v1/browse?key= 1 ...) 2 << Response(200, application/x-protobuf, 1.87m) 3 4 Intercepting https://youtubei.googleapis.com/youtubei/v1/browse?key 5 =... (Protobuf) 6 Found key 49399797 at position 4465 Found key 50195462 at position 4477 Notice how the Protobuf response payload is 1.87 MiB? As I said, Google makes it computationally expensive to decode, alter, and re-encode without the C++ source proto files, but a quick linear scan takes no effort at all. Walking backward from the ad markerWalking backward from the ad markerWalking backward from the ad marker Just a quick note, there is more than one field tag, but not all of them represent ads. That is why we need to backtrack from the /pagead / markers. Multiple identical field tags may be presentMultiple identical field tags may be presentMultiple identical field tags may be present Top - --------------------------------------------------------------------- Smoke Test: Remove Ads from Protobuf in O(n)-Time It works! In one pass with no additional memory, I'm able to scan a huge 1.8 MiB chunk of jibberish-looking Protobuf data, and in the screenshot below only at the 30,593^th byte (of 1.8 MiB) is our target found, and then backtracking ~600 characters yields our target field key to denature. Not only is this amazing, but I don't even need to block *.googleadservices.com or URLs with /pagead/ in them; Those requests are never made in the first place, anymore. Successfully able to remove ads from the Protobuf response Successfully able to remove ads from the Protobuf response Successfully able to remove ads from the Protobuf response Top - --------------------------------------------------------------------- Analysis of this Successful Adblocking Technique Summary By taking advantage of a feature (flaw?) in Protobuf that allows it to be backward compatible with schema changes, along with the fact that Protobuf is very sensitive to byte changes due to its compact nature, we can change a single byte in a critical location and tell Protobuf that an entire section of deeply-nested data is from a future schema version and it should be ignored. Timing Analysis Google returns huge responses in Protobuf (e.g. 1.8 MiB) - including even the layout of the iOS app - so only C++/Swift is fast enough to understand it all before the connection times out. I've shown that Python is several orders of magnitude too slow in decoding these Protobuf payloads, so connections do time out waiting on Python. With web-based JSON, the whole payload needs to be parsed, edited, and re-serialized; With my Protobuf technique, it takes microseconds thanks to a single linear scan and then ultra-quick backtracking. This technique is suitable for real-time adblocking without blocklists. Knock-On Benefits All those *.googleadservices.com and /pagead/* URLs on Apple devices originate from the Protobuf payload. This means they all go away for free - we don't need to block them. In fact, the YouTube app is zippier because fewer connections are made to ad URLs in the first place. This means we can avoid keeping a blocklist of YouTube ad URLs and stay on the sidelines of the whack-a-mole fun. Ads do not register for video location "slots" on the Apple devices and the content just plays. Future-Proof This is a heuristic technique that looks for two strings: /pagead/ and some calculated field tag nearby, so this technique is designed to be future-proof. Walking backward from the ad markerWalking backward from the ad markerWalking backward from the ad marker to find the field key Even if Google changes the field tag (and breaks millions of apps and Apple TVs before they upgrade), it's an academic exercise to enhance the following script to discover the new field tag(s) automatically. Should Google be Worried? No, not at all. This is a highly-specialized technique to block Apple-device YouTube ads (or Instagram, Whatsapp, Facebook, etc. tracker blocking). The CPU requirements to decrypt and re-encrypt HTTPS traffic greatly exceed those available to Raspberry Pis. Even if some company takes my script and considers making and selling a NIC dongle, it would likely not be powerful enough. An Nvidia Shield could handle it, but if you already have Android devices, then just hack the binaries; My technique is for Apple device owners where we don't want to compromise the OS so that further reduces the audience of this technique. Top - --------------------------------------------------------------------- The MITMProxy YouTube Adblocking Script Here is the MITMProxy addon script that serves as a proof-of-concept to block YouTube ads on networked Apple devices. The script can be run as follows (note the prerequisites in the script and be sure to install them first). Name it youtube.py and run the following command: mitmdump --listen-port 8080 --listen-host 127.0.0.1 -s "youtube.py" Here is the script, including a fairness function to allow ads 5% of the time: [# -*- coding: utf-8 ] # -*- coding: utf-8 -*- # Copyright (c) 2021. Eric Draken (ericdraken.com) # Block YouTube ads on Apple devices by exploiting a Protobuf flaw # # FreeBSD Prerequisites: # pkg install protobuf # pkg install py38-pip # pip install jsonpath-ng # import hashlib import inspect import json import re import subprocess import sys import traceback from datetime import datetime from json import JSONDecodeError from typing import Final from google.protobuf.internal.encoder import TagBytes from google.protobuf.text_format import WIRETYPE_LENGTH_DELIMITED from jsonpath_ng import DatumInContext from jsonpath_ng import jsonpath, parse from jsonpath_ng.ext import parse from mitmproxy import ctx, http from mitmproxy.addons.next_layer import NextLayer from mitmproxy.flow import Error from mitmproxy.proxy import layer, layers TRUNCATE_LEN: Final[int] = 120 DEBUG_MODE: bool = False 1 class Logger: 2 """Helper to bypass the async logger loop to view logs in 3 real-time""" 4 5 def info(self, msg): 6 print(msg) if DEBUG_MODE else ctx.log.info(msg) 7 8 def warn(self, msg): 9 print(msg) if DEBUG_MODE else ctx.log.warn(msg) 10 11 def error(self, msg): 12 print(msg) if DEBUG_MODE else ctx.log.error(msg) 13 14 def alert(self, msg): 15 print(msg) if DEBUG_MODE else ctx.log.alert(msg) 16 17 18 logger = Logger() 19 20 21 def trunc(msg) -> str: 22 """Helper for viewing very long URLs""" 23 msg = str(msg) 24 if len(msg) > TRUNCATE_LEN: 25 return f"{msg[:TRUNCATE_LEN-3]}..." 26 else: 27 return msg 28 29 30 class KilledError(Error): 31 """Better logging messages than just 'Connection killed.'""" 32 33 def __init__(self, reason: str) -> None: 34 self._msg = Error.KILLED_MESSAGE 35 self.reason = reason 36 super().__init__(self._msg) 37 38 @property 39 def msg(self): 40 caller = inspect.stack()[1].function 41 # These are the only two methods that compare the msg 42 # with KILLED_MESSAGE to perform business logic 43 if "killable" in caller or "check_killed" in caller: 44 return self._msg 45 else: 46 return self.reason 47 48 # Needed to satisfy a flow setter 49 @msg.setter 50 def msg(self, msg): 51 self._msg = msg 52 53 54 class JSONPathReplacement: 55 """Helper class to organize JSON ad replacements""" 56 57 def __init__(self, tag: str, target_path: str, replacement: 58 any) -> None: 59 self.tag: str = tag 60 self.target_path = target_path 61 self.target: jsonpath.Root = parse(target_path) 62 self.replacement: any = replacement 63 64 def update(self, obj: object): 65 found = self.target.find(obj) 66 if found: 67 for index, item in enumerate(found): 68 self.target.update(item, self.replacement) 69 logger.warn(f"Replaced `{self.target_path} 70 [{index}]` with `{self.replacement}`") 71 72 if DEBUG_MODE: 73 found_again = self.target.find(obj) 74 replacement_json = json.dumps(self.replacement) 75 for index, item_ in enumerate(found_again): 76 item: DatumInContext = item_ 77 if json.dumps(item.value) != replacement_json 78 : 79 logger.error(f"Replacement of 80 `{self.target_path}` did not succeed. Found `{json.dumps 81 (item.value)}`") 82 83 else: 84 logger.info(f"-Skipping `{self.target_path}`") 85 86 87 class ProtobufDebugParser: 88 """Use the C++ protoc binary to parse raw Protobuf data. This 89 is for debugging.""" 90 91 cmd = ["protoc", "--decode_raw"] 92 url_re = re.compile(r"(https?://[^\s\\]+)", re.IGNORECASE) 93 94 def format_response(self, data: bytes) -> list: 95 protoc_proc = subprocess.Popen(self.cmd, shell=False, 96 stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess. 97 PIPE) 98 (stdout, stderr) = protoc_proc.communicate(data) 99 if stderr: 100 raise Exception(stderr) 101 102 return stdout.splitlines(keepends=False) 103 104 def parse_response(self, data: bytes) -> list: 105 protoc_proc = subprocess.Popen(self.cmd, shell=False, 106 stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess. 107 PIPE) 108 grep_process = subprocess.Popen(["grep", "https://"], 109 shell=False, stdin=protoc_proc.stdout, stdout=subprocess.PIPE) 110 111 (_, stderr) = protoc_proc.communicate(data) 112 (stdout, _) = grep_process.communicate() 113 114 if stderr: 115 raise Exception(stderr) 116 117 urls = stdout.splitlines(keepends=False) 118 matches = [] 119 for url in urls: 120 match = self.url_re.search(url.decode()) 121 if match: 122 matches.append(match.group(0)) 123 return matches 124 125 126 class YouTubeAdBlocker: 127 """Intercept certain YouTube domains and modify the JSON or 128 Protobuf to remove ads from YouTube""" 129 130 # Set this to the blackhole IP used by pfBlockerNG, or even a 131 partial blackhole IP 132 blackhole_ip_prefix: Final[str] = "10.10.10." 133 134 # Intercept only these wildcard domains and no others 135 intercept_hosts = r"\.youtube\.com|google\.(com|ca)| 136 googleapis\.com|googleadservices\.com|googlevideo\.com" 137 intercept_hosts_re = re.compile(intercept_hosts, re. 138 IGNORECASE) 139 140 ad_url_search_string = b"/pagead/" 141 ad_url_search_limit = 80_000 142 target_field_tag = 50195462 # This is just one of several 143 field tags 144 145 protobuf_parser = ProtobufDebugParser() 146 147 # Block requests from wildcard domains with any of the follow 148 URL strings 149 blocked_partials: dict = { 150 "youtube.com": [ 151 "pagead/", 152 "log_event?", 153 "stats/ads", 154 "stats/qoe?", 155 "ptracking?", 156 "generate_204", 157 "error_204", 158 "adformat=", 159 "activeview?", 160 "_ad_", 161 "ai?", 162 "sw.js", # Just say no to service workers 163 ], 164 "google.com": ["pagead/"], 165 "google.ca": ["pagead/"], 166 "googleapis.com": ["pagead/"], 167 } 168 169 # Strip sections of JSON that contain ad information (for web 170 YouTube) 171 json_replacements = [ 172 JSONPathReplacement("yt_ad", 173 "$.responseContext.serviceTrackingParams[*].params[?(@.key == 174 'yt_ad')].value", "0"), 175 JSONPathReplacement("adPlacements", "$..adPlacements", [] 176 ), 177 JSONPathReplacement("adPlacementRenderer", 178 "$..adPlacementRenderer", {}), 179 JSONPathReplacement("adPlacementConfig", 180 "$..adPlacementConfig", {}), 181 JSONPathReplacement("adVideoId", "$..adVideoId", ""), 182 JSONPathReplacement("playerAdParams", "$..playerAdParams" 183 , {}), 184 JSONPathReplacement("showCompanion", "$..showCompanion", 185 False), 186 JSONPathReplacement("showInstream", "$..showInstream", 187 False), 188 JSONPathReplacement("useGut", "$..useGut", False), 189 JSONPathReplacement("gutParams", "$..gutParams", {}), 190 ] 191 192 @staticmethod 193 def in_allowed_ads_window(): 194 """Allow ads 5% of the time to support content creators 195 which follows the CRTC rules for Student Radio. 196 REF: https://crtc.gc.ca/eng/television/publicit/ 197 publicit.htm""" 198 now = datetime.now() 199 return 0 <= now.minute <= 2 200 201 """The following methods are hooks that are called during the 202 normal flow of MITMProxy""" 203 204 # noinspection PyMethodMayBeStatic 205 def load(self, _): 206 # Do not allow piped requests we could miss 207 ctx.options.http2 = False 208 try: 209 ctx.options.update(http2=False, anticomp=True, mode= 210 "transparent", termlog_verbosity="debug") 211 except KeyError: 212 ctx.options.update(http2=False, anticomp=True, mode= 213 "transparent") 214 logger.warn(f"{self.__class__.__name__} loaded") 215 216 def running(self): 217 """Intercept requests only for YouTube domains""" 218 if ctx.options.allow_hosts != [self.intercept_hosts]: 219 ctx.options.allow_hosts = [self.intercept_hosts] 220 221 ctx.options.ignore_hosts = [] 222 next_layer_addon: NextLayer = ctx.master.addons.get( 223 "NextLayer".lower()) 224 next_layer_addon.configure("allow_hosts") 225 logger.warn(f"Updated interceptable YouTube hosts") 226 227 def next_layer(self, nextlayer: layer.NextLayer): 228 """Allow blocked domains that resolve the blackhole IP to 229 pass through""" 230 if nextlayer.context.server.address and nextlayer.context 231 .server.address[0].startswith(self.blackhole_ip_prefix): 232 nextlayer.layer = layers.TCPLayer(nextlayer.context, 233 ignore=True) 234 235 # noinspection PyMethodMayBeStatic 236 def error(self, flow: http.HTTPFlow): 237 """The TLS failed due to an ad-blocker info page with no 238 verified TLS""" 239 if flow.error and ("OpenSSL Error" in flow.error.msg and 240 "alert internal error" in flow.error.msg): 241 flow.kill() 242 243 def request(self, flow: http.HTTPFlow) -> None: 244 """Block ad URLs that pfBlockerNG and Pi-hole cannot 245 detect""" 246 # Skip inspecting certain requests 247 if flow.response or flow.error or (flow.reply and flow. 248 reply.state == "taken"): 249 return 250 251 # Occasionally skip blocking ads to support content 252 creators 253 if self.in_allowed_ads_window(): 254 return 255 256 test_url: str = flow.request.url 257 test_host: str = flow.request.pretty_host.lower() 258 for host, partials in self.blocked_partials.items(): 259 if test_host.endswith(host): 260 for partial in partials: 261 if partial in test_url.lower(): 262 msg = f" [{host} -> {partial}] blocking 263 {trunc(flow.request.pretty_url)}..." 264 logger.info(msg) 265 266 # Should be a connection refused error 267 flow.kill() 268 flow.error = KilledError(msg) 269 return 270 271 def response(self, flow: http.HTTPFlow): 272 """This is the main workhorse. Intercept JSON and 273 Protobuf responses and modify them 274 to remove or denature ad information""" 275 276 # Skip inspecting certain responses 277 if not flow.response or flow.error or not flow.response. 278 headers: 279 return 280 281 logger.warn(f">> {flow.request}\n<< {flow.response}\n\n") 282 283 # Occasionally skip blocking ads to support content 284 creators 285 if self.in_allowed_ads_window(): 286 return 287 288 test_path: str = flow.request.url.lower() 289 test_content_type: str = str(flow.response.headers.get( 290 "content-type")).lower() 291 292 # Examine the Protobuf payload 293 if "protobuf" in test_content_type: 294 logger.warn(f"Intercepting {trunc(test_path)} 295 Protobuf") 296 297 try: 298 # Capture rich Protobuf information to disk for 299 offline analysis 300 if DEBUG_MODE: 301 # TODO: This copying can be avoided, but this 302 is debug mode, so we allow it 303 body = bytearray(flow.response.get_content( 304 strict=False) or b"") 305 lines = self.protobuf_parser.format_response( 306 body) 307 filename = "".join(x for x in test_path[:100] 308 if (x.isalnum() or x in "._- ")) 309 filename = f"protobuf-{filename}-{hashlib.md5 310 (test_path.encode()).hexdigest()}" 311 with open(f"{filename}.formatted", mode="w", 312 buffering=True) as f: 313 for line in lines: 314 f.write(f"{line}\n") 315 316 with open(f"{filename}.raw", mode="wb") as f: 317 f.write(flow.response.raw_content or b"") 318 319 with open(f"{filename}.decoded", mode="wb") 320 as f: 321 f.write(body or b"") 322 323 # TODO: Use a memory view or some more efficient 324 search structure 325 body: bytearray = bytearray(flow.response. 326 get_content(strict=False) or b"") 327 328 # Find a telltale ad URL, but limit the search 329 distance = body[: self.ad_url_search_limit].find( 330 self.ad_url_search_string) 331 if distance < 0: 332 return 333 334 logger.warn(f"Found {self.ad_url_search_string} 335 at position {distance}") 336 337 # Search forward for an ad URL signature, then 338 backtrack to find the field tag 339 tag_bytes = TagBytes(self.target_field_tag, 340 WIRETYPE_LENGTH_DELIMITED) 341 new_bytes = TagBytes(self.target_field_tag - 1, 342 WIRETYPE_LENGTH_DELIMITED) 343 target_pos = body[: distance - 1][::-1].find( 344 tag_bytes[::-1]) 345 if target_pos > 0: 346 target_pos = distance - 1 - target_pos - len( 347 tag_bytes) 348 logger.warn(f"Found {self.target_field_tag} 349 at position {target_pos}") 350 assert body[target_pos] == tag_bytes[0] 351 assert body[target_pos + 1] == tag_bytes[1] 352 assert body[target_pos + 2] == tag_bytes[2] 353 for ind, b in enumerate(new_bytes): 354 body[target_pos + ind] = b 355 356 """NOTE: There are other field keys in different 357 sections, 358 and there may be multiple ad sections to 359 denature. What preceded 360 is a PoC of the technique that already blocks 90% 361 of ads.""" 362 363 # Example Protobuf path: 364 b"4 {" 365 b" 49399797 {" # Damage this key 366 b" 1 {" 367 b" ... /pagead/" 368 # Example Protobuf path b" 1 {" b" 50195462 {" # Damage this key b" 1 {" b" 153515154 {" b" ... /pagead/" # Put the contents back in the response body flow.response.set_content(bytes(body)) except Exception as e: _, _, exc_traceback = sys.exc_info() traceback_ = traceback.format_tb(exc_traceback) logger.alert(f"{e!r}, {traceback_}") elif "json" in test_content_type: logger.warn(f"Intercepting {trunc(test_path)} JSON") # Examine the JSON payload try: obj = flow.response.json() for replacement in self.json_replacements: replacement.update(obj) flow.response.set_content(json.dumps(obj, ensure_ascii=False).encode()) except (TypeError, JSONDecodeError): pass # Do not stop the show except Exception as e: _, _, exc_traceback = sys.exc_info() traceback_ = traceback.format_tb(exc_traceback) logger.alert(f"{e!r}, {traceback_}") # Register the addon addons = [YouTubeAdBlocker()] This script happens to work in Python for a TLS-decrypting man-in-the-middle proxy written in Python. As a working proof-of-concept, it's pretty rad. Of course, it can be rewritten in Rust or Go or anything but single-threaded Python, but as an intellectual exercise to defeat ads that are served from the same domain as content, it's elegant. Top - --------------------------------------------------------------------- YouTube Premium It's unknown if CAD [DEL:$9.99/mo:DEL] $11.99/mo ($13.43/mo with tax) is even reasonable: Do I personally incur CAD $11.99 of cost to advertisers each month? How much does YouTube advertising cost?How much does YouTube advertising cost?Source Since ads are auctioned, the CPV (cost-per-view) varies. Also, many ad campaigns have a capped daily budget, so theoretically there should be fewer ads in the evenings as budgets run out during the day. Experiment in Ad Viewing I watched YouTube on and off for a day on a clean notebook computer with private browsing. My history showed that I only "watched" 10 videos: * I fast-forwarded through a few of them to get past the "like and subscribe" runtime padding. * I jumped to the end of one just to get to the "top three" from a "top twenty" list. * Two were low quality so I left early. * The rest were music videos. In all, for watching parts of 10 videos, I was exposed to 8 ads, and only two were skippable (which I skipped). $0.15 as a Ballpark CPV Let's use USD $0.15 as a CPV. In one day, let's say, I incurred 8 x $0.15, or $1.20 to advertisers. Extrapolated to one month, that is roughly USD $36/mo. Do I really cost advertisers USD $36/mo for very casual YouTube viewing? That sounds terrible for advertisers. CPV from US Advertising Spend Divided by Total Views From Statistica, in 2019, US YouTube advertisers spent $15.1 billion dollars. Also in 2019, US residents had 916 billion views (ref). That works out to an average of $15.1B / 916B, or USD $0.0165 per view. Then for me, that is only USD 13 cents. Extrapolated to one month, I theoreticaly cost advertisers only USD $3.96/mo. Is YouTube Premium Worth It? YouTube Premium subscription fee as of Jan, 2022YouTube Premium subscription fee as of Jan, 2022 When I allowed ads for my experiment, I hit the hardware mute button. I also looked away because I have several computers with a lot going on. Ad spend is wasted on me, but I still want to support content creators. For me, CAD $13.48/mo is more than I incur on actual ads and more than I pay for a Netflix subscription. The only way to justify the cost is to have YouTube playing constantly in the background on a TV. However, I truly enjoy a handful of creators, so I may start watching them in the background on non-stop play. Let's give the three-month YouTube Premium trial a chance, and I will still be monitoring what they track about me. YouTube Premium network trafficYouTube Premium network trafficYouTube Premium network traffic Top - --------------------------------------------------------------------- DMCA, Sony, Viacom Recently I learned that due to abuses of the DMCA Act of 1998, YouTube content creators who make reaction videos and "easter egg" videos may have their videos claimed by big companies like Sony and Viacom. That means that from when a claim is made, all ad revenue goes to those big companies, and not even to the creators. That means in all likelihood I unknowingly may not even be supporting my favourite YouTube creators. Did you know? Many fair-use and video-game-commentary videos may have automated copyright claims against them, meaning that ad revenue goes to big companies with deep legal pockets and your favourite creators may get nothing, so more and more creators leave YouTube for Twitch. Top - --------------------------------------------------------------------- Summary of Accomplishments I rarely give up, so this is an example of going into an extreme problem-solving mode to solve a fun problem loosely using cryptography and reverse engineering. In the end, a single byte turned it all around, so it was all worth it to come to an elegant and satisfying solution. Success: We were able to set up a hardware router from scratch, segment LANs into trusted and untrusted zones, set up traditional DNS adblocking, add a transparent MITM proxy, and ultimately block YouTube ads on networked Apple devices. Note: This was a hard problem - now solved - so I am paying for YouTube Premium to give the CPU a rest. Top - --------------------------------------------------------------------- Notes: 1. Adios, My Friend - Posted in: Hardware, Linux, Privacy, Security, Tips & TricksTags: Adblocking, MITMProxy, pfBlockerNG, pfSense, Protobuf, YouTube --------------------------------------------------------------------- Post navigation Previous Next Eric Draken Recent Posts * Advanced 3D Printing Tips & Tricks * Compile Marlin (3D Printer Firmware) with Docker * Unblock Google Analytics: Prevent AdBlockers from Blocking Site Analytics * Safecracking: Open a Safe with Drained Internal Batteries * How to tell if "N52" Neodymium Magnets are Real Categories * Tips & Tricks (46) * Problem Stories (31) * Hardware (15) * Linux (15) * Quant (11) * Programming (9) * Testing (9) * APIs (7) * Server (7) * Website Dev (7) * Algorithms (7) * Databases (7) * Cluster Computing (6) * Software (3) * Security (3) * Privacy (3) * Just Thinking (2) * Hosting (2) * 3D Printing (2) * Windows (1)