https://blog.skyvern.com/how-we-accidentally-burned-through-200gb-of-proxy-bandwidth-in-6-hours/

Skyvern Blog

  * Home
  * Github
  * Discord
  * Try it out -

Sign in Subscribe

How we accidentally burned through 200GB of proxy bandwidth in 6
hours

  * Suchintan Singh

Suchintan Singh

Sep 18, 2024 * 2 min read
How we accidentally burned through 200GB of proxy bandwidth in 6
hours

Context

Skyvern is an AI agent that helps companies automate workflows in the
browser. We run leverage proxy networks and run headful browser
instances in the cloud to facilitate most of our automations.

 200GB of proxy bandwidth was approximately $500 burned over the
course of 6 hours

One fatal morning

I woke up one morning to an alarm on my phone - Skyvern's failure
rate was through the roof. Classic startup moment.

Looking through our alerts, I noticed that our proxy bandwidth alert
had fired off. That's weird - we just renewed our plan a few days
ago, and should have enough quota to last a month.

I took a look at our metrics and saw something that made my heart
sink - we just burned through 200GB of proxy bandwidth in the last 6
hours. HOW DID THAT HAPPEN?

[image-2]Proxy bandwidth going through the roof

My first thought was - did we get compromised? We give all new users
$5 of credits to play around with.. and have had users try to abuse
it before. Did someone figure out how to create millions of accounts
and burn our bandwidth?

Digging deeper

Taking a quick look at our usage stats.. no.. nothing out of the
ordinary.

OK. Let's dig a little bit deeper - where is this bandwidth going?

[image-3]

What's going on? Why is there a random call to
optimizationguide-pa.googleapis.com11:05 occupying 70MB of
bandwidth.. over and over and over again?

Doing a quick google search revealed this thread:

https://www.reddit.com/r/chrome/comments/truz5d/
the_chrome_downloads_for_no_reason_whatsoever/

and

https://support.google.com/chrome/thread/157884177/
chrome-appearing-to-download-without-me-downloading-anything?hl=en

OK this is starting to make sense. Google's downloading a machine
learning model in the background to cache, and we seem to be
downloading it over and over again

We don't currently persist browser state between sessions.. did we
end up in a state where google wants to cache the model, and we keep
resetting to the uncached version?

Potential Solution

We have a few options to fix this:

#1 - Run chrome locally and save the locally generated user_data_dir
which includes the model, effectively caching it.

Problem: if the model became stale it would re-trigger the download
again in the future

#2 - Introduce a rule into our sidecar proxy to block this specific
google URL - prevent Google from downloading the model in the first
place

- DOMAIN-SUFFIX,optimizationguide-pa.googleapis.com,REJECT

Going back to sleep

We decided to do both to solve this problem. We updated our
user_data_dir to get a quick fix out, and also updated our sidecar
proxy to block that specific URL from triggering downloads in the
future

Sign up for more like this.

 
Enter your email
Subscribe
The dark path to discovering our ICPs
 

The dark path to discovering our ICPs

How to find ICPs to go after? This is a question I ask myself all the
time. The answer is obvious, right? Who would find the most value out
of Skyvern? When we first launched our product, we had no clue what
kinds of things companies would want to automate
Sep 11, 2024 2 min read
 

Changelog - September 4 - New pricing page, Faster live streams, and
more!

New Pricing page Skyvern's landing page just got a minor update - we
now have a dedicated pricing page ! We have 2 main problems we think
we can solve with a pricing page: #1: Our pricing wasn't clear at
all! The only way to learn about our
Sep 4, 2024 3 min read
How we cut token count by 11% and boosted success rate by 3.9% by
using HTML instead of JSON in our LLM calls
 

How we cut token count by 11% and boosted success rate by 3.9% by
using HTML instead of JSON in our LLM calls

TL;DR We started using HTML instead of JSON to represent possible
actions that Skyvern could take and reduced our cost by 11.8% and
increased our success rate by 3.9% What's Skyvern? Skyvern is an open
source AI agent that helps companies automate browser-based workflows
with AI.
Aug 28, 2024 3 min read
Skyvern Blog (c) 2024

  * Try it out

Powered by Ghost