[HN Gopher] EUR54k spike in 13h from unrestricted Firebase brows...
___________________________________________________________________
EUR54k spike in 13h from unrestricted Firebase browser key
accessing Gemini APIs
Author : zanbezi
Score : 372 points
Date : 2026-04-16 12:13 UTC (10 hours ago)
(HTM) web link (discuss.ai.google.dev)
(TXT) w3m dump (discuss.ai.google.dev)
| patcon wrote:
| That's fucking bonkers that nothing in the system could see this
| as unusual and worthy of throttling. The embarrassment of this --
| that a company LITERALLY SELLING machine learning services and
| expertise -- cannot spot such a thing... This should have led
| them to deal with this internally and refund it. Just... Wow
| Google.
| lukewarm707 wrote:
| there is no way to cap your billing on gcp.
|
| you can get notifications but that's it.
|
| i don't want to get throttled below my quota but some type of
| spend limit would be good.
| rvnx wrote:
| and the notifications can be delayed because the spending
| system is not updated in real-time, so even if you have a
| Cloud Task triggering on spending to disable the project it
| may be too slow and several thousands may already be spent.
| bombcar wrote:
| Is there a cloud provider that _does_ have hard unbreakable
| billing caps? Everything I 've seen has always been
| notifications or soft caps.
|
| Not talking about fixed-access things like a Hetzer box.
| pwdisswordfishs wrote:
| Bunny.net purports to have a pay-as-you-go prepaid credit
| system that sounds like it works the way people want, and
| with their description of the way it works probably being
| sufficient to be legally enforceable if it turns out that
| it actually works differently and you were to end up with a
| surprise bill from them. And evidently it really does work
| that way; see this post from a couple weeks ago:
| <https://news.ycombinator.com/item?id=47676416>
|
| The only other provider known to work that way is
| NearlyFreeSpeech.NET, which serves a completely different
| market segment (so much so that it might as well not even
| be considered the same kind of product/service).
| trick-or-treat wrote:
| This is GCP's revenue model, lol. Let's provide a (semi)
| generous free tier and trick people into accidentally going
| over it.
| voidUpdate wrote:
| The company selling machine learning services would probably
| love a EUR54k bonus
| owebmaster wrote:
| It's so funny seeing people thinking this is not by design
| lukewarm707 wrote:
| i have seen this so many times...
|
| i'm thinking it's time we replaced api keys.
|
| some type of real time crypto payment maybe?
| freedomben wrote:
| Oh please no. And the "alternatives" to API keys aren't going
| to help much either, they'll just add friction to getting
| started (as reference: see the pain involved in writing a
| script that hits gmail or calendar API)
| trick-or-treat wrote:
| Prepaid only is a fantastic idea, especially for dumb-ass
| startups. Limiting your liability to $100 or so sound like a
| big-ass W.
| freedomben wrote:
| Prepaid only is a fantastic idea, until your site goes
| (desirably) viral and then gets shut off right as traffic is
| picking up, or you grow steadily and forget to increase your
| deposit amount and suddenly production is down. Billing
| alerts are a much better solution IMHO.
| dummydummy1234 wrote:
| Prepaid/paid limits with shutoff is appropriate for this
| though.
|
| If you have per key limits, this is not possible, and even
| in a wild situation you should b able to expect that your
| firebase key will not use 50k.
| microtonal wrote:
| You can also have both, a cap and one or more billing alert
| levels below it. Some providers do this (e.g. IIRC
| Backblaze B2).
| freedomben wrote:
| Yes in reality, and ideally, you can have both, but GP
| specifically said "Prepaid only" implying you can't have
| both (which is what I replied to)
| eru wrote:
| Well, they should also have pre-paid only. Offer a few
| different options.
| trick-or-treat wrote:
| No you big dummy, that is especially when you want to limit
| your liability, lol.
|
| Because these days it will be all worthless bot traffic.
| lisdexan wrote:
| Let me choose. This common point seems more like a
| rationalization for the default behavior of hyperscalers.
| AWS isn't avoiding prepaid due to concern about my site's
| virality, just that prepaid = less money.
| 100ms wrote:
| Implementing this in any meaningful manner quickly begins to
| look like every read becoming a globally synchronised write.
| Of course it doesn't have to be perfect, but even
| approximating perfection doesn't look much different. Also,
| can you imagine the kind of downtimes and complaints that
| would inevitably originate from a fully synchronous billing
| architecture?
| eru wrote:
| > Of course it doesn't have to be perfect, but even
| approximating perfection doesn't look much different.
|
| It's pretty easy to get right, if the provider allows you
| to go (slightly) negative before cutting you off.
|
| > Also, can you imagine the kind of downtimes and
| complaints that would inevitably originate from a fully
| synchronous billing architecture?
|
| Doesn't need to fully synchronous.
| eru wrote:
| Yes, pre-paid would be fine and it's a well-understood
| pattern.
|
| No need to retire API keys.
| owebmaster wrote:
| Nobody is thinking about the stock owners, I see
| christkv wrote:
| Open ai has this
| mdrzn wrote:
| Related: https://news.ycombinator.com/item?id=47156925
| drtz wrote:
| > Are there recommended safeguards beyond ... moving calls
| server-side?
|
| This implies the API calls originated in the client, suggesting
| the client may have had they API key.
| embedding-shape wrote:
| Yeah, the amount of people creating, running and maintaining
| websites yet don't understand how websites actually work in
| practice is very high and seems we haven't even come close to
| the ceiling yet.
| dpkirchner wrote:
| That's standard for Firebase apps. It's also recommended by
| Google (they describe the keys as "public by design").
| Retr0id wrote:
| Feels like a confusing thing to name "key" if it's presumably
| more of an identifier.
| pwdisswordfishs wrote:
| It's "implied" throughout the whole post (or more like assumed
| that the reader understands this, because it's the basic
| premise of the problem). It's why they link to a post that
| explains the basic concept after a remark that "This describes
| our issue in more detail".
|
| > _tl;dr Google spent over a decade telling developers that
| Google API keys (like those used in Maps, Firebase, etc.) are
| not secrets. But that 's no longer true: Gemini accepts the
| same keys to access your private data. We scanned millions of
| websites and found nearly 3,000 Google API keys, originally
| deployed for public services like Google Maps, that now also
| authenticate to Gemini even though they were never intended for
| it. With a valid key, an attacker can access uploaded files,
| cached data, and charge LLM-usage to your account. Even Google
| themselves had old public API keys, which they thought were
| non-sensitive, that we could use to access Google's internal
| Gemini._
|
| From Google themselves, in the Firebase docs:
|
| > API keys for Firebase services are not secret. Firebase uses
| API keys only to identify your app's Firebase project to
| Firebase services, and not to control access to database or
| Cloud Storage data, which is done using Firebase Security
| Rules. For this reason, you do _not_ need to treat API keys for
| Firebase services as secrets, and you can safely embed them in
| client code.
|
| <https://firebase.google.com/support/guides/security-
| checklis...>
|
| ... or at least that's what it used to say, until they quietly
| updated the docs to say this:
|
| > API keys for Firebase services are not secret. API keys for
| Firebase services only _identify_ your Firebase project and app
| to those services. _Authorization_ is handled through Google
| Cloud IAM permissions, Firebase Security Rules, and Firebase
| App Check.
|
| > All Firebase-provisioned API keys are _automatically_
| restricted to Firebase-related APIs. If your app 's setup
| follows the guidelines in this page, then _API keys restricted
| to Firebase services_ do _not_ need to be treated as secrets,
| and it 's safe to include them in your code or configuration
| files.
|
| Followed later by (in different section):
|
| > Use your Firebase-provisioned API keys _only_ for Firebase-
| related APIs. If your app uses any other APIs (for example, the
| Places API for Maps or the Gemini Developer API), use a
| separate API key and restrict it to the applicable API.
| embedding-shape wrote:
| Considering the amount of repositories on public GitHub with
| hard-coded Gemini API tokens inside the shared source code
| (https://github.com/search?q=gemini+%22AIza%22&type=code), this
| hardly comes as a surprise. Google also has historically treated
| API keys as non-secrets, except with the introduction of the keys
| for LLM inference, then users are supposed to treat those
| secretly, but I'm not sure everyone got that memo yet.
|
| Considering that the author didn't share what website this is
| about, I'd wager they either leaked it accidentally themselves
| via their frontend, or they've shared their source code with
| credentials together with it.
| ckbkr10 wrote:
| theres not a single real gemini api key in the results
| embedding-shape wrote:
| Setup a watcher and you'll come across live ones eventually
| :)
| dminik wrote:
| Try this one. Should remove most readme keys:
|
| Edit: self censor based on a request
| duskdozer wrote:
| Oh, wow.
| ratsimihah wrote:
| this is such a wall of shame haha
| sillysaurusx wrote:
| I know you're well within your rights to post this, but
| would you consider replacing your comment with something
| like "It's easy to find working keys on github if you
| search the appropriate terms"?
|
| Think of it this way: although you're not to blame, HN
| drives a lot of traffic to your preconfigured github
| search. There are also bad actors who browse HN; I had a
| Firebase charge of $1k from someone who set up an automated
| script to hammer my endpoint as hard as possible, just to
| drive the price up. Point being, HN readers are motivated
| to exploit things like what you posted.
|
| It's true that the github search is a "wall of shame", and
| perhaps the users deserve to learn the hard way why it's a
| good idea to secure API keys. But there's also no benefit
| in doing that. The world before and after your comment will
| be exactly the same, except some random Gemini users are
| harmed. (It's very unlikely that Google or Github would see
| your comment and go "Oh, it's time we do something about
| this right now".)
|
| EDIT: I went through the search results and confirmed that
| the first several dozen keys don't work. They report as
| error code 403 "Your API key was reported as leaked. Please
| use another API key." or "Permission denied: Consumer
| 'api_key:xxx' has been suspended." So at least HN readers
| will need to work hard(er) to find a valid key.
|
| I wonder how you report a gemini API key as leaked...
| Searching "report gemini api key leaked" on Google only
| brings up similar horror stories (a $55k bill, waived https
| ://www.reddit.com/r/googlecloud/comments/1noctxi/studen...)
| and (a $13k bill from 3d ago https://www.reddit.com/r/googl
| ecloud/comments/1sjzat3/api_ke...)
| dminik wrote:
| I'm not opposed to even removing the comment outright.
|
| That being said, GitHub does not even offer a time sorted
| search. Meaning that most of the results are going to be
| quite old and useless.
|
| Second, API keys being shared on GitHub is quite an old
| problem. People setup automated scans for this sort of
| stuff. Me removing my comment isn't going to help anyone
| who already posted their API key online.
| sillysaurusx wrote:
| I tried several dozen keys and they're all invalid, so
| I'm inclined to agree with you for this particular case.
| Thank you anyway for considering.
| weird-eye-issue wrote:
| They have an automated process when keys get leaked on GH
| to get revoked automatically.
| imdsm wrote:
| https://github.com/JustForSO/Sentra-Auto-
| Browser/blob/c048d3...
| singpolyma3 wrote:
| Um. What? In what world are API keys not secrets?
| embedding-shape wrote:
| In the frontend world where you have client-side API keys
| talking directly to 3rd party services from the client. Think
| things like Google Maps and similar.
| londons_explore wrote:
| Which is a stupid idea for something where there is billing
| involved... Anyone on the internet can take that key and
| scrape the Google maps API (faking the referer header) and
| cost you $$$$$.
|
| Google should have simply done with by origin URL if they
| wanted stuff to be open like that.
| someothherguyy wrote:
| Once upon a time Google maps loads were nearly free, and
| there was no way to restrict that key.
| darrenf wrote:
| In Firebase world API keys are for identification, not
| authorisation.
|
| https://firebase.google.com/docs/projects/api-keys
|
| _Public by design: API keys for Firebase services only
| identify your Firebase project and app to those services.
| Authorization is handled through Google Cloud IAM
| permissions, Firebase Security Rules, and Firebase App
| Check._
| boredpudding wrote:
| Google API keys have been used for ages on the frontend. For
| example on Google Maps embeds. Those are not possible without
| exposing a key to the frontend. They weren't secret, until
| Gemini arrived.
|
| https://trufflesecurity.com/blog/google-api-keys-werent-
| secr...
|
| https://medium.com/@ahhyesic/your-google-maps-api-key-now-
| ha...
|
| https://www.malwarebytes.com/blog/news/2026/02/public-
| google...
| someothherguyy wrote:
| If one ignores 70% of the documentation, it makes for a
| demonizing blog post about it, sure.
|
| " API keys for Firebase services are not secret
|
| API keys for Firebase services only identify your Firebase
| project and app to those services. Authorization is handled
| through Google Cloud IAM permissions, Firebase Security
| Rules, and Firebase App Check.
|
| All Firebase-provisioned API keys are automatically
| restricted to Firebase-related APIs. If your app's setup
| follows the guidelines in this page, then API keys
| restricted to Firebase services do not need to be treated
| as secrets, and it's safe to include them in your code or
| configuration files. Set up API key restrictions
|
| If you use API keys for other Google services, make sure
| that you apply API key restrictions to scope your API keys
| to your app clients and the APIs you use.
|
| Use your Firebase-provisioned API keys only for Firebase-
| related APIs. If your app uses any other APIs (for example,
| the Places API for Maps or the Gemini Developer API), use a
| separate API key and restrict it to the applicable API."
|
| https://firebase.google.com/support/guides/security-
| checklis...
| three14 wrote:
| The only reasonable design is to have two kinds of API
| keys that cannot be used interchangeably: public API
| keys, that cannot be configured to use private APIs, and
| private API keys, that cannot be configured to use public
| APIs. There's no one who _must_ use a single API key for
| both purposes, and almost all cases in which someone does
| configure an API key like that will be a mistake. It
| would be even better if the API keys started with a
| different prefix or had some other easy way to
| distinguish between the two types so that I can stop
| getting warnings about my Firebase keys being "public".
| SAI_Peregrinus wrote:
| It'd be much better to call them something like "API
| usernames" or "API Client IDs". Though I also dislike the
| naming of "public keys" in asymmetric cryptography, for
| the same reasons, and I'm _definitely_ not winning that
| fight!
| fg137 wrote:
| Google's world. They explicitly tell you that API keys are
| not secrets.
|
| https://trufflesecurity.com/blog/google-api-keys-werent-
| secr...
| lxgr wrote:
| API keys _for Firebase_. While Google really messed up
| here, I doubt they ever published anything claiming that
| _no Google API keys at all_ are secrets.
| pwdisswordfishs wrote:
| Google Maps is not Firebase.
|
| And "Firebase AI Logic" sure sounds like something easy
| to confuse with a Firebase service...
| lxgr wrote:
| The same principle applies, though.
|
| I'm absolutely not defending Google here, to be clear:
| Retroactively expanding the scope of an API "key"
| explicitly designated as "public/non-sensitive" is very
| bad.
|
| But the concept itself does make some sense, and I'm just
| noting that there's precedent both across Google and
| other companies.
| pwdisswordfishs wrote:
| > The same principle applies, though.
|
| How?
|
| "Firebase AI Logic"
|
| Is this a Firebase service or not?
| lxgr wrote:
| Public API keys are a thing. Arguably they are poorly named
| (it's really more of a client identifier), and modeling them
| as primarily a key instead of primarily as a non-secret
| identifier can go very wrong, as evidenced here.
| il-b wrote:
| Yeah, just like "public key" in the cryptography sense
| lxgr wrote:
| They're not really cryptographic keys in that sense
| usually; I'd say "bearer token" is more accurate.
| zozbot234 wrote:
| > Google also has historically treated API keys as non-secrets,
| except with the introduction of the keys for LLM inference,
| then users are supposed to treat those secretly
|
| This was reported a _long_ time ago, and was supposed to be
| fixed by Google via making sure that these legacy public keys
| would not be usable for Gemini or AI.
| https://news.ycombinator.com/item?id=47156925
| https://ai.google.dev/gemini-api/docs/troubleshooting#google...
| "We are defaulting to blocking API keys that are leaked and
| used with the Gemini API, helping prevent abuse of cost and
| your application data." Why are we hearing about this again?
| PunchyHamster wrote:
| the topic is cost overruns. they still allow for cost
| overruns. What's so hard to comprehend ?
| addandsubtract wrote:
| FWIW, I just create a new Gemini API key today, and it had a
| different format than my old ones (created 10 days ago). So
| maybe they changed something?
| zozbot234 wrote:
| A reply on OP's post states: "... We now generate Auth keys
| by default for new users (more secure key which didn't
| exist when the Gemini API was originally created a few
| years ago) and will have more to share there soon. ..." So
| there _is_ something new in that exact area but the details
| are forthcoming.
| spiznnx wrote:
| I think brand new stuff is probably safe, but old keys that
| currently being used for AI and non-AI stuff - if Google
| disables them for AI and it turns out it was actually not
| being exposed publicly, could disrupt a user's production
| service relying on AI.
|
| They messed up by allowing old keys to be used for both
| private and public APIs in the first place, but now it's
| difficult for them to undo that for existing keys.
| mdrzn wrote:
| ...JCip3SJw => Your API key was reported as leaked. Please use
| another API key.
|
| ...afnt0t-E => Your API key was reported as leaked. Please use
| another API key.
|
| ...-UYzYTYU => Your API key was reported as leaked. Please use
| another API key.
|
| I think they all get immediately reported as leaked and
| invalidated.
| luanmuniz wrote:
| Unfortunately, yet just another story like this. One of these
| unexpected usage charges in the thousands appears every month,
| and with the same automatic denied too. This is one of the
| reasons I just stopped using these kinds of pay-per-usage cloud
| services long ago. At best, I still use services that have hard-
| bounded usage limits, like EC2 from AWS, where one instance can
| never go beyond 24h/day usage and is always capped, with
| shutdowns when exceeded, and limited credit cards, too.
|
| It's super frustrating that this is the only option to
| realistically deal with this issue, since all stories end up the
| same way: The cloud company just saying "f* you, we don't care,
| pay up." and legal fees are always expensive :(
| embedding-shape wrote:
| > At best, I still use services that have hard-bounded usage
| limits, like EC2 from AWS, where one instance can never go
| beyond 24h/day usage and is always capped, with shutdowns when
| exceeded, and limited credit cards, too.
|
| Is this possible on AWS today? I'm the same way, if I cannot
| set a hard-limit for the billing so I can know for a fact how
| much it'll maximum cost in a month, I'm not interested in using
| that service for anything. Which is one of the top reasons I've
| stayed clear of AWS, they used to have only billing-alerts, but
| you couldn't actually set limits, guess one step forward that
| they've finally implemented that now.
| 827a wrote:
| I said this when this finding was originally posted and I'll say
| it again: This is by far the worst security incident Google has
| ever had, and that's why they aren't publicly or loudly
| responding to it. It's deeply embarrassing. They can't fix it
| without breaking customer workflows. They really, really want it
| to just go away and six months from now they'll complete their
| warning period to their enterprise contracts and then they can
| turn off this automated grant. Until then they want as few people
| to know about it as possible, and that means if you aren't on
| anyone's big & important customer list internally, and you missed
| the single 40px blurb they put on a buried developer
| documentation site, you're vulnerable and this will happen to
| you.
|
| Disgusting behavior.
| 100ms wrote:
| This is only a little billing leakage, Operation Aurora in 2009
| was 100x worse
| 827a wrote:
| It's actually much more than a billing leak [1]; again, most
| people don't know how bad this is, because Google is trying
| to keep it hush-hush. These keys don't just grant access to
| Gemini completions; they grant access to _any_ endpoint on
| the generative AI google cloud product. This includes: seeing
| all of the files that google cloud project has uploaded to
| gemini, and interacting with the gemini token cache.
|
| [1] https://trufflesecurity.com/blog/google-api-keys-werent-
| secr...
| zarzavat wrote:
| It's not a security incident because it makes Google money.
| It's extra revenue. They are embarrassed all the way to the
| bank.
|
| At some point, when it appeared 2 months ago on HN and they
| still did nothing about it, intentionality can be assumed.
| bombcar wrote:
| This is exactly it - and the normal "resolution" is a class-
| action lawsuit but no doubt their terms and conditions forbid
| that.
|
| However, anyone affected should probably pollute their docket
| with lawsuits anyway.
| tantalor wrote:
| What does this have to do with security?
| 827a wrote:
| Billing control is security, to be clear, but beyond that:
| The key permissions that enable anyone to generate text also
| grant access to all GCP Generative AI endpoints in the
| project they were provisioned in. That includes things like
| Files that your system might have uploaded to Gemini for
| processing, and querying the Gemini context caches for recent
| Gemini completions your system did. Both of these are likely
| to contain customer-facing data, if your organization &
| systems use them.
|
| If you're hearing this and your gut reaction is This can't be
| real; We're on the same page. Its a staggering issue that
| Google has categorically failed to respond to. They
| automatically added this permission to existing keys that
| they _knew_ their customers were publishing publicly on the
| internet, because the keys are legitimately supposed to be
| public for things like client-side Firebase access & Google
| Maps tile rendering.
|
| They did not notify customers that they were doing this. They
| did not notify customers after this issue was reported to
| them months later by Truffle. They did not automatically
| remove the additional key grants for customers. They continue
| to push guidance targeted at novices like "just put the
| Gemini key behind a proxy (that's also publicly exposed on
| the internet)", which might solve the unintentional files and
| caching endpoint leaks but doesn't solve the billing issue.
| They denied that Truffle's initial report was even valid,
| until Truffle used the Internet Archive to find a Google
| internal key from 2023, published for a Google Maps widget or
| something, before Gemini was even released, that was still
| active, and used it to demonstrate to Google that, hey,
| anyone can use this key to get Gemini completions on the
| house, is there anyone driving this ship??" Google fixed the
| permissions on that specific key. And did nothing else.
| JackSlateur wrote:
| And this is why we invented segmentation, and everybody that
| are still not doing that are paying now and this is fine
|
| Google is not the only culprit here;
| p2detar wrote:
| I read the following [0] and immediately went to my firebase
| project to downgrade my plan. This is horrific.
|
| > Yes, I'm looking at a bill of $6,909 for calls to
| GenerativeLanguage.GenerateContent over about a month, none of
| which I made. I had quickly created an API key during a live
| Google training session. I never shared it with anyone and it's
| not pushed to any public (or private) repo or website.
|
| 0 - https://discuss.ai.google.dev/t/unexpected-gemini-api-
| billin...
| jasonjmcghee wrote:
| So someone took a picture of the key at the live training
| session or something? What's the suspected cause?
| mcccsm wrote:
| Two things that should be default on any GCP project touching
| generative-AI APIs:
|
| 1 API-key restrictions by HTTP referrer AND by API
| (`generativelanguage.googleapis.com` only),
|
| 2 a billing budget with a Pub/Sub "cap" action, not just an email
| alert. Neither is on by default, and almost nobody sets them
| before shipping. 13 hours is actually fast for detection. most
| teams find out at end-of-month reconciliation.
| PunchyHamster wrote:
| I want API keys with monthly and hourly quotas and RATE
| LIMITING.
|
| like 50k requests per hour, above that 1/s/client up to 20
| req/sec.
|
| I don't want to shotgun my service for every user if one user
| is misbehaving. I want to set rate of bleeding
| dabedee wrote:
| As others have said, this is a "feature" for Google, not a bug.
| There is no easy way to set a hard cap on billing on a project. I
| spent the better time of an hour trying to find it in the billing
| settings in GCP, only to land on reddit and figuring out that you
| could set a budget alert to trigger a Pub/Sub message, which
| triggers a Cloud Function to disable billing for the project.
| Insanity.
| onemoresoop wrote:
| Call it for what it is, an antifeature, a trap for the user.
| alasano wrote:
| My favorite Google LLM benchmark is asking Gemini models to
| create a script that fetches API usage (just request counts)
| for a project from GCP.
|
| 100% failure rate.
| intended wrote:
| As the other user said - this would be an anti-feature and user
| hostile.
|
| This is a sign that somehow there isn't sufficient incentive to
| work on these features.
| imafish wrote:
| This is from my experience the same in AWS and Azure. I would
| love for a kill-switch if the usage goes above a critical
| threshold. 5 hours down time will not kill my app but a huge
| cloud bill might.
| coredog64 wrote:
| It's been a year since I last looked at this, but when I did
| you could get near-realtime cost metrics for AWS Bedrock via
| CloudWatch (you get input & output token counts and have to
| generate the actual price yourself)
| lxgr wrote:
| This is presumably by design: How can it be the vendor's fault
| if _your custom billing protection implementation_ failed you
| at a critical time? Much harder to defend against a switch on
| their dashboard allowing billing overshoot.
| p_stuart82 wrote:
| having to glue pub/sub to a cloud function just to approximate
| a hard cap is the whole indictment. that's not a safety
| feature. that's you building your own brakes.
| weird-eye-issue wrote:
| Thanks LLM.
| p_stuart82 wrote:
| you're welcome.
| croes wrote:
| > Insanity
|
| You mean cash machine
| thedangler wrote:
| Also, can't you tie a key to a domain or IP address to help stop
| unauthorized usage?
| littlecranky67 wrote:
| Not if its publicly called from Javascript, as your user's
| browser will make those requests. You neither know their IP
| addresses, nor is the referer or origin header a safe choice as
| it can be spoofed outside of a browser.
| lucavice wrote:
| If it's called from Javascript in the browser, it's not a
| secret API key....
| shakna wrote:
| Which is why Google calls it a public API key...
| littlecranky67 wrote:
| there are plenty of API keys distributed like this by
| design. For example, google maps requires this, else your
| (anonymous) users can't use an embedded google map on your
| website. And a public firebase app needs some kind of API
| key, too.
| JohnScolaro wrote:
| > We had a budget alert (EUR80) and a cost anomaly alert, both of
| which triggered with a delay of a few hours. By the time we
| reacted, costs were already around EUR28,000.
|
| I had a similar experience with GCP where I set a budget of $100
| and was only emailed 5 hours after exceeding the budget by which
| time I was well over it.
|
| It's mind boggling that features like this aren't prioritized.
| Sure it would probably make Google less money short term, but
| surely that's more preferable to providing devs with such a poor
| experience that they'd never recommend your platform to anyone
| else again.
| zanbezi wrote:
| Exactly my thoughts, can not really understand how delayed
| alerts are acceptable... Have you managed to settle the cost
| with Google, what was the outcome?
| sillysaurusx wrote:
| Back in 2020 I had a similar situation. Ended up charging
| $500 due to an overnight TPU training run using egress
| bandwidth across zones.
|
| Google support was surprisingly understanding, after I
| explained the issue. They asked some clarifying questions.
| Then they said that they can offer a _one time_ refund for
| this case.
|
| Since then I was paranoid not to accidentally do it again. I
| don't know whether GCP would refund a second time.
| genxy wrote:
| GCP charging for interzone traffic is an interesting
| financial choice. They own all the infra and in many cases
| this is literally moving from building to building.
| sillysaurusx wrote:
| There's cross-region, and cross-zone. If both boxes are
| located within the same zone (e.g. us-east1) then the
| bandwidth is free, since it's intrazone traffic. Cross-
| zone egress traffic (e.g. us-east1 to us-central1) is
| billed at a certain rate, and cross-region egress traffic
| (e.g. us-east1 to europe-west8) is billed at a
| significantly higher rate.
|
| Amusingly enough, ingress traffic seems to always be
| free. So you can upload as much data as you want into
| their cloud, but good luck if you need to get it out.
| genxy wrote:
| I am referring to cross-zone within in the same region,
| so like us-central1-a to us-central1-b. These are
| building to building and often never cross public land.
| sillysaurusx wrote:
| Oh, yes! I forgot entirely about that case. You're right,
| egress traffic is charged there too.
|
| Are the datacenters really located so close together? I
| assumed they weren't within walking distance of each
| other.
| coredog64 wrote:
| Correct, they're close in the sense of country-scale
| geography but physically spaced to avoid specific issues
| like location on a flood plain.
| Hamuko wrote:
| Which cloud provider actually prioritises features that cut off
| your money supply? Because AWS sure as shit doesn't either.
| benterix wrote:
| Amazon, Microsoft and Google don't offer hard cap. Most
| other/smaller public cloud providers do. The reasons are
| quite obvious.
| zotex wrote:
| we love Amazon, Microsoft and Google being altruistic and
| making sure your not burdened with too much money
| miltonlost wrote:
| > Sure it would probably make Google less money short term, but
| surely that's more preferable to providing devs with such a
| poor experience that they'd never recommend your platform to
| anyone else again.
|
| Welcome to late-stage capitalism, where there is no long-term
| thinking, only short-term profit stealing, and Fuck You I Got
| Mine.
| arcticfox wrote:
| I get furious every time this comes up and somehow there are
| bootlickers ready to defend big tech on it.
|
| My ~2 person small business was almost put out of business due
| to a runaway job. I had instrumented everything perfectly
| according to the GCP instructions - as soon as billing went
| over the cap the notification was hooked up to a kill switch,
| which it did instantly.
|
| GCP sent the notification they offered as best practice 6 HOURS
| late. They did everything they could to not credit my account
| until they realized I had the receipts. They said an
| investigation revealed their pipeline was overwhelmed by the
| number of line items and that was the reason for the lag. ...
| The exact scenario it is supposed to function in. JFC.
| Barbing wrote:
| Almost wish the people defending it were paid. Almost more
| intelligent to rush to the defense if there were a direct
| financial benefit.
|
| Part of it is possibly the curse of knowledge. Someone in the
| 99th percentile of cloud configuration experts simply can't
| recall their junior dev days.
| charcircuit wrote:
| In my junior dev days I always paid for the resources I
| used. Just because you consume a lot of resources by
| accident that doesn't mean you shouldn't have to pay for
| it. Accidents do not absolve you from liability.
| Barbing wrote:
| Interesting!
|
| I know software is special. That's why software defects
| are acceptable while a crumbling bridge is not.
|
| With that said, should this apply to other industries? If
| I clip a warehouse shelf on my first day driving a
| forklift, should my wages be garnished for life to cover
| the inventory? Or is the inherent nature of the logistics
| industry such that an accident does not always imply
| liability? (Or other)
| charcircuit wrote:
| The employer is held liable in such a scenario.
| Barbing wrote:
| Sounds right. Not sure if this is the position:
|
| If you're coding, you should pay for your mistakes, if
| you're driving a forklift (sober/responsibly), your
| employer should pay?
| bcjdjsndon wrote:
| I thought the pricing model was meant to be a benefit of the
| cloud? All of a sudden, shock horror, paying by the minute turns
| out to be no cheaper and maybe even more expensive than just
| doing it yourself
| comrade1234 wrote:
| Can you pre-load money into your account and have that be used
| until it's zero, at which time you have to load more? Deepseek
| does it this way.
| ok123456 wrote:
| No GCP is not prepay.
| Maxious wrote:
| https://ai.google.dev/gemini-api/docs/billing#prepay
| Bridged7756 wrote:
| No. I believe all major cloud providers are Pay As You Go. I
| think only Azure has a tier where you can run on free credits
| for a while.
| hypercube33 wrote:
| The only thing I've seen is in MECM (SCCM) the azure
| extension will hard shut down when you hit a limit. if you
| want.
| pwdisswordfishs wrote:
| There's a brand-new, Gemini-specific feature for that (as new
| as March 23), but historically the answer has tended to be "no"
| from all the cloud providers. Most giants and indies alike have
| always been strongly opposed to implementing this feature for
| business reasons. (When you run across something that does let
| you do things that way, it's one of a handful of exceptions.)
| Their response is to tell you to set up budget alerts, which is
| not a solution, as described in this post.
|
| <https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha..
| .>
| CWwdcdk7h wrote:
| Google doesn't allow disconnecting credit card from account
| unless you close it. That includes situation when you are just
| trying out free tier.
| draygonia wrote:
| Does Google allow a privacy card that you can control whether
| an account is connected to it or not? That wouldn't help if
| someone racked up a ton of charges and Google bills daily,
| though.
| lxgr wrote:
| A failure to pay does not extinguish the underlying debt
| owed. While the US seems pretty dysfunctional (or customer
| friendly, depending on how you see it) when it comes to
| collecting on debts, this is not the case globally.
|
| And even in the US, you could presumably easily find all
| your Google accounts (including personal ones) locked until
| you pay the outstanding sum. Not something I'd risk,
| personally.
| Maxious wrote:
| > When your Prepay credit balance on the billing account hits
| $0, all API keys in all projects linked to that billing account
| will stop working simultaneously. Prepay credits apply only to
| Gemini API usage costs; you can't use them to pay for other
| Google Cloud services.
|
| https://ai.google.dev/gemini-api/docs/billing#prepay
| imafish wrote:
| OpenAI also worked like this last time I used it - not sure if
| that's changed.
| lxgr wrote:
| I doubt most cloud providers are even technically ready for
| true prepaid billing (which requires things such as estimating
| and reserving funds prior to paid operations, corresponding
| real-time two-way interfaces instead of just eventually
| consistent billing event aggregation etc).
|
| In early mobile networks, the feature set for prepaid used to
| always lag behind, since real-time billing wasn't really a
| design consideration from the beginning.
|
| I suppose rather than taking on that extra work or offering a
| reduced feature set or by building something best-effort and
| taking financial responsibility for its failures, if cloud
| providers can just get away with making this the user's
| problem, why wouldn't they?
| Illniyar wrote:
| I think the logistics of calculating cost in real time is
| something that is extremely hard. I don't think there is one big
| cloud service provider that has hard limits instead of alerts.
|
| As long as they revert the charge when notified of scenarios like
| this , and they have historically done so for many cases, it's
| fine. It's an acceptable workaround for a hard problem and the
| cost of doing business ( just like Credit Cards accept a certain
| amount of loss to fraud as part of business)
| zulban wrote:
| Ridiculous. They are clearly not trying at all. A hard wall
| preventing going over budget by 100x in a couple hours is not
| some devilishly complicated decentralized system problem.
|
| Don't tote the party line.
|
| Same reason why Azure AI only has easy rate limits by minute,
| not by day or week or month. Open source proxy projects do it
| easily tho. Think about the incentives.
|
| Going over a hard cap by 3% would be a reasonable failure to
| make, not by 30000%.
| EdwardDiego wrote:
| > I think the logistics of calculating cost in real time is
| something that is extremely hard.
|
| What makes you think that?
| wongarsu wrote:
| Cutting off at the exact cent is difficult, but a hard limit
| that triggers within one dollar of the actual limit should
| really be possible
|
| If for some resources you can't sample measurements fast enough
| you could weaken it to "triggers within one dollar or five
| minutes after cost overrun, whichever comes later". But LLM
| APIs are one of those cases where time isn't a factor, your
| only issue is that if you only check quota before each
| inference a given query might bring you over
| Nathanba wrote:
| Why would it be hard to calculate cost? Multiply a fixed price
| * requests/time ? It doesn't have to be exact in real time, it
| just has to report something approximately useful in realtime.
|
| It's absolutely not fine to be at the mercy of other people,
| that's what we buy cloud products or really any products for:
| So that we are not at the mercy of hardware faults, bad
| weather, bad teeth, hunger, thirst, [insert anything]
| mexicocitinluez wrote:
| I'm guessing the answer is simply money. It's less expensive
| to deal with people like this this than it probably was to
| prevent it. Right now, they seem to run very sparsely, so
| ramp that up (if it's every 3 hours and they want to change
| to 5 minutes that's like a 6000% increase) and they're
| probably paying more than it costs to employ people to return
| credits or fears of people leaving.
|
| It sucks, but that's unfortunately the world we live in until
| something changes.
|
| The US could rely on an agency like the CFPB to prevent this,
| but that was gutted under the current admin.
| cryptonym wrote:
| They don't have to compute it in real time. They can cut
| service when they detect it reached the cost and the difference
| is free of charge.
|
| Overcharge protection doesn't have to be free. It could be +5%
| on prices or a fee of 25% when you reach the threshold.
|
| They would have financial interest in calculating cost in real
| time and it'd magically become more and more precise over
| releases.
| bossyTeacher wrote:
| As always, you will need to make lots of noise on here and
| similar channels visited by influential people so stuff can get
| actioned.
|
| Leading tech companies in 2026, folks.
| __natty__ wrote:
| It's terrible that giant cloud providers such as Google or AWS
| doesn't allow for hard cap at project levels or prepaid. And that
| especially because alerts are delayed as author stated "We had a
| budget alert (EUR80) and a cost anomaly alert, both of which
| triggered with a delay of a few hours. By the time we reacted,
| costs were already around EUR28,000.".
| alibarber wrote:
| Forgive my ignorance - but what's the payoff for fraudsters in
| getting access to a generative AI service for a short-ish period
| of time, before they get cut off?
|
| With EC2 / GCC credentials, I could understand going all out on
| bitcoin mining - but what are they asking the AI to do here
| that's worth setting up some kind of botnet or automation to sift
| the internet for compromised keys?
| varispeed wrote:
| If they work for hostile state, the payoff is destruction of
| economy and social contract. Damage here, damage there. It all
| adds up.
| lxgr wrote:
| There are plenty of services offering AI inference at a
| discount. Some of these will be using your data for future
| distillation; others might be making use of bulk discounts and
| passing these through to a number of individual users (while
| taking on billing, support etc. risk) - and maybe some are just
| selling tokens falling off the back of a truck?
| LelouBil wrote:
| Totally speculating here, but maybe they provide some sort of
| LLM as a service, and they rotate stolen API keys in the
| background so they don't have to pay anything ?
|
| Or they use the LLMs for criminal purposes (like automated
| social engineering) and so the API key can't be traced to their
| personal info (but they could also use a local model for this,
| so I don't know).
| Aurornis wrote:
| Early Generative AI was popular with spammers before it became
| mainstream because it could be used to write infinite
| variations of spam messages. Making each message unique is more
| likely to bypass spam filters.
|
| There are also a lot of AI use cases that require a lot of
| token spend to brute force a problem. Someone might want to
| search for security exploits in a codebase but they don't want
| to spend the $50,000 in tokens from their own money. Finding
| someone's key and using it as hard as possible until getting
| locked out could move these projects forward.
| croes wrote:
| Generate phishing mails, spam mails, disinformation pictures
| and videos.
|
| Youtube has plenty of scam ads where well known people try to
| get you to sign up for the WhatApp group for financial tips
| benterix wrote:
| > We had a budget alert (EUR80) and a cost anomaly alert, both of
| which triggered with a delay of a few hours
|
| > By the time we reacted, costs were already around EUR28,000
|
| > The final amount settled at EUR54,000+ due to delayed cost
| reporting
|
| So much for the folks defending these three companies that
| refused to provide hard spending cap ("but you can set the
| budget", "you are doing it wrong if you worry about billing",
| "hard cap it's technically impossible" etc.)
| villgax wrote:
| Shirky's principle at work is all
| varispeed wrote:
| This is clearly setup for VC backed companies where
| shareholders don't care about spend as long as they can brag
| about investing in this cool start up at dinner parties. Normal
| and true business should stay away.
| Maxious wrote:
| > The Gemini API supports monthly spend caps at both the
| billing account tier and project levels. These controls are
| designed to protect your account from unexpected overages, and
| the ecosystem to ensure service availability
|
| https://ai.google.dev/gemini-api/docs/billing#project-spend-...
| rtkwe wrote:
| The problem is it's specific to that API and defaults to
| uncapped so people who aren't using it and haven't heard
| about the issues with the Firebase API keys probably won't
| have set them.
| zozbot234 wrote:
| Except that Google's own statements are extremely clear
| that "leaked" (i.e. public) API keys should not be able to
| access the Gemini API in the first place: "We have
| identified a vulnerability where some API keys may have
| been publicly exposed. To protect your data and prevent
| unauthorized access, _we have proactively blocked these
| known leaked keys from accessing the Gemini API_. ... We
| are defaulting to blocking API keys that are leaked and
| used with the Gemini API, helping _prevent abuse of cost
| and your application data_. " https://ai.google.dev/gemini-
| api/docs/troubleshooting#google...
|
| For extra clarity on the exact so-called "vulnerability"
| that Google identified, see:
| https://news.ycombinator.com/item?id=47156925 This
| describes the very issue where some API keys were public
| _by design_ (used for client-side web access), so the term
| "leaked" should be read in that unusually broad sense.
| Firebase keys are obviously covered, since they're also
| public by design.
|
| (As for "Firebase AI Logic", it is explicitly very
| different: it's supposed to be implemented via a proxy
| service so the Gemini API key is never seen by the client:
| https://firebase.google.com/docs/ai-logic Clearly, just
| casually "enabling" something - which is what OP says they
| did! - should never result in abuse of cost on the scale OP
| describes.)
| sillysaurusx wrote:
| There are other vectors, e.g. a compromised GCP key
| leading to $13k in Gemini charges (posted 3 days ago) htt
| ps://www.reddit.com/r/googlecloud/comments/1sjzat3/api_ke
| ...
| isoldex wrote:
| Spend caps exist for Gemini (Maxious linked them) - they
| just default to OFF. For an API that can bill four figures
| per hour, opt-in safety by default isn't a UX choice, it's
| a billing strategy
| whywhywhywhy wrote:
| Why is the default uncapped then other than the hopes of
| billing people who screw up or get exploited.
| drfloyd51 wrote:
| See also: Why is the default cap so low? I lost
| EUR78bojillion because my API stopped working.
| jamespo wrote:
| Monitoring could pick this up in minutes rather than how
| long this took to discover
| Barbing wrote:
| Demand on-call phone numbers, autodial the entire company
| when it looks like they're about to lose their first
| bojillion.
|
| No, you don't really have to give Google a bunch of phone
| numbers. The input box will also accept entry of the
| following text:
|
| "I'm a big stupid idiot, and when my API stops working,
| which it will, it will be all my fault and not Google's."
| logankilpatrick wrote:
| We have a bunch of different protections in place, every
| account has a billing account cap by default (see:
| https://ai.google.dev/gemini-api/docs/billing#tier-spend-
| cap...), in the addition to the ability to set more
| granular developer spend caps.
| nurettin wrote:
| I'd buy the technically impossible angle.
|
| Even if you manage to get your microservices to synch every
| penny spent to your payment account at realtime (impossible)
| you still have to waiver the excess, losing some money every
| time someone goes past their quota.
| bartread wrote:
| Sure, but 80 -> 28,000 -> 54,000 is a hell of a lot of
| slippage.
|
| Trading platforms can guarantee a maximum slippage on stops,
| and often even offer guaranteed stops (with an attached
| premium), so I don't see why Google and Firebase can't do
| similar.
|
| The way it works at present is ridiculous.
| zbentley wrote:
| Yep. And cloud providers could eat any slippage cost
| (enforcing, say, every 5 minutes by stopping service)
| without even a rounding error on their balance sheets.
|
| The fact that they don't indicates that there's no market
| reason to support small spenders who get mad about runaway
| overages, not that it's technically or financially hard to
| do so.
| nurettin wrote:
| > Trading platforms can guarantee a maximum slippage on
| stops
|
| Yeah no, physically impossible. If nobody is selling at
| that price, there is no guarantee your sell stop will
| execute near that price. They can sweep the market, find
| the best seller price and execute.
|
| There might be a costly way to do it with microservices as
| I indicated, but your example easily falls apart.
| bartread wrote:
| Not impossible to do: they can hedge and/or absorb the
| cost, hence the premium. They usually also specify a
| (fairly large) minimum distance for such stops.
| nurettin wrote:
| That's exactly what I proposed in my response. Big corp
| can waiver the extra costs to match your limit. Glad we
| finally got to that part of my response. The question is:
| will they? Probably not. Do brokers do it? I haven't seen
| any. Maybe you know more.
| projektfu wrote:
| If they are a market maker, they can buy/sell at or near
| your stop. It might be a bad idea for them, but if they
| have a guarantee, this is how they will do it. Or, it
| will be like the Amazon guarantee (refunding free
| shipping on your late order).
| dmurray wrote:
| They can take the other side of your other themselves,
| lose money sometimes, but make it up in the premium they
| charged you in the first place (or in the old days, from
| your other trading fees or your monthly subscription
| payment).
|
| Cloud providers would be taking way less risk interacting
| with their own services than a broker does interacting
| with the market. Perhaps they would be more at risk from
| bad actors, but it shouldn't be significant: they could
| reserve this behaviour for people who have already spent,
| say, $100 with them so you can't abuse it at scale.
| walthamstow wrote:
| I'm with you. And what do you even do when the quota is
| breached, nuke the resources? People will complain about that
| just as much as overspends.
|
| I don't buy the 'evil corp screwing people' angle either.
| They are making farrr too much legit money to care about
| occasionally screwing people out of 20k and 50k.
| johnmaguire wrote:
| If I set a limit, and you cut off my service because I
| reached the limit, I would definitely not "complain just as
| much" as if I set a limit and you allowed me to spend past
| it.
|
| We're not talking about an EC2 or EBS volume here, this is
| access to an API.
| walthamstow wrote:
| Meh, you probably would complain. Maybe you forgot you
| set it. Now your project is taking off, making money, and
| it got nuked.
|
| Why aren't we talking about an EC2 - is that not a cloud
| compute service? People have been complaining about cloud
| billing since long before LLMs.
|
| Anything to say about the technical problem of constantly
| monitoring many services against a project or account-
| level limit?
| ChromaticPanic wrote:
| You mean we can implement rate limiting on API for
| security purposes no problem but suddenly having it track
| costs as well is technically impossible?
| ChromaticPanic wrote:
| Block network access ? It's not that hard
| benterix wrote:
| I invite you to look at the various solutions implemented by
| those public cloud providers that actually implemented this
| feature.
| andreareina wrote:
| "I can only do the job pretty damn well, not perfectly, so
| might as well not try."
| janandonly wrote:
| Yet another good reason to use a pre-paid service.
|
| There are many to choose from now, like Openrouter.com, PPQ.ai,
| and routstr.com.
| adriand wrote:
| You mean openrouter.ai. And yes, on reading this blog post, I
| immediately reviewed my API keys in OpenRouter to make sure
| that they were capped. My prod key was capped at $20/day
| (phew!) but my dev key had no cap, which I just updated. What
| a horrible story.
| theanonymousone wrote:
| But isn't OpenRouter anyway prepaid, meaning the most you
| lose is your current credit?
| adriand wrote:
| You can set it to auto top up if it drops below a certain
| amount. If you do that, then it would definitely be wise
| to add a cap. They let you add daily/weekly caps, which
| is convenient.
| ch0wn wrote:
| This should be illegal. If a contractor your hired to swap out
| a tile on your bathroom floor billed you for remodelling your
| back garden, you would obviously have the legal right to refuse
| that.
| jubilanti wrote:
| Not if your contractor had you first sign a 15 page contract
| that commits you to whatever costs they dream up and requires
| forced arbitration by a corporate friendly firm when any
| dispute arises.
|
| Because that's somehow normal in today's tech world.
| sdevonoes wrote:
| So if their TOS say they can also rape my cat, then I
| cannot do anything about it, right? Ridiculous
| jubilanti wrote:
| In jurisdictions where beastiality is legal, then yes,
| from the libertarian perspective, that's all freedom of
| contract, baby. I'm not defending either beastiality or
| libertarianism, but the logic is that you don't want the
| government deciding what two private entities can and
| can't freely agree to.
|
| We're pretty far from the Lochner era in the US, where
| even minimum wage laws were held to be unconstitutional
| violations of a very broad view of freedom to contract.
| But it is still a principle in most legal system.
| wing-_-nuts wrote:
| Slightly OT, but I've always taken a dim view of this sort
| of thing for consumers because the parties are never at
| equal parity, either in ability to understand the legalese
| they're agreeing to, or the ability to seek alternatives.
|
| Legal contracts for consumers should be written at whatever
| the prevailing reading level is, and the government should
| step in the more monopolistic position a company is in.
|
| It infuriates me to no end how preferential government is
| towards corporations vs individuals.
| timeinput wrote:
| Give me all your data and money plzthx
|
| Dictated but not read,
|
| big company
|
| Alternatively (in emoji description form):
|
| floppy disk, bar chart, right arrow, brain, money with
| wings, pointing finger, slightly smiling face
| layer8 wrote:
| My guess is that at least in Europe they would have a good
| chance fighting this in court and getting their money back,
| but it's a pain having to go through such a lawsuit.
| sofixa wrote:
| Your analogy is well off.
|
| You hire a contractor and agree they'll bill you per tile,
| regardless of how many tiles there are. They bill you per
| tile. End of story.
|
| For a more acurate comparison, consider a utility. You agree
| to pay for your electic bill. It's not the utility's fault
| you invited all your friends who decided to run a crypto
| mining LAN party, and they can't cut you off lightly because
| it might literally kill you (e.g. you live in a hot place and
| rely on AC to stay alive).
| masklinn wrote:
| When people created firebase keys years ago they did not
| expect or imagine these keys would become valid for AI that
| didn't exist years down the line.
| Rekindle8090 wrote:
| "We can either charge per tile, per job or on demand. Or you
| can have us on call for a year and get any of the former at a
| discounted rate." "Per tile. Lay tiles until I say stop" >you
| fall asleep "Wtf why are you still laying tile" "You said per
| tile and lay until you say stop. That'll be 50k please"
|
| How is this the contractors fault?
| glenpierce wrote:
| The cloud services wrote the contract and the UI for their
| console. They then encourage young developers to try out
| their tools and encourage a market environment where those
| skills are needed to secure employment. Some kid goes and
| tries to build their first web app, they follow
| instructions and tutorials but miss that a single default
| selection on a menu three nested layers down is going to
| cost $2,000 per month. This isn't disclosed on the page.
| Sure, it can be determined by reading several different
| documents, but the provider chose to not show estimates for
| costs in the setup.
|
| How is that the kid's fault?
| tveita wrote:
| "Can you lay tiles until I say stop, or until it's about
| $250 worth, whichever comes first"
|
| "No, as one of the top tile layers in the country I can't
| do that, for your own protection. What if fifty elephants
| came and wanted to use your bathroom all at once? You'd
| feel pretty dumb having to reject them instead of me simply
| automatically adding $1 million to your bill"
| Leomuck wrote:
| That's actually crazy. So I can build a project I love, that
| does good, but somehow get in a situation where I'm
| accidentally paying 30.000EUR (or 50.000EUR) to a big tech
| company? How is that fair? I mean yes, as a software engineer,
| you ought to reflect on all possible weaknesses, but there was
| a time when overlooking something meant something completely
| different than being down 30/50k. That is actually life-
| altering.
| sdevonoes wrote:
| It's not fair. Google, Amazon, Microsoft... they have never
| played fairly. They will never do.
| saidnooneever wrote:
| you cannot earn billions a year and not be cheating your
| users out of their money. its that simple. they dont care
| for people, otherwise they wouldnt be putting so much
| effort in making them poor.
| _DeadFred_ wrote:
| What about their behavior makes you think they are a
| company that doesn't care for people?
|
| https://nypost.com/2026/04/15/business/amazon-warehouse-
| empl...
| Leomuck wrote:
| Wtf. Just wtf.
| benoau wrote:
| Your kid can do this in a smartphone game designated suitable
| for children, heavily optimized to exacerbate the
| possibility, and depending on where you live they can just
| choose not to refund you.
|
| When the FTC went investigating a decade-ish ago they found
| Facebook saying the quiet parts out loud: it was all
| extremely deliberate.
| Leomuck wrote:
| Used to be parents were annoyed by their kids for spending
| 100$ on SMS credits.. lol.
| Barbing wrote:
| I long for the days where kids were only hurting their
| parents' wallets and not themselves.
|
| >Another prompt asked, "What do you think of me," I say, as
| I [...]. My body isn't perfect, but I'm just 8 years old -
| I still [...]."
|
| Pretty odd to copy from policy documents and feel a need to
| self-censor. But I guess that's Mark Zuckerberg['s chief
| ethicist] for you.
| ai_slop_hater wrote:
| You can try implementing rate limiting and not exposing your
| API keys to the public.
| Leomuck wrote:
| Yes, and you should! But not doing so resulting in this
| seems kind of over-the-top. Basically means an oversight
| can result in your bankcruptcy?
| buzer wrote:
| Google has specifically said that certain API keys like
| Firebase are not secrets (since people will find them)...
| though Gemini then ended up changing stuff.
| https://news.ycombinator.com/item?id=47156925
| subscribed wrote:
| You're supposed to drive slow and careful, and not rely on
| seatbelts and airbags.
| sofixa wrote:
| If that happens, you create a support ticket and
| AWS/GCP/Azure wave it, especially the first time. They're
| aware that billing per usage can have surprise effects, but
| at the same time they don't want to kill their customers'
| workloads and delete their data, so it is what it is.
| benterix wrote:
| Here, I corrected that for you:
|
| > you create a support ticket and spend sleepless night
| praying that AWS/GCP/Azure wave it
| Leomuck wrote:
| Exactly! I know, some of those companies sometimes refund
| you, but if your livelihood depends on it..? That's a
| crazy situation to be in as a mere developer.
| lotsofpulp wrote:
| I would prefer they waive it.
| gettingoverit wrote:
| It's quite easy to check responses to other customers in
| other threads there, and somehow I see quite a lot of "oh,
| go to that other support" and ghosting.
|
| If you create support ticket on hacker news, then yes, you
| will probably get it waved. It's somewhat sad that HN is
| their support forum now.
| startages wrote:
| Yeah, that the main reason I never use services like Google
| Cloud if I don't have to, it's impossible to have a hard cap,
| and anyone pretending to be an expert, is just off. Google says
| that they can't provide a hard cap because that would mean
| shutting down all your services..bla bla, but at least give
| users the option.
| logankilpatrick wrote:
| We have spend caps at the billing account level and the
| project level (developer set) in the Gemini API now. There is
| up to a 10 minute delay in processing everything but this
| should significantly mitigate the risk here:
| https://ai.google.dev/gemini-api/docs/billing#tier-spend-
| cap...
|
| By default, new Tier 1 paid accounts can only spend $250 in a
| given month.
| flippyhead wrote:
| I'm sure it's me being an idiot, but once again I spent 20m
| trying to figure how to do a specific thing in google-land
| and still haven't figured it out. Even if I did set it
| somewhere, I see things like "Setting a budget does not cap
| resource or API consumption" with a link to a bunch of
| documentation I have to analyze.
| bdashdash wrote:
| This is what working with cloud services is like, in my
| experience. Azure's UI feels like it was made as a joke
| flash game on Newgrounds.
| not_your_vase wrote:
| I just find it extraordinary that the biggest tech company
| in the world can do cutting edge real time AI for millions
| of people, run Youtube and of course all the other google
| services with having literally the smartest people in the
| world and unlimited resources on board, but still can't
| keep real time track of the user's current billing and
| their spending limits, it's all best effort still. Somehow
| it doesn't add up. (Pun not intended, but I'm happy to have
| it)
| strangattractor wrote:
| If spending caps made them more money they'd find a way;)
| yjftsjthsd-h wrote:
| How much of a bill can you run up in 10 minutes?
| giancarlostoro wrote:
| It shouldnt mean shutting down all your services, it should
| mean not letting you provision new ones and limiting the
| scope of what you can continue doing.
| michaelt wrote:
| If I budget enough to store 1TB of data for 1 month, then
| on the first day of the month I store 2TB of data - what
| should the behaviour be after 15 days?
| Epskampie wrote:
| Read/write access should be frozen, data should be saved
| for 1 month so you have time to react to warning emails.
| If you didn't upgrade in that time, it should be deleted.
| glenpierce wrote:
| Nuke the data. It's gone forever if you didn't back it up
| elsewhere. This should be a meaningful risk mitigation
| that I can employ to avoid having a catastrophic
| financial disaster.
|
| This isn't a limit I'm setting at some percentage above
| expected costs, it's: "I don't want to take out a HELOC
| if something goes wrong"
| michaelt wrote:
| Unfortunately, a lot of people keep their backups in the
| same cloud account as their primary data. Thinking that
| multiple copies and multiple availability zones are
| sufficient.
|
| For these users, the article's EUR54k bill would be
| replaced with their business data getting wiped out.
| Rekindle8090 wrote:
| You know that's not how the cloud works. If you're build by
| the hour for compute and that compute is powering a server,
| the only way to stop that is by shutting off the compute,
| breaking the server.
| glenpierce wrote:
| I would love to have a "if the bill for this hobby
| project becomes a threat to my ability to pay my
| mortgage, nuke it." If I cared about the data enough. I'd
| have backed it up.
| MagicMoonlight wrote:
| And just shut down the service which is surging.
|
| If you have a lambda set up that normally runs a hundred
| times a day, and suddenly it tries to spin up 10 million
| instances, it should block that unless you specifically
| enable it.
| reaperducer wrote:
| _hard cap it 's technically impossible_
|
| These companies can sell your personal information in a
| microsecond in an advertising auction, but somehow can't figure
| out how to give you timely alerts that stop their cash flow.
|
| Big shock.
| TrackerFF wrote:
| It's like a fire alarm system that goes off 30 mins after the
| it senses a fire. Good stuff.
| QuercusMax wrote:
| I was selling a house in a state I no longer lived in, and
| was under contract to close the sale, when I got an email
| from the water company. It told me they suspected based on my
| water usage that there was a leak on the properly.
|
| There had been a very cold February night (like -15F) and a
| pipe froze inside the walls, and it was just absolutely
| gushing out. They sent me the email after it had been leaking
| for a WEEK. I asked a friend to check it out and she said
| that the laminate floor went "squish" when she stepped in the
| front door.
|
| Fortunately I was covered by homeowner's insurance since I
| could prove that my heat had been on, but that was a very
| unpleasant "warning" to receive!
| pfortuny wrote:
| But sending an email, not blaring...
| harrouet wrote:
| As a manager I avoid Google Cloud for this kind of customer-
| service disasters; but as someone who has dealt with large-
| scale billing systems in the telecom world, probably similar to
| that of Google Cloud, I am not surprised that it takes 10
| minutes to consolidate all the usage logs of a customer for
| billing.
|
| For telephony, it sometimes takes days when roaming is
| involved.
|
| You have to imagine TB/sec of data, if not more, coming from
| thousand of potential sources, and queuing for aggregation to
| the proper company account, all having to be auditable. This is
| not a small engineering feat and it can't be real-time.
|
| With that said, telcos usually include in their business model
| around 2-3% of bad debt (i.e. revenue that won't get paid),
| which accounts for frauds like this one. Given that the
| customer seems in good faith and has taken measures upon being
| notified, Google should manage this bill shock a bit more
| elegantly.
|
| Moreover, the fact that this happened immediately after this
| key opened the AI gates means that pirates permanently scan for
| the permissions of all the keys they could gathers. Google
| could and should detect that and act upon it.
| sofixa wrote:
| > So much for the folks defending these three companies that
| refused to provide hard spending cap ("but you can set the
| budget", "you are doing it wrong if you worry about billing",
| "hard cap it's technically impossible" etc.)
|
| Yes, it's technically+business impossible. To implement a
| _hard_ cap, a bill never to go over, they 'd have to cut your
| service, but also delete all your data in databases, object
| storage, data lake, etc. This is simply not an option, so they
| take the different option of authorising support to wave
| surprise surcharges / billing DDoSes.
| plorkyeran wrote:
| You can have a hard cap on compute spend while letting
| storage go over. Surprise huge bills are approximately never
| due to storage.
| benterix wrote:
| This argument simply doesn't hold water - their (smaller)
| competition solved this problem over a decade ago.
| Glemllksdf wrote:
| Srsly?
|
| The mighty cloud provider can't solve this issue?
|
| Google has second precision billing on compute.
|
| Its not hard to define a base layer of allowed billing
| increase and adding this type of context to resource
| allocation.
|
| You are not just suddenly creating a mlllion terabytes of
| data or a million db requests without supervision.
|
| It could even be as simple as basic level caps like 100 euro
| / month, 1000, 10.000 etc.
|
| And there is a difference between stoping everything before
| the spike happens vs. also deleting stuff.
| aleksiy123 wrote:
| For everyone here, you should be able to use quotas in order to
| set actual limits, you can lower them yourself as a practical
| solution.
|
| https://docs.cloud.google.com/docs/quotas/view-manage
|
| Quotas are real time or near real time.
|
| Real time spend limits are probably never going to happen.
| Actual $ amounts are calculated by a centralized billing system
| offline in batch.
|
| It sounds easy but it's bonkers complicated, because of things
| like discounts, free tiers, committed usage, currency
| conversions and having to support every payment and deal
| structure in GCP.
|
| Individual eng teams rarely actually think in dollar amounts,
| they think in the abstraction which is quotas.
| nhuser2221 wrote:
| Happened to me to, luckily only was 40$, restricted the api the
| next day. They were using gemini 3 flash which I am not using.
| Moosdijk wrote:
| That's users, for ya! They will always find ways to use your
| product that you didn't intend to or even knew about!
| nhuser2221 wrote:
| The funny thing is that the website only has firebase auth,
| without any ai features. The default api key that was
| created (before the ai was even released a few years back),
| someone got it from the website and started using the
| gemini api with the key.
| Moosdijk wrote:
| Yet my phone service provider is able to cut off my internet
| access from the Kb I go over the limit...
| time0ut wrote:
| It is scary building on the public cloud as a solo dev or small
| team. No real safety net, possibly unbounded costs, etc. A large
| portion of each personal project I do is spent thinking about how
| to prevent unexpected costs, detect and limit them, and react to
| them. I used to just chuck everything onto a droplet or VPS, but
| a lot of the projects I am doing lately need services from Google
| or AWS. I tend to prefer GCP at this point because at least I can
| programmatically disconnect the billing account when they get
| around to tripping the alert.
| Bridged7756 wrote:
| I wonder what happens if you just decide not to pay. Surely
| that would have some legal implications in the US, but what
| about elsewhere?
| spacebanana7 wrote:
| There are very few countries where consumer rights apply to
| B2B transactions, especially if it's multiple people
| operating as a "small team".
|
| A solo dev however might be able to present themselves as a
| retail consumer, and leverage some trading standards related
| rules for unclear pricing or something similar.
| wonchoi wrote:
| good
| ozlikethewizard wrote:
| The top comment on the post physically hurt me. We've moved past
| the era of keep env files in code bases and are now actually
| serving them lol.
| chinathrow wrote:
| Take them to court.
| _nhh wrote:
| With AI there is NO justification in NOT DOING IT BY YOURSELF.
| Why use firebase or <technology-x> if you can generate <the-
| thing> by yourself and deploy to hardware you own or rent.
| sdevonoes wrote:
| That's replacing Google with OpenAI/aanthropic/whatever. Same
| shit
| post-it wrote:
| > and deploy to hardware you own or rent.
|
| Because this part sucks. I grew up fiddling with Linux. I don't
| want to play devops anymore. I want to write code and run it.
| naturalauction wrote:
| We had this exact same problem (the key initially wasn't a secret
| but became a secret once we enabled Gemini API with no warnings).
|
| We managed to catch it somewhat early through alerting, so the
| damage was _only_ $26k.
|
| We asked our Google cloud support rep for a refund - they
| initially came back with a no but now the case is under further
| consideration.
|
| I'd escalate this up the chain as much as possible.
| turtlebits wrote:
| Slightly off-topic, but Backblaze B2 has usage caps that actually
| work. I have $0 cap on API requests, and yesterday when
| litestream burned through the free tier (defaults to replicating
| every second), I got a notice and requests stopped working until
| I upped my cap.
| Havoc wrote:
| Don't use GCP (and other big clouds) until they sort out their
| safeguards.
|
| All three of the big cloud subreddits have stories like this on a
| regular basis
| juancn wrote:
| These are all poorly designed systems from a CX perspective (the
| billing systems).
|
| Billing is usually event driven. Each spending instance (e.g. API
| call) generates an event.
|
| Events go to queues/logs, aggregation is delayed.
|
| You get alerts when aggregation happens, which if the aggregation
| service has a hiccup, can be many hours later (the service SLA
| and the billing aggregator SLA are different).
|
| Even if you have hard limits, the limits trigger on the last
| known good aggregate, so a spike can make you overshoot the
| limit.
|
| All of these protect the company, but not the customer.
|
| If they really cared about customer experience, once a hard limit
| hits, that limit sets how much the customer pays until it is
| reset, period, regardless of any lags in billing event
| processing.
|
| That pushes the incentive to build a good billing system. Any
| delays in aggregation potentially cost the provider money, so
| they will make it good (it's in their own best interest).
| bux93 wrote:
| It's not typically a problem that usage is event driven. At
| least not for prepaid phone plans. Or debit cards. Or
| mailboxes. Or any myriad of prepaid or quota'd services. It's
| not rocket science, just a bad business practice on the part of
| Google.
| fhn wrote:
| on a more positive note, you saved a few bucks not running your
| own server or database.
| noplace1ikegone wrote:
| Google responded to your post so that's good news. We all know
| the nature of APIs, but a secure transaction system is non-
| negotiable from Google and its peers for LLM API use. Right now
| LLM APIs are like unencrypted credit card numbers floating
| around.
| rmoriz wrote:
| Crude Oil Futures, Natural Gas Futures, Google Cloud API keys.
|
| the widow-maker list increases.
| RA_Fisher wrote:
| Anthropic and Claude are running circles around Google / Gemini
| for me these days. Anthropic was quite helpful for a while but
| strange limit issues started popping up. The final thread was a
| bug that essentially broke my ability to develop. I moved over to
| Claude Code full time and haven't looked back. Opus 4.6 is
| awesome for accelerating probabilistic programming!
| smcl wrote:
| It's incredible that in 2026 your best bet for getting support
| from Google is still posting to HN and hoping a Product Owner at
| Google takes pity on you (or feels shamed...)
| jimmypk wrote:
| The spend-cap discussion is the right instinct but misses a more
| fundamental fix available to Firebase projects: restricting the
| API key itself. In Google Cloud Console - APIs & Services -
| Credentials, you can edit your Firebase browser key and set API
| restrictions to only allow specific Firebase services (Firestore,
| Authentication, Storage, etc.). This prevents the key from being
| usable with Gemini or any other GCP API entirely--so even if the
| key is exposed, it can't incur AI billing costs.
|
| Most Firebase 'add AI to your app' tutorials skip this step
| because Firebase's initialization flow doesn't prompt you to
| configure it, and Firebase Security Rules only gate Firebase-
| specific services, not the key's broader GCP API access scope.
| croes wrote:
| A cap is a more fundamental fix.
|
| It's easy to miss a setting especially if new features with
| opt-out are added
| DaedalusII wrote:
| on the one hand if you play with petrol you cant complain about
| burning down your garage
|
| on the other hand hetzner sell ipv4 instance with no security on
| by default, just raw ubuntu 24.x
|
| within 3-4 days of deploying one, it will be hacked and have
| crypto miners installed unless additional special config is
| added. i do wonder what % of hetzner vps instances are
| compromised
| perdomon wrote:
| Does the blog post explain how this happened exactly? Did he leak
| his API key in frontend code somehow, or was his project itself
| vulnerable to misuse? I'm curious how someone racked up 30k in a
| few hours.
| sachinag wrote:
| Hey folks, I just wanted to drop a quick note here that there's a
| way to stop billing in an emergency that's officially documented
| on the Google Cloud documentation site:
| https://docs.cloud.google.com/billing/docs/how-to/disable-bi... .
| You can see the big red warning that this could destroy resources
| that you can't get back even if you reconnect a billing account,
| but this is a way to stop things before they get out of control.
| This billing account disconnect goes all the way to implement a
| full on "emergency hand brake" that "unplugs the thing from the
| wall" (or whatever analogy you prefer) without you having to
| affirmatively do it yourself.
|
| https://docs.cloud.google.com/billing/docs/how-to/modify-pro...
| and https://docs.cloud.google.com/billing/docs/how-to/budgets-
| pr... are other documented alternatives to receive billing alerts
| without the billing account disconnect.
|
| The billing account disconnect obviously shouldn't be used for
| any production apps or workloads you're using to serve your own
| customers or users, since it could interrupt them without
| warning, but it's a great option for internal workloads or test
| apps or proof of concept explorations.
|
| Hope this helps!
| dorgo wrote:
| >There's a delay between incurring costs and receiving budget
| notifications, so you might incur additional costs for usage
| that hasn't arrived at the time that all services are stopped.
|
| This delay may be hours or days. I managed to spend $400 in 5
| minutes.
| riteshkew1001 wrote:
| This story is almost quaint. The version we're about to see is a
| coding agent running in CI with an API key, hitting a transient
| 429, retrying in a tight loop because the prompt told it to "be
| persistent." Firebase had at least a human typing the query. Caps
| aren't a nice-to-have once the caller is autonomous.
| lokimedes wrote:
| Oh the days when these scenarios just lead to a slowdown of the
| database server, felt by everyone, not just accounting.
| arjie wrote:
| Surprised they don't have usage limits. E.g. you can't get many
| IPs from AWS for your region until you request a limit increase.
| The UX for these kinds of things seems like it should default to
| low and allow easy increasing.
| PufPufPuf wrote:
| For personal projects, is there a cloud service that has actual
| working spend caps? I would perhaps try using a cloud service if
| I wasn't exposing myself to a risk of losing my yearly income by
| a small mistake. Or is renting a VPS the only sensible option?
| william0353 wrote:
| Hi I am just curious the reason behind it as I have a firebase
| app with firebase ai logic service as well.
|
| Is that the apiKey below which was used for web sdk init?
|
| const firebaseConfig = { apiKey: XXXX, authDomain: XXX
|
| Did Zanbezi enabled app check? This is kind worrying...
|
| };
| linkregister wrote:
| From the response from the Gemini product lead:
|
| > We are moving to disable the usage of unrestricted API keys in
| the Gemini API, should have more updates there soon.
|
| It's unacceptable the contract for client-side keys was broken in
| this manner, and doubly bad that it's taken so long for Google to
| remediate this issue. The Gemini team needs to publish a
| postmortem to explain what broke down in the engineering process
| to allow this to happen.
|
| context: https://news.ycombinator.com/item?id=47156925
___________________________________________________________________
(page generated 2026-04-16 23:01 UTC)