[HN Gopher] Down the Cloudflare / Stripe / OWASP Rabbit Hole
___________________________________________________________________
Down the Cloudflare / Stripe / OWASP Rabbit Hole
Author : mdhb
Score : 232 points
Date : 2023-02-20 09:05 UTC (13 hours ago)
(HTM) web link (www.troyhunt.com)
(TXT) w3m dump (www.troyhunt.com)
| mpalfrey wrote:
| I wonder what is actually causing the error here. Looking at the
| 2 payloads diff on the site I can't spot anything. It's a weird
| one that's for sure.
| tyingq wrote:
| Something real subtle probably, like enough of the redacted
| areas accidentally containing something like looks like SQL.
| Like WALTER contains ALTER, CASEY contains CASE, etc...enough
| to add up the score to fail.
|
| Hopefully more convoluted than simple substring matches like
| that, but I've seen WAFS do that. If it is adding up substring
| matches, there is stuff like "subscription_update", "null",
| "created", "discountable", "total_count", etc, that would set a
| high floor.
| hislaziness wrote:
| Definitely Stripe should have done a better job here. However,
| also points to the risk of allowing service providers to handle
| your security. Maybe there should have been a back test and or a
| "maker" "checker" control for pushing new security rules.
| dncornholio wrote:
| Cloudflare is blocking the shit out of legitimate requests, how
| is this Stripes fault?
|
| Putting the webhook callback behind CloudFlare was the mistake
| IMO.
| bouke wrote:
| Indeed, in the end Troy disabled a certain rule for incoming
| requests from Stripe but not all. Just before publishing the
| article he discovered more (webhook) requests being blocked
| by a different rule. This seems like a continuous source of
| issues where valid (webhook) requests are being (arbitrarily)
| blocked. Why wouldn't you just disable all rules for requests
| coming from Stripe?
| michaelt wrote:
| Yes - Stripe is even nice enough to provide a list of IP
| addresses their webhook requests could come from [1]
|
| And Cloudflare is well known for randomly throwing up
| "Checking your browser before accessing example.com"
| interstitials and captchas and suchlike - of course you don't
| want bot detection on your callback API.
|
| [1] https://stripe.com/docs/ips?locale=en-GB#webhook-
| notificatio...
| a2tech wrote:
| A customer I work with has a large Drupal site hosted with
| a big time Drupal provider. Cloudflare has decided that
| image uploads from the browser via the JS uploader needs to
| be checked and injects itself into that...but the browser
| has no way to display it so the page just hangs. Its awful.
| yellow_lead wrote:
| WAF blocking things randomly isn't a good look for Cloudflare. I
| wonder how many outages this has caused for their customers.
| AlfeG wrote:
| From my experience randomly blocking rules in WAF is essentialy
| a feature of the WAF. I have random issues with WAF all over
| the places from random providers on different projects.
|
| And yep. Disabling WAF were always a good enough solution
| untill we figure out which of the request parts are triggering
| rules
| davedx wrote:
| "I look at the managed WAF Cloudflare provides more favourably
| than I did before simply because I have a better understanding of
| how comprehensive it is. I want to write code and run apps on the
| web, that's my focus, and I want someone else to provide that
| additional layer on top that continuously adapts to block new and
| emerging threats. I want to understand it (and I now do, at least
| certainly better than before), but I don't want managing it day
| in and day out to be my job."
|
| My takeaway from this is actually that you can't just have a
| managed WAF that gets out of your way. This looks like it took
| easily a day or more of work. What's the advantage of using a
| CloudFlare "managed" WAF versus running your own within AWS? I
| guess the "infrastructure" is managed, but the operations
| isn't...
|
| I'm a pretty experienced engineer, and I honestly don't know if
| I'd have been able to solve this issue personally. Most likely I
| would have just whitelisted all of Stripe's IP's, like the
| initial hotfix did.
| pixl97 wrote:
| Working around WAFs for years in enterprise support this kind
| of crap happens all the time. In my current work we'll have a
| client try to access our app in some way that oddly explodes.
| In general a browser HAR file is very useful. Then we have to
| check our app (hosted on the customer's servers), then we'd
| have to look at the load balancer, and when that doesn't bear
| results it's quite often we find a WAF in network path. Most of
| the time it's near impossible to find the team that manages it
| and then get helpful information out of them about the issue.
| aaroncloud wrote:
| I have used a few different WAFs to date, AWS WAF, CloudFlare WAF
| and Akamai WAF. I have had issues with all but usually the
| frequent issue is an unexpected blocking of request. Would be
| interesting to benchmark a set of WAFs to see which ones block
| threats vs which block valid traffic.
| tyingq wrote:
| A funny one that comes up is many WAFs either block, or only
| examine the first X bytes of large-sized post requests, for
| performance reasons.
|
| So your choice of unpredictable lost requests, or a loophole
| where the WAF is blind.
| billpg wrote:
| Suppose I use Stripe in a similar way to HIBP and a customer
| comes along just as my server has gone down, would Stripe try
| again later? I think it should wait a little while but I'd hope
| it'd automatically try again later as long as there's data that
| needs to be sync'd but isn't yet.
|
| I can maybe imagine 403 being a "Don't try again" signal in a way
| that 500 or timed-out isn't.
| philipwhiuk wrote:
| If it doesn't you can manually re-trigger it. That does require
| you to monitor it of course.
| __float wrote:
| Yes, it retries for up to three days according to
| documentation: https://stripe.com/docs/webhooks/best-
| practices#retry-logic
| talkingtab wrote:
| I wonder what the take away from this is? The simple one is "bad
| cloudflare" or "bad stripe" or even "bad hibp". Or maybe all in
| conjunction. Or maybe none.
|
| But that seems simplistic to me. The smell of this is a system
| that is so poorly made that it has layer upon layer of obscure
| hacks to protect it. It appears that no one can understand why
| this happened and the best guess is that it had something
| legitimate that was misunderstood. Maybe the word "alter" and
| "table"? This is the equivalent of you walking into a bank,
| telling the person "Hi my name is Rob and I came to the bank
| today to ..." And then the bank goes into automatic shut down.
|
| This is broken. IMHO.
| jsnell wrote:
| From the information given, bad Cloudflare. These kinds of
| content-matching rules should be triggering deterministically,
| and testable in a hermetic test environment. They also have
| sample payloads that get blocked vs. ones that gets through,
| despite being essentially identical. It should be about as easy
| to debug as it gets.
|
| That it's tricky to debug suggests there's something totally
| different just badly understood rules. Maybe a server with a
| hardware fault that's making it return bogus results (though
| that should be easy to find in monitoring), maybe some kind of
| race condition, or running of different rules in parallel +
| having global or request-scoped state such that the order in
| which the rules finish running matters.
| ghoshbishakh wrote:
| I solved the stripe webhook issue for https://pinggy.io by adding
| a refresh button that calls the stripe API to double check all
| subscriptions for a user.
| ricardo81 wrote:
| Came across similar situations before [beyond previous issues
| with being a trusted bot by Cloudflare]
|
| The destination website user was wondering why our service didn't
| visit their website and why we were receiving 503s in return,
| with CF in the middle. Turns out in this case it was an
| additional bot blocking service they'd installed on their own
| server, which I'd presume is hard for them to debug if the
| request never reaches them (but in this case it did, just added
| confusion for them).
|
| Far from ideal to have a middle man arbitrate a web request or
| decide what is trusted. On the flip side, lots of rogue traffic
| that should be blocked. Think in some cases the WAFs are just a
| bit too aggressive with the grey area and UAs they don't know. It
| creates a barrier of entry to new players.
| perfecto_maduro wrote:
| We whitelisted the stripe IPs completely after getting burned
| once. If Stripe gets hacked so that the hackers jump off to our
| site, we have far bigger problems to worry about.
| danpalmer wrote:
| It seems like Cloudflare should be doing this for you. It
| wouldn't be hard for them to keep a list of IPs from common
| known-good integrations. They could prompt on first hit to ask
| you if you want to allow-list those companies, or even just do
| it by default.
| JohnFen wrote:
| Can you imagine the firestorm that would happen if CF was
| found to be allowing traffic from certain other entities, no
| matter how trustworthy they're perceived to be, to bypass
| security controls by default? And the firestorm would be
| entirely warranted.
| mac-chaffee wrote:
| I've been running the OWASP coreruleset in production for about a
| year now and it has been a big pain. The way I made it manageable
| was 1) training users that "if you see a 403 error, tell me ASAP"
| and 2) learning the ModSecurity rule syntax to be able to create
| rule exceptions for users very quickly. This is not possible to
| do at Cloudflare's scale.
|
| Even then, users who didn't know the intricate details of the Web
| Application Firewall (like Troy in this case) would waste hours
| hunting down the issue. Since less popular sites often have more
| illegitimate traffic than legitimate traffic, there was really no
| good way for me to proactively fix WAF issues.
|
| The conclusion I have drawn is that WAFs really only have a few
| very narrow use-cases.
|
| The main use-case is when you want to write your own rule to
| protect hosts from a specific zero day while they are being
| patched. Like a simple rule to detect Log4J [1] was an effective
| band-aid while we scrambled to implement real patches. But WAFs
| have an inherent weakness: clever attackers can pretty much
| always circumvent rules, or force to you write a rule that is so
| complex that it causes slowness or blocks legitimate traffic.
|
| Another use-case is when you have to deploy some untrusted code
| that is likely vulnerable to common (>1 year old)
| vulnerabilities. Like running an old/archived wordpress instance.
| This is the only time when the coreruleset makes sense IMO.
|
| As I see it, WAFs are a tool created in a simpler time when the
| number of possible attacks and applications was small. In the
| modern era where there is a constant deluge of zero-days, huge
| attack surfaces, tons of variability in applications, and lots of
| sites where RCE/SQLi is a feature (think CI job definitions,
| Juptyer notebooks, custom query languages), WAFs have lost their
| effectiveness.
|
| [1]: https://github.com/coreruleset/coreruleset/issues/2331
| Liquidor wrote:
| I'm not expert and this was really nice and easy to read and
| understand. Thanks for sharing Troy.
|
| Question: Should we even use Cloudflare for server-server
| communication such as webhooks, API calls etc?
| jacquesm wrote:
| If your API calls are so generic that you don't need to
| identify the caller using an API key or some other token then
| you might be able to get some mileage out of it. But personally
| I'd like all API calls to go to _my_ servers and not someone
| else 's. If only because I don't like 3rd parties having the
| ability to log the payloads of requests, and whoever terminates
| your SSL connections has the ability.
| berkle4455 wrote:
| Damn this is absurd. Is there not a way on Clouldflare to flag
| webhooks with a "if request contains this api key, let it
| through, block the rest"?
| tyingq wrote:
| It looks like "WAF - add exceptions" supports Wireshark style
| expressions, so there's things like:
|
| (http.request.uri.query contains "some-string")
|
| Or similar for checking headers, post body content, etc.
| a2tech wrote:
| You can create custom rulesets, which is what the article's
| author did. This allowed the API calls to essentially bypass
| the WAF.
| blibble wrote:
| just remember sometimes cloudflare add new features which
| aren't controlled by existing rules
|
| "security off for this ip?" "I guess you don't mean our new
| shiny feature!"
| jamespo wrote:
| Running a WAF with a dynamic ruleset between you and your payment
| provider seems a bit risky to me.
| danuker wrote:
| Using a third-party's WAF and trusting it will not log your
| data also shows some risk.
|
| By centralizing all data going through the web you paint a
| massive target on yourself (for black hats).
| Perseids wrote:
| What I find bleak about the situation, is that it is a glaring
| design-fail, even though everyone involved should have the
| necessary expertise to do better. A callback from your payment
| provider should never go through a best-effort WAF. Instead, as
| you already have a strong business relationship, you could easily
| exchange/store/configure strong credentials with stripe [1]. When
| even a security professional doesn't do that, what does it say
| about the state of documentation of this feature?
|
| Looking at the documentation directly, what they advise you to do
| is kind of the worst idea they could come up with:
| https://stripe.com/docs/webhooks/signatures - you need custom
| logic[2] to verify that their MAC ("signature" they call it
| incorrectly) is valid and you need to configure a _different
| secret_ for _each_ of your endpoints. And then you still need to
| handle replay attacks somehow, which is its own nightmare to do
| correctly. It 's no wonder the WAF can't do that for you.
|
| From a few years old personal experience, I'm really irritated by
| stripes web-hook approach overall. Payment process information is
| such a vital business concern that "let's try to call them and if
| that fails... well we tried" is broken on principal alone. The
| obvious approach is to have an event list which you as customer
| long-poll or just poll every few seconds if your framework
| doesn't support async well. This is also trivial to do securely:
| You're HTTPS library already authenticates stripe during the TLS
| handshake and that is all that is necessary.
|
| [1] Best case scenario: Let stripe authenticate with mutual TLS,
| but I know this is quite a long way away from typical web server
| configurations.
|
| [2] Stripe's approach very much reminds of DPoP
| https://news.ycombinator.com/item?id=31266575 which shall in now
| way be construed as a compliment.
| davedx wrote:
| > The obvious approach is to have an event list which you as
| customer long-poll or just poll every few seconds
|
| This is also _much much easier_ for the payments integrations
| developers to test, compared to all the messing about and
| running dodgy proxies that testing webhooks involves.
|
| Webhooks for critical application paths just seem like a bad
| idea all around really
| bmizerany wrote:
| > Webhooks for critical application paths just seem like a
| bad idea all around really
|
| Well, put. I've been deep in the Stripe API for a while now,
| and the invoices API provides up-to-date account status
| information and what a customer is paying for. This can be
| referenced as needed and, if desired, cached with some TTL
| and referenced as the "truth." Then webhooks can be viewed as
| a convenient way to bust the cache quicker than waiting for
| the TTL to expire.
|
| It could be better, but it provides the necessary information
| without relying on webhooks.
|
| An example of how we use this API as a form of entitlement
| checks at Tier can be found here: https://github.com/tierrun/
| tier/blob/f7c32426d30ca314706ca7e...
| philipwhiuk wrote:
| Yeah, Stripe is well documented but very ugly to actually
| handle in practice.
| lvice wrote:
| I just would like to give my opinion on some of your points,
| which I don't agree with:
|
| > Looking at the documentation directly, what they advise you
| to do is kind of the worst idea they could come up with:
| https://stripe.com/docs/webhooks/signatures - you need custom
| logic[2] to verify that their MAC ("signature" they call it
| incorrectly) is valid and you need to configure a different
| secret for each of your endpoints
|
| It certainly help that I use their official SDK, but it's one
| line of code to add the signature validation. Also, I'm not
| sure why you would want to create a lot of endpoints to listen
| to these webhook. I simply have one, and the Stripe SDK helps
| me in determine the event type, its deserialization, etc.
|
| > Payment process information is such a vital business concern
| that "let's try to call them and if that fails... well we
| tried" is broken on principal alone
|
| That's not how it works. The webhooks keep retrying with
| exponential backoff until they succeed. You can also manually
| retrigger them for individual events.
|
| > The obvious approach is to have an event list which you as
| customer long-poll or just poll every few seconds if your
| framework doesn't support async well
|
| Nothing is preventing you to do that. In fact, in my codebase I
| do polling to the Stripe API as a fallback to check if payment
| is successful in case there are issues with webhooks. But it's
| nice to have the webhook telling you immediately if a payment
| fails/succeed, in order to give feedback to the user fast about
| the status of his payment (and not wait the next long polling
| iteration)
|
| Not everything on Stripe is perfect, but I do find it really
| pleasant to work with in general
| Perseids wrote:
| Thanks for your remarks and corrections.
|
| While what you say is correct, it doesn't apply to the
| problem Troy Hunt faced. What he needs is DDOS protection on
| his API. The request authentication Stripe provides is too
| complicated to be checked by the web application firewall.
|
| The (edit:) pragmatic approach is to
|
| a) not use webhooks or
|
| b) let Stripe connect to you via HTTPS (to prevent replay
| attacks and leakage of the secret URI), give Stripe a secret
| URI, whitelist the secret URI in the WAF _and_ verify the
| payload MAC via the official SDK.
|
| > in order to give feedback to the user fast about the status
| of his payment (and not wait the next long polling iteration)
|
| Nitpick: The long poll / Server Sent Event should respond
| immediately once there is new data available, so it should
| not be slower than the webhook.
| PeterisP wrote:
| > b) let Stripe connect to you via HTTPS (to prevent replay
| attacks and leakage of the secret URI), give Stripe a
| secret URI, whitelist the secret URI in the WAF and verify
| the payload MAC via the official SDK.
|
| IMHO the long term best architecture would be HTTPS client
| certificates / mutual TLS auth- you would just whitelist
| that only clients signed/approved by Stripe can connect to
| that Stripe-callback endpoint.
| robertlagrant wrote:
| From memory, I think that's how Twilio callbacks work.
| eis wrote:
| Ignoring the question of who is actually doing something wrong
| and the fact that most customers on lower tiers wouldn't even be
| able to get most of the presented information, it seems absurd to
| me that a customer on an Enterprise plan that costs thousands of
| dollars a month can't get a simple "this is why those requests
| trigger the firewall" answer from CloudFlare. The diff between
| the request that passed and the one that triggered the rule is
| simple enough and the WAF is not some opaque AI I hope.
|
| Why can't the triggering request be replayed through the WAF with
| output that shows the scores for each bit?
| arpa wrote:
| It's like complex systems have complex failure modes, who'd
| thunk?
| cromulent wrote:
| From someone else in the article's comments:
|
| > if we treated the customer's phone number as a hex
| representation of ASCII, it spelled something that was
| recognisable as a command.
|
| And the WAF team suggested they ask the customer to change their
| phone number.
|
| Goodness me.
| NovemberWhiskey wrote:
| This is absolutely not the first (or second) time I've seen an
| outage triggered by a well-meaning security rules update on a
| WAF.
|
| To be honest, a lot of security-related deployment processes
| would be regarded as unacceptable, wild-west level shit if they
| occurred in the software lifecycle - like difficulty to identify
| that a change had _even occurred_ , inability to see before/after
| for the change, release processes effected manually via consoles,
| change deployed directly to production without going through a
| lower environment, and big-banged as opposed to canaried etc.
| etc.
| a2tech wrote:
| This boils down to 'Cloudflare did something' and without the
| Enterprise plan you'd never be able to pull the data required to
| diagnose the problem. Oh also, Cloudflare knows they did
| something but plays the blame game until you get to someone that
| openly acknowledges that they know something is broken.
|
| I've said it before and I'll say it again--Cloudflare is not
| making the Internet safer, its just making it less open. I know
| that is not the overwhelming sentiment of HN and me complaining
| about it isn't going to change anyones mind.
| JohnFen wrote:
| > Cloudflare is not making the Internet safer
|
| I am certainly not a fan of Cloudflare and woudn't use their
| services, but I think this is not an accurate statement. Their
| services do objectively provide a security benefit. The only
| question is really whether or not the cost/benefit ratio is
| favorable.
| fmajid wrote:
| Only if you discount the impact on users who are blocked like
| those on VPNs, or myself because Cloudflare seems to dislike
| my ISP, and have no recourse.
| throwawayapples wrote:
| Try getting a fresh IP from your ISP. I had a single IP
| that was blocked (incredibly annoying, and apparently
| impossible to have cloudflare fix it), but triggering a
| DHCP change caused me to get a good one.
| JohnFen wrote:
| That sort of thing falls into the "cost" category.
| carapace wrote:
| > Cloudflare is not making the Internet safer, its just making
| it less open.
|
| They're doing both, eh?
|
| I switched my family's home DNS to Cloudflare's family-safe DNS
| ( https://blog.cloudflare.com/introducing-1-1-1-1-for-
| families... ) to protect them from malware and porno (I'm not a
| prude, they don't use the Internet for that (no really! We are
| weirdos.) and porn sites are often a malware vector anyway.)
|
| A few months ago I noticed that you can't browse weed stores
| through their DNS anymore.
|
| I don't really blame them, I'm assuming it's due to pressure
| from the US Federal Gov. (who still consider pot to be some
| insanely dangerous narcotic!) But it was definitely a personal
| "until they come for you" moment.
| JohnFen wrote:
| This sounds like perfectly reasonable behavior, not needing
| any pressure from government agencies.
|
| I would assume that anything purporting to filter the
| internet to be "family safe" would exclude weed stores, as
| well as liquor stores, tobacco stores, and anything else that
| most people would consider inappropriate for children.
| cuuupid wrote:
| As a bit of a meta point it's a bit astonishing how good
| Cloudflare is at PR and community management. The sentiment
| I've seen here in the past is overwhelmingly pro-Cloudflare.
| Every now and then they'll publish a hit piece on AWS here and
| there'll be an AWS hate thread, but no one seems to question
| Cloudflare's motives for promoting bad press about their
| competitor directly to their target market.
|
| Then again in most things, the underdog trying to upstage the
| incumbent is always a popular narrative. You're right that
| talking about it likely isn't going to change any sentiment.
| throwaway2847 wrote:
| HN is like any other community and is thoroughly captured by
| some hype machines (Cloudflare, the language that shall not
| be named) and virulently opposed to others (cryptocurrency)
| jcuenod wrote:
| What are you talking about? Java doesn't have a hype
| machine...
| groffee wrote:
| > it's a bit astonishing how good Cloudflare is at PR and
| community management
|
| It's really not, look at any cloudflare support thread here
| and how they crawl out of the woodwork for damage control
| because they failed in basic customer service
|
| it's a huge red flag, and everyone in the comments always
| sees it.
|
| cloudflare is cancer and needs to die.
| Jamie9912 wrote:
| Cloudflare isn't really to blame here when the customer has
| FULL control over all security settings - they can define rules
| as they please, and have all the tools (including the API) to
| do this
| davedx wrote:
| Huh? In this case, _CloudFlare_ updated the OWASP ruleset,
| causing Troy 's payments integration to break.
|
| That's not nice DX in my opinion.
| Jamie9912 wrote:
| Customer should be aware that they are relying on
| Cloudflare defined rulesets and should understand that they
| change
| aniforprez wrote:
| This sounds like victim blaming honestly. If Troy Hunt,
| one of the most well known security researchers around
| and who's been running HIBP for almost a decade now ends
| his blog post with effectively a shrug saying "I dunno
| what happened" how is any reasonable customer who doesn't
| have access to the Enterprise plan supposed to debug any
| of this especially when Cloudflare themselves barely
| admitted fault? They even tried rolling back the OWASP
| ruleset and it didn't change a thing. He had to manually
| add the exceptions to the firewall. This is arguably
| terrible DX
| aestetix wrote:
| And don't forget, if they don't like you, they will happily
| deactivate your account with no notice and no reason given.
| weird-eye-issue wrote:
| Source?
| imwillofficial wrote:
| [flagged]
| sudhirj wrote:
| Bit hyperbolic, no? Maybe rephrase to "if you engage in Neo-
| nazi hate crime and promote real world harm, they'll
| deactivate your account after much internal deliberation on
| their role as gatekeepers and will publish a post with
| clearly stated rationale"?
|
| I'm not advocating Cloudflare, but I do think we need to be
| fair when judging stuff like this.
| jaywalk wrote:
| Your description of what happened is a bit hyperbolic as
| well. I remember this event, but the details are pretty
| fuzzy. IIRC, it stirred up so much controversy precisely
| because no US laws were broken. So saying "engage in Neo-
| nazi hate crime and promote real world harm" is inaccurate.
| The content was garbage, no doubt about it. But let's not
| get hyperbolic in the other direction either.
| JohnFen wrote:
| "no US laws were broken" != lacking in harmful
| activities.
| ASalazarMX wrote:
| > no US laws were broken
|
| People keep confusing private organizations and
| governments. They don't need to break any laws to become
| undesirable customers, and any private company that
| tolerates these kind of customers does so at their own
| discretion. The world doesn't owe them a right to be
| awful people, but they can still be awful if they accept
| the consequences.
| jaywalk wrote:
| I'm not confusing anything. Cloudflare's stance was that
| they'd serve anything that wasn't illegal, and one day
| they decided to change that. Of course that's their
| prerogative, but I'm just explaining why it stirred up so
| much controversy.
| quanticle wrote:
| The problem I have with CloudFlare is that their
| definition of "undesirable customer" includes forums
| engaging in hateful, virulent, but ultimately _legal_
| speech, but _does not_ include forums that are selling
| methamphetamine and stolen credit card numbers.
| aestetix wrote:
| "Let me be clear: this was an arbitrary decision. It was
| different than what I'd talked talked with our senior team
| about yesterday. I woke up this morning in a bad mood and
| decided to kick them off the Internet."
|
| https://gizmodo.com/cloudflare-ceo-on-terminating-service-
| to...
|
| Can you clarify where the "much internal deliberation on
| their role as gatekeepers" was?
| Lt_Riza_Hawkeye wrote:
| Overall I agree with you - the only caveat I have to offer is
| Cloudflare's support of eSNI. My opinion on CF used to be quite
| black and white, but there is at least someone in there (for
| who knows how long) contributing to the actual security of the
| web. Not mutually exclusive with doing harm in other ways.
| zX41ZdbW wrote:
| Case 1. It blocks my SQL playground due to SQL injections:
| https://play.clickhouse.com/play?user=play
|
| - solution: disable WAF.
|
| Case 2: It damages my presentations by removing whitespaces in
| HTML elements styled as "white-space: pre" at
| https://presentations.clickhouse.com/
|
| - solution: disable auto minification.
|
| Case 3: It makes the debian packages repository inconsistent
| https://packages.clickhouse.com/
|
| - solution: disable caching.
|
| In fact, Cloudflare is an amazing service - it is powerful and
| easy to use, you only have to take care when enabling and
| disabling its features.
| dt3ft wrote:
| I had the same issue with one of my sites. I activated CF owasp
| ruleset and all hell broke lose (it took a while to backtrack,
| aka down the rabbit hole I went). I deactivated it and the
| problems remained. It took several hours before things were back
| to normal. All in all - CF is too big now, they want to have
| _ALL_ the features, but the only feature I want is DDoS
| protection...
| technion wrote:
| There are multiple places where he uses a feature only available
| to enterprise users (read: Not me) to resolve this. I guess the
| "normal" thing here would be to disable the WAF and be done with
| it.
| e1g wrote:
| This jumped out at me too. Cloudflare keeps many diagnostic and
| debugging tools available only on the Enterprise plan
| (typically >$3k per month). When something like this happens to
| most small teams and startups, they are working in the dark
| even on the Business plan at $200/mo.
| tills13 wrote:
| FWIW CloudFlare has worked with us time and time again to get
| pricing in line with our expectations. We have numerous
| "Business Plus" or "Free Plus" plans across various accounts
| and zones that include features that are, as listed in the
| marketing, only available to Enterprise.
|
| Just reach out to someone at Cloudflare. I'm sure they'd love
| your business.
| a2tech wrote:
| And if you have to disable WAF, what exactly is Cloudflare
| doing for you?
| danielheath wrote:
| Edge Cache with unmetered egress?
| perfecto_maduro wrote:
| unmetered up to a point, but yeah. Also network-level DDoS
| protection
| tills13 wrote:
| If you are reaching the point where they are sending you
| emails asking about what you are doing, you should be
| paying for it.
| r1ch wrote:
| Even with the WAF disabled (at least as much as I can disable
| it without the Enterprise plan, i.e. "Essentially Off"), I've
| found it will still block legitimate requests. Tainted CGNAT
| or dynamic IPs are my guess.
|
| The WAF doesn't really matter for my use case as the route is
| handled by a CF worker, in fact I'd prefer it doesn't get in
| the way.
| perfecto_maduro wrote:
| yeah, but sadly there aren't many CDNs that offer WAF, even
| the most basic one. I literally begged bunnyCDN to build one
| so we can switch from CF. It's on their roadmap for like
| forever.
| Tijdreiziger wrote:
| In the article, the author disables the WAF only for Stripe
| outbound IPs, which can be presumed to be safe (unless
| Stripe's machines/IP space gets hacked). The WAF still works
| for traffic from all other IPs
| mpalfrey wrote:
| Would it still do DDOS even without the WAF?
| mstrem wrote:
| Our DDoS mitigation is separate yes and will still work.
| raverbashing wrote:
| Yeah, while I would have tried to investigate this a bit, this
| seems to be an issue on Cloudflare
|
| Bypassing the rules is a workaround, but not a fix
| gz5 wrote:
| And WAF isn't the only design option for webhooks. WAF = define
| what to block. Another options is to define what to accept
| (block all else by default).
|
| OpenZiti approach (1) for example:
|
| a. enroll each side of the webhook w/ X.509 identity
|
| b. X.509 gates a network overlay between the servers
|
| c. each server initiates outbound sessions to the overlay
|
| d. block everything else (deny-all inbound on both servers)
|
| (1) disclosure: i am a maintainer of the openziti foss, and you
| can only (fully) use the technique above if you have enough
| control of both sides, e.g. use a Lambda function:
| https://blog.openziti.io/my-intern-assignment-call-a-dark-we...
| judge2020 wrote:
| The problem is that you hope this doesn't encourage the
| developers to lapse on RBAC and let an authenticated client
| do everything.
| jeroenhd wrote:
| Mutual TLS auth seems like such a no-brainer for APIs like
| these.
|
| I do wonder, though, how does OpenZiti deal with client
| certificates expiring? Do you not set/enforce an expiration
| date or do you have some kind of automated renewal mechanism?
| robertlagrant wrote:
| Cert renewal is why it's not used everywhere, I think.
| While OAuth2 / client credentials is complicated, at least
| .jwks endpoints make cert rotation easy.
| ufmace wrote:
| My first thought is - why is this traffic going through
| CloudFlare at all? There's no caching benefit and it's all going
| to be coming from a datacenter anyways.
|
| Maybe it's more trouble than it's worth to bother setting up a
| separate non-CDNed DNS for API routes for this particular site.
| But then how much time was spent trying to sort out why those
| requests were being blocked?
| tiffanyh wrote:
| > About Page: "I'm Troy Hunt, an Australian _Microsoft Regional
| Director_ and Microsoft Most Valuable Professional for Developer
| Security. _I don 't work_ for Microsoft, but they're kind enough
| to recognise my community contributions by way of their award
| programs which I've been a part of since 2011."
|
| Can someone explain what this Microsoft Regional Director role is
| because it sounds like he works for Microsoft but then says he
| does not.
| mpalfrey wrote:
| Microsoft Regional Director is effectively someone who is an
| advisor to Microsoft - https://rd.microsoft.com/en-us/about/ .
| philipwhiuk wrote:
| That's a really stupid choice of title for essentially a "Key
| Partner"
| amluto wrote:
| I've generally thought that POST we hooks are the wrong
| abstraction for this type of payment confirmation, and reading
| this makes me believe it even more:
|
| This whole process needs something like a message queue. Stripe
| should publish and event to the queue, and HIBP should receive
| that event. A message queue is still subject to network failures,
| but any sensible implementation will notice and recover missed
| events.
| danpalmer wrote:
| See also: Cloudflare in front of Mastodon.
|
| A lot of naive Mastodon admins are putting default Cloudflare
| configurations in front of their instances. The problem is that
| inter-instance requests necessary for federation then get caught
| as bots (because they essentially are) and connections in the
| network degrade, causing eventual de-federation.
|
| Cloudflare is magic, in the good and bad ways. I'd only use it
| for very small personal sites or things that can afford downtime
| and don't need integrations, or for large businesses that can
| afford the enterprise plans and will actively manage the account
| and respond to business and tech needs with config changes.
| pbronez wrote:
| I wonder if their Wildebeast ActivityPub implementation will
| have similar issues.
| dbetteridge wrote:
| We had a very similar issue recently but with s3 presigned url
| uploads, cloudflare WAF and OWASP rules.
|
| Worked fine one day on 'high', next day started blocking random
| file uploads with a 'blocked by cloudflare page'.
___________________________________________________________________
(page generated 2023-02-20 23:01 UTC)