[HN Gopher] Tunnel Vision: CloudflareD AbuseD in the Wild
___________________________________________________________________
Tunnel Vision: CloudflareD AbuseD in the Wild
Author : redm
Score : 42 points
Date : 2023-08-09 13:45 UTC (9 hours ago)
(HTM) web link (www.guidepointsecurity.com)
(TXT) w3m dump (www.guidepointsecurity.com)
| filleokus wrote:
| Hmm, risking sounding like the dropbox-is-only-ftp-and-svn guy
| but shouldn't any threat actor worth their salt be able to use
| any array of open source tools to do this? Especially if you
| already have access on the machine and the possibility to place
| binaries there?
|
| The biggest risk I see is if the target is already using this for
| legit use cases, since I guess it would be really difficult to
| discern between the two.
| j16sdiz wrote:
| > I guess it would be really difficult to discern between the
| two
|
| I guess renaming the malware binary to chrome .exe would have
| the same risk?
| filleokus wrote:
| The article mentions other signals, like port numbers and DNS
| lookups for outgoing connections to Cloudflare when setting
| up the tunnel.
|
| If you use this Cloudflare feature for other stuff, you
| obviously can't use those as high SNR indicators.
| redm wrote:
| I find that more and more CloudFlare is so ubiquitous that we
| have to use CloudFlare tools to protect ourselves from other
| people attacking us via CloudFlare. Here's an example of a rule
| we had to put in place to block people using CloudFlare workers
| to scrape pages and bypass security. CloudFlare doesn't seem to
| care about this kind of abuse (or maybe they do but aren't
| talking about it publicly).
|
| (!cf.bot_management.verified_bot) and (cf.bot_management.score lt
| 10 ) and len(cf.worker.upstream_zone) gt 0 and not
| cf.worker.upstream_zone in {"<zone>"} and (not ip.geoip.asnum eq
| <exception as>)
| kentonv wrote:
| We definitely care about abuse generated by Workers.
|
| A request coming from someone else's Worker should not "bypass"
| any security -- it should be treated the same as if they were
| running their bot on any other cloud provider and making
| requests across the internet. If you find otherwise, please
| file it as a security vulnerability:
| https://hackerone.com/cloudflare
|
| (Actually, it should be slightly harder to run this kind of
| abuse from Workers, because any request coming from a Worker
| has the cf-worker header identifying the domain that owns the
| worker. It looks like your block rule is actually taking
| advantage of this. Other cloud providers usually cannot inject
| headers like this so it's harder to trace and block abuse from
| them.)
|
| (I'm the tech lead for Workers.)
| redm wrote:
| Thank you for your response. Since I have your ear, I'll
| provide some direct feedback.
|
| "We definitely care about abuse generated by Workers."
|
| That may be the case internally but it's not as evident
| through support. I base this on their ability to answer
| security-related questions, diagnose odd behavior, or
| mitigate problems. FYI, we are paying for your highest tier
| of support.
|
| "it should be treated as if they were running their bot on
| any other cloud provider and making requests across the
| internet."
|
| Part of this is how Site Analytics and WAF analytics
| represent worker data (or don't). Even though the worker
| modifies the contents, the IP is passed through as the end-
| user IP (at least the way we use them). These analytics
| systems need to do a better job of identifying anything
| worker or tunnels related. If this were being done through
| AWS it would be evident. We can use various mechanisms to
| mitigate this, but I wanted to clarify why abuse through
| Workers is different for our implementation.
|
| Also, the process to block non-zone workers was undocumented,
| and it took a couple of weeks for someone to dig it up and
| only after several trials and errors. Support wasn't sure it
| would work, and there was no other documentation (at the
| time).
|
| We have yet to get an answer about the high volume of tunnel
| requests from specific goes.
|
| Combine this with the opaque stack you have that causes odd
| situations. For example, I added a worker that automatically
| retries 500 errors from the origin. Simple enough. This
| breaks Zero Trust, though, and causes a redirect loop. There
| may be documentation somewhere, but there isn't a good
| breakdown of how Zero Trust fits into a customer-defined
| worker, and a redirect loop is undoubtedly a poor failure
| mode.
|
| There is a world of difference between the information you
| have and may provide as the Technical Lead of Workers, and
| what I'm getting via support or raw documentation.
|
| I could provide you more information offline if you want.
|
| Edit: Let me just add the other recent thread [1] regarding
| redirect loops in verification is par for the course. We had
| the same issue and tried for weeks for progress or
| resolution. The issue was never resolved, nor could we get
| information on why the failure mode was so poor, etc. That
| ambivalence adds up to expecting general ambivalence or
| inability to make meaningful progress. At some point, you say
| to yourself, "Why waste my time talking to support and
| putting in much effort and getting the run-around."
|
| [1] https://news.ycombinator.com/item?id=37049016
| kentonv wrote:
| Sorry to hear about your difficulties getting good answers
| from support. I will pass along your feedback.
|
| > Even though the worker modifies the contents, the IP is
| passed through as the end-user IP (at least the way we use
| them).
|
| That should only be true when your own worker makes a
| request to your own origin. It should never be true for
| requests coming from someone else's worker. Again, please
| file a security report if you found a case where that is
| happening.
| oefrha wrote:
| Well, I would say that's a pretty good ad for legitimate uses of
| Cloudflare Tunnel.
| Daviey wrote:
| As an enterprise cloudflare customer, we were interested in using
| this product for legitimate purposes but also block internal non-
| legitimate access and asked them for advice how to do this..
| Support wasn't able to offer any guidance.
|
| EDIT: Toned down the language.
| sam-at-cf wrote:
| Hi from the Cloudflare Product team - blank stares shouldn't be
| the response there. Can you send me a note, srhea AT
| cloudflare, and I'll make sure you get a response?
| leetrout wrote:
| I vouched for your comment. You might want to email hacker
| news mods so they can make sure your account is in good
| standing.
| mike_d wrote:
| I'm actually glad HN is putting some effort into combating
| the Cloudflare employee vote brigading.
| leetrout wrote:
| I was unaware there was an issue. any prior info about
| this?
| Daviey wrote:
| Hi Sam, thank you for the constructive reply. I've toned down
| what I said. I'm not able to talk about the specific example,
| but are you able to provide guidance for this generic issue
| of how organisations could block entirely this service at the
| egress edge and/or make it selective for approved/unapproved
| usage?
|
| Thanks
| ehhthing wrote:
| I don't understand why this is news for anybody. This is just the
| cloudflare version of ngrok...
| [deleted]
| TechBro8615 wrote:
| It's a bit more pernicious than ngrok just because it's more
| difficult to isolate and block only malicious usage of the
| feature. But yeah I agree it's not breaking news that implants
| will find ways to tunnel out of your infrastructure when
| exfiltrating data...
| mike_d wrote:
| (Context: part of my job is breaking into stuff all day)
|
| This is a technique commonly referred to as "living off the land"
| where an attacker makes use of a tool like cloudflared to conduct
| an action that would otherwise be blocked by security tools. It
| makes the defenders job so much harder because you now need to
| differentiate between your devops team being cool and a
| legitimate threat inside your network by looking at the exact
| same indicators generated by the two. Looking for things like
| unsigned applications making outbound network connections are
| removed from the defenders toolbox.
|
| Yes, cloudflared does the same thing as ngrok. You'll also find
| that ngrok is blocked in most corporate environments as well for
| posing an equal risk. As an attacker, you have a good chance of
| setting off alarms that (should) specifically detect ngrok.
|
| I think the point of this post it to highlight that cloudflare
| tunnels need to be block by default as well and only allowed when
| there are specific approved use cases.
| rafaelturk wrote:
| If an attacker is capable of installing apps on your server...
| than anything is possible. Don't know why mention Cloudflared
| other than clickbait. O Overall I'm huge fan of CF Zero Trust and
| tunnels. I wish documenation and examples were clear, but form a
| security stand point CF is one of the best security solutions we
| use.
| strangescript wrote:
| Yes, their docs, everywhere, leave something to be desired. For
| example, don't put worker code snippets with little context.
| Always post an entire file that the user can expand and
| collapse. Snippets assume the user knows things that they might
| not.
| Cyykratahk wrote:
| I feel like this is just another case of "It rather involved
| being on the other side of this airtight hatchway" [0]
|
| If an attacker is capable of installing apps on your server...
| you've already lost.
|
| 0.
| https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
| mike_d wrote:
| > If an attacker is capable of installing apps on your
| server... you've already lost.
|
| That is a deep misunderstanding of how security works.
|
| The defender's dilemma states that breaches are inevitable
| because defenders have to be right 100% of the time whereas
| attackers only have to be right once. This is why you have to
| defend in as many layers as possible.
|
| A better way to approach it is to assume that at any given time
| an attacker can execute code on your systems (because they have
| knowledge and access to unreleased exploits) and work on ways
| to detect anomalies and behaviors indicative of compromise as
| well as limiting the blast radius.
| mschuster91 wrote:
| > If an attacker is capable of installing apps on your
| server... you've already lost.
|
| You're right, but cloudflared is a whole lot easier to get
| running than previous RATs, and attackers don't need to operate
| their own c&c infrastructure - you can't defend yourself from
| such attacks by blanket banning Chinese, Russian, Iranian and
| North Korean IP addresses (which I honestly advise everyone to
| do if they don't have business in that country), and you can't
| easily block outbound to Cloudflare either as half the Internet
| is hiding behind them.
|
| Basically, cloudflared lowers the price and effort for
| attackers _dramatically_ , while the effort to defend against
| this threat model has now risen significantly.
|
| If Cloudflare were to actually be willing to do something
| against being used by scammers, they'd put the ingress IPs for
| the C&C infrastructure on dedicated IP ranges and publish these
| in a machine-readable format so every reasonable person that
| does not use Cloudflare tunnels can ban them.
|
| (Side note: "zero trust" needs to die)
| josephcsible wrote:
| > If Cloudflare were to actually be willing to do something
| against being used by scammers, they'd put the ingress IPs
| for the C&C infrastructure on dedicated IP ranges and publish
| these in a machine-readable format so every reasonable person
| that does not use Cloudflare tunnels can ban them.
|
| I don't like the idea of making it easier to block certain
| services, because it goes both ways: it'd also be easier for
| bad guys to block good guys from using said services.
| eigenmeister wrote:
| I'm not seeing how the GP's idea could be used by bad guys
| to block access to services - it'd be something you could
| add to your firewall, if you don't use this service, to
| prevent it's use in gaining persistence.
|
| If the bad guys can modify your firewall config, you
| already have a problem.
| mschuster91 wrote:
| They're meaning governments like China's or Russia's with
| "bad guys" here, and he has a point there.
| theamk wrote:
| This kinda surprises me, what is so "dramatic" about
| cloudfared specifically? This seems just like another reverse
| tunnel tool, and there are plenty of them.
|
| I am not working with malware specifically, but in the past
| I've used ssh tunnels, random one-off websocket thingy we
| wrote, wireguard tunnel, frp proxy, and even AWS SSM agent to
| get access to machines with all incoming connections blocked.
| They are pretty simple to setup and generally cannot be
| blocked with whitelist block already.
|
| (and I bet that for malware, they are worse than cloudfared.
| Based on CF's reputation, they take security reports
| seriously, so I would not be surprised if they take down
| malicious tunnels fast. While random VM on low-cost hosters
| will probably takes days to respond.)
| mschuster91 wrote:
| > This seems just like another reverse tunnel tool, and
| there are plenty of them.
|
| These are known though, you can block them without causing
| issues.
|
| > and even AWS SSM agent to get access to machines with all
| incoming connections blocked
|
| SSM is awesome, but the ways it works... I have no idea
| how. I _think_ though that it uses outbound connections,
| but unfortunately AWS SGs can 't do deny rules.
| sophacles wrote:
| "Winning" and "Losing" are not a great framework for this sort
| of analysis. In that binary framework, there's no difference
| between "1 bit of sensitive data was leaked" and "the medical
| and financial records of every person ever, nuclear launch
| codes, and the backdoor that causes every nuclear reactor to go
| critical simultaneously were leaked" - both are just "losing".
|
| This is why various compartmentalization techniques exist
| throughout the stack - principle of least privilege, network
| segmentation, acls, and so on. If something is compromised,
| limit the "blast radius" of what can be done with that
| compromise.
|
| The other part is containment. By analogy alarm systems and
| guards. If the alarm system goes off and says an intruder is in
| the building, well by the "winning and losing" framework, the
| defender has already lost. (Of course by the binary framework,
| you don't even need an internal alarm system - once the breach
| occured you've lost so why bother?). Or you could try to
| contain the problem and send guards to stop the intruder before
| they do too much damage.
|
| This is where the cloudflared usage described in the article is
| problematic. It's a tool that is less likely to trip alarm
| systems, and further can provide a wide range of access while
| doing so[1]. If it doesn't trip alarms, a small breach leaking
| a little data can turn into a big breach leaking lots of data.
| It absolutely should be considered when designing networks,
| security methods, etc.
|
| I don't know what the solution here is, that probably depends
| on the individual/org doing the threat assessment, and probably
| varies by environment within that domain. Should the tool be
| blanket banned? Probably not it's also a really useful tool for
| a lot of people in a a lot of situations. Maybe cloudflare can
| split the tool in to individual binaries for functionality and
| have the tool called cloudflared just be a wrapper. Maybe they
| can change it some other way.
|
| [1] this tool is used for a lot of stuff and may already be on
| the systems in question, so it's not even installing a program
| -- just execing it. Since it is used for a lot of different
| things, it may just be on a blanket "allow this app" list. And
| so on.
___________________________________________________________________
(page generated 2023-08-09 23:02 UTC)