[HN Gopher] AS13335 doing SSH scanning
___________________________________________________________________
AS13335 doing SSH scanning
Author : jimsi
Score : 148 points
Date : 2021-09-25 08:45 UTC (14 hours ago)
(HTM) web link (encryp.ch)
(TXT) w3m dump (encryp.ch)
| zinekeller wrote:
| Update: they apparently also have a paid product called
| "Cloudflare for Teams", which also uses the same infrastructure
| and are targeted for companies. Probably the reason they don't
| block out SSH is that companies do want SSH access - but I'm
| pretty sure that RDP access would be higher on the list.
|
| My friend working in an ISP has cleared the mystery for me - it's
| Cloudflare's Warp product (a semi-VPN (in the "teleport to other
| countries" sense, not the original definition)), which can be
| used for free, and I'm not shocked that it is being abused.
| Cloudflare has also categorically denied that their Worker
| product allows SSH connections - it is simply not designed for
| that, apparently.
|
| So I don't know how to respond to this specifically except that I
| didn't know Cloudflare offers a VPN (apparently a limited
| knocked-down one though) and I'm pretty sure that other people in
| other ASes scan the default SSH port - in fact, I see more and
| more scans from ASes of cloud providers than residential ASes,
| whereas five years ago I mainly saw Chinese and Brazilian
| residential ISPs. This is why I often move the SSH port - it's
| not that it improves my security posture, just so that the noise
| in the logs are minimised.
| TechBro8615 wrote:
| FYI the WARP VPN is _not anonymizing_ and doesn't claim to be.
| If you connect through WARP to a website behind Cloudflare,
| they will include your real IP in a header.
|
| The main point of WARP is to circumvent ISP traffic shaping
| (which works quite well btw).
| userbinator wrote:
| _a semi-VPN (in the "teleport to other countries" sense, not
| the original definition)_
|
| It's weird how the "VPN providers" have taken that acronym and
| turned it into what is basically another synonym for "proxy".
| kentonv wrote:
| > Cloudflare has also categorically denied that their Worker
| product allows SSH connections - it is simply not designed for
| that, apparently.
|
| Hi, I'm the tech lead of Workers, so I'll clarify.
|
| At present, Workers can only generate HTTP traffic, because the
| only API we've given Workers for network communications is
| `fetch()`, which is HTTP-specific. This similar to how in-
| browser JavaScript today cannot open arbitrary TCP connections,
| because there's no API for it. It's likely we'll add an API for
| arbitrary TCP eventually, but at the moment it's not possible
| that this traffic comes from Workers.
|
| (As far as where it _did_ come from, I don 't have any internal
| knowledge, but WARP traffic (that's our VPN-like service) seems
| like an obvious explanation to me.)
| tyingq wrote:
| His dump of packets shows an MSS of 1380 versus the more
| common default 1460, which sounds like it's coming from
| something with tunneled traffic. MSS 1380 would correspond to
| a source MTU of 1420, which is a typical default for a GRE
| tunnel in IPv4sec Tunnel mode. Pretty sure that's what WARP
| is.
| jpgvm wrote:
| WARP is Wireguard but yeah, otherwise this is pretty much
| on point. Traffic is tunneled, likely just a random WARP
| user abusing Cloudflares free VPN service to do some
| scanning.
| tyingq wrote:
| Ah, okay. Google searches suggests the default MTU on
| Wireguard is also 1420, which would produce an MSS of
| 1380. So that makes sense.
| specto wrote:
| Yeah lots of baddies using workers. I blocked cloudflare a long
| time ago because of workers.
| [deleted]
| cuu508 wrote:
| In case Cloudflare monitors the keyword "Cloudflare" in HN
| comments, I'll mention it here - Cloudflare.
| jimsi wrote:
| Previously this post was named in such way, but post get
| flagged... So I removed its mentioning and replaced with ASN
| johnklos wrote:
| I'm beginning to suspect that in addition to Cloudflare fanbois
| who'll downvote anything that paints Cloudflare in a bad light,
| Cloudflare may even have bots of their own. There're just too
| many downvoters to be explained by simple fanbois.
|
| I get that some companies have fans, and I get that some of
| those fans become fanbois - fans regardless of the truth, with
| a ridiculous us-versus-them mentality. But I seriously doubt
| there are enough to explain what we see here on HN.
|
| It'd be nice if Cloudflare weren't such assholes. They refuse
| to answer questions directly, they intentionally use form
| responses to be vague and uncommunicative when it suits them,
| and they clearly don't care about doing the right thing. They
| COULD be a much better company if they just communicated
| without being smug assholes.
|
| Even a straightforward answer that isn't what we want to hear,
| like, "These are our VPN endpoints, and we have no plans to do
| anything about VPN customers who are attempting to brute force
| ssh." would still make it easy for us to respect Cloudflare.
| Instead, they facilitate abuse, ignore abuse complaints, and
| refuse to communicate, which, at least in my opinion, makes
| them assholes.
| evgen wrote:
| Cloudflare is all over HN and their CTO, jgrahamc, will
| frequently appear to answer questions. California is just
| waking up so I expect a few ho it a before we get some solid
| answers to this issue.
| cpach wrote:
| As others have pointed out, disabling password-based logins for
| sshd is a must.
|
| You can also go further if you like: Put the sshd service behind
| WireGuard (or spiped). Then malicious actors will never find your
| SSH ports.
| [deleted]
| nickdothutton wrote:
| Just use something like fail2ban to drop these at the packet
| filter level, either courtesy of your cloud provider or on the
| host itself. Make sure your sshd config is watertight.
| loeg wrote:
| You know, or just don't use fail2ban:
| https://research.securitum.com/fail2ban-remote-code-executio...
| . It's adding extra attack surface for a cosmetic benefit.
| tempodox wrote:
| Good catch, thanks!
|
| Remedy: Don't let fail2ban send mail, or at least remove the
| whois part.
| megous wrote:
| nftables allows to automatize blocking of IP addresses (even
| with a automatic timeout) without any userspace involvement.
|
| Pretty nifty feature.
| nousermane wrote:
| Are there reputable places where people can share the IPs
| tripping fail2ban? Like spamhaus.org, but for scanners?
| giantrobot wrote:
| So...drop 8.0.0.0/8 traffic and move on? Port scans happen all
| the time, some are malicious and others not.
| kristianpaul wrote:
| I run OSSEC that comes with automatic responses for common SSH
| authentication events
| vadfa wrote:
| Cloudflare has a VPN product, WARP. It's possible that they are
| clients of theirs doing the scanning.
|
| In any case I wouldn't think much of it. If you obsess over any
| garbage traffic you get you will go insane.
| jimsi wrote:
| Maybe that's a case, but their abuse team hasn't replied
| anything in a 2 weeks about that after I gave them all
| timestamps and both source/destination IP addresses
| vadfa wrote:
| They won't disconnect clients over SSH scanning. That's a
| ridiculous expectation. It's 2021. Nobody cares.
| brohee wrote:
| What kind of answer would you expect, in all seriousness? The
| thing you are reporting is not illegal or even dodgy.
| marginalia_nu wrote:
| VPS and VPN providers should be very mindful of their
| reputation in this regard. If they get a reputation as a
| "black hole" where complaints vanish and nothing ever
| happens, the effect may be that other customers start to
| find themselves blocked or throttled on a subnet level.
| TechBro8615 wrote:
| I don't think anyone is about to block or throttle
| traffic from Cloudflare IP ranges.
| marginalia_nu wrote:
| Incoming connections? Why not. I have at times, it's
| pretty great.
| eli wrote:
| I don't know of any provider that would take action on
| reports of ssh scanning.
| TechBro8615 wrote:
| A little bird told me a story that AWS will forward abuse
| reports to customers performing outbound nmap scans.
| jimsi wrote:
| I expect to hear who (and why) generates that kind of
| traffic from cloudflare owned subnets.
| tux3 wrote:
| I can understand your frustration with background
| internet noise, but please note Cloudflare is not known
| for broadcasting their customers' names to the first
| abuse report with a pcap of a TCP handshake.
|
| There may be more realistic ways to go about protecting
| people's SSH servers that trying to dox Cloudflare VPN
| users.
| that_guy_iain wrote:
| I can't understand it. There doesn't appear to be any
| downside or even abuse happening. The fact OP expects a
| company to explain who and why a customer of theirs did a
| legal non abusive act is just an outstanding level of
| entitlement.
| Hrundi wrote:
| Who's trying to dox those users?
| tux3 wrote:
| I assumed OP wants to know the identity of the Cloudflare
| users scanning their SSH ports.
|
| I think OP guessed it was probably not Cloudflare
| themselves scanning their ports, so I think that's what
| they meant by "hear who and why".
|
| Maybe dox is too strong a word. My point is, from what
| I've heard, the general sentiment is that you're unlikely
| to get any information about customers just by sending
| abuse reports to Cloudflare.
| chmike wrote:
| OP obviously simply expected at least an explanation on
| the cause of these ssh connection probes. He got the
| explanation here (VPN).
| [deleted]
| miyuru wrote:
| If your ISP and the server support IPv6, just disable SSH on
| IPv4.
|
| Some of my servers don't even have any IPv4 connectivity and
| there haven't been any failed SSH logins over IPv6.
| jimsi wrote:
| My OpenSSH is located on a non standard port, 22/tcp is
| going to the endlessh honeypot.
| zinekeller wrote:
| Not to disappoint you, but except for logging SSH
| honeypots are becoming useless (most bots automatically
| disconnect when they detect a long login banner).
| sudobash1 wrote:
| So should I add a long banner to my server to disguise it
| as a honey pot, just in case?
| throw0101a wrote:
| > _endlessh honeypot._
|
| *tarpit
|
| A honeypot lets people "in" to see/research malware
| that's in the wild:
|
| * https://en.wikipedia.org/wiki/Honeypot_(computing)
|
| A tarpit just takes up the attacker's resources:
|
| * https://en.wikipedia.org/wiki/Tarpit_(networking)
| judge2020 wrote:
| Not what most people run, but SSH honeypots are also
| useful:
|
| https://lwn.net/Articles/848291/
| jagger27 wrote:
| I like this solution a lot!
| mcwhy wrote:
| It could also be used for IP reputation, running ssh can be a
| sign of a proxy or other server/service that website owners
| might want to block.
| [deleted]
| [deleted]
| jimsi wrote:
| I think single SYN packet would be enough to know whether
| this port open or not, but they trying to gather full banner.
| sowhatw wrote:
| Who cares? Get a better hobby.
| LinuxBender wrote:
| This is not unique to Cloudflare, Level3 or any of the other
| networks I've seen mentioned in this thread. The entire internet
| scans all the common ports and this should be expected to
| continue indefinitely. If this is causing alerts from your
| logging, there are a few options:
|
| - Implement a firewall on your instances.
|
| - Implement port knocking for sshd.
|
| - Move sshd to a non standard port to avoid the nmap/bot noise.
|
| - Only log successful logins.
|
| - Any combination of the above.
|
| There are pros and cons to each item so you would have to decide
| which combination is the most appropriate and least friction for
| your organization. If leaving the port exposed consider enforcing
| key based authentication and disabling password authentication.
| For high risk accounts such as monitoring accounts that use
| passwordless unrestricted sudo, you can even restrict what
| networks the keys are valid from.
| akira2501 wrote:
| I've been happy with having my firewall count SSH connection
| attempts, and if you make too many attempts in too short a
| period of time, it just blacklists that IP for 24 hours.
| silisili wrote:
| My first 'defense' was just moving off of port 22. I had
| planned other things, but to be honest, the attempts went
| from hundreds a day to zero, so I never bothered doing more.
| t0mas88 wrote:
| I do the same, but only 10 minutes after 5 failed logins.
| That's enough for bots while not really getting in your way
| if it accidentally blocks the hotel you're in.
|
| Combined with only allowing key based login, password is
| disabled.
| _moof wrote:
| I've recently added 2FA as well. Super easy to do.
| alias_neo wrote:
| I wrote a blog on how to do it a couple years back if
| anyone is interested in a "tutorial":
| https://2byt.es/post/totp/
|
| It's for the Pi as that's the target audience but it
| should apply generally for OpenSSH.
|
| There's also a sister post about improving your "first
| factor" for those still using passwords:
| https://2byt.es/post/totp2/
| bayindirh wrote:
| The first package I install on public facing servers is
| fail2ban.
|
| After starting it, everything becomes much calmer. Nice thing
| is, fail2ban can protect much more than SSH.
| airocker wrote:
| My experience was similar. I opened a public SSH once and there
| were many unsuccessful login attempts. The way we resolved it
| was by installing VPN and closing the IP for public access.
| seized wrote:
| Or Fail2Ban. I have mine setup so that it does an API call to
| my OPNSense firewall. A few wrong logins to my self hosted
| things (SSH included) and that IP is permanently blocked from
| everything.
| hacker_newz wrote:
| If you expose sshd to the internet in the first place you are
| doing something wrong.
| jftuga wrote:
| https://github.com/jftuga/gofwd
|
| I created gofwd, a cross-platform TCP port forwarder with Duo
| 2FA and Geographic IP integration. Its use case is to help
| protect services when using a VPN is not possible. While it's
| use cases are limited, it works great for me when I need to
| connect to my home network.
|
| The overall elegance of this solution is that no additional
| software is needed. As long as you are within your predefined
| geo-ip location, have your phone, and know your hostname/ip
| address (and port number), then you will be able to access your
| system remotely.
| rollcat wrote:
| Personally, I just whitelist SSH access on the firewall (UFW
| and/or AWS SecGrps). Everyone on the admin team has a static
| residential address. On the road, we use ZeroTier.
| vgb2k18 wrote:
| Adding: [-] whitelist allowed ip's for port 22. Requires static
| ip's client-side obviously [-] fail2ban if static ip's are not
| available.
| codexon wrote:
| > - Move sshd to a non standard port to avoid the nmap/bot
| noise.
|
| There are scanners like shodan that will scan every single port
| you have now so moving it to a non-standard port doesn't stop
| all the attackers.
| jms703 wrote:
| The parent post didn't say to do this to stop attackers. They
| said to do this to reduce alerts in the logs.
| LinuxBender wrote:
| Shodan scans a handful of common ports [1] but you are right,
| a targeted attack will not be stopped by moving ports. Along
| the lines of what jms703 mentioned, moving ports just removes
| the noise from the bots so you can have better alerting on
| real attacks and not suffer from alert fatigue.
|
| [1] - https://en.wikipedia.org/wiki/Shodan_(website)
| twodai wrote:
| My favorite is to implement a tarpit while moving the ssh port.
| rualca wrote:
| > My favorite is to implement a tarpit while moving the ssh
| port.
|
| For those who, like me, are unfamiliar with the concept of a
| SSH tarpit, it's a technique consisting of inserting delays
| into SSH connection attempts.
|
| Old HN discussion on Endlessh:
|
| https://news.ycombinator.com/item?id=24491453
| hda111 wrote:
| Isn't this easily detectable for any malicious script? Just
| a small timeout and try the next port.
| themulticaster wrote:
| Does that mean that you automatically block IPs that try to
| login on port 22 (since they are obviously malicious/port
| scannners)? If yes, is there any specialized daemon for that
| or are you using firewall rules? Or are you running a SSH
| honeypot on port 22?
| c0wb0yc0d3r wrote:
| The post is referring to this:
| https://github.com/skeeto/endlessh
|
| I'm guessing that the person you replied to is referring to
| the same.
| ben_bai wrote:
| A tarpit is a technique to waste your opponents time and
| network sockets. Usually by accepting the connection but
| only sending or receiving data at like 1byte/s.
| jrockway wrote:
| This is probably Cloudflare Workers?
|
| I don't think that demanding that every ISP inspect and censor
| outgoing packets is ever going to work. Every network operator
| has their own definition of good and bad, and it's probably up to
| that operator to enforce it themselves. Regarding SSH, you can
| basically assume that 4.2 billion IPs will concurrently try every
| password database that exists against your server 24/7. Passwords
| are a failed experiment. People can't remember them, and
| attackers can easily guess them. The solution is: don't accept
| password logins, and ban particular IPs after a certain number of
| failed attempts. (You should do the same for any unauthenticated
| resource -- if someone gets 1000 pages from your website in a
| second, you should probably cut off access to them for a while.
| Some script has gone awry, and it's costing you money, not the
| author of the script.)
|
| The topics that HN discusses can show you the breadth of opinions
| on this issue. One day it's "I hate cloud provider X for blocking
| my abusive service" with 3000 comments agreeing. The next day
| it's "I hate cloud provider Y for not blocking someone's abusive
| service" with a different 3000 commeters agreeing. No consensus
| will ever be reached, and there is no Internet police force you
| can bully into agreeing with you. I would just take precautions
| and move on, rather than appealing to a mob or administrative
| body for some meaningless justice.
| jimsi wrote:
| Probably yes, but hey! Why do Cloudflare Worker would need SSH
| connection establishment? I do not asking Cloudflare to block
| 22/tcp entirely, but all this situation is very odd - I am
| seeing anomaly, reported abuse to them, but no explanation why
| this is happening.
|
| About password authentication I totally agreeing with you, but
| this is a bit out of scope of this thread.
|
| The only purpose of this article is to know the truth what is
| really happening. I have never seen such many connections to
| the SSH even from researchers.
| dapids wrote:
| I think you need to spend more time looking at logs and
| network traffic in general, this is standard. I bet your
| public home IP will be scanned at least a couple times just
| today.
| Symbiote wrote:
| Judging my my home network, they can probably expect an SSH
| connection attempt on a home IP about every 5 seconds or
| so, or 18,000 times a day.
|
| For some reason, this is 6x more than a server I have on an
| university network.
| stefan_ wrote:
| Please name the cloud providers that have a policy of allowing
| malicious actors to do SSH scans? Do you mean Cloudflare?
| judge2020 wrote:
| Let me know if you get banned for doing nmap -p 22 on AWS or
| GCP.
| dleslie wrote:
| They used to block port scanning, back in 2015:
|
| > When unauthorized port scanning is detected by AWS, it is
| stopped and blocked.
|
| Page 13
|
| https://web.archive.org/web/20160331233541/https://d0.awsst
| a...
|
| It's not clear, now.
| tinus_hn wrote:
| Or it's not detected. They're trying to avoid being
| blocked anyway.
| dilyevsky wrote:
| We get obnoxious security cgi scanner from palo alto
| networks tripping alerts all the time coming from gcp
| ranges
| brohee wrote:
| What is the issue exactly? Is there any kind of exploitation
| attempts or is someone fishing for banners to do statistics upon?
| The second case is research, and legitimate...
| _moof wrote:
| Just curious/not furious: is there a convincing argument for
| why one should enable this particular kind of research at the
| cost of potentially diminished security? I recognize that this
| question relies on an assumption that is itself arguable, and
| that it might vary with context.
|
| (Aside: I really wish English had a better way to distinguish
| between an earnest question and a rhetorical challenge.)
| jimsi wrote:
| I have asked Cloudflare about this traffic, but support says
| "Cloudflare helps protect sites, and accelerate them. We do not
| attack sites, and our network can't be used to generate attack
| traffic." They deny any malicious outgoing traffic, but after I
| have provided pcap dump they just ignoring me and do not reply
| anything. Very odd behaviour for researchers.
| r1ch wrote:
| Cloudflare likes to use the "We're just a proxy" defense when
| it suits them, but with the rapid release of other products
| they are very capable of generating arbitrary traffic and
| hosting content without an origin.
| Bluecobra wrote:
| > What is the issue exactly?
|
| The OP has too much time on their hands. :)
|
| You can play this whack-a-mole game indefinitely. If you have
| SSH open to the Internet, expect that others will try to
| connect 24/7.
| sowhatw wrote:
| Why do you think this matters in any way whatsoever? Ports are
| there to be connected to. I don't see the problem.
| gtirloni wrote:
| Is SSH scanning considered "abuse" now?
| chmike wrote:
| Some ssh scanning is obviously abuse. Is it hard to make the
| difference ?
| nickdothutton wrote:
| Port scanning was always considered abuse until researchers
| decided they needed additional blog post material. I worked for
| a "business ISP" in the early 90s and we had no qualms kicking
| someone off the network for it, although usually the originator
| of the scan had themselves been compromised... because... as I
| say it was considered at least a breach of terms of service.
| tptacek wrote:
| I ran engineering for what was at the time the most popular
| ISP in Chicago from 1995-1997 (the company later sold to RCN)
| and port scanning has never been considered abuse, though
| there have always been people who wished it was and loudly
| proclaimed it so.
| annoyingnoob wrote:
| I'd say that port scanning is attack prep. When you port scan
| me I take an adversarial stance. Certainly volume plays a role
| in determining whether scanning is abusive or not.
|
| There are differences between walking by and noticing a door,
| knocking on a door, trying a key in a door, trying to defeat
| the lock. Before you can break the lock you have to notice the
| door. I haven't met any friendly door knockers on the internet.
| tptacek wrote:
| Yes you have. Censys is a good example (you have assuredly
| met Censys, whether you know it or not.)
| loloquwowndueo wrote:
| Just install sshguard or some such and move on with your life. Or
| - block the entire IP range, and see who complains.
| sammy2244 wrote:
| Cloudflare warp is essentially a free vpn, this is going to
| happen. Iirc they use completely separate subnets for this so you
| can filter it/block it. Ip starts with 8. Something
| benjojo12 wrote:
| To the best of my knowledge, the IP blocks in the 8.0.0.0/8 range
| announced by cloudflare are their VPN exit blocks.
|
| I had similar issues with them in the past spewing weird traffic.
| Didn't really ever reach a conclusion on it, other than just
| reducing logging for that error condition a scanner was
| triggering.
| jimsi wrote:
| How did you discover this? Anyway thanks for additional
| information
| joshenders wrote:
| jimsi, employees are also extremely active on Twitter. It's
| not the most scalable but that's usually my goto for eyebrow
| raising Cf related questions.
| aaron42net wrote:
| Enabling Warp via the "1.1.1.1" Android app gets me an
| 8.x.x.x VPN address, at least. This /24 appears to be routed
| to my city's Cloudflare node, so presumably there's a /24 per
| city they run this service in.
|
| Running a quick port scan from my phone against one of my
| machines works, so it doesn't look like they are restricting
| this too heavily.
|
| And I'm not logged into this app and haven't granted it
| additional permissions, so I'm not sure they have any idea
| who I am here.
| benjojo12 wrote:
| via support ticket with cloudflare. They didn't explicitly
| confirm it, but my experience is that they would have
| corrected me if that was wrong :)
___________________________________________________________________
(page generated 2021-09-25 23:01 UTC)