[HN Gopher] Endlessh-go: a Golang SSH tarpit that traps bots/sca...
___________________________________________________________________
Endlessh-go: a Golang SSH tarpit that traps bots/scanners
Author : fastily
Score : 152 points
Date : 2024-03-28 06:23 UTC (16 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| schlonger0009 wrote:
| Does it matter, though? You can easily scan out the correct SSH
| port.
| nilsherzig wrote:
| Scanning all 65k ports takes time. Those aren't targeted
| attacks, just bots connecting to every 22 ports they can find
| nilsherzig wrote:
| Scanning all 65k ports takes time. Those aren't targeted
| attacks, just bots connecting to every 22 ports they can find
| dugite-code wrote:
| Depends on how well programed the bot is I guess. Personaly I
| use the encrypted packet port knocking package fwknop on my
| home server to hide the ssh port until I need it.
| scsh wrote:
| The point of this isn't to hide your actual SSH service, but to
| tie up resources for those who are somewhat blindly
| scanning/connecting to any open SSH port.
| bayindirh wrote:
| You can setup a VPN (or head/tailscale) and confine your "real"
| sshd there, and leave one of these tarpits in the open for fun
| and profit.
| Svip wrote:
| The original endlessh hints at this, but doesn't go further into
| details, and the endlessh-go's README doesn't mention it at all.
| Am I suppose to have endlessh run on port 22 and then have my
| real SSH server run on an obscure port? In none of the examples
| does it run on port 22. I feel like I'm missing something
| obvious, that the READMEs simply take for granted I know.
| nilsherzig wrote:
| That's how I have it setup
| agilob wrote:
| Yes, check this distro
| https://hub.docker.com/r/linuxserver/endlessh
| nubinetwork wrote:
| I don't think it matters, ssh bots will try any port that sends
| back the ssh banner.
| freedomben wrote:
| The more targeted/sophisticated ones will, but there's a
| crapload of bots that just scan all publicly addressable IPs
| for port 22 and attempt to connect. If your goal is to trap
| as many bots as possible in the tarpit, you'll get a lot more
| if you run on port 22.
| skrause wrote:
| But they don't scan every port. I've been running my SSH
| server on a non-standard port for a long time, it took four
| years until I had the first bot with login attempts. About a
| year ago I changed the port and haven't seen any bots since
| then.
| qwertox wrote:
| You can surround your custom port by a couple of ports on
| which a simple server listens for connection attempts. Any
| connection attempt is considered hostile and the ip will
| then be blacklisted in iptables. This prevents portscans
| from reaching your port.
| metadat wrote:
| Only works for sequential scans, most scanners are more
| targeted towards specific services.
| AlecSchueler wrote:
| If they're targetting SSH specifically how are they going
| to guess i'm running it on port 1690 and not port 22
| other than by scanning up in sequence?
| kevindamm wrote:
| Different quality of locks in the ever-escalating arms
| race. Probably there are many many more sequential
| scanners out there. For the persistent actors who are
| doing random ordering or shuffle then you could add port-
| knocking for the real sshd... but then they just have to
| find a working client and sniff the connection
| requests... to which you add a TOTP step for determining
| which ports to use, and so on...
| dambi0 wrote:
| There is a known upper bound they could randomise the
| guesses from the range.
| yard2010 wrote:
| Excuse the old school metaphor - you put a lock on your
| door so your house is harder to break into, not to
| prevent anyone from breaking into your house.
| metadat wrote:
| Absolutely agree, when I wrote this I was thinking more
| of defending against the low hanging fruit - mass
| scanners.
|
| Once someone has deemed you a worthwhile target and is
| carefully proving all ports, these more nuanced
| approaches become more worthwhile. Even then, a
| sophisticated adversary may have many unique src IPs at
| their disposal.
| michaelcampbell wrote:
| That's the theory. I have an internet facing box using ssh on
| a weird port with fail2ban on it just in case.
|
| In over 10 years I've never had a single probe on that port
| with ssh.
| AlecSchueler wrote:
| Same, I went from logging thousands of attempts per day to
| zero per decade with a simple switch of ports.
| michaelcampbell wrote:
| They CAN, but they don't.
| ycombinatorrio wrote:
| I run endlessh on the port 2222 and I configured fail2ban to
| redirect the source ip addresses who did X failed attempts from
| the dest port 22 to the dest port 2222 transparently. I use the
| table NAT and prerouting to achieve that, you can use ipset to
| match the source ip addresses.
| withinboredom wrote:
| I do something similar except send them bytes from
| /dev/random, providing free protocol fuzzing.
| pdimitar wrote:
| Oh nice, do you have a blog post detailing it step by step?
| dylan604 wrote:
| Isn't the point of a honeypot that it's not a real server? What
| guarantees are there that there won't be an exploit that allows
| escaping the honeypot into the real data? Personally, I do not
| believe anything is 100% secure. So inviting the vampire into
| your facade home, and then getting upset when the vampire sees
| the charade and walks into your real home is just one of those
| "well of course that happened" situations.
| rolph wrote:
| if you use port knocking, the first hit on your honeypot, can
| be the trigger to lockdown or redirect, a lot of other ports
| to somewhere away from your actual.
| gnfargbl wrote:
| Golang works well for this application because it can easily cope
| with very large numbers of idle goroutines.
|
| What the author may be missing is that golang also works well for
| bots and scanners, for exactly the same reason. Attackers' time
| isn't being "wasted" by this, their goroutines are just sitting
| idle for longer.
| da768 wrote:
| Though for a large amount of HTTP bots, the authors don't even
| bother changing the default Python User-Agent. I'd assume a
| large proportion of these bots still can't run concurrently.
| aesh2Xa1 wrote:
| I think it still works. You have two scenarios where the
| attacker is efficiently using goroutines; (1) you also use
| goroutines or (2) you do not. In the latter, the attack is more
| expensive for you.
|
| Another detail is that an attacker with many idle connections
| to your host might not instantiate any new ones.
|
| Of course, in the scenarios where the attacker is not using
| goroutines then you have the upper hand as well.
| gnfargbl wrote:
| This isn't a real ssh server, so the "cost" to you of the
| attack isn't really relevant. You can choose not to run this
| software at all, and the additional cost to you is zero.
| daghamm wrote:
| But the bots can easily detect these, cant they? As long as there
| is a timeout on socket read, this shouldn't waste that much of
| the scanners time.
|
| Or am I misunderstanding this?
| c0wb0yc0d3r wrote:
| Yes a bot can, and sophisticated bots do.
|
| At the same time it's much easier to write code that just died
| the bare minimum. Imagine you're a bot herder, if your bot net
| consists of stolen CPU cycles what difference does it make if
| your bots are slowed down. It doesn't cost you money.
| throw10920 wrote:
| > if your bot net consists of stolen CPU cycles what
| difference does it make if your bots are slowed down. It
| doesn't cost you money.
|
| This is wrong. It _does_ cost you money - either directly,
| because you paid money to use someone else 's botnet, or as
| an opportunity cost, in that you can't use your bots on as
| many targets.
| LysPJ wrote:
| Endlessh periodically sends data so the read timeout won't
| trigger. Specifically, it draws out the crypto negotiation
| stage indefinitely by exploiting a feature of the SSH protocol.
|
| (Of course, the bot author could detect that behaviour too.)
|
| There's more info from the author of Endlessh:
| https://nullprogram.com/blog/2019/03/22/
| gnfargbl wrote:
| You're understanding perfectly. The way this works is that it
| sends a slow drip of junk before the SSH version banner string.
| A scanner running at any real scale is going to have an overall
| timeout beyond which it doesn't bother waiting any longer for
| the banner string.
|
| This is going to _very slightly irritate_ some of the extremely
| low-level actors. Is setting up a tool to do that a good use of
| time?
|
| If you want to effectively deter attackers using a sand-trap
| approach, you need to find some kind of task with asymmetric
| cost in your favour. This isn't that.
| eddd-ddde wrote:
| What is a good example of assymetrical cost in your favour?
| gnfargbl wrote:
| CAPTCHAs are an example. Although I am not sure if they're
| a _good_ example.
| arsome wrote:
| You could probably achieve better here by providing fake
| weak credentials and then getting an actual human to
| connect and check out the honeypot on as many IPs as
| possible.
| HumblyTossed wrote:
| > Unfortunately the wonderful original C implementation of
| endlessh only provides text based log, but I do not like the
| solution that writes extra scripts to parse the log outputs, then
| exports the results to a dashboard, because it would introduce
| extra layers in my current setup and it would depend on the
| format of the text log file rather than some structured data.
| Thus I create this golang implementation of endlessh to export
| Prometheus metrics and a Grafana dashboard to visualize them.
|
| " I didn't like the logging, so I re-implemented the entire
| thing."
|
| I'm not mocking, I just see this often (and have done it
| myself!). It's interesting the things we do to get around the
| little things we don't like.
| ffsm8 wrote:
| The thing about coders is that they're often creators at heart.
|
| The stated reason is likely only the excuse they told
| themselves to justify the project. But the real reason was
| likely that they wanted to create something, and this was a
| good justification.
|
| Might just be me projecting though, because I do that all the
| time
| yard2010 wrote:
| I've stopped lying to myself, I'm making cool stuff just for
| the sake of it. It's like playing a video game, it doesn't
| have to be productive if I enjoy it.
| ajsnigrutin wrote:
| > " I didn't like the logging, so I re-implemented the entire
| thing."
|
| And did that in the "language of the week" :)
|
| The stuff that was reinvented in eg. ruby a few years ago is
| now reinvented in go and rust.
| pantsforbirds wrote:
| People build in languages they are comfortable with. Golang
| is fairly easy to ship with and the performance is fine for
| something like this. Seems like a good tool for the job
| 0cf8612b2e1e wrote:
| In fact, Go is one of the better tools for the job given
| its ability to easily spin up network connections.
| rijoja wrote:
| > It's interesting the things we do to get around the little
| things we don't like.
|
| Yeah and perhaps pick up valuable skills, that might help us
| down the road in ways that are hard to quantify.
| ok123456 wrote:
| Following the SSH hardening guide stops 99% of bots and scanners
| because they can't negotiate a cipher using whatever ancient ones
| their SSH implementation is set up to use.
| LinuxBender wrote:
| This, and a handful of simple firewall rules in the raw table
| can block about 90%+ of that remaining 1% just looking at the
| _spoofable_ banner that none of the bots seem to spoof I assume
| due to being lazy like me.
|
| _In the raw table:_ -A PREROUTING -i eth0 -p
| tcp -m tcp --dport 22 -d [my server ip] -m string --string
| "SSH-2.0-libssh" --algo bm --from 10 --to 60 -j DROP -A
| PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip]
| -m string --string "SSH-2.0-Go" --algo bm --from 10 --to 60 -j
| DROP -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d
| [my server ip] -m string --string "SSH-2.0-JSCH" --algo bm
| --from 10 --to 60 -j DROP -A PREROUTING -i eth0 -p tcp
| -m tcp --dport 22 -d [my server ip] -m string --string
| "SSH-2.0-Gany" --algo bm --from 10 --to 60 -j DROP -A
| PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip]
| -m string --string "ZGrab" --algo bm --from 10 --to 60 -j DROP
| -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server
| ip] -m string --string "MGLNDD" --algo bm --from 10 --to 60 -j
| DROP -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d
| [my server ip] -m string --string "amiko" --algo bm --from 10
| --to 60 -j DROP
|
| _Adding the server IP minimizes risks of also blocking outbound
| connections as raw is stateless_
|
| I rarely do this any more given they rotate through so many LTE
| IP's. Instead I get the bot operators to block me by leaving
| SSH on port 22 and then giving them a really long
| VersionAdendum that seems to get the bots feeling _broken,
| sticky and confused_. There are far fewer SSH bot operators
| than it appears. They will still show up in the logs but that
| can be filtered out using drop patterns in rsyslog.
| VersionAddendum " just put in a really long sentence in
| sshd_config that is at least 320 characters or more"
|
| Try it out on a test box that you have console access to just
| in case your client is old enough to choke on it. Optionally
| use offensive words for the bots that log things to public
| websites. Only do this on your hobby nodes, not corporate owned
| nodes unless legal is cool with it, _in writing_.
| Snawoot wrote:
| Why use PREROUTING chain? You could have achieved same with
| INPUT chain without specification of ingress interface and
| server IP address.
| LinuxBender wrote:
| Anything I explicitly drop I do so in the raw table to keep
| them out of the state table. The state table is more CPU
| expensive _especially at high packet rates_ and runs the
| risk of depleting the default state table limits especially
| for anything that now has a broken state on purpose _like
| these poor lil bots_. Since I brought it up, here is how to
| increase the state table limits.
|
| Create /etc/modprobe.d/nf_conntrack.conf
| cat /etc/modprobe.d/nf_conntrack.conf options
| nf_conntrack expect_hashsize=256400 hashsize=256400
|
| And then in /etc/sysctl.conf: # from
| /etc/sysctl.conf: increase state table limits. #
| Requires 1/4 mem to hash table plus 400 overhead because I
| am the cargo culting king: # cat
| /etc/modprobe.d/nf_conntrack.conf # options
| nf_conntrack expect_hashsize=256400 hashsize=256400
| net.nf_conntrack_max = 1024000
|
| _Should people use default state table memory allocations
| on a busy node, everyone can be locked out of it regardless
| of how many TB of RAM are free. The node can appear
| "down"._
| myself248 wrote:
| include EICAR.TXT ;)
| nubinetwork wrote:
| I don't know if this is still the case, but -m string used to
| be resource intensive, because it has to parse each packet
| for the string before passing it on to other rules.
| LinuxBender wrote:
| It can be. This this case however it is limited to eth0,
| tcp, port 22. If any of those don't match there will be no
| parsing and thus no impact. Another mitigating factor is
| that we are only looking at specific byte regions of the
| packet so parsing is minimized. On busy SFTP servers I
| would probably avoid using such rules if CPU load is
| becoming a problem. For most people this will not even
| register in htop or vmstat. There are also ways to use this
| string check in combination with ipset and/or xt_recent to
| minimize the times we see a packet from a bot. Here is an
| example using an IPSet called "bots" that we drop early on
| in the raw table and also use in the filter outbound rules
| to reset openssh trying to respond the first time we see
| the bad string so we close the socket earlier.
|
| In a startup / init script / systemd unit file:
| # IPv4 ipset flush bots 2>/dev/null ipset
| create bots hash:ip hashsize 2048 maxelem 65536 timeout
| 604800 netmask 24 2>/dev/null # IPv6
| ipset flush bots6 2>/dev/null ipset create bots6
| hash:ip hashsize 2048 maxelem 65536 timeout 604800 netmask
| 64 family inet6 2>/dev/null
|
| _In this example I am using a bigger netmask much in the
| way name servers rrl rate limit._
|
| In the raw table, drop bots we saw for a week:
| -A PREROUTING -i eth0 -p tcp -m tcp --dport 22:80 -d
| [server ip] -m set --match-set bots src,dst -j DROP
| -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [server
| ip] -m string --string "SSH-2.0-libssh" --algo bm --from 10
| --to 60 -j SET --add-set bots src --exist --timeout 604800
|
| In the filter table outbound rules: -A
| OUTPUT -o eth0 -p tcp -m tcp --sport 22 -m set --match-set
| bots dst -j REJECT --reject-with tcp-reset
|
| _This should only be performed on servers that one has
| console / out-of-band access to, after exhaustive testing._
| sandos wrote:
| I think you could employ the same tactics that advanced fuzzers
| do with these tarpits: then mutate the responses randomly, to try
| get "new" responses from the attackers, instead of new coverage
| in the code as in the fuzzer. Unless they are using static
| scripts, which would be boring.
|
| I have understood that most attacks are super-simple sort of, so
| probably not much to learn there. But an interesting project!
| INTPenis wrote:
| Funny but my first thought wasn't wasting their time at all,
| that's easily fixed with a few code adjustments on their client
| end. My thought was to harvest their IPs and publish them in
| blocklists.
| rijoja wrote:
| > My thought was to harvest their IPs and publish them in
| blocklists.
|
| Please do, it would mean good karma.
| lez wrote:
| For other usecases there is an ipfilter target TARPIT, that does
| a similar thing on the TCP level.
| hwbunny wrote:
| But why? Just hide your ssh port. And port knock in. Or put it @
| tor/wg/whatever.
___________________________________________________________________
(page generated 2024-03-28 23:01 UTC)