[HN Gopher] I looked through attacks in my access logs
___________________________________________________________________
I looked through attacks in my access logs
Author : thunderbong
Score : 312 points
Date : 2024-01-28 13:50 UTC (9 hours ago)
(HTM) web link (nishtahir.com)
(TXT) w3m dump (nishtahir.com)
| velcrovan wrote:
| I make a point of running fail2ban on my servers and will even
| add custom jails to catch attacks specific to the types of
| functionality I may be exposing on the site(s) hosted on them.
| But it's been a long time since I checked whether fail2ban's
| other defaults are still comprehensive enough to block the most
| common attacks. I guess I'll bookmark this link for when I get
| around to doing that.
| Palomides wrote:
| if your systems have any of these easily targeted
| vulnerabilities exposed, fail2ban won't save you
| velcrovan wrote:
| fair point, the main reason I use fail2ban is to limit
| traffic from malicious activity rather than letting the
| attempts run rampant and unchecked.
| creeble wrote:
| If it makes you feel good, do it. It can also cut down on
| log noise a bit, for when you're really looking for
| something.
|
| But in general, I've given up on caring about the routine
| "attacks" listed in all the logs. If you have good
| security, they don't matter. And if you don't, they don't
| matter either.
| geraldhh wrote:
| caring for pointless attacks is more work and bears more
| risk
| jbverschoor wrote:
| In addition, you can have nginx filters to check for simple
| patterns (php on a non-php site? -> instant ban). Too many
| 404s? -> instant ban.
| mkoryak wrote:
| Sometimes I think it might be fun to setup an express server that
| correctly responds to one of these attacks just so I can waste
| someones time.
|
| But doing that would also waste my time.
| klysm wrote:
| Honeypots are good fun
| azinman2 wrote:
| It's all automated; not you're not really wasting an actual
| persons time.
| pnw wrote:
| It does waste their time if your honeypot is constantly
| responding with legitimate looking but fake credentials.
| Presumably the hacker is going to try to use them?
|
| It's the same idea used by anti-spam activists back in the
| day with software that would flood spam website forms with
| fake but realistic looking info, so the real data would be
| buried in the noise.
| cyberlurker wrote:
| https://en.wikipedia.org/wiki/Tarpit_(networking)
| emmanueloga_ wrote:
| Ah, tarpit refers to a system that purposely slows down
| answers, while honeypot is a system that _looks like_ it's
| delivering the goods but it is just a trap.
|
| I'm sure they mostly refer to the same thing, though.
|
| --
|
| https://en.wikipedia.org/wiki/Honeypot_(computing)
| gnyman wrote:
| It's mostly wasted time but I feel it's slightly more
| beneficial than playing video games (which I also do) so I do
| it for fun sometimes. [1] [2]
|
| [1] https://infosec.exchange/@gnyman/109318464878274206
|
| [2] https://nyman.re/super-simple-ssh-tarpit/
| warkanlock wrote:
| If anyone is receiving these types of logs on AWS, please do
| yourself a favor and place AWS WAF in front of your VPC.
|
| It's not expensive and can significantly help you, saving you
| from many headaches in situations like this. While it might not
| block everything that arrives at your service, it can be a great
| help!
| anamexis wrote:
| This is a good suggestion, but careful with the default
| rulesets. We turned on AWS WAF (in our case, the motivation was
| just SOC 2 compliance).
|
| There were a few overzealous rules that subtly broke parts of
| our app.
|
| There were request body rules that did things like block
| requests that contained "localhost" in the request _body_.
| There was also a rule that blocked requests without a User-
| Agent header, which we were not previously requiring on API
| requests, so we broke our entire API for a few users until we
| figured that out.
| everfrustrated wrote:
| In my experience WAFs are not something that one should ever
| "just turn on".
|
| Complete due diligence is required to fully understand and
| realise the impact of the rules and should be tested like any
| software change by going through a testing phase.
|
| Ideally software teams should be fully trained and be
| responsible for their lifecycle.
| macNchz wrote:
| Yes you need to be familiar with the rulesets being
| applied, and prepared to closely monitor what is being
| blocked. Ideally I think I'd roll it out one ruleset at a
| time to limit the number of potential issues being
| introduced at once.
|
| Had a fun one after turning on the AWS WAF with some
| default rules-a small number of users reported they
| couldn't upload new logo images anymore. Turned out some
| Adobe product was adding XML metadata to the image files,
| which the WAF picked up and blocked.
| marcosdumay wrote:
| Just to add, but only testing will never work well enough
| for something like this.
|
| This is one of the cases where you must understand what you
| are doing. There's no technique for doing it
| mindlessnessly.
| arter4 wrote:
| Agree. There are so many ways WAF rules can unintentionally
| block legitimate traffic. From very long URLs (is that a
| DoS attempt?), to special characters in a POST with a file
| upload (is that = part of a SQL injection attempt or is
| that just part of a base64 encoded file?) and so on.
| nullindividual wrote:
| I use the Azure equivalent of the AWS WAF but I have no
| direct experience with AWS WAF. Azure WAF leverages the OWASP
| ruleset[0] and many of those rules throw false-positives,
| SQL-related rules being one of the top offenders.
|
| As you note, it requires adjustment due to overzealous rules.
| OWASP has Paranoia Levels[1] which allow you to be more
| targeted.
|
| [0] https://github.com/coreruleset/coreruleset
|
| [1] https://coreruleset.org/20211028/working-with-paranoia-
| level...
| bogota wrote:
| It is not that easy as using the AWS WAF with default rules for
| our application led to many valid requests and IPs being
| blocked. You need to know what is being blocked and verify at
| first or you will in some cases be losing customers.
| CubsFan1060 wrote:
| Your best plan is to start with all the rules in count mode.
| Let that sit for awhile and analyze anything that was
| counted. As you feel good about it, slowly start to move
| things into block.
| fabian2k wrote:
| None of the attacks listed in the post would be an issue for
| any kind of modern web application. Why should I add a WAF for
| this?
| remram wrote:
| WAF tends to ban widely, sometimes for dubious reasons. For
| example, researchers at my university study Twitter data, and
| the mere fact of following links from a small random sample of
| tweets means that our university's IPs are blocked by most WAF.
| mac-chaffee wrote:
| In actuality, WAFs hurt more than help. They give a false sense
| of security since they are so easily bypassable, plus they have
| a significant performance cost and a significant chance of
| blocking legitimate traffic:
| https://www.macchaffee.com/blog/2023/wafs/
| SamuelAdams wrote:
| What are some realistic, self hosted mitigation strategies for
| defending against these attacks?
| azeemba wrote:
| Update your software frequently.
| klysm wrote:
| But not too frequently
| Tijdreiziger wrote:
| I'd rather update it too frequently and potentially bork
| something, than not update it frequently enough and
| potentially get pwned.
| mfashby wrote:
| Updates as the other commenter says. Also isolation technology
| like docker containers, chroots, bsd jails, protections that
| systemd offers, or virtual machines. While not perfect, it
| means that the attackers must have the ability to chain
| exploits in order to break out of the compromised application
| to the rest of the host system.
| ggpsv wrote:
| Docker is great but it is easy to shoot yourself on the foot
| if you use it conveniently but don't actually understand it.
|
| A common mistake is to publish the Docker ports unknowingly
| to all interfaces (e.g `5432:5432`), which makes your Docker
| container available to everyone. It is common to see this in
| Docker tutorials or pre-made Docker Compose files. Coupled
| with UFW, it may give you a false sense of security because
| Docker manages its own iptables rules.
| elashri wrote:
| I do make the habit of not expose ports and just use
| reverse proxy for the container. Of course, you will need a
| bridged network between the reverse proxy container and the
| target container, but that's fine. I'm sure there is more
| clever ways around that.
| ggpsv wrote:
| I prefer to run the webserver using systemd on the host
| so publishing the container port to 127.0.0.1 is enough
| for me.
| mfashby wrote:
| Yes I've made this mistake with docker and UFW before :(
| Such a footgun.
| ggpsv wrote:
| Don't expose your services publicly unless it is necessary. If
| you're self-hosting services that are meant to be accessed only
| by you then consider accessing them exclusively over a VPN like
| Wireguard (Tailscale is nice) and firewall everything else.
| yjftsjthsd-h wrote:
| Or, depending on practical constraints, even http simple auth
| via nginx/apache reverse proxy.
| ggpsv wrote:
| Yes, though I'd say _and_ not _or_. Just because you're
| using a VPN it doesn't mean you should drop all forms of
| authentication.
| yjftsjthsd-h wrote:
| Well, I did mean or; sometimes just sticking httpd in
| front of the application with a user:pass over https is
| fine, and also _much_ easier if the client can 't run a
| VPN client or doesn't want to.
| Aachen wrote:
| I am not aware of a vulnerability in popular web server
| software that affected a basic auth login screen for over
| a decade. Assuming you use a proper password and don't
| typo the domain and end up on someone who wants to
| specifically phish you through typo squatting, it's about
| as solid as SSH or Wireguard
|
| They serve different use-cases but I wouldn't say that
| VPN is strictly better than HTTP auth or vice versa.
| Recommending to double up for a self-hosted little
| something, not a big target like 4chan or Gmail, is
| overkill
| ggpsv wrote:
| What I'm saying is that security is about layers. I agree
| with you, a VPN and HTTP auth is not apples to apples
| where one is better than the other.
| achairapart wrote:
| Also, be sure your web server is not serving
| dotfiles/dotfolders (apart for a few legitimate reasons, ie
| .well-known).
| Aachen wrote:
| Why? Code is open source and I don't check config changes
| into .git. Do you mean like .htpasswd which is already
| default disallowed (on web servers that make use of it in the
| first place; I think Nginx doesn't block it by default but
| also doesn't use it so it wouldn't grant any access)?
| wetbaby wrote:
| Nord Meshnet, ZeroTier, Cloudflare Tunnels.
|
| Instead of exposing your applications externally, you create a
| private network that uses UDP hole punching.
|
| This isn't completely self-hosted, as you need some server to
| auth / broadcast connection details with. Self-hosting might be
| possible on ZeroTier, but I'm not familiar enough to say for
| sure.
| tamimio wrote:
| Yeah you can self host zerotier, there's even quick docker
| for it.
| lazyeye wrote:
| Tailscale is ridiculously easy to install and get running.
| ufmace wrote:
| IMO, you shouldn't do anything special. They're all very low-
| skill automated attacks. Just design and deploy stuff well, do
| the basic stuff correctly before you make anything publicly
| accessible, and don't worry about the noise. If you're doing
| things properly, none of it will work. Whatever fancy thing
| anyone suggests to try to reduce the noise likely won't slow
| down any actual determined human attacker much, and will just
| cause you more hassle to deploy and maintain.
| layer8 wrote:
| Use packages from a distribution like Debian and run
| _unattended-upgrades_ or equivalent for the security-updates
| repository. They usually fix newly reported vulnerabilities in
| less than a day.
| cantSpellSober wrote:
| > the user agents for these attacks mention _Mozlila_ /5.0
|
| The linked write up on why is interesting, finds attacks for many
| of the same files, and recommends blocking the user agent.
| woodruffw wrote:
| The author says they aren't a security person, so to correct a
| minor thing: the first examples are credential and configuration
| discovery, not directory traversal. The latter, to the best of my
| knowledge, is reserved for techniques where the attacker
| "escapes" the webroot or otherwise convinces the server to serve
| things outside of its normal directories.
| asynchronous wrote:
| It's technically both if they're performing an include file
| that wasn't supposed to be hosted- ex. "/../../passwd/etc"
| foofie wrote:
| I also check the access logs collected by my self-hosted
| services, and I think there's a detail that's conspicuously
| absent from this analysis: the bulk of these malicious requests
| are made by regular people running plain old security scanners
| that are readily available from sites such as GitHub. These are
| largely unsofisticated attacks that consist of instances of one
| of these projects just hammering a server without paying any
| attention to responses or even if they have been throttled or
| not.
|
| Some attacks don't even target the IP and instead monitor domains
| and its subdomains, and periodically run the same scans from the
| exact same range of IP addresses.
|
| For example, on a previous job we had a recurring scan made over
| and over again from a single static IP address located in Turkey
| that our team started to refer to it as "the Turkish guy", and
| our incident response started featuring a preliminary step to
| identify weird request patterns that was basically checking if it
| was the Turkish guy toying with our services.
| patja wrote:
| Fail2ban doesn't help when the attacker/abuser rotates their IP
| address constantly. I now look for aberrations in a few http
| headers they often neglect to spoof in their attempt to act like
| an honest human.
| boringuser2 wrote:
| These attackers don't have inexhaustible resources.
| logifail wrote:
| Indeed, but neither do you.
|
| Be very careful that fail2ban isn't actually exhausting your
| own resources faster than you believe you're exhausting the
| attackers' resources...
|
| https://www.google.com/search?q=fail2ban+resource+usage
| quesera wrote:
| I've recently logged credential stuffing attacks coming from
| over 100K distinct IPv4 addresses to a small backend API.
| vntok wrote:
| Rate-limit after x failed logins on _either_ source IP,
| username and password. Just provide a realworld sidechannel
| escape hatch for legitimate users (ex: phone or email).
| Barely anyone will actually use it.
| quesera wrote:
| We had to go further. The attacks were from arbitrary
| IPs, and were working through a huge list of leaked
| username/password combinations.
|
| The attacker even went to the trouble of spoofing some
| custom headers in our API client.
|
| Our eventual solution was to a) attack our own auth
| credentials first, identify any users with leaked creds
| (from other services) and force a password reset for
| them. b) disallow users from setting common leaked
| passwords. c) make the auth checking request as low-cost
| as possible, and scalable separately from the main
| application. d) when an attack request is detected,
| bypass the relatively-expensive real cred check but
| return the same failure response (including timing) as a
| real failure. e) build a secondary requirement in the
| auth flow that can be transparently enabled when under
| high volume attack.
|
| This works, so far. It sheds the volume to the
| application, and has low-to-zero impact on legit users.
| This took a couple weeks away from feature development
| though!
| Aachen wrote:
| This interests me a lot as we do security but don't run a
| big service ourselves and so don't have data on what
| motivated attackers' behavior is exactly.
|
| How many active users (to an order of magnitude; no need
| for precise numbers of course) does this service have? 100k
| IPs sounds pretty costly to burn, so I'm curious how
| important one needs to be before that's considered worth
| it.
|
| And could you say what type of IP addresses those were? Did
| it look residential such as from compromises computers
| (botnet), do they rent lots of IP addresses temporarily
| from aws/netcup/alibaba, or is it a mix such that neither
| category has the overwhelming majority?
|
| If it's all server IP ranges for a service where end users
| normally log in, you could apply entirely different rate
| limits to those than to residential IPs, for example. Hence
| I'm wondering how these cred stuffing attacks are set up
| quesera wrote:
| It was a mix. Lots of apparent botnets (residential
| service scattered across many providers and networks),
| but also healthy chunks of colo/SaaS netblocks as well.
|
| And of course a ton of open proxies.
| finnjohnsen2 wrote:
| fail2ban has calmed attacks (ssh dictionary mostly) from
| 100-1000s per day to about a dozen, on my private rpi thing. I
| assume most attackers are looking for low hanging fruit with
| little (cheap) effort.
| lofaszvanitt wrote:
| When will people forget about fail2ban already? It's an old,
| cumbersome, useless tool.
| mianos wrote:
| Bit they don't. The most of the ssh attacks come from the same
| 10 addresses from China Telecom. Sometimes they don't change
| for.months. Maybe it is some country sponsored attack
| organisation or maybe just carrier grade nat.
| azinman2 wrote:
| Why protect the IPs of the those trying to attack you?
| bshacklett wrote:
| Most of them are probably bots running without the knowledge of
| the IP owner. There's little benefit to sharing those IPs with
| anyone other than the provider who owns them.
| nothis wrote:
| Hyper-naive take: Couldn't nearly all of these attacks be blocked
| by a white-list approach, essentially hiding every file or
| directory from the internet except a very controlled list of
| paths and escaping all text sent so it can't contain code?
|
| I somehow always imagine these types of hacks to be more clever,
| like, I dunno, sending harmless-looking stuff that causes the
| program receiving it to crash and send some instructions into
| unprotected parts of RAM or whatever. This all looks like "echo ;
| /bin/cat /etc/passwd" and somehow the server just spitting it
| out. Is that really the state of web security?
| dartos wrote:
| You're probably right, but consider that not every person is
| even aware of the security risks of running servers.
|
| Someone might be trying to play with self hosting or a
| contractor at a company did a bad job and accidentally exposed
| stuff they shouldn't.
|
| This attacker is likely just trolling lots of IPs hoping for
| low hanging fruit that can be exploited with simple/well known
| attacks.
| Zetobal wrote:
| Security through obscurity is like a ninja tiptoeing in a room
| full of laser beams; make one loud move and you'll reveal that
| your entire protocol hinges on no one sneezing!
| akerl_ wrote:
| How is strictly controlling exposed server resources to only
| URIs you've confirmed should be an exposed an example of
| "security through obscurity".
| scherlock wrote:
| Yup, 99.999% are script kiddies running bots that look for
| unsecured servers or indicators for known exploits.
| quesera wrote:
| > _Couldn 't nearly all of these attacks be blocked by a white-
| list approach, essentially hiding every file or directory from
| the internet except a very controlled list of paths and
| escaping all text sent so it can't contain code?_
|
| This is basically how things work.
|
| For convenience, instead of itemizing each filename, the
| webserver root is a subdirectory and anything underneath is
| fair game. The webserver uses the OS "chroot" facility to
| enforce this restriction. What you are seeing is ancient
| exploitation strings from 30 years ago that haven't worked on
| any serious webserver since that time, but a) keeping the test
| in the attackers lib is essentially free, and b) there are some
| unserious webservers, typically in cheap consumer hardware.
|
| Webservers pass plain text to the app server. It is the app
| server/framework's responsibility to understand the source of
| the request body and present it to the application in a clear
| way, possibly escaped. But the app needs to process this and
| sometimes through poor coding practices, fails to respect the
| untrusted nature of the data. This again is more typical in
| historical systems and low-cost consumer products where
| software is not a marketing advantage.
| Thorrez wrote:
| >This all looks like "echo ; /bin/cat /etc/passwd" and somehow
| the server just spitting it out. Is that really the state of
| web security?
|
| It's attempting to exploit a vulnerability in bash that was
| discovered and fixed in 2014:
|
| https://en.wikipedia.org/wiki/Shellshock_(software_bug)
| 1B05H1N wrote:
| I work in application/product security and have managed WAFs for
| multi-billion dollar companies for many many years.
|
| Move DNS to Cloudflare and put a few WAF rules on your site
| (managed challenge if bot score less than 2 / attack score == x).
| I doubt you'll even pay anything, and it will resolve a lot of
| your problems. Just test it before moving it to production please
| (maybe setup a test domain). Remember, a WAF is not an end-all be
| all, it's more of a band-aid. If you app isn't hardened to handle
| attacks, no amount of advanced WAF/bot protection will save it.
|
| Message/email me if you need help.
| asabla wrote:
| Usually I only manage internal facing applications these days,
| which makes the attack surface greatly reduced compare to
| public ones.
|
| But since you seem to have a lot of knowledge in this area.
| Have you manage solutions which also includes infrastructure in
| Azure combined with Cloudflare?
|
| And if so, any suggestions on things people usually miss?
| except for the usual stuff of OWASP and what not
| CharlesW wrote:
| I was unfamiliar with this, so for anyone who's in a similar
| position: https://blog.cloudflare.com/waf-for-everyone/
|
| The Free Managed Ruleset appears to be deployed by default, and
| Cloudflare keeps a changelog here:
| https://developers.cloudflare.com/waf/change-log
| 418tpot wrote:
| Yes, that's just what the internet needs is even more websites
| centralized behind Cloudflare. Why do we even bother with TLS
| anymore if we're going to give them unencrypted access to
| practically all of our internet traffic.
|
| Hacker news is so funny, they complain about the amount of
| power we've allowed Google, Amazon, and Microsoft to have, and
| then go right around and recommend putting everything behind
| Cloudflare.
|
| Once Cloudflare starts using attestation to block anyone not on
| Chrome/iOS Safari it'll be too late to do anything about it.
| solumunus wrote:
| > Hacker news is so funny, they complain about the amount of
| power we've allowed Google, Amazon, and Microsoft to have,
| and then go right around and recommend putting everything
| behind Cloudflare.
|
| It's almost as if those saying contradictory things are
| actually different people despite being on the same website.
| But it can't be that, surely? Truly a perplexing phenomenon
| that I hope someone can one day explain.
| 418tpot wrote:
| Fair, although I know quite a few people that hold both of
| these opinions simultaneously because I've met them in
| person. It's only after I point out their hypocrisy do they
| even realize what a danger Cloudflare poses to the free and
| open internet.
|
| I suspect it's because hating on Google is in vogue, and so
| is recommending Cloudflare.
| jopsen wrote:
| Given how Cloudflare works I imagine that there are
| alternative services offering the same thing.
|
| Probably not as cheap. AWS can put a WAF and CDN infront
| of your site too.
|
| And migrating from one service to another isn't much more
| work than moving DNS records.
|
| Just saying, it's not the same level of vendor lockin as
| using dynamodb or whatever.
| esafak wrote:
| You criticize but don't offer suggestions. What do you use
| instead of Cloudflare?
| chaxor wrote:
| Agreed
|
| We should be suggesting self hosted and decentralized
| solutions to website hosting and file hosting.
|
| On that note, does anyone have any _secure_ methods of
| providing serving a file from your computer to anyone with a
| phone /computer that doesn't require them
| downloading/installing something new? Just a password or
| something? Magic-wormhole almost seems great, but it requires
| the client to install wormhole (on a computer, not phone),
| and then type specific commands along with the password.
|
| Is there a simple `iroh serve myfile.file` from server and
| then client goes to
| https://some.domain.iroh/a086c07f862bbe839c928fce8749 and
| types in a password/ticket you give them?
|
| That would be wonderful.
| NicoJuicy wrote:
| > Once Cloudflare starts using attestation to block anyone
| not on Chrome/iOS Safari it'll be too late to do anything
| about it.
|
| That's just plain bs...
|
| Eg
|
| 1) they have customers and their customers want protection,
| with minimal downsides.
|
| 2) Cloudflare is the only one with support for Tor. I'm 100%
| sure you didn't knew that.
|
| What "examples" do you have to blame them for something they
| aren't doing? Based on what?
|
| I'm getting tired of people blaming Cloudflare for providing
| a service that no one else can provide for free to small
| website owners => DDOS protection.
| pid1wow wrote:
| What do you mean? On Tor I get a Cloudflare block just from
| clicking 2 links on the front page of HN:
|
| http://forums.accessroot.com/index.php?showtopic=4361&st=0
|
| >Please wait while your request is being verified...
|
| I can't remember any day I didn't get a Cloudflare block.
| Even on bare IP sometimes. WAFs are security theater.
| ozim wrote:
| Putting WAF on app and calling it a day is indeed putting
| lipstick on a pig.
|
| I can imagine that might be needed if some company for some
| reason has to run some not really up to date stuff but yeah it
| is just a bandaid.
| pferde wrote:
| An interesting thing that I've noticed is that some of the
| attackers watch the Certificate Transparency logs for newly
| issued certificates to get their targets.
|
| I've had several instances of a new server being up on a new IP
| address for over a week, with only a few random probing hits in
| access logs, but then, maybe an hour after I got a certificate
| from Let's Encrypt, it suddenly started getting hundreds of hits
| just like those listed in the article. After a few hours, it
| always dies down somewhat.
|
| The take-away is, secure your new stuff as early as possible,
| ideally even before the service is exposed to the Internet.
| elashri wrote:
| It is also useful to rely more on wildcard certs, as it makes
| it difficult to determine from CT logs the specific subdomains
| to attack.
| bombcar wrote:
| There's really no reason to avoid wildcard certs for your
| domains, unless you have so many subdomains that are managed
| by various business interests.
|
| I use LE wildcard certs and they're great, you can use them
| internally.
| couchand wrote:
| It seems like the principle of least power would apply
| here. There's value in restricting capability to no more
| than strictly necessary. Consider the risk of a compromised
| some-small-obscure-system.corporate.com in the presence of
| a mission-critical-system.corporate.com when both are
| issued wildcard certs.
|
| Wildcard certs are indeed a valuable tool, but there is no
| free lunch.
| baobun wrote:
| You'd usually put a reverse proxy exposing the services
| and terminating TLS with the wildcard cert.
|
| The individual services can still have individual non-
| wildcard internal-only certs signed by an internal CA.
| These don't need to touch an external CA or appear in CT
| logs - only the reverse proxy/proxies should ever hit
| these, and can be configured to trust the internal CA
| (only) explicitly.
| nullindividual wrote:
| A compromised wildcard certificate has a much higher
| potential for abuse. The strong preference in IT security is
| a single-host or UCC (SAN) certificate.
|
| Renewing a wildcard is also unfun when you have services
| which require a manual import.
| dfc wrote:
| Renewing any certificate that requires a manual import is
| not fun. Why are wildcard certs less fun to manually import
| than individual certificates?
| nullindividual wrote:
| Presumably one purchases a wildcard for multiple distinct
| systems.
| maccam912 wrote:
| Same! As soon as a new cert is registered for a new subdomain,
| I get a small burst of traffic. It threw me off at first
| assuming I had some tool running that was scanning it.
| supriyo-biswas wrote:
| These aren't attackers - they're usually services like
| urlscan.io and others who crawl the web for malware by
| monitoring CT logs.
| joshspankit wrote:
| The thread is specifically talking about logs of attacks
| heywoodlh wrote:
| Was looking into Certificate Transparency logs recently. Are
| there any convenient tools/methods for querying CT logs? i.e.
| search for domains within a timeframe
|
| Cloudflare's Merkle Town[0] is useful for getting overviews,
| but I haven't found an easy way to query CT logs. ct-
| woodpecker[1] seems promising, too
|
| [0] https://ct.cloudflare.com/
|
| [1] https://github.com/letsencrypt/ct-woodpecker
| j0hnyl wrote:
| https://certstream.calidog.io/
| H8crilA wrote:
| https://crt.sh/
| simonw wrote:
| Steampipe have a fun SQLite extension that lets you query
| them via SQL:
| https://til.simonwillison.net/sqlite/steampipe#user-
| content-...
|
| It uses an API provided by https://crt.sh/
| high_priest wrote:
| Querying crt.sh helped me identify a dev service I was
| supposed to take down, but forgot about it. Nice
| alternative use case :D
| KronisLV wrote:
| > The take-away is, secure your new stuff as early as possible,
| ideally even before the service is exposed to the Internet.
|
| Honestly it feels like you'll need at least something like
| basicauth in front of your stuff from the first minutes it's
| publicly exposed. Well, either that, or run on your own CA and
| use self-signed certs (with mTLS) before switching over.
|
| For example, when some software still has initial install/setup
| screens where you create the admin user, connect to the DB and
| so on, as opposed to specifying everything initially in the
| environment variables, config files, or other more specialized
| secret management solutions.
| p_l wrote:
| Generally I'd recommend not exposing anything unless you
| deployed the security for it.
|
| Just SSH scanning can be a big issue.
| psanford wrote:
| A big issue how? If you block password auth, ssh scanning
| is a nonissue.
| aunderscored wrote:
| Unless an attack on the sshd is employed. Which is also
| possible
| bradknowles wrote:
| DDoS attack on your sshd?
| indigodaddy wrote:
| One option could be to lock the port down to only your
| jump/bastion server source IP.
| bradknowles wrote:
| That fails to when you lose that IP address, or you lose
| that server.
| indigodaddy wrote:
| I suppose if you don't have console access, sure. But
| inconvenient at worst imv.
| bradknowles wrote:
| If you have a way around that bastion server, then at
| least you've got a backup. But then you also have to
| worry about the security of that backup.
| woleium wrote:
| or get your certificates using dns auth a week or so prior to
| exposing the service
| Filligree wrote:
| > something like basicauth
|
| I wish. I use basicauth to protect all my personal servers,
| the problem is Safari doesn't appear to store the password! I
| always have to re-authenticate when I open the page.
| Sometimes even three seconds later.
| burgerquizz wrote:
| I am wondering if there was any reports (on large scale systems)
| on what hackers were looking for on servers?
| CharlesW wrote:
| Cloudflare's WAF managed rulesets changelog might be helpful
| for this: https://developers.cloudflare.com/waf/change-log
|
| For example, they did an emergency update on 1/22 in response
| to CVE-2023-22527.
| simpaticoder wrote:
| Thank you! I've been self-hosting for about a year running a
| 400-line http/s server of my own design, and it's remarkable all
| the attacker traffic my 3 open ports (22, 80, 443) get, although
| I've never taken the time to analyze what the attackers are
| actually trying to do! This post fills in a LOT of blanks.
|
| Would be cool to do the same thing for the weird stuff I see in
| /var/log/auth.log!
|
| It's crazy that attackers would bother with me since the code is
| entirely open source and there is no server-side state. The best
| outcome for an attacker would be root access on a $5/mo VPS, and
| perhaps some (temporary) defacement of the domain. A domain no-
| one visits!
| epcoa wrote:
| These are all automated bots. No one is "bothering". You open
| the 3 most well known ports you're going to get connections.
| They don't know what you're running nor do they care.
| simpaticoder wrote:
| By "bothering with me" I mean "add my IP to the long list of
| IPs they are scanning".
|
| By the way, I find it annoying that my logs get filled with
| this kind of trash. It has the perverse effect of making me
| long for something like Google Analytics since they rarely if
| ever bother running a javascript runtime.
| epcoa wrote:
| That long list isn't curated, it's every publicly routable
| IPv4 address. It really does not take long to run some
| canned probes against 3.7 billion addresses. Making your
| service IPv6 only tends to cut down on this traffic. You're
| anthropomorphizing a script on some botnets.
| ericpauley wrote:
| This isn't entirely true. Many scanners do preference
| specific IP ranges such as cloud providers. Cloud IPs
| receive substantially more scanning traffic than darknet
| IPs or even random corporate IPs.
| iramiller wrote:
| IPv6 only? If you have a DNS record for that your are
| still not making it very difficult for scripts to find
| you.
| ozim wrote:
| No one is maintaining a list they just scan all. Scanning
| every IPv4 there is on a single port takes minutes.
| dotancohen wrote:
| Access to your VPN is a great way to launch attacks on other
| machines, and to add another layer to covering his tracks. Not
| to mention hosting malware to be downloaded to other places,
| and even a crypto miner.
| mianos wrote:
| I set up a honeypot once and logged the passwords of created
| accounts. I then used 'last' to find the incoming ip.
|
| I then used ssh to try and connect to the originator (from an
| external box). I went back 5 jumps until I got to a windows
| server box on a well known hosting service that I could not
| get into.
|
| Lots of what looked like print servers and what looked like
| linux machines connected to devices. Maybe just the exploit
| at the time.
| icameron wrote:
| Consider blocking 22 except whitelist your own IP. My ISP
| changes my IP rarely in practice and when they do I can log
| into the hosting web admin panel and update the rule.
| petee wrote:
| Just accept key-only logins, everything else becomes noise.
|
| I also limit concurrent connections, which significantly
| reduces data usage during aggressive attacks
| sph wrote:
| SSH is constantly hammered by bots as well: %
| ssh example.com Last failed login: Sun Jan 28 16:59:35
| UTC 2024 from 180.101.88.233 on ssh:notty There were 5385
| failed login attempts since the last successful login.
| Last login: Sat Jan 27 13:33:30 2024 from xxx.xxx.xxx.xxx
|
| 5.3k failed attempts in ~30 hours. I know, I should be setting up
| fail2ban.
| SoftTalker wrote:
| fail2ban will reduce your log noise but it's another thing to
| manage, you can end up locking yourself out also, and if you're
| using good passwords (or better, public key auth) it's not
| really providing any additional safety.
|
| Ideally you don't need ssh open to the whole world anyway, and
| can restrict it to a certain subnet or set of addresses. Then
| your attacks will drop to nearly zero.
| NavinF wrote:
| > I should be setting up fail2ban.
|
| No, that does nothing if you followed best practices and
| disabled password login. fail2ban is just another Denial of
| Service risk that has the added bonus of bloating your firewall
| table and slowing down all your new connections
| idoubtit wrote:
| fail2ban is not very performant and it will only reduce the
| amount of attempts. An alternative is to add a nftables rule
| (or iptables or whatever firewall). Something like :
| table inet filter { chain input { tcp dport ssh
| accept comment "Accept SSH" tcp dport 22 ct state new
| limit rate over 2/minute drop
|
| But even with rate limiting, the logs are still polluted by
| auth attempts. Changing the port does little. The only solution
| we found was to configure port knocking (with direct access
| from a few whitelisted IPs).
| miyuru wrote:
| Most of my servers are IPv6 only and there are no failed ssh
| attempts on those. I install fail2ban just in case and firewall
| the IPv4 address, since I don't SSH via IPv4.
| nerdponx wrote:
| I set my SSH port to something with a high number that is not
| used by any other known service. Drive-by attacks dropped to 0.
| vldb7 wrote:
| I'm not a fan of fail2ban. A simple but quite effective
| approach would be permitting remote login only from certain IP
| ranges. I know that it looks like a bad trade for self hosted
| web apps, but it is very easy to setup on many cloud providers.
|
| Also I normally set up a jump host first - a smallest instance
| that only runs ssh and everything else would not open ssh port
| to the outside at all. One nice effect is having to search just
| one auth log if something about ssh looks concerning.
| emmanueloga_ wrote:
| In case you are using AWS, I learned that you can close port 22
| on your EC2 instances, and connect trough the Systems Manager
| (SSM):
|
| https://cloudonaut.io/connect-to-your-ec2-instance-using-ssh...
| rsolva wrote:
| If someone has experience with CrowdSec (a kind of crowdsourced
| fail2ban), I would like to hear your opinion!
| creeble wrote:
| Don't know CrowdSec but have used abuseipdb.com a bit to reduce
| log noise.
| dangus wrote:
| I would be interested in reading something similar but focused on
| less common ports and services, like game servers that run on
| high ports and more typically use UDP.
|
| I wonder if that's more or less of a "safe" situation.
| jbverschoor wrote:
| You could and should alway monitor protocol errors (any
| protocol).
| snowwrestler wrote:
| Back when I started managing self-hosted sites, I would look
| through access logs as well. We even had an IDS for while that
| would aggregate the data and flag incoming attack attempts for
| us.
|
| Eventually I stopped proactively reviewing logs and stopped
| paying for the IDS. It was a waste of time and a distraction.
|
| It's not hard to find really useful content that summarizes
| common vulnerabilities and attacks, and just use that to guide
| your server management. There are a ton of best practices guides
| for any common web server technology. Just executing these best
| practices to 100% will put you way ahead of almost all attackers.
|
| And then the next best use of your time and resources is to
| prioritize the fastest possible patching cadence, since the vast
| majority of attacks target disclosed vulnerabilities.
|
| Where logs are super helpful is in diagnosing problems after they
| happen. We used log analysis software to store and search logs
| and this was helpful 2-3 times to help find (and therefore
| address) the root cause of attacks that succeeded. (In every case
| it turned out to be a known vulnerability that we had been too
| slow to patch.)
| dandrew5 wrote:
| > And then the next best use of your time and resources is to
| prioritize the fastest possible patching cadence, since the
| vast majority of attacks target disclosed vulnerabilities.
|
| Just curious, do you leverage any tools to decide when to patch
| or is it time-interval based? We currently attempt[0] to update
| our packages quarterly but it would be nice to have a tool
| alert us of known vulnerabilities so we can take action on them
| immediately.
|
| [0] "Attempt" meaning we can't always upgrade immediately if
| the latest version contains difficult-to-implement breaking
| changes or if it's a X.0.0 release that we don't yet trust
| eyegor wrote:
| Besides pointing pentester tools like metasploit at yourself,
| there are some nice scanners out there. Examples in no
| particular order:
|
| https://github.com/quay/clair
|
| https://github.com/anchore/grype/
|
| https://github.com/eliasgranderubio/dagda/
|
| https://github.com/aquasecurity/trivy
|
| So then you set up something like a cron job to scan
| everything for you and email the results once a week or
| whatever if you don't want to monitor things actively.
| dandrew5 wrote:
| Thanks for these. I should have clarified, I'm more
| interested in something that will alert me when newly-
| discovered vulnerabilities surface. The systems I maintain
| are protected [enough] today but a new hack could drop that
| doesn't make mainstream media and I may not hear about it.
| We have annual security audits but it would be nice to
| patch things immediately. Aside from subscribing to a
| security forum/discord/slack, I'm wondering what other
| methods folks are employing to solve this.
| bradknowles wrote:
| Keep your pentesting tools up-to-date. Run them against
| yourself on every single deployment, if you can. Don't
| just run them quarterly because that's all that your PCI-
| DSS requirements say you have to do.
|
| Integrate security code scanning tools into your CI/CD
| process. Tools like Dry Run Security, or something
| comparable.
|
| There's much more, but that has to do with how to run
| your CI/CD systems and how to do your deployments in
| general, and less to do with security aspects thereof.
| layer8 wrote:
| The simplest is to only use packages from a distribution like
| Debian and run _unattended-upgrades_ or equivalent for the
| security-updates repository. They usually fix vulnerabilities
| in less than a day.
| gnyman wrote:
| > In every case it turned out to be a known vulnerability that
| we had been too slow to patch.
|
| Yes. This is why relying on "patching" is a bound to fail at
| some point. Maybe it's a 0-day, or maybe the attackers are just
| quicker.
|
| The solution to this is defence in depth, and it's very easy
| for most services, especially when self-hosting personal
| things. Few tips most people can do is.
|
| Put up a firewall in front or put it behind VPN/tailscale.
|
| Hide it in a subfolder. The automated attacks will go for
| /phpmyadmin/ , putting it in /mawer/phpmyadmin/ means 99.9% of
| the attackers won't find it. (This is sometimes called security
| by obscurity and people correctly say you should not rely on
| it, but as a additional layer it's very useful).
|
| Sandbox the app, and isolate the server. If the attackers get
| in, make it hard or impossible for them to get anywhere else.
|
| Keep logs, they allow you to check if and how you got attacked,
| if the attack succeeded and so on.
|
| Depending on the service, pick one or more of these. Add more
| as necessary.
|
| The key thing is that you should not rely on any ONE defence,
| be it keeping it patched or firewalled, because they will all
| fail at some point.
| citizenpaul wrote:
| >security by obscurity
|
| I detest this phrase right up there with "fake it till you
| make it". All security is by definition obscurity. Just a
| meaningless platitude that rhymes.
|
| I suppose un-formalized un-proofed security practices will
| eventually be broken or counting on hackers not to do any
| investigation of your system will get you hacked, doesn't
| roll off the tongue though.
| jmb99 wrote:
| > This is sometimes called security by obscurity and people
| correctly say you should not rely on it, but as an additional
| layer it's very useful.
|
| Anyone who thinks "security by obscurity" is useless should
| try reverse engineering some properly obfuscated executables
| (or even code). Obscurity is absolutely useful; definitely
| not a complete solution by itself, but a very useful
| component to a security solution.
| bradknowles wrote:
| The term "security by obscurity" or "security through
| obscurity" implies that ONLY obscurity is being used to
| provide the security in question. Like leaving a totally
| unsecured server on your network with root access available
| with no password, and telnet or sshd open on a high
| numbered port instead of the regular ones.
|
| Obscurity is a useful tool to be added on top of real
| security, and can help reduce the random baseline doorknob
| jiggling attacks, where people are just scanning the
| standard ports. But obscurity by itself is not enough to
| provide any real security beyond that.
| Gibbon1 wrote:
| Reminds me of a friend that does security stuff. He says
| never store or transmit keys in a useful form. Bonus using
| munged keys will trigger alarms.
| evantbyrne wrote:
| The elephant in the room is that-at least in my experience-a lot
| of these attacks come from hostile nation states. This is going
| to be controversial, but one may find it useful to block entire
| IP ranges of problematic states that you cannot do business with.
| I was able to block 100% of probes to one of my new services by
| doing this.
| macintux wrote:
| 15+ years ago I was reviewing server logs for small, local
| businesses we hosted and came to the conclusion we should just
| block all APNIC IP addresses.
| acherion wrote:
| APNIC includes Australia, New Zealand, Japan, Taiwan and
| Singapore - did you block addresses from there too?
| macintux wrote:
| Given the length of time since then, I have no
| recollections beyond making the decision
| nerdponx wrote:
| It's not an elephant, it's just that many people aren't willing
| to block legitimate users from those regions.
| evantbyrne wrote:
| American websites don't typically have customers dialing in
| from North Korea. Just saying that IP blocking is something
| more businesses should consider. Traffic can also be routed
| to different subdomains for businesses that need to provide a
| subset of services to a region.
| Erratic6576 wrote:
| I once got a message in my logs "you have been powned" :/
| ijhuygft776 wrote:
| what did you do about it
| Erratic6576 wrote:
| I don't remember well. I guess I Worried and stopped reading
| logs.
|
| Aah. I diverted my domain to cloudflare, allowing only
| traffic from 1 country in.
|
| So instead of exposing my IP publicity through my registrar,
| I set the firewall to only allow traffic in from cloudflare.
|
| I would have loved to install PfSense as well but it was out
| of my budget
| ijhuygft776 wrote:
| so, you, are, still, hacked.
| iboisvert wrote:
| As someone who knows very little about security, this is really
| interesting, thanks! A question though: how would one know if
| there has been a breach? These examples look relatively easy to
| detect, but I guess there would be more complex cases?
| kevindamm wrote:
| This is why some people run a honeypot in their network... and
| even those won't necessarily catch everything if the honeypot
| only mimics services that the attacker isn't probing for. You
| can set up tripwires on access and egress of sensitive data but
| that's only part of the surface area (and if the system gets
| attacked those tripwires could be disabled, if the attacker
| either knows what to look for or has a plan for a side channel
| for exfiltrating data).
|
| Really the only good answer is defense in depth and keep
| looking for any indicators of odd behavior, and wall out
| unrelated systems entirely from each other, keep the DMZ and
| public facing bits as simple as possible.
| macintux wrote:
| I also know very little, but something that struck me upon
| reading your question: if a breach is successful, the logs
| can't be relied upon for detection/analysis if they're on the
| same server. It's important to ship them elsewhere.
| tamimio wrote:
| IOC or indicators of compromise, but if you know little it is
| always advisable to hire someone to go through it on demand or
| periodically as there's no one trick to rule them all.
| emmanueloga_ wrote:
| WAF seems to be an essential piece for any website with even a
| little bit of visibility / traffic on the net. Some questions:
|
| * Comparison of AWS WAF vs Cloudflare vs Others?
|
| * Many services like EC2 charge for data transfer [1], so how
| much of your monthly/yearly costs of hosting goes toward fending
| scans like these? Does AWS count any traffic blocked by the WAF
| toward the transfer limits?
|
| --
|
| 1: https://aws.amazon.com/ec2/pricing/on-demand/
| rfmoz wrote:
| With Fastly WAF - SignalSciences, Distil Networks - Imperva,
| Akamai - StackPath. There are a few companies that started
| alone in this ecosystem and found the evolution path with the
| mayor CDN networks, those who have failed themselves in provide
| a good internal alternative service.
|
| Human security - PerimeterX, Haproxy WAF, Datadome, are other
| players to different target audience.
|
| If you have a good control over the app exposed, maybe you only
| need a WAF in the sense of stop stuff outside your infra. The
| sql injections, weird urls trying the way to /etc/passwd or
| related things look from the past and only makes noise
| nowadays. The real issue is when someone hits you in a rate
| impossible to manage with your resources or when it cost you
| more than the securing layer.
| nickjj wrote:
| If you're on AWS the AWS WAF is pretty low cost. You can expect
| to pay less than $10 / month and still get an ok amount of
| value on a decently popular site.
|
| The problem is you have to manually configure a lot, the rate
| limiting aspect is way worse than Cloudflare and while the AWS
| WAF can geolocate an IP address and block by country it does
| not send the country code back to you in a header where as
| Cloudflare does. The last one stings because it's super handy
| to have an accurate country code attached to each request,
| especially if it's something you don't need to think about or
| waste I/O time calling out to a 3rd party service in the
| background to backfill in that data later.
| m0rissette wrote:
| Man people don't use shadow anymore and expose creds through
| /etc/passwd; craziness
| SebFender wrote:
| Interesting, well prepared and presented - a necessary read for
| many.
| urbandw311er wrote:
| As a thought experiment - if there was public money to back this,
| would it be making us all safer to run a series of honeypot
| servers that automatically start DDOSing the various C&C servers
| that attempt to compromise them?
| PeterisP wrote:
| Such a response would generally be a crime; existing cybercrime
| legislation generally does not have any clauses permitting
| retaliation as "self defence", and also very often you'd be
| DDoSing an innocent third party, another victim whose
| compromised device is abused to route traffic, and also
| affecting their neighbors on the same network.
| qaq wrote:
| Tangentially the sad thing about whole cyber sec space is that
| well resourced APTs like say APT-29 Cozy Bear. Have enough
| resources and actually run labs where they deploy all top
| Endpoint solutions and validate their offensive tools against
| them.
| yawaramin wrote:
| Does anyone else have the experience that almost every single one
| of the 'attacks' is using HTTP and not following a 301 redirect
| to HTTPS? I have my internet-facing web server set up to redirect
| all HTTP -> HTTPS and this thwarts almost _every_ 'attack'. I
| have to say, they're not very smart about it.
| gary_0 wrote:
| I do that as well, but nothing gets through anyways because I'm
| not running Wordpress or a version of Apache from 2012. I also
| have nginx set up to reject connections without the expected
| 'Host' header, and that quickly bounces a few attackers. But
| many still end up getting all the way to a 404 or 400 for their
| attack URL.
| aborsy wrote:
| I'm looking for a secure authentication/proxy application to put
| in front of the webservers that have to be exposed to the
| internet. The application will authenticate the user with SSO or
| hardware key, before forwarding the traffic to the right internal
| address.
|
| Cloudflare Tunnels are great, but CF doesn't allow the TLS pass
| through. CF man in the middles and decrypts the traffic.
|
| So far I have looked into Teleport, authelia, authentik, and
| keycloak, perhaps combined with Traefik.
|
| Any feedback on the level of security of these tools for being
| exposed to the public internet?
| lukax wrote:
| If your auth provider supports is OAuth 2 OIDC you should check
| oauth2-proxy. It's just one binary or a container sidecar.
|
| https://github.com/oauth2-proxy/oauth2-proxy
| FuriouslyAdrift wrote:
| Granted we are a Microsoft-based shop, but Microsoft Entra
| Application Proxy has worked out great for exposing our
| internal web based apps to the outside for mobile/home workers.
|
| https://learn.microsoft.com/en-us/entra/identity/app-proxy/a...
| aborsy wrote:
| This seems to be very similar to the Cloudflare tunnels. They
| are reverse proxies in the cloud, with TLS termination. The
| traffic is terminated and scanned in the cloud.
|
| https://learn.microsoft.com/en-us/entra/identity/app-
| proxy/a...
|
| A version with end to end encryption will be great!
| zetalemur wrote:
| Interesting. Came to similar conclusions when analyzing my
| (httpd) access logs for https://turmfalke.httpd.app/demo.html ...
| but so far nothing really out of the ordinary.
| pid1wow wrote:
| > directory traversal
|
| The correct term is directory enumeration. Traversal usually
| means something about ../../
| daneel_w wrote:
| Curious enumerations on the most common items. There's definitely
| a topical bias. By far the most common attack attempt I see on
| all the various webhosts I administer is WordPress-oriented
| (despite not present on any of the hosts), which doesn't even get
| an honorable mention by the author. Perhaps he hosts WordPress
| content and didn't discern attacks from legitimate traffic.
| tamimio wrote:
| Yeah some thing in websites I have, even the personal one, they
| get on average tens of these attacks, and again, it seems they
| are after .env too and other php related directories. Big portion
| of these attacks come from TOR networks too.
___________________________________________________________________
(page generated 2024-01-28 23:00 UTC)