[HN Gopher] Gitlab Project Hit by Spambot
___________________________________________________________________
Gitlab Project Hit by Spambot
Author : nielsole
Score : 122 points
Date : 2021-02-08 13:17 UTC (9 hours ago)
(HTM) web link (gitlab.com)
(TXT) w3m dump (gitlab.com)
| sushikokk wrote:
| Maybe they flood the mailbox to hide legit mails in between all
| the spam mails, like password changes or other important
| notifications?
| CatTheHacker wrote:
| Happens on GitHub as well, fortunately not with such massive
| flood
| arusahni wrote:
| Here's their incident status:
| https://status.io/pages/incident/5b36dc6502d06804c08349f7/60...
|
| Looks like the immediate issue was resolved a few minutes ago.
| boleary-gl wrote:
| GitLab team member here.
|
| We're actively working on mitigating this spam attack:
| https://twitter.com/gitlabstatus/status/1358777539624198145
| xeeeeeeeeeeenu wrote:
| It's interesting that the nickname of the spammer is "fixspam
| gitlab". It seems that their intention is to force gitlab to
| introduce anti-spam measures.
| viraptor wrote:
| Could be... I've seen this before - people annoyed by
| continuous low effort spam and the service not fixing it,
| deciding to show what a targeted attack will look like to force
| a response. (Not saying that GL doesn't care)
| markmacardle wrote:
| what is the motivation behind attacks like this? Like is whoever
| put in the effort to do this just trying to annoy Gitlab's
| customers? What do they gain from that?
| talhah wrote:
| Judging from the example email the name of the spammer is
| "fixspam gitlab", I'm guessing they want gitlab to fix spam.
| jancsika wrote:
| I run my own instance of Gitlab which basically means that I
| handle spam manually. It is a pain.
|
| Gitlab recently added a band-aid of a feature to require admin
| approval for new sign-ups, which comes with its own obvious
| problems. But even the UI for that is clunky-- you have to click
| a button to reveal some options, find the option to delete the
| spammer, type (read: paste) the users handle as a failsafe, after
| which Gitlab takes you back to the main user view requiring you
| to click _again_ on the unverified users tab.
|
| Last time I talked about this problem someone from Gitlab claimed
| they were working on a feature to send an email with a simple
| link to click to approve/deny a user. That feature still has not
| shipped.
|
| They _did_ fix an unrelated catastrophic bug of keeping around
| artifacts on each branch even if you set an expiry. On a gitlab
| instance with even moderate development rate and moderate-sized
| artifacts this fills up the hard-disk with artifacts that the
| admin must manually delete (or manually create their own script
| to delete them, which is still a pain). However, their fix was to
| add a checkbox that _defaults_ to the buggy behavior. _Per
| project_. So for every new user 's project I have to manually
| tell them to turn this anti-feature off to keep from filling up
| the hard-drive with artifact spam. Or else write a damned script
| to go in and turn off this feature periodically.
|
| The community edition of Gitlab feels a lot like the early Gnome
| 3 touch-screen stuff. Great in theory, but one quickly gets the
| sense that no one is dogfooding-- if they did these kinds of
| problems wouldn't exist.
| jeanlucas wrote:
| Is it spam or a "mail ddos"?
| boleary-gl wrote:
| Both?
| jdsalaro wrote:
| Strictly speaking the former. You could, however, say it's both
| since submitted issues trigger an e-mail being sent to people
| following the project.
| jitbit wrote:
| We are a small SaaS vendor and fighting spam is, like, 20% of the
| technical work that we do. Bad actors register trial accounts and
| then reverse engineer your front-end and forge millions of
| requests. Just 2 days ago we had a very similar issue [1]
|
| We have all sorts of protection: rate limiting, crippling non-
| paid accounts, detecting if a trial signup comes from a VPN/Tor
| (+5 to "suspicious" score!) etc... But they still manage to find
| ways. It's a never-ending battle. And the saddest part is - all
| this hard work is completely invisible to our existing paying
| customers :(
|
| If your app has an email-sending module of some sort it WILL be
| abused. Even a trivial "reset password" form is a target.
|
| My support goes to the GitLab team. Good luck and no hard
| feelings.
|
| [1] https://www.jitbit.com/news/5354-spammer-attack-post-mortem/
| ruffrey wrote:
| Small SaaS vendor checking in. This comment rings true and it's
| a constant battle.
| freyfogle wrote:
| Can't upvote this comment enough. Have wasted years of my life
| on these invisible, but sadly necessary, features.
| tutfbhuf wrote:
| Use hCaptcha for sign up and other sensitive parts of your
| application.
| meibo wrote:
| I'd rather not use the service if it has hCaptcha OR
| ReCaptcha.
|
| hCaptcha has bad UX, their dataset is way harder to decipher
| and commonly takes me three or four tries to get around on
| Cloudflare, if it doesn't keep failing with a server error
| which I've had happen on multiple, distinct occasions.
|
| At least for Google, I was done in one turn when lucky, if I
| had no other choice (e.g. banking).
| contravariant wrote:
| It seems to have gotten better but I still haven't forgiven
| ReCaptcha for regularly making me spend ages retrying the
| unending and excruciatingly slow loading tests.
|
| At least hCaptcha has a decent chance of letting me through
| if I answer correctly.
| stanislavb wrote:
| I feel you. Fighting SPAM is eating a serious amount of my
| energy, too.
| nonbirithm wrote:
| I wonder if effective DDoS protection today is inextricably
| tied to having to route your traffic through centralized third
| parties like Cloudflare.
| rectang wrote:
| Imagine a world where spammers could be identified, prosecuted,
| and convicted reliably. The amount of economic energy unleashed
| would be staggering.
|
| We are still in the early days of the internet. It is a
| frontier territory where crime is rampant.
| ZWoz wrote:
| Thats about initiatives. Many daily drivers break some laws,
| like speeding or not using turn signals. Does that make
| traffic frontier territory? :) Try something serious, like
| operating black drug market or treatening high level
| politician (in internet): you going to be surprised, how well
| different countries co-operate. For example operation
| involving 30 countries: https://www.europol.europa.eu/newsroo
| m/news/%E2%80%98avalanc...
| jarym wrote:
| So these 'bad actors' are they just out to find a way to DoS
| your service or do they have some other motive for forging
| millions of requests. (I get that email sending is a target -
| that has been the case since forever - but I'm asking about
| 'what else'?)
| jdsalaro wrote:
| It varies: generating inconvenience(for fun or hate towards a
| particular service), spreading scam URLs(leveraging existing
| trust relationships), SEO and improving discoverability of
| their projects (porn, streaming of pirated content, etc) and
| many more.
| jitbit wrote:
| Yes, they sure do have a motive - to send spam via a 3rd
| party service.
|
| Like in this case (GitLab) - create a fake "issue" with
| spammy links in it and then "notify" thousands of users.
|
| In our case - we are a helpdesk ticketing app, so bad actors
| sign up for a trial, then add "users" to their account and
| then, say, create "tickets" on users' behalf (customizing the
| email template by inserting spammy links).
| [deleted]
| waihtis wrote:
| What do the bad actors technically do? Map your api endpoints
| and spam them with traffic? Something else also?
| malinens wrote:
| I work for an e-mail provider. This is very painful part of our
| work. Part of our services are free and ad supported. We are
| disabling, queueing thousands of mailboxes per day. Captchas
| and user scoring does not help that much. There are very cheap
| services were people manually will solve recaptchas/hcaptchas.
| Even required phone numbers (normal users are pissed when we
| ask for a phone number to use account fully but we don't have
| much choice...). There are phone number farms which are used
| for account verification also
| jdsalaro wrote:
| > This is very painful part of our work
|
| GitLabSec team-member here, ^ this 100%.
|
| Our Trust and Safety team works really hard to keep up with
| spam. For those interested in how they do so, you can read
| more here[1]. We are also working on making our spam
| detection engine more effective at mitigating high-volume
| spam incidents such as this.
|
| For current, working product improvements to detect and
| mitigate spam you and others can check out MRs labeled "spam
| fighting"[2]. We've had quite a few internal conversations
| about strategies and best-practices. For more information on
| how to contact our Trust & Safety Team, including tips on how
| to deal with abuse in your own instance or steps to suggest
| spam/abuse related features, see our handbook[3]
|
| [1] https://about.gitlab.com/blog/2020/10/29/how-we-work-to-
| dete...
|
| [2] https://gitlab.com/gitlab-
| org/gitlab/-/merge_requests?scope=...
|
| [3] https://about.gitlab.com/handbook/engineering/security/se
| cur....
| zoomablemind wrote:
| Maybe introduce a Moderator role to the multi-user
| projects?
|
| That would require the project's moderator to clear the
| newly posted issues (whatever is possibly a spamming route
| there) before they propagate. As this is an additional
| friction point, some bad actors may be intercepted there
| (say, trying to clear a million issues at once or other
| equally ridiculous actions).
|
| Of course, by itself it won't stop someone from clearing a
| single issue to a million of "users". But may still help in
| id'ing such actors.
| LunaSea wrote:
| Have you tried differentiating ISP vs hosting provider traffic?
|
| Also, you could try to detect headless browsers like Headless
| Chrome.
| pcarmichael wrote:
| Lots of VPNs route through hosting providers. Some hosting
| providers are more prevalent with spammers than others. But
| in the end you can only really use it as a scoring measure.
| It's not nearly binary enough to straight filter using it.
| walrus01 wrote:
| google "residential proxies for sale" - there is a whole huge
| grey market in selling proxies from peoples' actual
| residential internet connections, usually to be driven by
| bots or people in click-farms somewhere in a low labor cost
| location.
|
| If you want to appear to be a legitimate comcast, charter,
| centurylink, frontier end user on a last mile broadband
| connection in a house somewhere in the US48 states, that's a
| standard service that fraudulent organizations make use of
| now.
|
| Usually using software that people have been tricked into
| installing on their computers.
| technion wrote:
| I don't need to Google them. "Residential proxies for sale"
| is one of the most common Facebook ads I see. There was one
| on my screen just five minutes ago. It's staggering that
| the industry is so blatant.
| darkwater wrote:
| Exactly, and this is why "in theory" Google reCaptcha, the
| that is just a checkbox, is a needed solution. If they only
| used it to just stop spam and not also to track legitimate
| users...
| gogopuppygogo wrote:
| Sounds like something a lot of companies need. Could be a good
| lifestyle startup company.
| nathancahill wrote:
| Require a credit card on signup.
| wu_187 wrote:
| That won't work as the same bad actors typically dabble in
| CC fraud as well. They will typically already have
| thousands of CC numbers at their disposal.
| dec0dedab0de wrote:
| Credit card and a delay.
| applecrazy wrote:
| That would basically destroy user acquisition metrics. As
| a user, I don't want to wait _days_ to get access to
| something.
| f430 wrote:
| My solution was to block IP addresses from the following
| regions who were never our target customers anyways:
|
| India, Midde East, Turkey, Russia, China, South East Asia
| (except Singapore), All of Eastern Europe, Africa and
| island nations.
|
| It immediately reduced the amount of fraud and spam. Sure
| you might miss out on the <1% of your revenues but its a
| tradeoff I'm okay with.
| sergiotapia wrote:
| Yep, totally agree. We blocked China (a country we do
| zero business in) and saw a drastic double digit
| percentage decrease.
| saurabhnanda wrote:
| What is this going to do to their Sendgrid/SES bill this month, I
| wonder!
| nielsole wrote:
| I would be more concerned with deliverability. Gitlab email
| notification now has a Spam score of 6.4 (up from 0.0) in
| Fastmail
| oakesm9 wrote:
| For context, the default settings are to move anything with a
| score above 5 to the spam folder.
| bennyp101 wrote:
| Interesting that both Gitlab and Github had issues today -
| coincidence? Maybe Bitbucket will be next!
___________________________________________________________________
(page generated 2021-02-08 23:01 UTC)