[HN Gopher] Rubygems.org AWS Root Access Event - September 2025
___________________________________________________________________
Rubygems.org AWS Root Access Event - September 2025
Author : ilikepi
Score : 216 points
Date : 2025-10-09 17:48 UTC (5 hours ago)
(HTM) web link (rubycentral.org)
(TXT) w3m dump (rubycentral.org)
| xer0x wrote:
| The Rubygems take over drama continues!
| terracatta wrote:
| That email screenshot is pretty bad for Arko. It clearly shows
| intent to sell PII data to a third party during a time when Ruby
| Central had diminished funds and needed help affording basic
| services.
|
| What the fuck.
| mikey_p wrote:
| Why do they need money? What happened to their funding?
| dismalaf wrote:
| It was after a big sponsor pulled out and presumably before
| Shopify stepped in...
| krowek wrote:
| One of the major sources of funding was cut because they
| sided with the devil...
| baggy_trough wrote:
| The dastardly devil who made the whole thing popular in the
| first place! Quite a devil indeed.
| ioasuncvinvaer wrote:
| Yes? What's your point?
| eutropia wrote:
| They buried the lede...
|
| Arko wanted a copy of the HTTP Access logs from rubygems.org so
| his consultancy could monetize the data, after RC determined they
| didn't really have the budget for secondary on-call.
|
| Then after they removed him as a maintainer he logged in and
| changed the AWS root password.
| JeremyNT wrote:
| What a truly wild situation.
|
| In a certain sense this post justifies _why_ RC wanted so badly
| to take ownership - I mean, here you have a maintainer who
| clearly has a desire to sell user data to make a buck - but the
| _way_ it all played out with terrible communication and rookie
| mistakes on revoking access undermines faith in RC 's ability
| to secure the service going forward.
|
| Not to mention no explanation here of who legally "owned" the
| rubygems repo (not just the infra) and why they thought they
| had the right to claim it, which is something disputed by the
| "other" side.
|
| Just a mess all around, nobody comes off looking very good
| here!
| andrewguenther wrote:
| In 2025 there's no reason for anyone to be logging into an AWS
| account via the root credentials and this should have been
| addressed in the preventative measures.
|
| There's no actual control improvements here, just "we'll follow
| our procedures better next time" which imo is effectively doing
| nothing.
|
| Also this is really lacking in detail about how it was determined
| that no PII was accessed. What audit logs were checked? Where was
| this data stored?
|
| Overall this is a super disappointing postmortem...
| nerdjon wrote:
| > In 2025 there's no reason for anyone to be logging into an
| AWS account via the root credentials and this should have been
| addressed in the preventative measures.
|
| I am curious what preventative measures you expect in this
| situation? To my knowledge it is not actually possible to
| disable the root account. They also had it restricted to only 3
| people with MFA which also seems pretty reasonable.
|
| It is not unheard of that there could be a situation where your
| ability to login through normal means (like lets say it relies
| on Okta and Okta goes down) and you need to get into the
| account, root may be your only option in a disaster situation.
| Given this was specifically for oncall someone having that
| makes sense.
|
| Not saying there were not failures because there clearly are,
| but there have been times I have had to use root when I had no
| other option to get into an account.
| flumpcakes wrote:
| I'm now questioning my sanity but I thought you could disable
| login for the root account in AWS.
| the_mitsuhiko wrote:
| Since there are certain operations that can only be done
| with the root account, there is no way to disable access to
| it.
| jhealy wrote:
| Since 2024 you can disable the root credentials on all
| accounts except the Organization management account:
| https://aws.amazon.com/blogs/aws/centrally-managing-root-
| acc...
|
| I don't think the post mortem details whether the root
| access was on the org management account or an org member
| account.
| the_mitsuhiko wrote:
| Oh wow. I completely missed that change.
| oneplane wrote:
| You don't need the root account, unless you need to bypass
| all policies. In such a scenario, you a use the root access
| reset flow instead, reducing standing access.
|
| As for other flows (break glass, non-SSO etc), that can all
| be handled using IAM users. You'd normally use SAML to assume
| a role, but when SSO is down you'd use your fallback IAM user
| and then assume the role you need.
|
| As for how you disable the root account: solo accounts can't,
| but you can still prevent use/mis-use by setting a random
| long password and not writing it down anywhere. In an Org,
| the org can disable root on member accounts.
| nerdjon wrote:
| To me that sounds like security by obscurity not actual
| security.
|
| If you have the ability to go through the reset flow than
| then why is that much different than the username and
| password being available to a limited sets of users. That
| would not have prevented this from happening if the
| determination was made that all 3 of these users need the
| ability to possibly get into root.
|
| As far as having an IAM user, I fail to see how that is
| actually that much better. You still have a user sitting
| there with long running credentials that need to be saved
| somewhere that is outside of how you normally access AWS.
| Meaning it is also something that could be easily missed if
| someone left.
|
| Sure yes you could argue that the root user and that IAM
| user would have drastically different permissions, but the
| core problem would still exist.
|
| But then you are adding another account(s) on top of the
| root account that must exist that you now need to worry
| about.
|
| Regardless of the option you take, the root of the problem
| they had was 2fold. Not only did they not have alerts on
| the usage of the root account (which they would still need
| if they switched to having long running IAM users instead,
| but now they would also need to monitor root since that
| reset flow exists) and their offboarding workflow did not
| properly rotate that password, which a similar problem
| would also exist with a long running IAM user to delete
| that account.
|
| At the end of the day there is not a perfect solution to
| this problem, but I think just saying that you would never
| use root is ignoring several other issues that don't go
| away just by not using root.
| zrail wrote:
| Resetting the root password requires proving access to
| the email address associated with the root account. It
| also leaves a massive papertrail.
| akerl_ wrote:
| By "massive papertrail" do you mean "a pair of emails to
| the associated address"?
| oneplane wrote:
| No, a massive amount of CloudTrail logs.
| akerl_ wrote:
| Does it? Pretty sure that logging in as root generates
| one cloudtrail per action, regardless of whether or not
| you did it with a saved password or you reset the
| password. Resetting the password doesn't generate a
| cloudtrail event as far as I've seen.
| oneplane wrote:
| Not using root means not bypassing policies. There is no
| way to not bypass all policies. So yes, never using root
| makes that issue go away completely.
|
| As for all the other stuff: what it does is it creates
| distinct identities with distinct credentials and
| distinct policies. It means that there is no multi-party
| rotation requires, you can nuke the identity and
| credentials of a specific person and be done with it. So
| again, a real solution to a real problem.
| nerdjon wrote:
| It solved "a" problem buy not "the" problem.
|
| It depends on what the goal of all of this was, which is
| unclear. If the goal was simply to get the data that they
| originally wanted it does not solve that problem and it
| would have just happened a different way.
|
| According to the article there was 11 days between the
| first actions taken and them finding out it happened.
|
| If instead of a root account you have a long running IAM
| user that you can then assume into the role you normally
| use through SSO. If you also do not monitor that account
| with proper alerts and proper offboarding procedures than
| they could have logged into that account and retrieved
| the data they wanted.
|
| Which again is the reason I am saying they just saying
| not using root is not a magic bullet that would have
| avoided problems. Maybe the situation would have been
| different but they still could have done a lot in 11
| days.
| codegeek wrote:
| "The root account credentials, essentially the highest level of
| administrative control, are stored in a shared enterprise
| password manager in a shared vault to which only three
| individuals had access: two current Ruby Central staff members
| and one former maintainer, Andre Arko"
|
| I am wondering. Did they at least have MFA enabled on the root
| login or not ?
| terracatta wrote:
| Yes because they state under the section "Root Cause Analysis"
|
| > Ruby Central failed to rotate the AWS root account
| credentials (password and MFA) after the departure of personnel
| with access to the shared vault.
| sersi wrote:
| If both password and MFA are stored in the same shared vault
| then MFA's purpose is compromised. Anyone getting access to
| that shared vault has the full keys to the kingdom the same
| as if MFA wasn't enabled.
|
| Also in this day and age, there's no reason to have the root
| account creds in a shared vault, no-one should ever need to
| access the root account, everyone should have IAM accounts
| with only the necessary permissions.
| blibble wrote:
| > If both password and MFA are stored in the same shared
| vault then MFA's purpose is compromised. Anyone getting
| access to that shared vault has the full keys to the
| kingdom the same as if MFA wasn't enabled.
|
| absolutely
|
| > no-one should ever need to access the root account
|
| someone has to be able to access it (rarely)
|
| if you're a micro-org having three people with the ability
| to get it doesn't seem that bad
|
| everything else they did is however terrible practice
| mikey_p wrote:
| So they failed to properly protect their credentials?
|
| This sure doesn't reflect all this supposed professionalism and
| improvements RC was supposed to make.
|
| Years ago, I decided with all the DHH drama, that using Rails
| was too much of a liability and this shit just makes the whole
| Ruby ecosystem a liability to anything build in that ecosystem.
| andrewmcwatters wrote:
| Ethical and legal boundaries? RubyGems Privacy Notice already
| tells you that they share information with a number of large
| firms and notably ClickHouse... for "Customer Data Processing."
|
| All this proposal does is request from one of the maintainers/on-
| call providers? another entry in this Privacy Notice as a part of
| a payment deal.
|
| This is a mess, but it also unnecessary smears both sides. It
| calls out that RubyCentral had poor cloud management in place,
| and it trashes an on-call provider.
|
| This is a terrible postmortem and all it does is advertise to
| users that RubyCentral doesn't know what it's doing.
| RhythmFox wrote:
| Seems like there is a coordinated effort to boost support for
| comments that accept their narrative and downvote opinions that
| question it... or Hacker News comment sections are just very
| biased toward corporate narratives... possible for sure ;)
| jaredcwhite wrote:
| I have no doubt a certain contingent who tend to flock on a
| certain former bird-themed network are happily spreading
| their votes around.
| andrewmcwatters wrote:
| I think the point being that whatever was going on with
| access controls here was blatant gross negligence, because
| what is this guy doing with access who doesn't literally own
| the organization or the intellectual property?
|
| Set that aside, which obviously stinks, but then why is said
| obviously incompetent organization sharing confidential
| corporate emails with the public, saying this guy proposing a
| corporate data access plan in exchange for consulting fees is
| crossing "ethical and legal boundaries?"
|
| Are you serious? This reeks of someone or a number of people
| who have no idea what they're doing. Have these people never
| worked with a business before? Never spoken with a software
| firm? A marketing firm? Any consultancy whatsoever?
|
| Who owns the registry? What is going on here? This is insane.
|
| It makes the entire Ruby ecosystem look like it's run by
| children.
| RhythmFox wrote:
| If only the Ruby ecosystem was really run by children, that
| sounds way more for than whatever this is. Joking aside
| though I think you are very right to question the
| professionalism behind this post. The narrative spin here
| is so toxic to good faith arguments around how we could
| change the 'supply chain' of Gems for the better. Wasn't
| that what this was all about Ruby Central? Can we get an
| update on that instead of this attack on your former admin?
| andrewmcwatters wrote:
| Apparently the guy isn't even just a former admin, he
| actually seems to be a pivotal person around the whole
| formation of the RubyGems ecosystem, which is even
| weirder.
|
| This isn't some random consultant who happened upon an
| admin position.
| krmbzds wrote:
| All my comments on the previous thread has been retroactively
| flagged so I can attest to this.
| ttfvjktesd wrote:
| > failed to rotate the AWS root account credentials ... stored in
| a shared enterprise password manager
|
| Unfortunately, many enterprises follow the poor practice of
| storing shared credentials in a shared password manager without
| rotating them when an employee with prior access leaves the
| company.
| phoronixrly wrote:
| So we have DHH with his unhinged posts on one side, and Arko
| wanting to sell PII on the other. Great!
|
| I think we need an f-droid-like project for Rubygems that builds
| the gems from source, and takes care of signing, and is backed by
| a non-profit that is independent from Rails/Shopify
| dismalaf wrote:
| Gem can pull in gems from any repository, even straight from a
| git server like GitHub. And most of the time gems _are_ built
| from scratch on your computer, Nokogiri is the only one I can
| think of that isn 't.
| zrail wrote:
| The problem, as with every package manager, is transitive
| dependencies. It's all well and good to set up direct
| dependencies to only pull from git repositories, but bundler
| still needs a way to resolve those gems' dependencies.
|
| You could pre-resolve every dependency in your chain to a git
| repository, even to a fork under your own control, but that
| will end up being a maintenance nightmare.
| Imustaskforhelp wrote:
| Can't a middle compromise happen as it happens in something
| like golang?
|
| Can some vps/serverless provider not do this like fly.io as
| an recent example with kurt got got? or hetzner?
|
| I think that golang's model can actually be sort of
| cheaper/ more cost effective for servers as compared to how
| ruby might be doing it right now and so cheaper might mean
| that a new non profit can be created which can work on less
| money/outside funding/drama overall
| zrail wrote:
| Retrofitting Go's dependency model into Ruby is not
| trivial. Go has used full URLs for dependencies from the
| jump, making a central package repository irrelevant.
| Ruby doesn't have that. At best you might have a source
| code URL in the gem source that you can access from a gem
| server, but that doesn't really anything. Someone still
| has to provide the index.
| phoronixrly wrote:
| > I think that golang's model can actually be sort of
| cheaper/ more cost effective for servers as compared to
| how ruby might be doing it right now and so cheaper might
| mean that a new non profit can be created which can work
| on less money/outside funding/drama overall
|
| It also means no code signing and the natural capture of
| most of the ecosystem by Microsoft (due to devs
| preferring to host their code on github, a bundler that
| lacks package hosting will be entirely at the whim of MS)
| Imustaskforhelp wrote:
| If you are worried about github/MS capture... Then my
| suggestion is to just create mirrors of golang projects
| you like on gitlab/codeberg
|
| But this is so so much better than having arko or
| somebody having your PII.
|
| Like I hate github but I am pretty sure that people there
| aren't actively looking for my PII when I download go
| projects or that a single person couldn't really access
| it I suppose
|
| I am not really familiar but if I remember the heads
| project related to coreboot isn't there a way to sign
| your github repository with your ssh key or something
| related (I can be wrong, I usually am)
|
| Like I know it could be a pain in the ass but if you are
| so worried about github, what if we could optionally have
| everything be gpg'd via ssh keys & the project could only
| work if someone shares a ssh key
|
| And something like rubygems could just have a name <->
| github mapping <-> gpg mapping and it might require some
| additional software right now but I am just giving ideas
| maybe for new languages as well I am not sure
|
| What are your thoughts? And what do you think the ideal
| way could be. I have heard from many people (like
| primagen) that golang is the best package model and I
| also resonate with that statement but yeah github is a
| bit of menace/threat to open source
|
| All the more reason to use something like codeberg!
| Imustaskforhelp wrote:
| Yes!
|
| Not even sure why you are being downvoted, this is such a great
| idea actually.
|
| F-droid has been so professional and they are just so
| professional
|
| There was this developer (axet) who recently accused f-droid of
| somehow convincing the users "maliciously" that the funds are
| going to the the creator and f-droid when in reality it was
| going to f-droid and he name called them and what not..
|
| Do you know what f-droid team still said?
|
| They said that they can help him in the donation process and
| remove theirs and they actually took some feedback from what I
| know...
|
| They clarified that the donations in their about page that the
| money that you donate through f-droid in their website's
| homepage donate goes to f-droid only which should be obvious
| but for some it wasn't
|
| they also had f-droid donate in the website links of apps and I
| am not sure when they stopped it but they also stopped it and I
| deeply deeply respect it.
|
| Like, okay maybe mistakes happen but f-droid is seriously good
| corporation. We might need something like that for sure. I
| genuinely think that out of thinking about open source so much,
| I realized that we need to have priorities to share things
| about open source.
|
| F-droid is on the top of the list, its just that great, then
| there is signal/grapheneos or maybe all 3 are on top...
|
| F-droid as an organization is something that I deeply
| appreciate and its a shame of google's attestation. I genuinely
| love f-droid nowadays.
| phoronixrly wrote:
| > Not even sure why you are being downvoted, this is such a
| great idea actually.
|
| Expressing negative opinions about DHH is not well-received
| here.
|
| Oddly enough the Ruby community includes both the most
| thoughtful and gentle people and the biggest assholes I
| know... I refuse to believe the latter are not fringe.
| Imustaskforhelp wrote:
| Yes I absolutely hate DHH as well
|
| and in fact I was using omarchy but then migrated over to
| cachyos hyprland
|
| https://jakelazaroff.com/words/dhh-is-way-worse-than-i-
| thoug...
|
| DHH is not a good guy but the hype around him made me feel
| so. He's weird and racist and fascist.
|
| Stop the hype around dhh and everyone please read the
| article everybody here's DHH reality
|
| Let's ditch the superlatives and review David's post
| objectively: He thinks that even if you
| were born in the UK, you only count as British if you're
| white. He wouldn't consider living in London
| specifically because it has too many people of color.
| He uses racist tropes to accuse Asian men of being
| dangerous predators who attack white women. He
| pushes debunked conspiracy theories about immigrants
| replacing white people. He finds a march where
| speakers called for banning all non-Christian religions and
| ethnically cleansing immigrants "heartwarming".
| Finally -- and maybe most alarmingly -- he argues that all
| of the above is normal and not extreme.
|
| You can use whatever word you want to describe all that.
| But if you, like me, didn't realize that this is who DHH
| is, we can probably agree that he's way worse than we
| thought.
|
| The above lines were from the article
|
| This guy shouldn't remotely be talked about in a good light
| imo, yes I appreciate open source but I can't seperate the
| art from the artist.
|
| I genuinely don't know why they defend this guy.
|
| HN literally flaged this post in literal minutes when it
| had come out but I was lucky to have read it and I will
| continue to spread this word because HN's moderators seem
| to flag anything like this and its kinda sick and enabling
| behaviour really
|
| They will allow the post that cf is sponsoring omarchy
| promotion thing or omarchy links in general but not a dhh-
| is-way-worse-than-i-thought/and I was surprised by how
| quickly they deleted the post that after I had read that
| post, it got flagged and I couldn't even write a comment.
|
| A little bit Shocking if I can be honest.
|
| I was on omarchy but now I am on cachyos hyprland and I
| learnt some custom live iso stuff too, I might make an
| article about it... I edited this because maybe I got a
| little angry towards DHH but I genuinely don't like the
| guy. I genuinely admired him as a person untill I found
| about it and I have strong opinions on him.
|
| I think its the paradox of tolerance, should we as a
| society be tolerant to the intolerant people?
| brunosutic wrote:
| Does anyone know if Arko can have legal troubles (like being
| sued) if it can be proven he "removed authorized users" from
| RubyCentral AWS account?
| hokumguru wrote:
| https://www.justice.gov/jm/jm-9-48000-computer-fraud
|
| at least a misdemeanor. most of the time its prosecuted, a
| felony.
| dismalaf wrote:
| RubyCentral is a legal entity and they paid him for his work so
| they definitely could take action against him if he was shown
| to have harmed users or the company in any way.
| heartbreak wrote:
| Legal recourse is an interesting omission from their update,
| especially considering this was surely approved by their
| attorney(s).
| hokumguru wrote:
| Even if Arko wasn't the unauthorized access, this absolutely
| tarnishes his reputation for the rest of his career, damning
| email notwithstanding.
|
| Horrible time to own/run a consultancy. Can't imagine what his
| other customers are thinking right now.
| dismalaf wrote:
| Lol and in previous posts everyone was acting like Arko is above
| board...
|
| I brought up multiple times that his actions were suspicious, was
| downvoted. Now proof of that plus an email trying to low-key
| extort RubyCentral into allowing him to sell user data...
| ameliaquining wrote:
| It's not extortion, because refusing his offer would have left
| them no worse off than if he hadn't gotten involved in the
| first place. You can still object on other grounds, of course.
| busterarm wrote:
| Props to Ruby Central for taking all of the smears and
| reputational damage on the chin silently while they mitigated an
| actual security incident, made absolutely sure and wrote up a
| proper post-mortem. All of that in line with their original
| statement that their actions taken were in the interest of
| security/integrity of their platform.
|
| If there's any evidence that you need to know who the proper
| stewards of Ruby's gems are, it's this.
| software_writer wrote:
| 100%
| xaxaxa123 wrote:
| odd timing for such an incident..
| busterarm wrote:
| Not really. Someone abusing their access after being removed
| is exactly when such events occur.
|
| Your post suggests conspiratorial thinking when there
| shouldn't be.
| xaxaxa123 wrote:
| my post suggests critical thinking when there should
| be..your post is just defamatory. all of this looks like a
| "problem, action, solution" scheme. what was the takeover
| then, a honest and transparent move by RC? you must be
| kidding. looked pretty much conspiratorial.
| Mystery-Machine wrote:
| An entity that promised security had a security incident due of
| their incompetence to properly secure their production
| environment root access?
| busterarm wrote:
| If somebody is going to abuse their accidentally-retained
| access after being removed from my organization, than the
| incompetence was in having that person in my organization in
| the first place. It turns out they were perfectly justified
| in removing him!
|
| First of all, it's criminal, and second of all, it absolutely
| lights a torch to any credibility they have. I expect people
| don't want to become unhireable.
|
| I've had access/credentials to organizations that I've left
| and never abused them even once.
| blibble wrote:
| it's not that clear cut, because there is no "rubygems
| company" or clear ownership of any of this stuff
|
| it would be quite easy to argue that ruby central had never
| had a right to remove these people at all
|
| > I've had access/credentials to organizations that I've
| left and never abused them even once.
|
| yes, likewise
|
| and if I was Andre I wouldn't have even have ATTEMPTED to
| do this, as it looks terrible regardless of the eventual
| legal position
| queenkjuul wrote:
| I mean I'd much rather it be people who remember to rotate
| their passwords after firing high profile staff
| ctoth wrote:
| AWS account root access on a language package registry for 11
| days. Not EC2 root - AWS account root. Complete control over IAM,
| S3, CloudTrail, every-damn-thing.
|
| They're claiming "no evidence of compromise" based on CloudTrail
| logs that AWS root could have deleted or modified. They even
| admit they "Enabled AWS CloudTrail" after regaining control -
| meaning CloudTrail wasn't running during the compromise window.
|
| You cannot verify supply chain integrity from logs on a system
| where root was compromised, and you definitely can't verify it
| when the logs didn't exist (they enabled them during
| remediation?).
|
| So basically, somebody correct me here if I'm wrong but ... Every
| gem published Sept 19-30 is suspect. Production Ruby applications
| running code from that window have no way to verify it wasn't
| backdoored. The correct response is to freeze publishing, rebuild
| from scratch (including re-publishing any packages published at
| the time? Ugh I don't even know how to do this! ) , and verify
| against offline backups. Instead they rotated passwords and
| called it done.
| yargzeblog wrote:
| IMO the only way to avoid doing a total rebuild is to have
| Andre Arko:
|
| 1. Admit that he was the unauthorized actor (which means he's
| probably admitting to a crime?) 2. Have him attest he didn't
| exfil or modify the integrity of service while committing a
| crime.
|
| If I was Ruby Central I would give clemency on #1 in exchange
| for #2 and I think #2 helps Andre Arko.
| ctoth wrote:
| So you would expect people to accept that the entire root
| chain of custody for the Ruby supply chain is attested by ...
| A guy saying he didn't do anything bad? I have a cool
| cryptocurrency you might wanna check out that I definitely
| don't have a backdoor to!
| LamaOfRuin wrote:
| If Andre doing that was criminal, it seems quite possible
| that their original takeover of the github organization was
| also criminal?
|
| I have been waiting to hear if there would be any civil
| action on it since it's not at all clear they had any rights
| to do most of what they did.
| SAI_Peregrinus wrote:
| Ruby Central isn't capable of giving clemency. They could
| refuse to testify in any prosecution, but they don't get to
| pick whether a relevant attorney general or district attorney
| decides to prosecute.
| ctoth wrote:
| Thinking about this a bit more... it sure is interesting that
| around the time of a competing project launch that something
| just happens which might reasonably completely compromise trust
| in the previous incumbent, isn't it? Odd!
| dmix wrote:
| That was intentional according the Joel Drapper who leaked
| this incident, he wanted to make Ruby Central look bad
|
| https://www.reddit.com/r/ruby/comments/1o2bxol/comment/ninly.
| ..
|
| >> Why did Joel give so little time of advance notice before
| publishing his post revealing Andre's production access? That
| struck me as irresponsible disclosure, but I may have missed
| something.
|
| > I decided to publish when I did because I knew that Ruby
| Central had been informed and I wanted the world to be
| informed about how sloppy Ruby Central were with security,
| despite their security _posturing_ as an excuse to take over
| open source projects.
|
| > What I revealed changed nothing about Ruby Central's
| security, since Andre had access whether I revealed that he
| did or not. When you have security information that impacts
| lots of people, you publish it so they can take precautions.
| That is responsible disclosure.
| akerl_ wrote:
| Given the context of the post, it seems like "Enabled AWS
| CloudTrail, GuardDuty, and DataDog alerting" means "enabled
| alerts via CloudTrail, GuardDuty, and Datadog", not "enabled
| Cloudtrail logging". Otherwise the comment about reviewing
| Cloudtrail wouldn't make sense.
| ctoth wrote:
| So the attacker turns logging off (was log file validation
| enabled? usually isn't in Terraform ) which does not fire an
| alert because there is no alerting. Then does their bad stuff
| ... Then modifies the logs (which are in an S3 bucket on the
| compromised account, remember!) Then they turn logging on?
| The whole point is alerts go outside AWS. They go to like,
| your inbox or pagerduty or whatever. If they had no alerts
| then what use are their logs, which could have been modified?
| Do you think they set up cross-account logging or had
| enable_log_file_validation set to true?
| shevy-java wrote:
| You are actually right, I didn't think of this before.
|
| How can they ensure that nobody else did any tampering?
|
| It seems RubyCentral did not think this through completely.
| busterarm wrote:
| You can set up EC2 instances in a way that that just having
| AWS root access doesn't give you ssh/console access to the
| instances. You can still do things like Run Command but that
| leaves a very obvious trail (although even this is
| preventable with enough effort).
|
| Also you can enable cloudtrail log validation which can
| ensure you know if you're looking at tampered logs or not.
|
| Really it all depends on how their accounts are set up.
| Unless you know the operational details you can't make a call
| here.
|
| I've run a multi-million dollar/year AWS Org for the last
| decade or so and setting things up this way is kind of brass
| tacks.
| ctoth wrote:
| So you almost certainly know that a lot of IaC tooling has
| terrible defaults like enable_log_file_validation being set
| to false. Based on the quality of their credential
| management and what else you can read from this blog post,
| how much would you wanna wager they did it right?
| busterarm wrote:
| I assume everyone is professional until they prove
| otherwise.
|
| Based on how things have been described on both sides, it
| actually sounds like they do a pretty good job.
| Oversights happen -- we're all human -- and this access
| was already limited to a small single-digit number of
| people. Given the history, it's reasonable that Arko
| would have had this high of a level of access and the
| oversight was in forgetting that when removing him.
|
| Also it's reasonable to assume that people with that
| access wouldn't do something criminal/malicious, and if
| they did, while annoying, the situation is very easily
| recoverable. Especially if you're using IaC tooling as
| you mentioned.
|
| If you're already taking the position that Ruby Central
| are "the bad guys" it's easy to assume that they're doing
| everything wrong, but that would be a mistake.
| blibble wrote:
| > It seems RubyCentral did not think this through completely.
|
| this is the problem when you fire all the maintainers who do
| anything
| frenchtoast8 wrote:
| I believe this is a scenario where AWS recommends multiple
| accounts.
|
| 1. Create another "management" AWS account, and make your
| other AWS account a child to that.
|
| 2. Ensure no one ever logs in to the "management" account, as
| there shouldn't be any business purpose in doing so. For
| example, you should require a hardware key to log in.
|
| 3. Configure the "management" account to force children
| account to enable AWS Config, AWS CloudTrail, etc. Also force
| them to duplicate logs to the "management" account.
|
| Step 2 is important. At the end of the day, an organization
| can always find a way to render their security measures
| useless.
| 0cf8612b2e1e wrote:
| 2) Surely, someone needs access to the account. How do you
| prevent those with access from using it? Security feels
| like turtles all the way down where you ultimately have to
| trust a few people to do the right thing.
| zrail wrote:
| The only reason someone would need access to the
| management account would be maintaining child accounts
| and IAM roles or reviewing logs, none of which should
| need root.
| arianvanp wrote:
| You can't enable or disable AWS Cloud Trail as far I know?
|
| You can enable the persistent storage of trails. But you can
| always access 90 days of events regardless of that being
| enabled
| frenchtoast8 wrote:
| This was my understanding as well, but earlier I couldn't
| find any documentation to prove this so I never wrote a
| comment.
|
| CloudTrail can be configured to save logs to S3 or CloudWatch
| Logs, but I think that even if you were to disable, delete,
| or tamper with these logs, you can still search and download
| unaltered logs directly from AWS using the CloudTrail Events
| page.
| ctoth wrote:
| Only management events, see
| https://news.ycombinator.com/item?id=45532772
| placardloop wrote:
| CloudTrail logs for the last 90 days are enabled by default,
| cannot be turned off, and are immutable, even by root. If you
| view this "event" as starting when Arko was supposed to have
| their access terminated, that's within the 90 day window and
| you can indeed trust the logs from that period.
| ctoth wrote:
| CloudTrail's 90-day immutable Event History only logs
| management events (IAM changes, instance launches, bucket
| creation). It does NOT log:
|
| * S3 object reads/writes (GetObject, PutObject) - these are
| "data events" requiring explicit configuration[0]
|
| * SSH/RDP to EC2 instances - CloudTrail only captures AWS API
| calls, not OS-level activity[1]
|
| With root access for 11 days, someone could modify gem files
| in S3, backdoor packages, SSH into build servers - none of it
| would appear in the logs they reviewed. Correct?
|
| [0] https://docs.aws.amazon.com/awscloudtrail/latest/userguid
| e/l...
|
| [1] https://repost.aws/questions/QUVsPRWwclS0KbWOYXvSla3w/clo
| ud-...
| placardloop wrote:
| SSH is totally irrelevant here. Having AWS root account
| access doesn't give you any ability to SSH to or otherwise
| access running instances. You could access data on those
| instances by cloning the EBS volumes or modifying build
| pipelines or changing network access or similar, but these
| would all show up in CloudTrail even without data events
| enabled.
|
| For S3 objects, you don't necessarily need data events to
| identify if tampering happened. S3 objects are immutable as
| well, so if any changed you would see that reflected in the
| creation date and new hashes that S3 attaches as tags,
| which you can correlate with application logs to see if
| they match up or not. It's not as simple as data logging,
| sure.
|
| But you're also missing the key component here that they
| did _not_ say they only just enabled CloudTrail logs,
| they're saying they just now enabled CloudTrail log
| alerting. We don't have any idea if data events were
| enabled or not, or if things like flow logs were enabled or
| not, or what other investigation tools they have running at
| the application layer. However, even if none of existed,
| there's still a lot more audit-ability of events that
| happen in an AWS account than you're implying, even the
| root account.
| knert wrote:
| I'm not sure this is true. The EC2 web console terminal
| drops me right into root on any of my instances.
| placardloop wrote:
| Ahh you're right, there are some that just initiate a
| connection via something like Session Manager, but those
| connections where AWS initiates the connection for you
| are logged in CloudTrail, even without data events, and
| root doesn't give you any ability to directly SSH into an
| instance outside of those methods (you cannot, for
| example, use root to find out what the private keys are
| for logging into an instance) so we're back to the fact
| that any such access would be auditable.
| coredog64 wrote:
| Not going to bother reading the article, but will chime in here
| that the recommendation from AWS is to have a separate security
| account within your organization that _only_ holds your
| CloudTrail logs. This does potentially double your cost, as you
| only get one CloudTrail for free, and it 's very useful to have
| an in-account trail for debugging purposes.
|
| Organizations are also useful because you can attach SCPs to
| your accounts that deny broad classes of activities even to the
| root user.
| tptacek wrote:
| Isn't the subtext of this post pretty clearly that the
| unauthorized actor was Andre Arko, who had until days prior all
| the same access to RubyGems.org already?
|
| The impression I have reading this is that they're going out of
| their way to make it clear they believe it was him, but aren't
| naming him because doing so would be accusing him of a criminal
| act.
| phoronixrly wrote:
| The other subtext is that they literally have no idea how to
| run rubygems securely... And what to do in case of a security
| incident...
| JoshTriplett wrote:
| The other other subtext is that this sure is an effective
| distraction from their governance problems, and muddies the
| waters. Given the utter lack of trust I have for anything
| the Ruby Central folks say at this point, given the amount
| of spin and misinformation they've spread already, my
| default assumption is that this is an excuse to malign
| someone who may well have had legitimate access, in the
| process of claiming that you're locking things down, which
| was always the excuse being made for kicking people out.
| hitekker wrote:
| At this point, it looks like all parties involved
| contributed to the governance problems over many years
| https://archive.md/SEzoV
|
| > Regarding Arko's blog post about his removal, McQuaid
| [Homebrew Maintainer] told me it's good that Arko is
| crediting other people for their contribution and that
| he's following open source principles of community and
| transparency, but that "his 'transparency' here has been
| selective to things that benefit him/his narrative, he
| seems unwilling or unable to admit that he failed as a
| leader in being unwilling or unable to introduce a formal
| governance process long before this all went down or
| appoint a meaningful successor and step down amicably."
| tptacek wrote:
| I'm addressing the question of whether we all had better
| assume all the RubyGems published after this incident were
| compromised, and my response is "that is probably not
| rational since the actor in this scenario had all this
| access legitimately just days beforehand". The rest, I
| don't care.
| phoronixrly wrote:
| Look, it's enough to know that Rubygems did not require
| 2FA before August 2022. There were gems with millions of
| downloads with owners without 2FA on their accounts. I
| think your initial assumption is pretty safe even without
| the ongoing fiasco.
| ctoth wrote:
| Let's say that they are 100% correct, we parse the subtext as
| text, it was totally him.
|
| We still do not know the critical details of how (and when)
| _he_ stored the root password he copied out of their password
| manager (encrypted in his own password manager? on his pwned
| laptop? in dropbox? we 'll never know!) therefore the whole
| chain of custody is still broken.
| deng wrote:
| If they really have ethical concerns regarding sharing data with
| third parties, maybe they should update their privacy policies
| accordingly?
|
| "We collect information related to web traffic such as IP
| addresses and geolocation data for security-relevant events and
| to analyze how and where RubyGems.org is used."
|
| (https://rubygems.org/policies/privacy)
|
| "We may share aggregate or de-identified information with third
| parties for research, marketing, analytics, and other purposes,
| provided such information does not identify a particular
| individual."
|
| (https://rubycentral.org/privacy-notice/)
| its-summertime wrote:
| They do seem to have found the perfect scapegoat to point
| everyone towards to deflect from all the other issues with their
| actions
| RhythmFox wrote:
| Not only that but they get to use an individual who they have
| philosophical differences with. You can say it was 'good
| security practice', tarnish his reputation, and get to switch
| the narrative to something sympathetic to yourself all in one
| go. Very convenient for them.
|
| I think they make a lot of overly strong claims here, even
| though there are plenty of alternative explanations possible.
| The mere fact that 3 people had AWS root access during this
| period but they only identify one and never question that it
| could have been one of the others is telling. They reallllly
| want you to just take it as obvious that 1) all these actions
| were taken by 1 individual and 2) that individual was
| malicious. Then they sprinkle in enough nasty sounding
| activities and info about Andre to get you to draw the
| conclusion that he is bad, and did bad things, and they had to
| do these things the way they did.
|
| Using what reads like a business strategy email as a 'nefarious
| backstory' is so bad faith. I bet if you got access to all the
| board's emails you would see a ton of proposals for ways to
| support RubyGems that may not all sound great in isolation.
| They are being just transparent enough to bad mouth Andre while
| hiding any motivations from their end as purely 'security'
| related.
| jmuguy wrote:
| Well this certainly clarifies a lot. I'm not sure I'm confident
| that they know nothing was compromised further or the extent of
| any data exfil, it sounds like a lot of logging was enabled
| _after_ the incident occurred.
| jnewland wrote:
| This is a pretty hilarious and long-winded way to say "we have no
| idea how to lock someone out of a web service:"
|
| > 1. While Ruby Central correctly removed access to shared
| credentials through its enterprise password manager prior to the
| incident, our staff did not consider the possibility that this
| credential may have been copied or exfiltrated to other password
| managers outside of Ruby Central's visibility or control.
|
| > 2. Ruby Central failed to rotate the AWS root account
| credentials (password and MFA) after the departure of personnel
| with access to the shared vault.
| TehCorwiz wrote:
| Right?! Did nobody there think to actually disable the
| accounts? These are the people who are harping about "security"
| being the reason for the ham-fisted takeover of the source
| repos, but they didn't secure the production infrastructure?
| colonwqbang wrote:
| It didn't occur to them that he might have written the password
| down? That's wild.
| shevy-java wrote:
| I'd recommend to people to wait for a response - RubyCentral
| spins up a gazillion accusations right now and has been in the
| last days (and, it is also incomplete, because why did they fire
| every dev here and placed Marty Haught in charge specifically?
| They never were able to logically explain this; plus, why didn't
| they release this write-up before? It feels very strange to wait
| here; they could have clarified things before, but to me it seems
| they kind of waited and then tried to come up with some
| explanation that, to me, makes no real sense).
|
| I also highly recommend to not accept RubyCentral's current
| strategy to post very isolated emails and insinuate that "this is
| the ultimate, final proof". We all know that email conversation
| often requires lots of emails. So doing a piecemail release
| really feels strange. Plus, there also were in-person meetings -
| why does RubyCentral not release what was discussed here? Was
| there a conflict of interest due to financial pressure?
|
| Also, as was already pointed out, RubyCentral went lawyering up
| already - see discussions on reddit. Is this really the
| transparency we as users and developers want to see? This is
| blowing up by the day and no matter from which side you want to
| look at it, RubyCentral sits at the center; or, at the very
| least, made numerous mistakes, tries to cover past mistakes by
| ... making more mistakes. I think it would be better to dissolve
| RubyCentral. Let's start from a clean state here; let's find
| rules of engagement that doesn't put rich corporations atop the
| whole ecosystem.
|
| Last but not least - this tactical slandering is really annoying.
| If they have factual evidence, they need to bring the matter to a
| court; if they don't, they need to stop slandering people. To my
| knowledge RubyCentral hasn't yet started a court case, and I have
| a slight suspicious that they also will not, because we, as the
| general public, would then demand COMPLETE transparency,
| including ALL of RubyCentral's members and their activities here.
| So my recommendation is: wait for a while, let those accused
| respond.
| dismalaf wrote:
| > let those accused respond.
|
| Literally all we've heard so far is from the other side...
|
| > If they have factual evidence, they need to bring the matter
| to a court
|
| I'd be surprised if they aren't. This post feels very much like
| the amount of disclosure a lawyer would recommend to reassure
| stakeholders.
|
| > rules of engagement that doesn't put rich corporations atop
| the whole ecosystem
|
| Right now the only thing stopping us all from being held
| hostage by rogue maintainers is a rich corporation.
| saghm wrote:
| The rogue maintainers have apparently been been successful
| enough with their stewardship for years to the point that
| people still use and care about the tools they had maintained
| today. On the other hand, the new maintainers sponsored by
| the rich corporation have managed to draw scrutiny
| immediately about how they became the new maintainers and
| apparently failed to effectively protect their new assets
| from a major breach within two weeks of acquiring them
| despite security being their main argument for why they
| should be in charge in the first place.
| saghm wrote:
| Yeah, this is incredibly confusing. The stance that Ruby
| Central has stated since the takeover of the RubyGems (offline)
| tooling on Github was that it was necessary for supply chain
| security, but if this happened literally within a couple of
| weeks of when they tried (and apparently failed?) to remove all
| of the previous maintainers, how does this add any amount of
| confidence in their ability to secure things going forward? If
| they can't even properly remove the people they already knew
| had access that they went out of their way to try to remove,
| it's hard to feel like consolidating their ownership over all
| of the tooling is going to be an improvement.
| rgreeko42 wrote:
| So is this a smear of Arko (and by extension Ruby Gems' sloppy
| security) but dressed up like a Security disclosure?
|
| If I'm reading it right, it seems quite petty (and a bit
| cowardly). Arko was a maintainer was he not? How is that a
| breach? Presumably his credentials were not misbegotten, or is
| that the accusation?
| xaxaxa123 wrote:
| interesting timing, isnt it?!
| nomdep wrote:
| > "Following these budget adjustments, Mr. Arko's consultancy,
| (...), submitted a proposal offering to provide secondary on-call
| services at no cost in exchange for access to production HTTP
| access logs, containing IP addresses and other personally
| identifiable information (PII). The offer would have given Mr.
| Arko's consultancy access to that data, so that they could
| monetize it by analyzing access patterns and potentially sharing
| it with unrelated third-parties."
|
| WTF. This is the same guy that is launched gems.coop, a competing
| index for Ruby gems recently.
|
| On the other hand, RubyCentral actions were truly incompetent, I
| don't know anymore who is worse
| tptacek wrote:
| Presuming, as a group full of security peers kibitzing about this
| in a chat right now all do, that the "unauthorized actor" here is
| Andre Arko, this is Ruby Central pretty directly accusing Arko of
| having hacked Rubygems.org; it depicts what seems to be a black
| letter 18 USC 1030 violation.
|
| Any part of this narrative could be false, but I don't see a way
| to read it and take it as true where Arko's actions would be OK.
| skywhopper wrote:
| The most notable thing about this post is the great lengths to
| which they are willing to go to detail every step of this
| process, something they never came close to doing in the fallout
| from the original stripping of rights from the longtime
| volunteers.
| fijiaarone wrote:
| At least it's not npm. Or cargo.
___________________________________________________________________
(page generated 2025-10-09 23:00 UTC)