[HN Gopher] Heroku Security Notification
___________________________________________________________________
Heroku Security Notification
Author : peterwallhead
Score : 411 points
Date : 2022-05-05 03:28 UTC (19 hours ago)
(HTM) web link (status.heroku.com)
(TXT) w3m dump (status.heroku.com)
| darig wrote:
| peterwallhead wrote:
| Cheers to admin for changing the title, but old title better
| reflected the last update (and I used it because Heroku doesn't
| have a permalink to each new update).
| kaushikt wrote:
| I feel saddened to see this as a heroku side-project user since
| 2011.
|
| We are a small team and were hoping to migrate all our infra to
| Heroku in the upcoming quarter.
| boesboes wrote:
| Seems like they don't have much of their notifications in order
| either.
|
| I haven't heard _anything_ from Heroku on this, my colleague has
| been getting updates since this started...
|
| We are both admins of our companies account.
| Narkov wrote:
| How does a company like Salesforce mess this up for such an
| extended period of time? I understand that companies can make
| mistakes early on in a critical incident but this has been
| going on for weeks?!
| xena wrote:
| Root cause analysis must have failed pretty bad. I worked at
| Heroku and with how much headcount they've lost my Salesforce
| starving the beast over the years, it's not surprising that
| it would take this long for them to react like this. When you
| scare away the best people with Salesforce policies, their
| Jira clone built on Salesforce, and overall being difficult
| to impossible to get more people on the team; yeah people
| will pack their shit up and leave.
|
| Given how much of a web of interdependent and undocumented
| pain heroku is, I'm not surprised it's taken this long. A
| security team without any context to Heroku must have had to
| trace through everything system by system. Especially if
| core-db got popped.
| anyfactor wrote:
| I have seen nothing but bad news about Heroku recently. Albeit
| nothing provides that much ease to use on a free or affordable
| tier when you have minimum computing demands. I use it as my
| fundamental cloud provider for proof of concept stuff, so does
| many people.
|
| Salesforce is doing a terrible job managing this lucrative
| platform. I have no idea why muck up a good service like that.
| They have some plugins and a postgres connector but the drive to
| innovate, the drive to even care stops there. All these news act
| as a reminder that I should move my code to a "real" cloud
| provider.
| onphonenow wrote:
| Ugh - I got the password reset email w silly password complexity
| - never a good sign.
| DethNinja wrote:
| That email made it clear that Heroku lacks fundamental
| knowledge about security. I'm sure they lost some enterprise
| customers, I know I don't open accounts on websites with silly
| password complexity requirements.
| onphonenow wrote:
| Google allowed 6 character passwords for a while, and didn't
| expire them when they increased minimum to 8 for google
| workspace accounts. This has been fantastic, as users can
| remember their password forever even if its higher complexity
| (google does a password strength eval). No rotations either.
|
| I'm pretty confident google will pick-up someone trying to
| brute force a 6 character password. That google will notice
| connections from new / different IPs or browsers. That's
| because google asks for my 2FA in various situations but
| doesn't annoy me by asking for 2FA all the time.
|
| I use one govt system that has something like a 14 character
| password requirement. For even more security if you don't log
| in for 90 days your account goes inactive and the password
| EXPIRES! Very secure you say? Well, to regain access you have
| to provide the answer to a security question - favorite pet!
| That's a 5 letter word that doesn't change (and is probably
| pretty guessable).
|
| Here is another example:
|
| "(b) Information systems must be designed to require
| passwords to be changed not less frequently than every sixty
| (60) days." - SBA IT Security Policy - 90 47 4
| vel0city wrote:
| For Workspace accounts, an admin can choose to enforce
| complexity requirements on next login after making changes
| to the complexity requirements.
| Mandatum wrote:
| Really, really bad form from the Salesforce Trust team here.
| Hopefully the Slack acquisition means better, quicker
| communications.
|
| On a scale of Slack to Oracle on breach notifications, this was
| definitely closer to Oracle.
| curuinor wrote:
| rule of thumb: company A acquires company B, then company A
| does not change to become like company B, company B changes to
| become like company A. there are exceptions, but few. I think
| Slack's communications are going to become more crap,
| personally.
| JorgeGT wrote:
| Boeing's acquisition of McDonnell Douglas being a sad
| exception to that rule of thumb.
| tossaway2134 wrote:
| This isn't the first time Salesforce Cybersecurity has left us in
| the lurch while they perform damage control.
|
| On 17 May 2019, Salesforce performed maintenance on their
| databases that clear permission sets for users. My team was able
| to piece together that the incident happened at about 0200 CDT,
| and Salesforce didn't take ANY noticeable action for at least 9
| hours when they locked all customers out of the platform.
| Salesforce "fixed" the issue, which meant our Admins had to go in
| and reapply a bunch of profile settings...no big deal, right?
| Just a little bit of work for everyone to fix their own accounts.
| Salesforce acted like it wasn't a big deal.
|
| Wrong.
|
| If you were a Salesforce customer that built a tool using the
| Portal or Community tools Salesforce provides for external users,
| there was a 9 hour window when a customer could log in and
| instead of seeing the data you were sharing with them, they would
| see all data for all users. The permissions that indicated that a
| user should only be able to see their own data was gone.
|
| The only reason we knew about this was because we were paying
| extra for advanced logging. We were able to see a few of our
| users logged in during this time and looked at customer records
| they should not have had access to.
|
| Salesforce stood fast that exposing data through their Community
| and Portal tools this did not constitute a breach or even a
| violation of their SOC-2 Type II compliance. We were lucky that
| the only people that had access at the time were licensed
| partners. Nevertheless, our users lost their jobs and were
| stripped of their licenses.
|
| Anyone that was using those tools at the time for any sort of
| direct customer interaction that shared order history, customer
| engagement, referral programs, etc. was not so lucky; doubly so
| if they weren't paying for advanced logging and/or didn't know
| what to look for. Salesforce was more concerned about covering up
| their mistakes than they were about telling their customers that
| there was a problem.
|
| Seeing the Heroku notification page gives me PTSD. This looks
| all-too-familiar to me and I sympathize with those affected by
| this. I still feel like they were negligent back then, and I wish
| I knew who to tell to warn others.
| sofixa wrote:
| > Nevertheless, our users lost their jobs and were stripped of
| their licenses
|
| Why? Maybe I'm assuming good intentions, but a) did they know
| they were seeing records they shouldn't be; b) did they report
| that? Even with yes and no, firing for that seems a little too
| much.
| stevebmark wrote:
| Salesforce's monitoring, availability, incident handling,
| transparency, and accountability, are abysmal. We too we're
| affected by the incident. I sincerely hope they lose some of
| their compliance certifications, because their behavior is
| unacceptable.
| tossaway2134 wrote:
| Coincidentally the SOC report that would have covered this
| incident is no longer available on their website, but I
| wouldn't be holding my breath.
|
| I don't recall there being any notes of material deficiency
| in their SOX reporting for the fiscal year either.
| [deleted]
| TobiK2020 wrote:
| I'm really wondering right now how hard or easy it was to
| suddenly change all of their customers' passwords in the system.
| henryaj wrote:
| Obviously Heroku have handled this horribly - but are any small
| startups out there considering replatforming? Still seems like a
| lot of hassle and the competition I've tried (Cloud66, excid3's
| thing) haven't been as good.
| sleepyhead wrote:
| What was the problem with Cloud66?
|
| I looked at Render but they move data out of the chosen region
| (so out of EU) and that is a huge issue for our clients. They
| also proxy through Cloudflare which is another big problem when
| you are dealing with sensitive data.
| anurag wrote:
| Re: Render + Cloudflare, all data is encrypted all the way to
| Render. What is the specific issue you're referring to?
| sleepyhead wrote:
| The main problem is that data is transferred out of EU, it
| is simply not acceptable for a EU company regardless of any
| DPA (history have shown here that any such DPA would be
| invalid in the near future).
|
| However some of our clients (for example in health or
| financial industry) would be concerned that data is proxied
| through a third-party, doesn't matter if it is encrypted
| (also it's unclear to me how keys etc are managed and what
| data Cloudflare can access).
|
| btw: Render looks awesome but at the current offering is it
| not an option for us.
| anurag wrote:
| Understood on data transfer out of the EU. It's high up
| on our list. Also very happy to share details on CF
| encryption over email (support@render.com).
| staindk wrote:
| I'd maybe use this as an opportunity to prioritise moving
| everything over to AWS.
|
| I'm sure for some it may be an unwieldy amount of work but for
| others (depending on tech stack etc.) it ought to be fairly
| doable. In the long run it'll save money too.
| ojame wrote:
| I'm interested in peoples experience with this and if it's
| relatively true.
|
| We (like many others I assume) pay more for Heroku than AWS
| as it allows us to "outsource" our dev ops. We are a small
| team (sub 15) with a decent sized, decade old app. We've had
| it on AWS before (and used platforms like BuildKite) but both
| required much more overhead (in terms of employee salary).
| Anecdotally I've heard the same from friends, though I
| understand AWS works well and is cheaper if you know AWS
| well.
| Dave3of5 wrote:
| > I'm interested in peoples experience with this and if
| it's relatively true.
|
| My personal experience is that AWS is always more
| expensive. You don't use AWS because of the cost saving you
| use it because:
|
| * Top of the line h/w * Always the first to bring out new
| features * Very high availability of resource, like
| seriously I've never had a time even during the pandemic
| where they struggled with resource availability *
| Resilience of AWS systems are very high * Very good support
|
| There was an article a few months back where a teams tried
| moving from Heroku to AWS to save money and ended up
| spending nearly 3x the amount. Heroku give more resource
| than you actually pay for by default and it turns out they
| were using the extra resource during normal operation. When
| they done their calcs for a switch to AWS they used the
| quoted resource that Heroku say you get and there system
| died due to being under resourced. They had to up the
| resource which pushed them well over their budget. I'll try
| and find the article.
|
| Please Note: Heroku servers are on AWS already
| t_sawyer wrote:
| I'm interested if application ENVs were stolen. If they stored
| Github APIs in plain text I'm sure our ENVs were too.
|
| I went ahead and reset my planetscale passwords and moved my app
| out of Heroku. But I only had two apps hosted there I feel for
| anyone with a large number of apps on there.
| productceo wrote:
| What are some good Heroku alternatives these days?
| zoomzoom wrote:
| Heroku alternatives have gone in a few different directions: -
| next-gen PasS that are more opinionated and offer wider range
| of services. Also can be cheaper. examples are Digital ocean,
| Railway or render - Performance-focused PaaS like fly.io -
| "Heroku in your own cloud" like porter.dev, architect.io or
| quovery - k8s tooling like garden.io, ReleaseHub, even gitlab -
| these are often geared more towards internal DevOps teams at
| larger orgs when compared to the very low-lift PasS providers -
| serverless providers - like cloudflare functions, AWS lambda,
| GCP Cloud Run - Vercel/Netlify - SPA + serverless with a great
| developer experience - Replit - kind of in a category of their
| own but they have integrated hosting/datastores/user auth
|
| Lots of awesome products here, I'd argue that only replit is a
| true 10x change from the Heroku innovations in terms of
| providing a next-gen developer experience.
|
| I'm the cofounder of a new company called Coherence, that we
| think creates a new direction and offers a better platform for
| the next leap forward. By integrating from dev to prod and
| capturing the whole SDLC, as well as by operating in your own
| cloud, we're focused on delivering the best developer
| experience possible, without compromising anywhere else. Check
| us out at https://www.withcoherence.com. We're in an early
| closed beta so not yet a fit for all teams, but feedback is
| welcome!
| beninsydney wrote:
| Digital Ocean's "App Platform" is pretty nice, although not as
| slick as Heroku -
|
| https://www.digitalocean.com/products/app-platform
|
| Railway.app is pretty nice, very slick interface, this has the
| most "heroku-feel" -
|
| https://railway.app
| bluehatbrit wrote:
| I've used the DO App platform, generally it's pretty solid
| but getting up and running was miserable. The documentation
| is really thin and there's a bunch I just had to discover
| along the way, in particular while getting up and running
| with a DB as it's not clear how the certs are delivered into
| the environment variables etc.
|
| It's been solid since I got up and running but took me about
| a day to get a simple CRUD app moved over to it.
|
| If they can improve the documentation it'll be much better.
| Right now I'm not considering it for future projects though
| just because platforms like Fly.io and Render seem to have
| better docs and additional functionality that DO Apps doesn't
| have yet.
| oxff wrote:
| I've seen fly/io and render/com mentioned in these Heroku
| discussions, but I have yet to test either of them.
| kenrose wrote:
| Render.com
| henryaj wrote:
| Stuff like this worries me -
| https://community.render.com/t/genuine-question-why-does-
| ren...
| anurag wrote:
| (Render founder) Stuff like ^ worries me even more. What
| keeps me sane is our engineering team obsessing over
| reliability, and learning from every single incident, no
| matter how small. We're improving every day.
| SkyPuncher wrote:
| Aptible (though geared more towards Healthcare/Compliance
| space).
|
| ^ Note: I use to work for Aptible. Great company. Great people.
| Now working for one of their spin outs.
| btoro wrote:
| This is super cool. I think you just saved me a massive
| headache, as I plan to roll out a healthcare app with PHI in
| the next few months.
| SkyPuncher wrote:
| A healthcare startup is exactly how I got started with
| them. Makes it dead simple to cover all of the basics.
| estas wrote:
| Porter.run
| ksajadi wrote:
| Cloud 66
| fatih-erikli wrote:
| Cloudflare workers are pretty good if you are running something
| in Nodejs stack.
| wiredfool wrote:
| TIL that I have a heroku account.
| completelylegit wrote:
| Pretty good alternative to "We and our customer data got hacked".
| [deleted]
| mac-chaffee wrote:
| I really feel like people don't value intrusion detection enough.
| Why was Heroku's intrusion detection system "rely on Github's
| intrusion detection system"?
| jlmorton wrote:
| As described, the attacker quietly retrieved OAuth tokens from
| a single Heroku database, and then very loudly scraped GitHub.
|
| I'm not saying it would have been impossible to detect the
| first action, but it's substantially easier to detect the
| second.
| flas9sd wrote:
| last I've seen the Oauth permissions for the Heroku Dashboard
| given by Github are excessive and include write access to all
| public repos - as read-only is not an option if I recall
| correctly, see https://github.com/dear-github/dear-
| github/issues/113#issuec...
|
| Newer integrations like Github Apps are more granular and can
| restrict the scope , also ssh deploy keys are an option for other
| purposes, but specifically the tokens issued for the Heroku
| Dashboard can write to the public repos of a user or org.
| oxff wrote:
| Why am I hearing about this on _checks the fucking date_ on May
| 5th instead of, like, month ago.
| monkeymonkey wrote:
| https://www.heavybit.com/library/video/every-minute-counts-c...
| bgentry wrote:
| Hi, speaker in that linked video here :)
|
| A lot has changed at Heroku in the past 8 years since I left,
| particularly in the direction of being subsumed into the
| greater Salesforce org. My working assumption is that
| everything left there is being done "the Salesforce way" at
| this point. Take that to mean what you will, but it seems
| pretty clear we're long past the days of openly communicating
| with customers as quickly as you have relevant/important
| information to share with them.
| stevebmark wrote:
| Because SalesForce owns Heroku, and SalesForce has notoriously
| poor incident handling.
| jweir wrote:
| I first received word about this on 4/15 via an email from
| Salesforce.
| [deleted]
| nightpool wrote:
| Salesforce informed you that their entire DB was compromised,
| and not just the Github OAuth tokens, as they've been saying
| for weeks? The first indication that anything except a Github
| specific DB was compromised was Tuesday, when they started
| telling people that seemingly-all non-DB non-addon
| credentials were going to roll.
| VWWHFSfQ wrote:
| My understanding is they didn't even know about this until
| GitHub told them on April 13th. I'm guessing something got
| triggered in GitHub's system by a flurry of tokens issued to
| Heroku trying to enumerate private repositories. If the
| attacker had just played it low and slow they might never even
| have known at all.
|
| Who knows how long Heroku's internal systems were compromised.
| cmeacham98 wrote:
| Even if that's the case, it's still way after April 13th.
| chii wrote:
| it might be that it took this amount of time to establish
| the facts of the events. If they recounted an incorrect
| version early, it might do more damage than not telling it.
|
| I dont know if the github disclosure "includes" heroku's
| disclosure : https://github.blog/2022-04-15-security-alert-
| stolen-oauth-u... - but it was at least april 15th - close-
| ish to when the event occurred.
| bigDinosaur wrote:
| Is it ever true that earlier indications that credentials
| should be rotated leads to worse outcomes, though, as
| just one example?
|
| I'm sure I've received emails of the form: we _suspect_
| there may have been a breach, so we 're forcing password
| resets, and have always taken that fine.
| dev_tty01 wrote:
| Heroku reported it on 4/15. Read the beginning of the
| string of updates on the notification page posted here.
| Also,
|
| https://news.ycombinator.com/item?id=31048646
| heartbreak wrote:
| And they reported that the credentials were leaked on
| 5/3. That took a long time.
| latortuga wrote:
| This has been an ongoing security incident communicated
| through multiple channels since the day that github
| announced it. I've got a dozen emails or more in my inbox,
| the heroku dashboard includes mention of it, and the status
| page includes information about it.
| dataspun wrote:
| It seems as though enterprise customers of Heroku that leverage
| data products tied to their Salesforce instances, such as Heroku
| Connect and Salesforce Connect, are at risk of serious data
| breach and possibly haven't realized it. Nearly a month since the
| initial 2413 incident, we're still learning of its full scope. If
| customers haven't rotated credentials -- mistakenly thinking the
| incident is isolated to web apps and GitHub -- the risk is still
| present. Can it go any more sideways? Yet the communications from
| Heroku and Salesforce are disjointed, even amateur.
| rapfaria wrote:
| It is weird to start the day with a Salesforce email in your
| inbox - had to go checking if it was valid.
| craigkerstiens wrote:
| "a Heroku database" was what was known as core-db internally for
| the longest time. I'm not sure if still the case or not today.
|
| But at one point was the source of everything for Heroku.
|
| Over time things were moved out, so this isn't an everything that
| exists has been leaked, but it is not a guarantee that attacker
| didn't move from one area to another.
|
| As someone with some apps on Heroku, having worked there, but no
| knowledge of the details of the incident more than others... I
| would:
|
| 1. Rotate all creds
|
| 2. Ensure logging all connections to the DB (I can't recall how
| much you can do this on Heroku)
|
| 3. Extra heavily audit Github commits and Heroku releases
|
| 4. Maybe keep rotating all creds?
| srinathkrishna wrote:
| I feel for the team working on this at this time. I hope this
| doesn't end up accelerating the culling off of Heroku by
| Salesforce. One of the smartest and nicest bunch of folks I've
| worked with.
| craigkerstiens wrote:
| Completely agree, heart goes out to the team. No harsh
| judgement of all the engineering team there, not a fun
| situation and hope they know a lot of folks in their corner.
| jeromegv wrote:
| Heroku is heavily connected with Salesforce now with Heroku
| Connect, I doubt this is part of the plan.
| bobx11 wrote:
| There was already a plan by salesforce to kill it and make
| it into salesforce functions. Someone else here called it
| project periwinkle.
| traderj0e wrote:
| That's frustrating. Heroku is one of the nicest cloud
| services I've ever used, and am currently using.
| ryanmcbride wrote:
| Salesforce has been holding heroku further and further out
| over a ledge they're definitely not planning on supporting
| it for much longer.
|
| Edit: I only had one project left on heroku and I migrated
| it to Render a few months ago.
| Abishek_Muthian wrote:
| > but it is not a guarantee that attacker didn't move from one
| area to another.
|
| The incident notification seems like the customers who are
| using GitHub integration are the ones who are compromised, If
| the attacker has gained access to other accounts then it needs
| to be clarified so that we could take repository level
| mitigations as you've mentioned; Else most will just reset
| account passwords and be done with it.
| craigkerstiens wrote:
| The latest report states about "a database" which is
| presumably the internal database. I don't want to speculate
| too much, but it seems attacher had access to internal
| systems. GitHub were the ones that detected and noticed it
| and reported to Heroku. Do not disagree that there should be
| more clarity, but best to follow up with Salesforce on that.
| mike_d wrote:
| "Access to the environment was gained by leveraging a
| compromised token for a Heroku machine account"
|
| This is the equivalent of saying "the car was stolen because
| the car keys were laying on the kitchen table." They still
| don't know how they got into the house to get the car keys.
|
| GitHub was just one branch that the attacker took to further
| access, another being the download of the accounts database.
| We don't know how many other things they did.
| martinald wrote:
| The initial update said that the Heroku internal code
| itself was accessed. I wonder if they grabbed that, then
| analysed it to find various exploits?
| smarx007 wrote:
| "a compromised token for a Heroku machine account" sounds
| more like a master key was stolen from a car dealer.
| xena wrote:
| You get the wrong auth credentials in the right hands and
| everything goes to hell and back.
| [deleted]
| samwillis wrote:
| Well, it only seems to be getting worse on this one.
|
| I'm keen to get off Heroku, but waiting for one of the newer
| alternatives (Render/Fly+others) to implement WAL point in time
| restore for Postgres. It's the only thing keeping me on Heroku
| now, but is indispensable.
|
| Anyone here from them have any update on when we could see that
| feature made available?
| ryanSrich wrote:
| I'm also interested in this. I have 4 decent sized applications
| I want to migrate, but I don't have the resources to self host
| on GCP or AWS. K8s is nice in theory, but I know from
| experience it requires too much hand holding (let alone other
| managed services like a database and cache).
|
| From what I've seen in the market, fly, render, and railway are
| the only real Heroku competitors. All seem to be missing a few
| critical pieces of functionality that is preventing me from
| migrating. Railway doesn't appear to allow me to remotely run
| Python commands, they also don't have automated nightly
| backups. Fly and Render respectively have a handful of missing
| features. It would be amazing if one of these services came out
| with 1:1 feature parity with Heroku. I'd move in a heartbeat.
| gizzlon wrote:
| Google Cloud Run or App Engine Standard might be what you
| want:
|
| https://cloud.google.com/run/docs/quickstarts
|
| https://cloud.google.com/appengine/docs/standard
|
| Unfortunately some things are more difficult than necessary
| =/ OTOH the GCP ecosystem, network, and data-center presence
| are vast.
| martinald wrote:
| Azure web apps is pretty similar to heroku and I find it very
| easy to use. It has had a Linux version (instead of Windows)
| for quite a while now and it's pretty solid. Haven't had many
| real issues with it in many years of use.
|
| They also have managed postgres db with very powerful tools.
| anurag wrote:
| We are actively working on Postgres HA and PITR at Render and
| hope to have them both available this summer.
| bpicolo wrote:
| I recently spun up Google Cloud Run as an alternative. GCP has
| PITR for postgres.
|
| The only weird hitch with Google Cloud Run (and their other
| serverless products) is you either need to either use public
| IPs for e.g. memory store cache or other things in your network
| (minus your database), or set up a VPC Access Connector [0].
| Admittedly, that was easy once I realized I needed it, but was
| very annoying to figure out (because django's default redis
| socket timeout is never...).
|
| You also don't get SSH access, as it's fully managed. K8s is
| still the easiest version where you get that, unfortunately.
|
| [0]: https://cloud.google.com/vpc/docs/configure-serverless-
| vpc-a...
| zoomzoom wrote:
| You should check out tools that help bring a Heroku quality
| experience into your own cloud, where there are lots of
| database options that meet your needs.
|
| Coherence (disclosure- I'm a cofounder) -
| https://www.withcoherence.com - is one option. A defined
| workflow for production-quality full-stack web apps with dev
| and production built in alongside automated test environments,
| including CI/CD and cloud IDEs - all configured with one high-
| level YAML. We're in a very early private beta on google cloud
| right now - if you're interested, please check out our site
| above and let us know!
| ngcc_hk wrote:
| Why not just say hacker ? Know we are on hacker news and hence it
| is semantically not exactly right. But threat actor ... sound
| more like threaten actor. Just a movie star or drama queen. If
| one say Heroku was hacked or just sales force ... I know one want
| to manage but somehow the title is not exact right. Too pr.
| tempfs wrote:
| Threat actor is industry standard terminology for a malevolent
| or blackhat brand of hacker.
|
| Not all hackers are threat actors.
|
| https://en.wikipedia.org/wiki/Hacker
| koolba wrote:
| > Access to the environment was gained by leveraging a
| compromised token for a Heroku machine account.
|
| Any idea if this involves AWS EC2 Instance Roles? It's incredibly
| convenient, but has got to be the scariest feature to enable on a
| platform that allows arbitrary user code to execute.
| lovelearning wrote:
| > On April 7, 2022, a threat actor obtained access...GitHub
| identified the activity on April 12, 2022, and notified
| Salesforce on April 13, 2022, at which time we began our
| investigation.
|
| Can some experienced security professionals weigh in on the
| cultural and organizational factors that allow this kind of major
| breach to go unnoticed for a week, that too in a reputed company
| like Heroku?
|
| I'm not asking this rhetorically or in bad faith. It's a genuine
| question I have based on a project I did. I researched cybersec
| tech like SOAR, XDR, security logging, and SIEM in depth. On
| paper, the marketing for such tech gives the impression that by
| using them, such breaches can be detected and prevented in real-
| time. But there seems to be a mismatch between the claims and
| ground realities. If so, why?
| everfrustrated wrote:
| Most security products are snake oil. The vendor selling them
| and the people buying them (always business execs never
| engineers) have no interest in how well it actually works as a
| company is never buying these tools to actually be secure only
| ever to give the impression to some higher exec within the same
| company or tick a box on some audit process (which requires
| something to be in place but doesn't require verifying it
| works).
|
| It's beyond a joke how the "cyber" industry operates.
| sfdcsecthrow wrote:
| Throwaway for obvious reasons. But in my experience Salesforce
| security org is plagued with incompetent leaders who chase
| arbitrary metrics that does not improve security at all. At one
| point in time, security team at Salesforce was stellar and did
| some awesome work. Dont get me wrong, there are many many smart
| security engineers still around but their population has been
| dwindling. This all started when a bunch of new leaders were
| hired. Instead of promoting tenured smart people, the security
| leaders decided to bring in their own gang of coworkers from
| previous employers. At the same time they slowly started
| pushing out tenured leaders.
|
| These new leaders are typical VPs who have long lost any
| technical chops and its a huge task to explain any complex
| technical topic to them. On top of that they don't bother
| understanding the fundamental business model and just want to
| push their agenda on to everyone. So they have added "security
| processes" which requires checking boxes. The more boxes you
| check the more metric it generates the better the leader looks.
| These security leaders are so disconnected from the ground
| reality that they don't even realize that all they are doing is
| adding hurdles in the path of engineers without improving any
| security.
| lovelearning wrote:
| Thanks for the insight!
| srinathkrishna wrote:
| I do remember the whole debacle that happened at DEFCON and
| people got fired for presenting something on stage.
|
| Agreed, there's a lot of interesting stuff that came out of
| the Security (or related orgs) at Salesforce. Red team tools,
| chaos tools and JA3 which we use at my current work as well
| for SSL/TLS fingerprinting.
| adrr wrote:
| Because Heroku/Salesforce doesn't have real security. Requiring
| special characters in passwords and sending out emails that
| have http and not https links to a password reset page. Their
| security is a joke.
| nameless912 wrote:
| As a former Herokai, let me color this a bit:
|
| Heroku _used_ to have their own security team which was quite
| good and had some scary talented people on it. However, over
| the last 3 years or so Salesforce has been forcing Heroku to
| adopt Salesforce's operations practices, and this has not
| only wrecked productivity but completely destroyed morale and
| caused many, many of those talented people to quit. I for one
| decided to quit after only working there for around 8 months
| due to a horrific overreach by Salesforce into Heroku's
| operations.
|
| Among other things, Salesforce forced us to adopt:
|
| - their internal ticket tracking system, which _runs in an
| instance of salesforce_ (barf)
|
| - their slack instance, which lost us many of our
| customizations and broke a bunch of integrations for weeks (I
| wouldn't be altogether surprised if this was one of the
| causes of the delay in notifying Herokai as to what was going
| on)
|
| - their incident management process, which requires us to
| notify "Salesforce ops HQ" anytime there's an outage that
| meets certain criteria.
|
| This last one was especially bad, and meant that we no longer
| had full agency to act during incident response situations. I
| had one incident I responded to where the problem got worse
| while we waited for Salesforce IM to spin up, so that we
| ended up having what would have been a 10 minute outage turn
| into a 2 hour outage because the issue got out of control.
|
| In short, the problem isn't the people trying to administer
| Heroku; they're great folks under a lot of pressure with very
| few resources. The problem is, and has always been,
| Salesforce's "leadership" deciding what's best for a cloud
| platform they couldn't give less of a damn about.
| sfdcsecthrow wrote:
| The integration of Heroku stuff into SFDC would have
| probably been easier if the Heroku folks got off their high
| horse.
| nameless912 wrote:
| I don't want to start a flamewar, but it wasn't an
| integration; it was a hostile takeover. Heroku was doing
| just fine without SFDC's interference, and when
| Salesforce not only refused to believe that employees
| might have negative opinions about integrating but
| actively prevented us from speaking up about it, people
| got rationally angry and left. I remember talking with an
| architect on the Heroku side about how he felt about all
| of it, and he told me that it wasn't presented to him as
| a choice, and senior leadership was convinced that
| Herokai would see this as a good thing despite his (and
| others') warnings.
| Rastonbury wrote:
| At this point, I'm not sure why SFDC even bought Heroku..
| Is there major overlap between CRM users/buyer
| (salespeople) and Heroku's who are mainly devs, hobbyists
| and startups (i'm guessing)? Surely they didn't try to buy
| hero to compete with the big 3 cloud providers?
| nameless912 wrote:
| Welcome to the existential question of Heroku from the
| inside :) No one knew what the point of the product was
| anymore. My vibe is that SFDC's goal was to use Heroku as
| their cloud provider for everything, and when that didn't
| work (more due to lack of focus than technical issues),
| they tried to shoehorn Heroku into the Salesforce
| platform as a tack on feature. It was all very weird, no
| one knew what we were doing, and everyone was upset about
| it. I originally joined the org because I was really
| excited to help the early startup/small
| business/hobbyist/student sector, and SFDC writ large
| just did not care about those customers IMO.
| sfdcsecthrow wrote:
| This is not true. Heroku actually had a pretty decent
| security team. They have lost most of that talent though.
| cube00 wrote:
| _> GitHub identified the activity on April 12, 2022, and
| notified Salesforce on April 13, 2022, at which time we began
| our investigation. As a result, on April 16, 2022, we revoked
| all GitHub integration OAuth tokens, preventing customers from
| deploying apps from GitHub through the Heroku Dashboard or via
| automation._
|
| The three days after being notified to actually revoke the
| tokens isn't ideal either. Surely if GitHub comes to you and
| warns you of suspected unauthorised access you'd spend a very
| limited amount of time and then revoke the credentials to be on
| the safe side.
| yuliyp wrote:
| I think (based on what I've read on the linked page) the
| notice was telling Heroku that "hey, someone used a
| compromised OAuth token to download your source code" not
| that "the tokens that you are using to read Github
| repositories of your users are compromised". Both are Github
| OAuth tokens, but playing different roles. Presumably the
| compromise of source code might have been used to help get
| access to the database that had the Github integration OAuth
| tokens, and realizing that might indeed have taken a couple
| days.
| Mandatum wrote:
| Salesforce has been unable to attract or retain security
| talent. When they acquire a company, they close down the
| department that does security for that company - and then move
| everyone into the Salesforce Trust team. Unlike engineering who
| they typically leave alone (unless they're integrating or
| rebranding).
|
| In doing so, they typically lose everyone that setup the SIEM
| and run the SecOps center. Everything "security" ends looking
| the same.
|
| They don't pay well, executives have pulled talks and fired
| speakers who do things they disagree with (the same executives
| are promoted and remain there - no accountability), they've got
| a pretty bad wrap within the industry.
| lovelearning wrote:
| Let's assume that prior to acquisition, Heroku sec had set up
| a very secure posture using such tech. Then they lost most of
| their experienced people after acquisition.
|
| Some questions:
|
| 1) Are these tech not enough to enable others - perhaps less
| experienced, or experienced but not on a particular product -
| to take over while maintaining the same posture?
|
| 2) What kind of additional (perhaps intangible) security does
| an experienced team add to the posture that gets lost when
| they leave?
|
| 3) As I understand them, things like risk frameworks, NIST
| CSF, security assessments are all supposed to anticipate
| people problems (resignations, malicious insiders, etc) and
| make the posture as independent of them as possible, probably
| relying on automated tools like XDR and SOAR to do their
| thing regardless of who's sitting at the console. Does it not
| work like that in reality?
|
| Btw, thank you for your reply and insights (and to everyone
| else who replies)! Pardon my probably naive questions. I'm an
| outsider looking in and having trouble understanding this
| phenomenon of data breaches in the face of all the tech
| marketing.
| milkshakes wrote:
| Fundamentally, a security analyst authors detections,
| reviews surfaced alerts, or identifies hypotheses to
| investigate. In the case of reviewing surfaced alerts (the
| firing of a detection which may or may not be authored by
| the security team), differentiating true positives from
| false positives is subtle and often requires context or
| further digging. Of course, this requires time, which costs
| money, so you can imagine the tension there.
|
| This process can often be subtle, and difficult to
| automate. In many cases, the issue is automating the
| economical delivery of enough context to the deciding
| function that a clean choice can be made. However, even
| with enough context, and enough documentation, escalating
| vs suppressing an alert can often be a judgment call.
| Humans are meat based pattern matchers, and a decade's
| worth of "ML" and "AI" advancements still not sufficiently
| precise (as in vs recall) enough to filter out "things that
| look bad" from "things that are bad, for our specific
| environment", that knowledge still lies with the security
| team.
| muraiki wrote:
| Only the slides are available, but the presentation "AI
| is Not Magic: Machine Learning for Network Security" at
| CMU's FloCon in 2020 was about this:
| https://resources.sei.cmu.edu/library/asset-
| view.cfm?assetid...
| richardfey wrote:
| > fired speakers who do things they disagree with
|
| Can you please make an example? I am genuinely curious about
| what things
| intern4tional wrote:
| https://tech.slashdot.org/story/17/08/10/1919204/salesforce
| -.... is probably the most well known incident. There have
| been others though.
| Mandatum wrote:
| Hi Jim Alkove!
| sfdcsecthrow wrote:
| He has retired.
| gruez wrote:
| >bad wrap
|
| https://www.merriam-webster.com/words-at-play/usage-bad-
| rap-...
| dataspun wrote:
| indeed, unpalatable
| abrkn wrote:
| I received an email yesterday asking me to change my password. I
| did, and updated our services with the regenerated Heroku API
| key.
|
| This morning, I was unable to log into my account and had to
| reset again. And update our services again.
| ryannevius wrote:
| The email I received clearly stated the following:
|
| > Due to the nature of this issue, you may be required to reset
| your passwords again in the future.
| owlninja wrote:
| Always interesting to me with events like this that the actions
| or intentions of the "threat actor" are never discussed. The
| conversation is always finding tiny holes in the victim's systems
| and admonishing them for not being prepared.
| subroutine wrote:
| Actions other than... (1) obtained access to a Heroku database,
| (2) downloaded customer GitHub integration OAuth tokens, (3)
| enumerated metadata on customer repos with the OAuth tokens,
| (4) downloaded some Heroku private GitHub repos containing
| source code, and (5) exfiltrated customer hashed and salted
| passwords?
| photon12 wrote:
| There's going to be a question about the expected probability of
| this across cloud service providers.
|
| I've done security work for multiple cloud service providers and
| know a lot of people in the industry. I'm not really privy to
| give details.
|
| I can say: dev teams face limits on what they can build securely,
| platform teams face limits on what secure by default and
| monitoring features they have time to implement, security
| operations teams have a lot of data points to look at, and in
| theory even changes in personnel in a couple of teams can have an
| impact on the threat posture for a given set of a company.
|
| People are trying. But if you do the math for introduction of
| attack surface over time versus risk mitigation effort over time
| for that attack surface you can derive some estimates for
| likelihood of attack.
|
| If your cloud provider isn't providing you that data as a
| customer, it's not simple to make a determination about
| likelihood of risk introduction and breach of your cloud
| provider.
|
| This hasn't really been something people talk about because there
| was a tacit assumption that the biggest companies are mostly
| getting this right.
|
| I'm not trying to say the cloud is inherently broken, I run a lot
| of workloads in the cloud and trust sensitive data to the cloud.
| But I do wish there were better ways for customers to have data
| points with which they could evaluate platforms besides
| extrapolating the history of public breach reports.
| wlll wrote:
| Unrelated to your main point, "I'm not really privy to give
| details.", that's not how you use privy. If you have the
| details but aren't allowed to share them then you _are_ privy
| to the details, but you can 't share them.
|
| Privy means "sharing in the knowledge of (something secret or
| private)", but it has nothing to do with _sharing that
| knowledge with others_.
| mathstuf wrote:
| The phrase to use would be "I'm not at liberty to give
| details".
| Cthulhu_ wrote:
| I have no experience in organizations or IT systems that big,
| but, I can very much imagine that it's not only a matter of not
| having enough time to check everything, but over time, your
| systems become so big that it's hard to maintain an overview
| or, for that matter, control.
|
| I mean, there's been numerous incidents of a random developer
| having copies of customer data on their systems, or
| accidentally opening up a database or an Elasticsearch instance
| to the world.
|
| And I'm afraid the only way to help mitigate that is to
| restrict what an individual can access and do on the one hand,
| and bureaucracy on the other. And full-time staff whose only
| job is to maintain security and juggle access rights around.
| rapind wrote:
| This is something I've been thinking about as a user
| (provisioner) of cloud services like Heroku, AWS, Google, and
| VMs from Linode and Digital Ocean. Especially with the
| potential of state actors trying to cripple large businesses (I
| don't actually know how serious a threat this is TBH).
|
| Sure they take security further (I hope) than I would
| provisioning my own hardware, but I also worry about how large
| a target they are and how much more complex their systems are
| (increasingly large attack surface).
|
| It's appealing to just let someone else handle my hosting,
| including security, but I also wonder if I'd be better off
| running colocated metal. I mean I still have to worry about the
| hosts network and physical security when colocating, but it's a
| much smaller attack surface, and also a far smaller target than
| massive hosts like AWS.
|
| I think cloud provider's might need to level up their security
| practices, even if it incurs some friction with their customers
| and staff. I don't even know if it is feasible to evolve as
| fast as potential attackers from a cost perspective over the
| long term for many (or all) of them.
|
| Not to single anyone out, but take Digital Ocean as an example.
| They're constantly adding features to their platform. I assume
| this increases their attack service? How much weight are they
| giving to security as they add features? I assume quite a lot,
| but I don't really know! Another example is AWS, who are
| constantly growing their features / services (to the point
| where it's near impossible to navigate all of their
| offerings!). Every cloud provider is doing the same.
|
| There's also no real accountability. If my database on Heroku
| (or another provider) is compromised it could have a massive
| impact on my business, but at most it might mean losing a
| customer for my hosting provider, and maybe some bad press (not
| really enough though IMO). So the incentives aren't perfect.
| fancyremarker wrote:
| (Founder of Aptible, a Heroku-like PaaS focused on security and
| compliance)
|
| > dev teams face limits on what they can build securely,
| platform teams face limits on what secure by default and
| monitoring features they have time to implement, security
| operations teams have a lot of data points to look at, and in
| theory even changes in personnel in a couple of teams can have
| an impact on the threat posture for a given set of a company.
|
| I couldn't agree more. It's too bad, because I believe most
| companies should be solving this by building on a battle-tested
| platform that provides a safe path for devs. In theory,
| platforms like Heroku improve cloud security by reducing margin
| for error. In practice though (as we're seeing), these
| platforms can introduce new security vulnerabilities in the
| layer they introduce on top of IaaS.
|
| I also very much appreciate your comment about having a better
| way to evaluate the security of platforms without relying on
| public breach reports, or implicitly trusting what platforms
| say. I think the best thing is for platforms to be 100%
| transparent in how they implement security, namely by:
|
| 1. Running alongside IaaS services instead of layering a black
| box on top of them (coordinating, not fully abstracting)
|
| 2. Providing clear accountability for security defaults: every
| security default enforced by the platform should be represented
| in a validation that end users can view (if not alter)
| pm90 wrote:
| Agreed. cloud providers' incentives are aligned with growth
| which naturally mean easy accessibility; hence all the defaults
| being generally "open". Its so easy to make a resource
| accessible to anyone, or an IP accessible from anywhere in the
| cloud without proper restrictions by internal teams in the
| company; and often the default is to give teams superadmin to
| "unblock their time sensitive project" rather than maintaining
| principle of least access which requires more discipline (and
| thus effort).
|
| Either cloud providers need to assume more responsibility for
| security or a Federal Agency like the FBI or NIST need to be
| more proactively engaged in improving the security posture of
| cloud hosted US corps.
| hnbad wrote:
| > cloud providers' incentives are aligned with growth which
| naturally mean easy accessibility; hence all the defaults
| being generally "open"
|
| So no different from every VC funded startup (or startup
| seeking VC funding) then?
|
| The sentiment of imposing tighter regulations around data
| security feels counter to the general idea that the lack of
| regulations around data security (e.g. strong data protection
| laws) are what allows the US tech industry to dominate
| compared to its EU equivalents.
|
| I'm not disagreeing with this, I'm just pointing out the
| contradiction and wondering how those who believe the latter
| would reconcile that belief with demanding the former.
| dgb23 wrote:
| It's quite simple. Doing the right thing has short term
| costs and long term benefits.
| stephenanand wrote:
| bspear wrote:
| The comms for this security breach is a textbook case study of
| what NOT to do
| bpeebles wrote:
| I had posted the most relevant paragraphs in
| https://news.ycombinator.com/item?id=31255450 but it probably
| should be here too:
|
| >On April 7, 2022, a threat actor obtained access to a Heroku
| database and downloaded stored customer GitHub integration OAuth
| tokens. Access to the environment was gained by leveraging a
| compromised token for a Heroku machine account. According to
| GitHub, the threat actor began enumerating metadata about
| customer repositories with the downloaded OAuth tokens on April
| 8, 2022. On April 9, 2022, the attacker downloaded a subset of
| the Heroku private GitHub repositories from GitHub, containing
| some Heroku source code.
|
| ...
|
| > Separately, our investigation also revealed that the same
| compromised token was leveraged to gain access to a database and
| exfiltrate the hashed and salted passwords for customers' user
| accounts. For this reason, Salesforce is ensuring all Heroku user
| passwords are reset and potentially affected credentials are
| refreshed. We have rotated internal Heroku credentials and put
| additional detections in place. We are continuing to investigate
| the source of the token compromise.
| woodrow wrote:
| I reset my password and then closed my Heroku account today. What
| a sad end for something that was once a model for software
| deployment and developer experience.
| VWWHFSfQ wrote:
| > Separately, our investigation also revealed that the same
| compromised token was leveraged to gain access to a database and
| exfiltrate the hashed and salted passwords for customers' user
| accounts.
|
| What else was in this database? Typically the password field is
| stored alongside the rest of the user record. So was this the
| entire customer database that was stolen? Usernames, emails,
| salted/hashed passwords, what else?
|
| I feel like at some point they're just going to go completely
| radio silent because the extent of the breach will become such
| that they'll have no choice but to just lawyer up.
| sofixa wrote:
| > I feel like at some point they're just going to go completely
| radio silent because the extent of the breach will become such
| that they'll have no choice but to just lawyer up
|
| They can't, they surely have EU customers so have to follow
| GDPR disclosure rules, which they might already be afoul of.
| majormajor wrote:
| The last couple large places I work specifically split out the
| auth/password DB from the rest of the user data. They're used
| for different things and they have different types of
| sensitivity around them.
| throwawayboise wrote:
| Yep. Same thinking lead to the split of /etc/shadow from
| /etc/passwd.
| gabereiser wrote:
| This is also why folks advocate for a separate auth system
| (keycloak was just mentioned here on HN) since they are
| different types of information.
|
| >I feel like at some point they're just going to go
| completely radio silent because the extent of the breach will
| become such that they'll have no choice but to just lawyer
| up.
|
| I feel like they are heading this route as well. Possibly
| even withholding information in order to save the company
| from mass exodus due to the incident. I'm sure they'll be
| fine.
| huijzer wrote:
| > Possibly even withholding information in order to save
| the company from mass exodus due to the incident. I'm sure
| they'll be fine.
|
| Trust takes long to build and is easy to break. For anyone
| able to convert the Heroku buildpack to a Docker and able
| to move the database, moving away from Heroku shouldn't be
| too hard. There are multiple similar services nowadays.
| donatj wrote:
| We moved basically everything but username into an entirely
| different db, went so far as to hash the username column so
| we don't even know what's it is until you log in.
| 3np wrote:
| > hash the username column
|
| Is this giving you any real security benefit? (I'd assume
| the usernames are indexed elsewhere and that it's a
| reasonable assumption that whoever gains access to this
| hashed data has access to the username list as well, making
| a lookup trivial - or are these not safe assumptions?)
| donatj wrote:
| It's a PII covering our butts thing more than anything.
| Wanting to absolutely minimize what we know about our
| users.
| couchand wrote:
| Not GP, but one architecture where there would be a
| difference is there's a distinct identity provider,
| responsible solely for exchanging the user's long-lived
| username/password credentials for a short-lived ticket.
| bigDinosaur wrote:
| I imagine the process going a bit like
| https://youtu.be/y8OnoxKotPQ
| christophilus wrote:
| I knew what that was before I clicked it. Never
| disappoints.
| oefrha wrote:
| For any individual/organization remotely competent at
| security (i.e. not using Passw0rd! as password and reusing
| elsewhere), a leak of hashed password is probably the least
| of concerns. Compared to this anyway:
|
| > According to GitHub, the threat actor began enumerating
| metadata about customer repositories with the downloaded
| OAuth tokens on April 8, 2022. On April 9, 2022, the attacker
| downloaded a subset of the Heroku private GitHub repositories
| from GitHub, containing some Heroku source code.
| glenngillen wrote:
| former Heroku employee, though long long long ago, with no
| specific knowledge about this incident, but...
|
| We did so much work in open source it was just easier to
| assume everything was always publicly viewable, or that
| what you were doing now might be open sourced in the future
| along with the full commit history. Whether something was
| private or public was more a business decision around
| competitive risks and not a security-led one. To that end
| I'm far more concerned about a database and passwords
| getting popped.
|
| But who knows, a lot can change in 10 years. Maybe private
| repos being exposed is also very bad.
| glenngillen wrote:
| Yeah, the comms around this has been very concerning. Do I need
| to rotate every config var on all of my apps? Re-install every
| add-on?
|
| While the nature of what limited things they had disclosed to
| date pointed to this situation part of me wanted to believe it
| wasn't as bad as I was assuming. And now the trendline on this
| suggests I should have already done everything I've outlined
| above. And I'm low confidence anybody is going to be proactive
| in telling me until it's absolutely obvious that these things
| have been compromised and exploited.
| xena wrote:
| IIRC the environment variable settings are encrypted in a
| physically separate database. However it may be a good idea
| to rotate your secrets anyways. My hunch would be that there
| are so many "juicy" targets on Heroku that you probably don't
| need to worry too much right now unless you are or work for a
| "juicy" target.
|
| This is gonna suck.
| t_sawyer wrote:
| I posted a similar worry about the ENV's.
|
| Why would the Github API keys not fall into the same
| separate database and be encrypted as well? It's especially
| baffling if they already have an example/process of doing
| this properly.
| jeremymcanally wrote:
| The bigger question to me is how did they leverage a GitHub
| OAuth token to gain access to an internal database unless
| they're storing that config in their codebase.
|
| If that's the case...yikes.
| jeremyjh wrote:
| They didn't say that happened. I'm reading it as their DB was
| compromised and it's contents included GH auth tokens.
| jeremymcanally wrote:
| How do you read this as the database had tokens in it?
|
| > Separately, our investigation also revealed that the same
| compromised token was leveraged to gain access to a
| database...
|
| EDIT: Ah yep you're right. Two tokens in play there: one
| Heroku API token, one GitHub token. Phew.
| benevol wrote:
| > What else was in this database? Typically the password field
| is stored alongside the rest of the user record.
|
| That is where you are supposed to have encrypted as many data
| fields of the record as possible - in addition to the
| conventional database encryption which encrypts the database as
| a whole.
___________________________________________________________________
(page generated 2022-05-05 23:02 UTC)