[HN Gopher] SAPwned: SAP AI vulnerabilities expose customers' cl...
___________________________________________________________________
SAPwned: SAP AI vulnerabilities expose customers' cloud
environments and privat
Author : todsacerdoti
Score : 232 points
Date : 2024-07-17 22:02 UTC (1 days ago)
(HTM) web link (www.wiz.io)
(TXT) w3m dump (www.wiz.io)
| mvandermeulen wrote:
| Excellent write up. This wasn't a sophisticated attack. Seems
| like there is very little discipline at Salesforce when it comes
| to deploying production systems.
| phoe18 wrote:
| How is this related to Salesforce?
| bloqs wrote:
| Salesforce, SAP, it's all the same sort of
| btown wrote:
| This is like saying Apple and Alphabet are the same. SAP is
| a 52 year old company and is the largest non-American
| software company in revenue, and has never been part of
| Salesforce: https://en.wikipedia.org/wiki/SAP
| f1shy wrote:
| They do software?
| friscas wrote:
| SAP is a software company, its the direct competitor of
| Oracle.
| f1shy wrote:
| LOL!!!
| cosmotic wrote:
| As security researchers, you think they might have known that
| pixelating text to redact it is a poor choice.
|
| https://www.bleepingcomputer.com/news/security/researcher-re...
| NotMichaelBay wrote:
| It looks to me like it was blurred, not pixelated.
|
| Edit: Nvm, I guess they blurred text in some places, pixelated
| in others
| FrostKiwi wrote:
| In both cases, yes and there are multiple projects able to
| reverse it for both the pixelation case [1]
|
| Blurring is like a hashing algorithm. If you know the font,
| size and placement that was used, you can try out reblurring
| and Thus brute forcing characters
|
| [1] https://github.com/spipm/Depix
| TeMPOraL wrote:
| In case of blurring, wouldn't it be easier to try and guess
| the parameters of blur operation used, and invert it?
| Filligree wrote:
| Depending on the specific type of blurring that may be
| impossible, but if you can do it, sure.
| chatmasta wrote:
| The reported bugs have all been patched, and presumably the
| compromised secrets have been rotated. The blurring/pixelation
| is arguably unnecessary, regardless of its effectiveness. The
| censored data looks like local host names and some image
| hashes.
| dotty- wrote:
| I hope SAP does a hard retrospective on why Wiz's research was
| not disrupted before they got full cluster admin. Like, I want to
| know from SAP's side whether they received any alerts for any of
| this activity and whether they investigated them properly. I
| wonder if there is any regulation SAP has to follow that requires
| them to have adequate alerting for suspicious network activity
| and whether this research can be used to show that they do not.
| SoftTalker wrote:
| Indeed. And if they did not detect it, how can they know that
| customer data have not been compromised?
| uaas wrote:
| Usually security researchers are required to reach out to the
| target before escalating further into the systems, asking for
| permissions to proceed. This is also something bug bounty
| programs require as per their rules for their targets in scope.
| I'd expect this to be the case here as well, given the
| researcher is employed by a security company.
|
| Researchers also usually mention which points they asked for
| additional permissions at in writeups, but now always.
| Propelloni wrote:
| Oh, they have rules and regulations, for sure. Take a look at
| their certification page: https://www.sap.com/about/trust-
| center/certification-complia...
|
| Question is, do they live it or is it just some binder sitting
| on a shelf.
| BodyCulture wrote:
| The problem is that people who do decisions don't understand
| the technology. Most IT managers in Germany do not even know
| how programming works. There are exceptions, but the biggest
| players are people flying in blindsight.
| j45 wrote:
| It would be a great post to see how they detect such things in
| AI.
| mac-chaffee wrote:
| Shocked that there was a tiller instance running. That's been
| deprecated since 2020: https://helm.sh/blog/helm-v2-deprecation-
| timeline/
| uaas wrote:
| You would be horrified if you would know how much pre-'20s or
| even pre-'10s software is still running in production out
| there. Here we are talking about a huge enterprise and a
| somewhat complex migration (from tiller) but you can easily
| find outdated software without these aggravating circumstances
| as well.
| hunter2_ wrote:
| Software from 2019 is horrifyingly outdated? If updates with
| security patches exist but haven't been applied, sure, but
| that's not really a default scenario depending on the stack.
| uaas wrote:
| I've only used 2020 because of the example in question.
| Security patches might or might not have been applied in
| both my imaginary example and in real world.
| ketzu wrote:
| In my experience, "deprecated" is often taken as "we can still
| use that, it is not removed yet", which I find somewhat
| disheartening sometimes.
| c0balt wrote:
| That's easy though, the removed part can also be ingored by
| mirroring package repositories for RHEL/ Debian-based
| systems.
| 1oooqooq wrote:
| the sad part is that all this is going to accomplish is promote
| that sap has ai product their clients can purchase. it's not like
| anyone using sap know or care about security other than signing
| with a company that has all the ISO and whatnot, which is the
| reason they went with sap to begin with
| j45 wrote:
| I would assume for the prices SAP charges, it mah start as some
| kind of bulletin of how to properly secure the AI, and failing
| that a feature update to tighten some defaults.
|
| But a security fist to a leaky side door? I'd bet that upsets
| some customers.
|
| Many of these accounting systems are starting to sell AI to
| automate transactions, which may explain the read+write nature
| of the access described in the comments.
| betaby wrote:
| Am I reading it correctly, customer's account data is exposed to
| the same customer? The exception is some logs as I see.
| waterproof wrote:
| Some logs... and some other customers' training data and
| code... and SAP's internal docker image repository (with read-
| write access!)
| betaby wrote:
| You are right! I missed that all NFS folders on the
| screenshot have rwxr-xr-x permissions.
| jaaron wrote:
| While I get that it's the AI product, the vulnerability here is
| the k8s configuration. It really has nothing to do with the AI
| product itself or AI training or anything related to machine
| learning or generative AI, it's more about poor cloud computing
| platform security.
| bilekas wrote:
| The article doesn't say that it is an issue with the product
| itself though. It explains very well than infact it's the
| isolation of the AI training models.
|
| > The root cause of these issues was the ability for attackers
| to run malicious AI models and training procedures, which are
| essentially code
|
| It's being researched and investigated, to my understanding,
| due to the prevalence of AI products and the need to be mindful
| of the infrastructure.
| j45 wrote:
| The brand that sells is the brand at fault.
|
| Securing it or knowing to secure it or testing it or never
| releasing it until it was secure is all things that are with
| the brand making the sale.
| cchance wrote:
| Which is possibly worse lol, the fact SAP a company as big as
| they are with as much critical information as they have,
| fucking up basic cloud security, they didn't even fuck up
| something new they fucked up common shit from the sound of it.
| sunaookami wrote:
| The bigger the company the less they care since no one will
| hold them accountable anyways.
| ec109685 wrote:
| This is really bad. They are running a single K8s cluster and
| expecting hard multi-tenancy guarantees?
|
| All the major clouds use vm boundaries and separate K8s clusters
| between customers. Microsoft was similarly bitten a few years ago
| with one of their function products that expected K8s to be the
| primary security boundary.
| bilekas wrote:
| > They are running a single K8s cluster and expecting hard
| multi-tenancy guarantees?
|
| Maybe I missed something in the article but where are they
| expecting any hard guarantees. If there is a model being
| trained for example (running arbitrary code) where does a multi
| K8 tenency play?
|
| The main issue I see is all internal network communication was
| trusted once behind the proxy/firewall (Istio) but I probably
| just don't understand k8 clusters too well.
| robertlagrant wrote:
| Istio is point to point between services. It's not a boundary
| in the sense you're thinking.
| bilekas wrote:
| I will admit I don't know a lot about Kubernets at all, but
| as I see the Istio is supposed to be the proxy layer
| between services ?
|
| https://istio.io/latest/docs/ops/deployment/architecture/
|
| Being able to run as the Istio user (1337) renders the
| proxy itself moot right ?
| gbrayut wrote:
| There are a lot of other ways to bypass the Istio sidecar
| proxy, which is not designed to be a general egress
| boundary/firewall. See
| https://blog.howardjohn.info/posts/bypass-egress/
| Arbortheus wrote:
| K8S done right is literally designed for multi tenancy. A
| separate cluster per customer would be insanely costly and
| terrible for the planet. Maybe in premium products where
| security is paramount, but a separate cluster per customer is
| basically setting your money on fire.
| blincoln wrote:
| Multi-tenant Kubernetes[1] is a strcpy-level footgun IMO.
| It's perfectly fine as long as everyone involved does
| everything correctly and never makes a mistake.
|
| Kubernetes itself is very complex. The "who needs a UI when
| you have configuration files and an API?" approach makes it
| even more opaque to the people who often end up responsible
| for it. The landscape changes very rapidly.[2] I'd trust
| Kelsey Hightower to set up a secure multi-tenant deployment,
| but probably not anyone else.
|
| Is it not practical to deploy clusters on top of
| virtualization? That should make efficient use of hardware
| while still giving each tenant their own cluster, therefore
| providing stronger isolation than the typical Kubernetes
| configuration tends to.
|
| [1] I am specifically referring to a Kubernetes deployment
| where different customers are running custom code on the same
| underlying hosts. Using Kubernetes to host a service that is
| multi-tenant at a higher level is not something I would
| recommend, but it's not as immediately dangerous as a model
| where customers run container-level custom code.
|
| [2] This is not surprising for a relatively new technology,
| especially one that's as paradigm-shifting as Kubernetes was.
| But most people are not going to rearchitect and redeploy
| everything every six months just because the Kubernetes
| developers decided to replace a pod security or network
| security model with a non-backward-compatible alternative
| _again_.
| outworlder wrote:
| > K8S done right is literally designed for multi tenancy
|
| No it is not. Not in the way they are using it.
|
| There are two main use-cases. One is multiple teams, in which
| case they are bound by their company's policies and
| guardrails
|
| The second is multiple customers. But that also assumes they
| have no direct access to the cluster. A vendor would,
| instead, deploy multiple instances of a workload; the
| _customers_ would not.
|
| Straight from the horse's mouth:
| https://kubernetes.io/docs/concepts/security/multi-tenancy/
|
| There's also nothing that says that multiple clusters need to
| be expensive, if they are sized right. They can be as small
| as you need both in number of instances and instance size.
| The overhead here is the control plane but, for small
| clusters, the resource requirements are similarly small.
|
| That said, if hard multi tenancy is what you need, then you
| need to use things like this: https://github.com/kubernetes-
| sigs/cluster-api-provider-nest...
|
| (for the control plane - you still need to worry about the
| data plane)
| outworlder wrote:
| Hard multi tenancy can't realistically be achieved in the same
| logical K8s cluster. And it is a moving target, which makes
| trying to secure it by admission controllers... not a great
| plan.
|
| One needs to look into things like VirtualClusters to even
| begin to consider hard multi-tenancy with potential hostile
| tenants(https://github.com/kubernetes-sigs/cluster-api-
| provider-nest...). That is just about the control plane. It
| doesn't even touch the data plane.
|
| How secure that is even with the extra layer, I do not know.
| Even in the VM land we have seen crazy VM escape exploits over
| the years.T
| tiffanyh wrote:
| Has anyone used Wiz?
|
| It's possibly the fastest rocket for an enterprise software
| company ever.
|
| $100M in just 1.5 years time
|
| $350M at end of 3-year
|
| https://www.wiz.io/blog/100m-arr-in-18-months-wiz-becomes-th...
| rtev wrote:
| About to be acquired by Google for $23 billion as well!
| nicce wrote:
| So... the road for maximizing the profits will begin?
| kchr wrote:
| Using it and loving it. Security aspects aside, it's also the
| best tool I've tried for proper asset management in multi-cloud
| scenarios. With the graph feature you can write queries for
| basically anything, across all accounts if you wish.
| tetha wrote:
| This makes me glad I finally talked people at work into running
| our annual pentests of our products on production, and putting
| the entire production infrastructure in scope. Focus may be on a
| specific product or system, but everything is in scope.
|
| And the first test is running, and no one is screaming yet, so
| fingers crossed.
| DyslexicAtheist wrote:
| when you say yearly I assume you're not conducting regular
| internal pentests?
|
| any pentesting companies that you could recommend which do more
| than just drive-by shooting with metasploit?
| darefalcon wrote:
| Companies that penetrate networks uninvited looking for
| vulnerabilities to create blog content should be prosecuted IMHO.
| This piece in particular sounds like a hit piece thinly vailed as
| a vulnerability disclosure.
|
| "We thanked them for their co-operation". Sounds kinda like
| extortion.
| ppbjj wrote:
| Are you serious?
| _dain_ wrote:
| way to shoot the messenger.
| dumpsterdiver wrote:
| > Companies that penetrate networks uninvited looking for
| vulnerabilities to create blog content should be prosecuted
| IMHO.
|
| Your comment could be rephrased as, "Companies who carelessly
| collect and store sensitive user data insecurely should not be
| closely scrutinized, and should be left alone to continue
| exposing innocent user data to malicious cyber criminals."
|
| Looks a lot different when you look at it from that angle,
| right?
___________________________________________________________________
(page generated 2024-07-18 23:06 UTC)