[HN Gopher] Reverse engineering a $1B Legal AI tool exposed 100k...
___________________________________________________________________
Reverse engineering a $1B Legal AI tool exposed 100k+ confidential
files
Author : bearsyankees
Score : 784 points
Date : 2025-12-03 17:44 UTC (1 days ago)
(HTM) web link (alexschapiro.com)
(TXT) w3m dump (alexschapiro.com)
| observationist wrote:
| I think this class of problems can be protected against.
|
| It's become clear that the first and most important and most
| valuable agent, or team of agents, to build is the one that
| responsibly and diligently lays out the opsec framework for
| whatever other system you're trying to automate.
|
| A meta-security AI framework, cursor for opsec, would be the
| best, most valuable general purpose AI tool any company could
| build, imo. Everything from journalism to law to coding would
| immediately benefit, and it'd provide invaluable data for post
| training, reducing the overall problematic behaviors in the
| underlying models.
|
| Move fast and break things is a lot more valuable if you have a
| red team mechanism that scales with the product. Who knows how
| many facepalm level failures like this are out there?
| croes wrote:
| > I think this class of problems can be protected against.
|
| Of course, it's called proper software development
| jeffbee wrote:
| The techniques for non-disclosure of confidential materials
| processed by multi-tenant services are obvious, well-known,
| and practiced by very few.
| venturecruelty wrote:
| And jail time for executives who are responsible for data
| leaks.
| marginalx wrote:
| Are you saying executives cannot make mistakes ever (ask
| because you didn't qualify your statement)?
| venturecruelty wrote:
| I'm saying that if executives get praise and bonuses for
| when good things happen, they should also have negative
| consequences when bad things happen. Litigate that
| further how you wish.
| dghlsakjg wrote:
| The key word in that is "responsible".
|
| The legal world has plenty of ways for determining if you
| are legally responsible for the outcome of an event.
| Right now the standard is civil punishments for provable
| negligence.
|
| It sounds like GP is proposing a framework where we
| tighten up the definition of negligence, and add criminal
| penalties in addition to civil ones.
| pbhjpbhj wrote:
| Are you saying the OP was just a single error,
| effectively an executives typo.
| dghlsakjg wrote:
| In this case, AI was a red herring.
|
| This was just plain terrible web security.
| canopi wrote:
| The first thing that comes to my mind is SOC2 HIPAA and the whole
| security theater.
|
| I am one of the engineers that had to suffer through countless
| screenshots and forms to get these because they show that you are
| compliant and safe. While the real impactful things are ignored
| latchkey wrote:
| SemiAnalysis made this a base requirement for being
| appropriately ranked on their ClusterMAX report, telling me it
| is akin to FAA certifications, and then getting hacked
| themselves for not enforcing simple security controls.
|
| https://jon4hotaisle.substack.com/i/180360455/anatomy-of-the...
|
| It is crazy how this gets perpetuated in the industry as
| actually having security value, when in reality, it is just a
| pay-to-play checkbox.
| chickensong wrote:
| You have to start somewhere though. Security theater sucks, and
| it's not like compliance is a silver bullet, but at least it's
| something. Having been through implementing standards
| compliance, it did help the company in some areas. Was it
| perfect? Definitely not. Was it driven by financial goals?
| Absolutely. It did tighten up some weak spots though.
|
| If the options mainly consist of "trust me bro" vs "we can
| demonstrate that we put in some effort", the latter seems more
| preferable, even if it's not perfect.
| quapster wrote:
| This is the collision between two cultures that were never meant
| to share the same data: "move fast and duct-tape APIs together"
| startup engineering, and "if this leaks we ruin people's lives"
| legal/medical confidentiality.
|
| What's wild is that nothing here is exotic: subdomain
| enumeration, unauthenticated API, over-privileged token, minified
| JS leaking internals. This is a 2010-level bug pattern wrapped in
| 2025 AI hype. The only truly "AI" part is that centralizing all
| documents for model training drastically raises the blast radius
| when you screw up.
|
| The economic incentive is obvious: if your pitch deck is "we'll
| ingest everything your firm has ever touched and make it
| searchable/AI-ready", you win deals by saying yes to data access
| and integrations, not by saying no. Least privilege, token
| scoping, and proper isolation are friction in the sales process,
| so they get bolted on later, if at all.
|
| The scary bit is that lawyers are being sold "AI assistant" but
| what they're actually buying is "unvetted third party root access
| to your institutional memory". At that point, the interesting
| question isn't whether there are more bugs like this, it's how
| many of these systems would survive a serious red-team exercise
| by anyone more motivated than a curious blogger.
| j45 wrote:
| It's a little hilarious.
|
| First, as an organization, do all this cybersecurity theatre,
| and then create an MCP/LLM wormhole that bypasses it all.
|
| All because non-technical folks wave their hands about AI and
| not understanding the most fundamental reality about LLM
| software being fundamentally so different than all the software
| before it that it becomes an unavoidable black hole.
|
| I'm also a little pleased I used two space analogies, something
| I can't expect LLMs to do because they have to go large with
| their language or go home.
| jimbokun wrote:
| My first reaction to the announcement of MCP was that I must
| be missing something. Surely giving an LLM unlimited access
| to protected data is going to introduce security holes?
| j45 wrote:
| Agree.
|
| It's assuming and estimating it will behave like other
| software before it when it's nothing like the software that
| came before it.
|
| LLMs today won't behave like the software we're used to
| where 1+1 will equal 2 every time.
| jimbokun wrote:
| It's like handing access to your sensitive data to a
| complete stranger and hoping for the best.
| RansomStark wrote:
| Maybe this is the key takeaway of GenAI: that some access to
| data, even partially hallucinated data, is better than the
| hoops that the security theatre puts in place that prevents
| average Joe doing their job.
|
| This might just be a golden age for getting access to the
| data you need for getting the job done.
|
| Next security will catch up and there'll be a good balance
| between access and control.
|
| Then, as always security goes to far and nobody can get
| anything done.
|
| It's a tale as old as computer security.
| stronglikedan wrote:
| Nitpick, but wormholes and black holes aren't limited to
| space! (unless you go with the Rick & Morty definition where
| "there's literally everything in space")
| j45 wrote:
| Not a nit pick at all friend, it is even more rabbit holes
| to explore.
| dogman144 wrote:
| Assuming a 101 security program past the quality bar, there
| are a number of reason why this can still happen at
| companies.
|
| Summarized as - security is about risk acceptance, not
| removal. There's massive business pressure to risk accept AI.
| Risk acceptance usually means some sort of supplemental
| control that's not the ideal but manages. There are very
| little of these with AI tools however - small vendors,
| they're not really service accounts but IMO best way to
| monitor them probably is that, integrations are easy, eng
| companies hate devs losing admin of some kind but if you have
| that random AI on endpoints becomes very likely.
|
| I'm ignoring a lot of nuance but solid sec program blown open
| by LLM vendors is going to be common, let alone bad sec
| programs. Many sec teams I think are just waiting for the
| other shoe to drop for some evidentiary support while
| managing heavy pressure to go full bore AI integration until
| then.
| j45 wrote:
| You missed risk creation vs reward creation.
|
| And then folks can gasp and faint like goats and pretend
| they didn't know.
|
| It reminds me of the time I met an IT manager who dint have
| an IT background. Outsourced hilarity ensued through sales
| people who were also non-technical.
| dogman144 wrote:
| What am I missing? Risk acceptance is what you're
| referring to - risk creation and reward creation.
|
| Sec lead might have a pretty darn clear idea of an out of
| whack creation of risk v reward. CEO disagrees. Risk
| accept and move on.
|
| When you're technical and eventually realize there's a
| business to survive behind the tech skills, this is the
| stuff you learn how to do.
|
| People "will know" as you say because it's all documented
| and professionally escalated.
| electric_muse wrote:
| [flagged]
| snapcaster wrote:
| What makes you think that? it would need some prompt
| engineering if so since ChatGPT won't write like that (bad
| capitalization, lazy quoting) unless you ask it to
| lazide wrote:
| "Chat, write me a blog article that seems like a lazy human
| who failed English wrote it"?
| legostormtroopr wrote:
| What's worse being accused of an AI post or being defended
| because your post is so bad that AI wouldn't have written
| it?
| FrustratedMonky wrote:
| Well then that's everything.
| simonw wrote:
| That comment didn't read like AI generated content to me. It
| made useful points and explained them well. I would not expect
| even the best of the current batch of LLMs to produce an
| argument that coherent.
|
| This sentence in particular seems outside of what an LLM that
| was fed the linked article might produce:
|
| > What's wild is that nothing here is exotic: subdomain
| enumeration, unauthenticated API, over-privileged token,
| minified JS leaking internals.
| samdoesnothing wrote:
| It's probably a list of bullet points or disjointed sentences
| fed to the LLM to clean up. Might be a non-English speaker
| using it to become fluent. I won't criticize it, but it's
| clearly LLM generated content.
| Aurornis wrote:
| The users' comment history does read like generic LLM output.
| Look at the first lines of different comments:
|
| > Interesting point about Cranelift! I've been following its
| development for a while, and it seems like there's always
| something new popping up.
|
| > Interesting point about the color analysis! It kinda
| reminds me of how album art used to be such a significant
| part of music culture.
|
| > Interesting point about the ESP32 and music playback! I've
| been tinkering with similar projects, and it's wild how much
| potential these little devices have.
|
| > We used to own tools that made us productive. Now we rent
| tools that make someone else profitable. Subscriptions are
| not about recurring value but recurring billing
|
| > Meshtastic is interesting because it's basically "LoRa-
| first networking" instead of "internet with some radios
| attached." Most consumer radios are still stuck in the mental
| model of walkie-talkies, while Meshtastic treats RF as an IP-
| like transport layer you can script, automate, and extend.
| That flips the stack:
|
| > This is the collision between two cultures that were never
| meant to share the same data: "move fast and duct-tape APIs
| together" startup engineering, and "if this leaks we ruin
| people's lives" legal/medical confidentiality.
|
| The repeated prefixes (Interesting point about!) and the
| classic it's-this-not-that LLM pattern are definitely
| triggering my LLM suspicions.
|
| I suspect most of these cases aren't bots, they're users who
| put their thoughts, possibly in another language, into an LLM
| and ask it to form the comment for them. They like the text
| they see so they copy and paste it into HN.
| LoganDark wrote:
| > I suspect most of these cases aren't bots, they're users
| who put their thoughts, possibly in another language, into
| an LLM and ask it to form the comment for them. They like
| the text they see so they copy and paste it into HN.
|
| Yes, if this is LLM then it definitely wouldn't be zero-
| shot. I'm still on the fence myself as I've seen similar
| writing patterns with Asperger's (specifically what used to
| be called Asperger's; not general autism spectrum) but
| those comments don't appear to show any of the other tells
| to me, so I'm not particularly confident one way or the
| other.
| balamatom wrote:
| That's ye olde memetic "immune system" of the "onlygroup"
| (encapsulated ingroup kept unaware it's just an ingroup).
| "It don't sound like how we're taught, so we have no idea
| what it mean or why it there! Go back to Uncanny Valley!"
|
| It's always enlightening to remember where Hans Asperger
| worked, and under what sociocultural circumstances that
| absolutely proverbial syndrome was first conceived.
|
| GP evidently has some very subtle sort of expectations as
| to what authentic human expression must _look like_ ,
| which however seem to extend only as far as things like
| word choice and word order. (If that's all you ever
| notice about words, congrats, you're either a replicant
| or have a bad case of "learned literacy in USA"
| syndrome.)
|
| This makes me want to point out that neither the _means_
| nor the _purpose_ of the kind of communication which GP
| seems to implicitly expect (from random strangers) are
| even considered to be _a real thing_ in many places and
| by many people.
|
| I do happen to find that sort of thing way more cough
| _interesting_ cough than the whole "howdy stranger, are
| you AI or just a pseud" routine that HN posters seem to
| get such a huge kick out of.
|
| Sure looks like one of the most basic moves of
| ideological manipulation: how about we solved the Turing
| Test "the wrong way around" by reducing the tester's
| ability to tell apart human from machine output, instead
| of building a more convincing language machine? Yay,
| expectations subverted! (While, in reality, both happen
| simultaneously.)
|
| Disclaimer: this post was written by a certified
| paperclip optimizer.
| balamatom wrote:
| Or maybe these are people who learned from a LLM that
| English is supposed to sound like this if you want to be
| permitted to communicate a.k.a. "to be taken into
| consideration"! Which is wrong and also kinda sucks, but
| also it sucks and is wrong for a kinda non-obvious reason.
|
| Or, bear with me there, maybe things aren't so far downhill
| yet, these users just learned how English is supposed to
| sound, _from the same place_ where the LLMs learned how
| English is supposed to sound! Which is just the Internet.
|
| AI hype is already ridiculous; the whole "are you using an
| AI to write your posts for you" paranoia is even more
| absurd. So what if they are? Then they'd just be stupid,
| futile thoughts leading exactly nowhere. Just like most
| non-AI-generated thoughts, except perhaps the one which
| leads to the fridge.
| Aurornis wrote:
| Or maybe the 2 month old account posting repetitive
| comments and using the exact patterns common to AI
| generated comment is, actually, posting LLM generated
| content.
|
| > So what if they are? Then they'd just be stupid, futile
| thoughts leading exactly nowhere.
|
| FYI, spammers love LLM generated posting because it
| allows them to "season" accounts on sites like Hacker
| News and Reddit without much effort. Post enough
| plausible-sounding comments without getting caught and
| you have another account to use for your upvote army,
| which is a service you can now sell to desperate
| marketing people who promised their boss they'd get on
| the front page of HN. This was already a problem with
| manual accounts but it took a lot of work to generate the
| comments and content.
|
| That's the "so what"
| balamatom wrote:
| Wasn't there some sort of escape hatch for situations
| like that - for when it becomes impossible to trust the
| agora?
|
| It would be _massively_ funny if that escape hatch just
| sort of disappeared while we were looking at something
| else.
|
| Your point stands, though.
|
| >exact patterns common to AI generated comment
|
| How can there be _exact_ patterns to it?
| jfindper wrote:
| We finally have a blog that no one (yet) has accused of being
| ai generated, so obviously we just have to start accusing
| comments of being ai. Can't read for more than 2 seconds on
| this site without someone yelling "ai!".
|
| For what it's worth, even if the parent comment was directly
| submitted by chatgpt themselves, your comment brought
| significantly less value to the conversation.
| probably_wrong wrote:
| It's the natural response. AI fans are routinely injecting
| themselves into every conversation here to somehow talk about
| AI ("I bet an AI tool would have found the issue faster") and
| AI is forcing itself onto every product. Comments dissing
| anything that sounds even remotely like AI is the logical
| response of someone who is fed up.
| jfindper wrote:
| Every other headline and conversation having ai is super
| annoying.
|
| But also, its super annoying to sift through people saying
| "the word critical was used, this is obviously ai!". not to
| mention it really fucking sucks when you're the person who
| wrote something and people start chanting "ai slop! ai
| slop!". like, how am i going to prove is not AI?
|
| I can't wait until ai gets good enough that no one can tell
| the difference (or ai completely busts and disappears,
| although that's unlikely), and we can go back to just
| commenting about whether something was interesting or
| educational or whatever instead of analyzing how many em-
| dashes someone used pre-2020 and extrapolating whether
| their latest post has 1 more em-dashes then their average
| post so that we can get our pitchforks out and chase them
| away.
| Despyte wrote:
| Cultural acceptance of conversation with AI should've
| come because of actual AI that are indistinguishable from
| humans, being forced to swallow recognizable if not
| blatant LLM slop and turn a blind eye feels unfair
| jfindper wrote:
| the original comment in this chain is not blatant llm
| slop.
| anonymous908213 wrote:
| LLMs will never get good enough that no one can tell the
| difference, because the technology is fundamentally
| incapable of it, nor will it ever completely disappear,
| because the technology has real use cases that can be run
| at a massive profit.
|
| Since LLMs are here to stay, what we actually need is for
| humans to get better at recognising LLM slop, and stop
| allowing our communication spaces to be rotted by slop
| articles and slop comments. It's weird that people find
| this concept objectional. It was historically a given
| that if a spambot posted a copy-pasted message, the
| comment would be flagged and removed. Now the spambot
| comments are randomly generated, and we're okay with it
| because it appears vaguely-but-not-actually-human-like.
| That conversations are devolving into this is actually
| the failure of HN moderation for allowing spambots to
| proliferate unscathed, rather than the users calling out
| the most blatantly obvious cases.
| jfindper wrote:
| Do you think the original comment posted by quapster was
| "slop" equivalent to a copy-paste spam bot?
|
| The only spam I see in this chain is the flagged post by
| electric_muse.
|
| It's actually kind of ironic you bring up copy-paste spam
| bots. Because people fucking love to copy-paste "ai slop"
| on every comment and article that uses any punctuation
| rarer than a period.
| anonymous908213 wrote:
| > Do you think the original comment posted by quapster
| was "slop" equivalent to a copy-paste spam bot?
|
| Yes: the original comment is unequivocally slop that
| genuinely gives me a headache to read.
|
| It's not just "using any punctuation rarer than a
| period": it's the overuse and _mis_ use of punctuation
| that serves as a tell.
|
| Humans don't needlessly use a colon in every single
| sentence they write: abusing punctuation like this is
| actually really fucking irritating.
|
| Of course, it goes beyond the punctuation: there is zero
| substance to the actual output, either.
|
| > What's wild is that nothing here is exotic: subdomain
| enumeration, unauthenticated API, over-privileged token,
| minified JS leaking internals.
|
| > Least privilege, token scoping, and proper isolation
| are friction in the sales process, so they get bolted on
| later, if at all.
|
| This stupid pattern of LLMs listing off jargon like
| they're buzzwords does not add to the conversation.
| Perhaps the usage of jargon lulls people into a false
| sense of believing that what is being said is deeply
| meaningful and intelligent. It is not. It is rot for your
| brain.
| jfindper wrote:
| "it's not just x, it's y" is an ai pattern and you just
| said:
|
| > _" It's not just "using any punctuation rarer than a
| period": it's the overuse and misuse of punctuation that
| serves as a tell."_
|
| So, I'm actually pretty sure you're just copy-pasting my
| comments into chatgpt to generate troll-slop replies, and
| I'd rather not converse with obvious ai slop.
| anonymous908213 wrote:
| Congratulations, you successfully picked up on a pattern
| when I was intentionally mimicking the tone of the
| original spambot content to point out how annoying it
| was. Why are you incapable of doing this with the
| original spambot comment?
| jfindper wrote:
| I'm not replying to your slop (well, you know, after this
| one).
|
| Anyways, if you think something is ai, just flag it
| instead so I don't need to read the word "slop" for the
| 114th fucking time today.
|
| Thankfully, this time, it was flagged. But I got sucked
| in to this absolutely meaningless argument because I lack
| self control.
| anonymous908213 wrote:
| Ironically, you were the first person in this thread to
| use the word "slop". You have become what you hate.
| jfindper wrote:
| jokes on you, I already hate me, that's why I spend so
| much time on HN arguing about nothing
|
| oh shit I'm supposed to be done replying
| vkou wrote:
| What? It doesn't read that way to me. It reads like any other
| comment from the past ~15 years.
|
| The point you raised is both a distraction... And does not
| engage with the ones it did.
| snapdeficit wrote:
| "This comment is AI" is the new "First Post" from /. days.
| Please stop unless you have evidence or a good explanation.
| rootusrootus wrote:
| That was literally the same thought that crossed my mind. I
| agree wholeheartedly, accusing everything and everyone of
| being AI is getting old _fast_. Part of me is happy that the
| skepticism takes hold quickly, but I don 't think it's
| necessary for everyone to demonstrate that they are a good
| skeptic.
|
| (and I suspect that plenty of people will remain credulous
| anyway, AI slop is going to be rough to deal with for the
| foreseeable future).
| lordnacho wrote:
| Also, an AI comment might have a worthwhile point to be
| addressed. Pointing out something was written in a new way
| is not addressing the point.
| Aurornis wrote:
| Spammers use AI comments to build reputation on a fleet
| of accounts for upvoting purposes.
|
| That may or may not be what's happening with this
| account, but it's worth flagging accounts that generate a
| lot of questionable comments. If you look at that
| account's post history there's a lot of familiar LLM
| patterns and repeated post fragments.
| syndacks wrote:
| or, they wrote it and asked an LLM to improve the flow
| koumou92 wrote:
| You are right, it's 100% AI written
| samdoesnothing wrote:
| Ya ur right, it's either LLM generated, LLM enhanced, or the
| author has been reading so much LLM output that its writing
| style has rubbed off.
| Conasg wrote:
| Yeah, you have a point... the comment - and their other
| comments, on average - seem to fit quite a specific pattern.
| It's hard to really draw a line between policing style and
| actually recognising AI-written content, though.
| tomhow wrote:
| Please don't do this here. If a comment seems unfit for HN,
| please flag it and email us at hn@ycombinator.com so we can
| have a look.
|
| We detached this subthread from
| https://news.ycombinator.com/item?id=46137863 and marked it off
| topic.
| kylecazar wrote:
| If they have a billion dollar valuation, this fairly basic (and
| irresponsible) vulnerability could have cost them a billion
| dollars. If someone with malice had been in your shoes, in that
| industry, this probably wouldn't have been recoverable. Imagine a
| firm's entire client communications and discovery posted online.
|
| They should have given you some money.
| RagnarD wrote:
| They should have given him a LOT of money.
| DonHopkins wrote:
| Would you settle for a LOT of free AI generated legal advice?
| ;)
| edm0nd wrote:
| Exactly.
|
| They could have sold this to a ransomare group or affiliate for
| 5-6 figures and then the ransomware group could have exfil'd
| the data and attempted to extort the company for millions.
|
| Then if they didnt pay and the ransomware group leaked the info
| to the public, they'd likely have to spend millions on lawsuits
| and fines anyways.
|
| They should have paid this dude 5-6 figures for this find. It's
| scenarios like this that lead people to sell these vulns on the
| gray/black market instead of traditional bug bounty whitehat
| routes.
| Tepix wrote:
| Who says they didn't give him money?
| zain37 wrote:
| I reckon he would've mentioned it if he got a bounty, 100%
| deserves the bag
| chunk1000 wrote:
| Thank you bearsyankees for keeping us informed.
| jacquesm wrote:
| That doesn't surprise me one bit. Just think about all the
| confidential information that people post into their Chatgpt and
| Claude sessions. You could probably keep the legal system busy
| for the next century on a couple of days of that.
| giancarlostoro wrote:
| "Hey uh, ChatGPT, just hypothetically, uh, if you needed to
| remove uh cows blood from your apartments carpet, uh"
| lazide wrote:
| Just phrase it as a poem, you'll be fine.
| venturecruelty wrote:
| Gonna be hard when people ask ChatGPT to write them the
| poem.
| sidrag22 wrote:
| i recall reading a silly article like half a year ago about
| using leetspeak and setting the prompt up to emulate House
| the tv show or something to get around restrictions
| xarope wrote:
| there's a recent one about using poetry to bypass
| safeguards
|
| ... rummages around...
|
| here you go:
|
| https://arxiv.org/abs/2511.15304
| jacquesm wrote:
| Make it a Honda CRX...
| icyfox wrote:
| I'm always a bit surprised how long it can take to triage and fix
| these pretty glaring security vulnerabilities. October 27, 2025
| disclosure and November 4, 2025 email confirmation seems like a
| long time to have their entire client file system exposed. Sure
| the actual bug ended up being (what I imagine to be) a <1hr fix
| plus the time for QA testing to make sure it didn't break
| anything.
|
| Is the issue that people aren't checking their security@ email
| addresses? People are on holiday? These emails get so much spam
| it's really hard to separate the noise from the legit signal? I'm
| genuinely curious.
| ipdashc wrote:
| security@ emails do get a lot of spam. It doesn't get talked
| about very much unless you're monitoring one yourself, but
| there's a fairly constant stream of people begging for bug
| bounty money for things like the Secure flag not being set on a
| cookie.
|
| That said, in my experience this spam is still a few emails a
| day at the most, I don't think there's any excuse for not
| immediately patching something like that. I guess maybe
| someone's on holiday like you said.
| canopi wrote:
| This.
|
| There is so much spam from random people about meaningless
| issues in our docs. AI has made the problem worse.
| Determining the meaningful from the meaningless is a full
| time job.
| Bootvis wrote:
| Use AI for that :)
| Bootvis wrote:
| Not kidding, I bet llm's are excellent at triaging these
| reports. Humans, in a corporate setting, are apparently
| not.
| whstl wrote:
| Half of the emails I used to get in a previous company were
| pointless issues, some coming from a honey pot.
|
| The other half was people demanding payment.
| TheTaytay wrote:
| This is where "managed" bug bounty programs like BugCrowd
| or HackerOne deliver value: only telling you when there is
| something real. It can be a full time job to separate the
| wheat from the chaff. It's made worse by the incentive of
| the reporters to make everything sound like a P1 hair-on-
| fire issue.
| horacemorace wrote:
| Training a tech support team of interns to solve all of
| them would be an enviable hacker or software dev training
| program.
| latchkey wrote:
| My favorite one is the "We've identified a security hole in
| your website"... and I always respond quickly that my website
| is statically generated, nothing dynamic and immutable on
| cloudflare pages. For some odd reason, I never hear back from
| them.
| Capricorn2481 wrote:
| > October 27, 2025 disclosure and November 4, 2025 email
| confirmation seems like a long time to have their entire client
| file system exposed
|
| I have unfortunately seen way worse. If it will take more than
| an hour and the wrong people are in charge of the money, you
| can go a pretty long time with glaring vulnerabilities.
| giancarlostoro wrote:
| I call that one of the worrisome outcomes from "Marketing
| Driven Development" where the business people don't let you
| do technical debt "Stories" because you REALLY need to do
| work that justifies their existence in the project.
| gwbas1c wrote:
| Not every organization prioritizes being able to ship a code
| change at the drop of a hat. This often requires organizational
| dedication to heavy automated testing a CI, which small
| companies often aren't set up to do.
| stavros wrote:
| I can't believe that any company takes a month to ship
| something. Even if they don't have CI, surely they'd prefer
| to break the app (maybe even completely) than risk all their
| legal documents exfiltrated.
| Aurornis wrote:
| > I can't believe that any company takes a month to ship
| something.
|
| Outside of startups and big tech, it's not uncommon to have
| release cycles that are months long. Especially common if
| there is any legal or regulatory involvement.
| Jolter wrote:
| It'd be pretty reasonable to take the whole API down in
| this scenario, and put it back up once it's patched. They'd
| lose tons of cash but avoid being liable for extreme
| amounts of damages.
| technion wrote:
| I can only say you havent worked anywhere i have.
|
| I remember heartbleed dropping shortly after a deployment
| and not being allowed to patch for like ten months because
| the fix wasn't "validated". This was despite insurers
| stating this issue could cost coverage and legal getting
| involved.
| stavros wrote:
| What? That's crazy, wow!
| Barathkanna wrote:
| A lot of the time it's less "nobody checked the security inbox"
| and more "the one person who understands that part of the
| system is juggling twelve other fires." Security fixes are
| often a one-hour patch wrapped in two weeks of internal
| routing, approvals, and "who even owns this code?" archaeology.
| Holiday schedules and spam filters don't help, but
| organizational entropy is usually the real culprit.
| throwaway290 wrote:
| It's not about fixing it, it's about acknowledging it exists
| Aurornis wrote:
| > A lot of the time it's less "nobody checked the security
| inbox" and more "the one person who understands that part of
| the system is juggling twelve other fires."
|
| At my past employers it was "The VP of such-and-such said we
| need to ship this feature as our top priority, no exceptions"
| whstl wrote:
| I've once had a whole sector of a fintech go down because one
| DevOps person ignored daily warning emails for three months
| that an API key was about to expire and needed reset.
|
| And of course nobody remembered the setup, and logging was
| only accessible by the same person, so figuring out also took
| weeks.
| bongodongobob wrote:
| I'm currently on the other side of this trying to convince
| management that the maintenance that should have been done
| 3 years ago needs to get done. They need "justification".
| jll29 wrote:
| Write a short memo that saying you are very concerned,
| and describe a range of things that may happen (from "not
| much" over medium to maximum scare - lawsuits,
| brand/customer trust destroyed etc.).
|
| Email the memo to a decision maker with the important
| flag on and CC: another person as a witness.
|
| If you have been saying it for a long time and nobody has
| taken any action, you may use the word "escalation" as
| part of the subject line.
|
| If things hit the fan, it will also make sure that what
| drops from the fan falls on the right people, and not on
| you.
| ChrisMarshallNY wrote:
| It could also be someone "practicing good time management."
|
| They have a specific time of day, when they check their
| email, and they only give 30 minutes to that time, and they
| check emails from most recent, down.
|
| The email comes in, two hours earlier, and, by the time they
| check their email, it's been buried under 50 spams, and near-
| spams; each of which needs to be checked, so they run out of
| 30 minutes, before they get to it. The next day, by email
| check time, another 400 spams have been thrown on top.
|
| Think I'm kidding?
|
| Many folks that have worked for large companies (or
| bureaucracies) have seen exactly this.
| eru wrote:
| The system would be mostly sane, if you could sort by some
| measure of importance, not just recency.
| Aurornis wrote:
| In my experience, it comes down to project management and
| organizational structure problems.
|
| Companies hire a "security team" and put them behind the
| security@ email, then decide they'll figure out how to handle
| issues later.
|
| When an issue comes in, the security team tries to forward the
| security issue to the team that owns the project so it can be
| fixed. This is where complicated org charts and difficult
| incentive structures can get in the way.
|
| Determining which team actually owns the code containing the
| bug can be very hard, depending on the company. Many security
| team people I've worked with were smart, but not software
| developers by trade. So they start trying to navigate the org
| chart to figure out who can even fix the issue. This can take
| weeks of dead-ends and "I'm busy until Tuesday next week at
| 3:30PM, let's schedule a meeting then" delays.
|
| Even when you find the right team, it can be difficult to get
| them to schedule the fix. In companies where roadmaps are
| planned 3 quarters in advance, everyone is focused on their
| KPIs and other acronyms, and bonuses are paid out according to
| your ticket velocity and on-time delivery stats (despite PMs
| telling you they're not), getting a team to pick up the bug and
| work on it is hard. Again, it can become a wall of "Our next 3
| sprints are already full with urgent work from VP so-and-so,
| but we'll see if we can fit it in after that"
|
| Then legal wants to be involved, too. So before you even
| respond to reports you have to flag the corporate counsel, who
| is already busy and doesn't want to hear it right now.
|
| So half or more of the job of the security team becomes
| navigating corporate bureaucracy and slicing through all of the
| incentive structures to inject this urgent priority somewhere.
|
| Smart companies recognize this problem and will empower
| security teams to prioritize urgent things. This can cause
| another problem where less-than-great security teams start
| wielding their power to force everyone to work on not-urgent
| issues that get spammed to the security@ email all day long
| demanding bug bounties, which burns everyone out. Good security
| teams will use good judgment, though.
| srrdev wrote:
| Oh man this is so true. In this sort of org, getting
| something fixed out-of-band takes a huge political effort
| (even a critical issue like having your client database
| exposed to the world).
| DrewADesign wrote:
| While there were numerous problems with the big corporate
| structures I worked in decades ago where everything was
| done by silos of specialists, there were huge advantages.
| No matter where there was a security, performance, network,
| hardware, etc. issue, the internal support infrastructure
| had the specialist's pagers and for a problem like this,
| the people fixing it would have been on a conference call
| until it was fixed. There was always a team of specialists
| to diagnose and test fixes, always available developers
| with the expertise to write fixes if necessary, always ops
| to monitor and execute things, always a person in charge to
| make sure it all got done, and everybody knew which
| department it was and how to reach them 24/7.
|
| Now if you needed to develop something not-urgent that
| involved, say, the performance department, database
| department, and your own, hope you've got a few months to
| blow on conference calls and procedure documents.
|
| For that industry it made sense though.
| eru wrote:
| Interesting. Wouldn't the performance department have
| their fingers in all the pies anyway, too, or how was
| that handled?
| rvba wrote:
| > Many security team people I've worked with were smart, but
| not software developers by trade.
|
| A lot are people who cannot code at all, cannot administer -
| they just fill tables and check boxes, maybe from some
| automated suite. They dont know what http and https is,
| because they are just paper pushers what is far from real
| security, but more like security in name only.
|
| And they joined the work since it pays well
| tietjens wrote:
| Great comment. Very true.
| bfxbjuf wrote:
| Well we have 600 people in the global response center I work
| at. And the priority issue count is currently 26000. That means
| its serious enough that its been assigned to some one. There
| are tens of thousands of unassigned issues cuz the traige teams
| are swamped. People dont realize as systems get more complex
| issues increase. They never decrease. And the chimp troupes
| response has always been a Story - we can handle it.
| bgbntty2 wrote:
| I'm a bit conflicted about what responsible disclosure should
| be, but in many cases it seems like these conditions hold:
|
| 1) the hack is straightforward to do;
|
| 2) it can do a lot of damage (get PII or other confidential
| info in most cases);
|
| 3) downtime of the service wouldn't hurt anyone, especially if
| we compare it to the risk of the damage.
|
| But, instead of insisting on the immediate shutting down of the
| affected service, we give companies weeks or months to fix the
| issue while notifying no one in the process and continuing with
| business as usual.
|
| I've submitted 3 very easy exploits to 3 different companies
| the past year and, thankfully, they fixed them in about a week
| every time. Yet, the exploits were trivial (as I'm not good
| enough to find the hard ones, I admit). Mostly IDORs, like
| changing id=123456 to id=1 all the way up to id=123455 and
| seeing a lot medical data that doesn't belong to me. All 3
| cases were medical labs because I had to have some tests done
| and wanted to see how secure my data was.
|
| Sadly, in all 3 cases I had to send a follow-up e-mail after ~1
| week, saying that I'll make the exploit public if they don't
| fix it ASAP. What happened was, again, in all 3 cases, the
| exploit was fixed within 1-2 days.
|
| If I'd given them a month, I feel they would've fixed the issue
| after a month. If I'd given then a year - after a year.
|
| And it's not like there aren't 10 different labs in my city.
| It's not like online access to results is critical, either. You
| can get a printed result or call them to write them down. Yes,
| it would be tedious, but more secure.
|
| So I should've said from the beginning something like:
|
| > I found this trivial exploit that gives me access to medical
| data of thousands of people. If you don't want it public, shut
| down your online service until you fix it, because it's highly
| likely someone else figured it out before me. If you don't,
| I'll make it public and ruin your reputation.
|
| Now, would I make it public if they don't fix it within a few
| days? Probably not, but I'm not sure. But shutting down their
| service until the fix is in seems important. If it was some
| hard-to-do hack chaining several exploits, including a 0-day,
| it would be likely that I'd be the first one to find it and it
| wouldn't be found for a while by someone else afterwards. But
| ID enumerations? Come on.
|
| So does the standard "responsible disclosure", at least in the
| scenario I've given (easy to do; not critical if the service is
| shut down), help the affected parties (the customers) or the
| businesses? Why should I care about a company worth $X losing
| $Y if it's their fault?
|
| I think in the future I'll anonymously contact companies with
| way more strict deadlines if their customers (or others) are in
| serious risk. I'll lose the ability to brag with my real name,
| but I can live with it.
|
| As to the other comments talking about how spammed their
| security@ mail is - that's the cost of doing business. It
| doesn't seem like a valid excuse to me. Security isn't one of
| hundreds random things a business should care about. It's one
| of the most important ones. So just assign more people to
| review your mail. If you can't, why are you handling people's
| PII?
| ghostly_s wrote:
| > I think in the future I'll anonymously contact companies
| with way more strict deadlines if their customers (or others)
| are in serious risk. I'll lose the ability to brag with my
| real name, but I can live with it.
|
| What you're describing is likely a crime. The sad reality is
| most businesses don't view protection of customers' data as a
| sacred duty, but simply another of the innumerable risks to
| be managed in the course of doing business. If they can say
| "we were working on fixing it!" their asses are likely
| covered even if someone does leverage the exploit first--and
| worst-case, they'll just pay a fine and move on.
| bgbntty2 wrote:
| Precisely - they view security as just one part of many of
| their business, instead of viewing it as one of the most
| important parts. They've insured themselves against a
| breach, so it's not a big deal for them. But it should be.
|
| The more casualties, the more media attention -> the more
| likely they, and others in their field, will take security
| more seriously in the future.
|
| If we let them do nothing for a month, they'll eventually
| fix it, but in the mean time malicious hackers may gain
| access to the PII. They might not make it public, but sell
| that PII via black markets. The company may not get the
| negative publicity it deserves and likely won't learn to
| fix their systems in time and to adopt adequate security
| measures. The sale of the PII and the breach itself might
| become public knowledge months after the fact, while the
| company has had a chance to grow in the meantime, and make
| more security mistakes that may be exploited later on.
|
| And yes, I know it may be a crime - that's why I said I'd
| report it anonymously from now on. But if the company sits
| on their asses for a month, shouldn't that count as a
| crime, as well? The current definition of responsible
| disclosure gives companies too much leeway, in my opinion.
|
| If I knew I operated a service that was trivial to exploit
| and was hosting people's PII, I'd shut it down until I
| fixed it. People won't die if I make everything in my power
| to provide the test results (in my example of medical labs)
| to doctors and patients via other means, such as via paper
| or phone. And if people do die, it would be devastating, of
| course, but it would mean society has put too much trust
| into a single system without making sure it's not
| vulnerable to the most basic of attacks. So it would happen
| sooner or later, anyway. Although I can't imagine someone
| dying because their doctor had to make a phone call to the
| lab instead of typing in a URL.
|
| The same argument about people dying due to the disruption
| of the medical communications system could be made about
| too-big-to-fail companies that are entrenched into society
| because a lot of pension funds have invested in them. If
| the company goes under, the innocent people dependent on
| the pension fund's finances would suffer. While they would
| suffer, which would be awful, of course, would the
| alternative be to not let such companies go bankrupt? Or
| would it be better for such funds to not rely so much on
| one specific company in the first place? That is to say, in
| both cases (security or stocks in general) the reality is
| that currently people are too dependent on a few singular
| entities, while they shouldn't be. That has to change, and
| the change has to begin somewhere.
| nl wrote:
| Don't do this.
|
| I understand you think you are doing the right thing but be
| aware that by shutting down a medical communication services
| there's a non-trivial chance someone will die because of
| slower test results.
|
| Your responsibility is responsible disclosure.
|
| Their responsibility is how to handle it. Don't try to decide
| that for them.
| jofzar wrote:
| > October 27, 2025 disclosure and November 4, 2025 email
| confirmation seems like a long time to have their entire client
| file system exposed
|
| There is always the simple answer, these are lawyers so they
| are probably scrambling internally to write a response that
| covers themselves legaly also trying to figure out how fucked
| they are.
|
| 1 week is surprisingly not that slow.
| perlgeek wrote:
| Another aspect to consider: when you reduce the amount of
| permission anything has (like here the returned token), you
| risk breaking something.
|
| In a complex system it can be very hard to understand what will
| break, if anything. In a less complex system, it can still be
| hard to understand if the person who knows the security model
| very well isn't available.
| londons_explore wrote:
| The security@ inbox has so much junk these days with someone
| reporting that if you paste alert('hacked') into devtools then
| it makes the website hacked!
|
| I reckon only 1% of reports are valid.
|
| LLM's can now make a plausible looking exploit report ('there
| is a use after free bug in your server side implementation of X
| library which allows shell access to your server if you time
| these two API calls correctly'), but the LLM has made the whole
| thing up. That can easily waste hours of an experts time for a
| total falsehood.
|
| I can completely see why some companies decide it'll be an
| office-hours-only task to go through all the reports every day.
| tryauuum wrote:
| My favorite was "we can trigger your website to initiate a
| connection to the server we control". They were running their
| own mail servers and were creating a new accounts on our
| website. Of course someone needs to initiate a TCP connection
| to deliver an email message!
|
| Of course this could be a real vulnerability if it would
| disclose the real server IP behind cloudflare. This was not
| the case, we were sending via AWS email gateway
| Invictus0 wrote:
| This guy didn't even get paid for this? We need a law that
| establishes mandatory payments for cybersecurity bounty hunters.
| imvetri wrote:
| Legal attacks engineering - font type license fee on japan
| consumers. Engineering attacks legal - AI info dump in above
| post.
|
| How does above sound like and what kind of professional write
| like that?
| sys32768 wrote:
| I work for a finance firm and everyone is wondering why we can
| store reams of client data with SaaS Company X, but not upload a
| trust document or tax return to AI SaaS Company Y.
|
| My argument is we're in the Wild West with AI and this stuff is
| being built so fast with so many evolving tools that corners are
| being cut even when they don't realize it.
|
| This article demonstrates that, but it does sort of beg the
| question as to why not trust one vs the other when they both
| promise the same safeguards.
| layer8 wrote:
| The question is what reason did you have to trust SaaS Company
| X in the first place?
| sys32768 wrote:
| Because it's the Cloud and we're told the cloud is better and
| more secure.
|
| In truth the company forced our hand by pricing us out of the
| on-premise solution and will do that again with the other on-
| premise we use, which is set to sunset in five years or so.
| ansgri wrote:
| Probably has more to do with responsibility outsourcing: if
| SaaS has security breach AND they tell in the contract that
| they're secure, then you're not responsible. Sure, there
| may be reputational damage for you, but it's a gamble with
| good odds in most cases.
|
| Storing lots of legal data doesn't seem to be one of these
| cases though.
| bonesss wrote:
| I see profits and outsourcing.
|
| Selling an on-premise service requires customer support,
| engineering, and duplication of effort if you're pushing
| to the cloud as well. Then you get the temptations and
| lock in of cloud-only tooling and an army of certified
| consultant drones whose resumes really really need time
| on AWS-doc-solution-2035, so the on premise becomes a
| constant weight on management.
|
| SaaS and the cloud is great for some things some of the
| time, but often you're just staring at the marketing
| playbook of MS or Amazon come to life like a golem.
| pm90 wrote:
| SaaS is now a "solved problem"; almost all vendors will try
| to get SOX/SOC2 compliance (and more for sensitive
| workloads). Although... its hard to see how these
| certifications would have prevented something like this
| :melting_face:.
| pstuart wrote:
| And nobody seems to pay attention to the fact that modern
| copiers cache copies on a local disk and if the machines are
| leased and swapped out the next party that takes possession has
| access to those copies if nobody bothered to address it.
| lupire wrote:
| This was the plot of Grisham's book The Firm in 1991
| mbesto wrote:
| > My argument is we're in the Wild West with AI and this stuff
| is being built so fast with so many evolving tools that corners
| are being cut even when they don't realize it.
|
| The funny thing is that this exploit (from the OP) has nothing
| to do with AI and could be <insert any SaaS company> that
| integrates into another service.
| pr337h4m wrote:
| FWIW this company was founded in 2014 and appears to have added
| LLM-powered features relatively recently:
| https://www.reuters.com/legal/transactional/legal-tech-compa...
| teej wrote:
| It doesn't sound like your firm does any diligence that would
| actually prevent you from buying a vendor that has security
| flaws.
| Aperocky wrote:
| Does SaaS X/Cloud offer IAM capabilities? Or going further, do
| they dogfood their own access via the identity and access
| policies? If so, and you construct your own access policy, you
| have relative peace of mind.
|
| If SaaS Y just says "Give me your data and it will be secure",
| that's where it gets suspect.
| whalesalad wrote:
| using ai vs not-ai as your litmus test is giving you a false
| sense of security. it's ALL wild west
| hughes wrote:
| While the FileVine service is indeed a Legal AI tool, I don't
| see the connection between this particular blunder and AI
| itself. It sure seems like _any_ company with an inexperienced
| development team and thoughtless security posture could build a
| system with the same issues.
|
| Specifically, it does not appear that AI is invoked in any way
| at the search endpoint - it is clearly piping results from some
| Box API.
| empiko wrote:
| There is none. Filevine is not even an "AI" company. They are
| a pretty standard SaaS that has some AI features nowadays.
| But the hive mind needs its food, and AI bad as we all know.
| lionkor wrote:
| > any company with an inexperienced development team and
| thoughtless security posture
|
| Point out one (1) "AI product" company that isn't described
| accurately by that sentence
| richwater wrote:
| Of course there will be no accountability or punishment.
| lupire wrote:
| Who is Margolis, and are they happy that OP publicly announced
| accessing all their confidential files?
|
| Clever work by OP. Surely there is automatic prober tool that
| already hacked this product?
| dghlsakjg wrote:
| > Who is Margolis, and are they happy that OP publicly
| announced accessing all their confidential files?
|
| Google tells me they are a NY law firm specializing in Real
| Estate and Immigration law. There are other firms with Margolis
| in the name too. Kinda doesn't matter; see below.
|
| I doubt that they are thrilled to have their name involved in
| this, but that is covered by the US constitution's protections
| on free press.
| valbaca wrote:
| Given the absurd amount startups I see lately that have the words
| "healthcare" and "AI", I'm actually incredibly concerned that in
| just a couple of months we're going to have an multiple, enormous
| HIPAA-data disasters
|
| Just search "healthcare" in
| https://news.ycombinator.com/item?id=46108941
| mattfrommars wrote:
| This might be off topic since we are in topic of AI tool and on
| HackerNews.
|
| I've been pondering a long time how does one build a startup
| company in domain they are not familiar with but ... Just have
| this urge to 'crave a pie' in this space. For the longest time, I
| had this dream of starting or building a 'AI Legal Tech Company'
| -- big issue is, I don't work in legal space at all. I did some
| cold reach on lawfirm related forums which did not take any
| traction.
|
| I later searched around and came across the term, 'case
| management software'. From what I know, this is what Cilo
| fundamentally is and make millions if not billion.
|
| This was close to two years or 1.5 years ago and since then, I
| stopped thinking about it because of this understanding or belief
| I have, "how can I do a startup in legal when I don't work in
| this domain" But when I look around, I have seen people who start
| companies in totally unrelated industry. From starting a 'dental
| tech's company to, if I'm not mistaken, the founder of hugging
| face doesn't seem to have PHD in AI/ML and yet founded
| HuggingFace.
|
| Given all said, how does one start a company in unrelated domain?
| Say I want to start another case management system or attempt to
| clone FileVine, do I first read up what case management software
| is or do I cold reach to potential lawfirm who would partner up
| to built a SAAS from scratch? Other school of thought goes like,
| "find customer before you have a product to validate what you
| want to build", how does this realistically work?
|
| Apologies for the scattered thoughts...
| jimbokun wrote:
| One approach is to partner with someone who is an expert in
| that space.
| strgcmc wrote:
| I think it comes down to, having some insight about the
| customer need and how you would solve it. Having prior
| experience in the same domain is helpful but is neither a
| guarantee nor a blocker, towards having a customer insight
| (lots of people might work in a domain but have no idea how to
| improve it; alternatively an outsider might see something that
| the "domain experts" have been overlooking).
|
| I just randomly happened to read about the story of, some
| surgeons asking a Formula 1 team to help improve its surgical
| processes, with spectacular results in the long term... The F1
| team had zero medical background, but they assessed the
| surgical processes and found huge issues with communication and
| lack of clarity, people reaching over each other to get to
| tools, or too many people jumping to fix something like a hose
| coming loose (when you just need 1 person to do that 1 thing).
| F1 teams were very good at designing hyper efficient and
| reliable processes to get complex pit stops done extremely
| quickly, and the surgeons benefitted a lot from those process
| engineering insights, even though it had nothing specifically
| to do with medical/surgical domain knowledge.
|
| Reference: https://www.thetimes.com/sport/formula-
| one/article/professor...
|
| Anyways, back to your main question -- I find that it helps to
| start small... Are you someone who is good at using analogies
| to explain concepts in one domain, to a layperson outside that
| domain? Or even better, to use analogies that would help a
| domain expert from domain A, to instantly recognize an
| analogous situation or opportunity in domain B (of which they
| are not an expert)? I personally have found a lot of benefit,
| from both being naturally curious about learning/teaching
| through analogies, finding the act of making analogies to be a
| fun hobby just because, and also honing it professionally to
| help me be useful in cross-domain contexts. I think you don't
| need to blow this up in your head as some big grand mystery
| with some big secret cheat code to unlock how to be a founder
| in a domain you're not familiar with -- I think you can start
| very small, and just practice making analogies with your
| friends or peers, see if you can find fun ways of explaining
| things across domains with them (either you explain to them
| with an analogy, or they explain something to you and you try
| to analogize it from your POV).
| airstrike wrote:
| I think if you have no domain expertise or unique insight it
| will be quite hard to find a real pain point to solve, deliver
| a winning solution, and have the ability to sell it.
|
| Not impossible, but very hard. And starting a company is hard
| enough as it is.
|
| So 9/10 times the answer will be to partner with someone who
| understands the space and pain point, preferably one who has
| lived it, or find an easier problem to solve.
| joshvm wrote:
| I would also split the concerns:
|
| 1. Compliancy with relevant standards. HIPAA, GDPR, ISO,
| military, legal, etc. Realistically you're going to outsource
| this or hire someone who knows how to build it, and then
| you're going to pay an agency to confirm that you're
| compliant. You also need to consider whether the incumbent
| solution is a trust-based solution, like the old "nobody gets
| fired for buying Intel".
|
| 2. Domain expertise is always easier if you have a domain
| expert. Big companies also outsource market research. They'll
| go to a firm like GLG, pay for some expert's time or
| commission a survey.
|
| It seems like table stakes to do some basic research on your
| own to see what software (or solutions) exist and why
| everyone uses them, and why competitors failed. That should
| cost you nothing but time, and maybe expense if you buy some
| software. In a lot of fields even browsing some forums or
| Reddit is enough. The difference is if you have a _working_
| product that 's generic enough to be useful to other domains,
| but you're not sure. Then you might be able to arrange some
| sort of quid pro quo like a trial where the partner gets to
| keep some output/analysis, and you get some real-world
| testing and feedback.
| fallinditch wrote:
| > ... after looking through minified code, which SUCKS to do ...
|
| AI tends to be good at un-minifying code.
| a_victorp wrote:
| Legit question: when working on finding security issues, are
| there any guidelines on what you can send to LLMs/AI?
| fallinditch wrote:
| I got downvoted, so maybe that means someone thinks un-
| minifying code is not advised for dealing with security
| issues? But on reflection surely you can just use the 'format
| code' command in the ide? I am no expert but surely it's ok
| to use AI to help track down and identify security issues
| with the usual caveats of 'don't believe it blindly, do your
| double checking and risk assessing.'
| CER10TY wrote:
| Personally, I'd just use common sense and good judgment. At
| the end of the day, would you want someone to hand your
| address, and other private data to OpenAI just like that?
| Probably not. So don't paste customer data into it if you can
| avoid it.
|
| On the other hand, minified code is literally published by
| the company. Everyone can see it and do with it as they
| please. So handing that over to an AI to un-minify is not
| really your problem, since you're not the developer working
| on the tool internally.
| nodesocket wrote:
| Doesn't Chrome Developer tools automatically un-minify?
| yieldcrv wrote:
| I've worked in several "agentic" roles this year alone (I'm very
| poachable lol)
|
| and otherwise well structured engineering orgs have lost their
| goddamn minds with move fast and break things
|
| because they're worried that OpenAI/Google/Meta/Amazon/Anthropic
| will release the tool they're working on tomorrow
|
| literally all of them are like this
| trollbridge wrote:
| Old school blue chip type of companies are like this too.
| They've thrown all the process and caution they used to have to
| the wind so that they can... apply AI to their IT org which
| isn't even their core business?
| 2ndatblackrock wrote:
| now that's just great hacking
| corry wrote:
| "Companies often have a demo environment that is open" - huh?
|
| And... Margolis allowed this open demo environment to connect to
| their ENTIRE Box drive of millions of super sensitive documents?
|
| HUH???!
|
| Before you get to the terrible security practices of the vendor,
| you have to place a massive amount of blame on the IT team of
| Margolis for allowing the above.
|
| No amount of AI hype excuses that kind of professional
| misjudgement.
| me_again wrote:
| I don't think we have enough information to conclude exactly
| what happened. But my read is the researcher was looking for
| demo.filevine.com and found margolis.filevine.com instead. The
| implication is that many other customers may have been
| vulnerable in the same way.
| corry wrote:
| Ah, I see now that I read too quickly - the "open demo
| environment" was clearly referencing the idea that the vendor
| (Filevine) would have a live demo, NOT that each client
| wanted an open playground demo account that is linked to a
| subset of their data (which would be utterly insane).
| etamponi wrote:
| I don't disagree with the sentiment. But let's also be honest.
| There is _a lot_ of improvement to be made in security software,
| in terms of ease of use and overcomplicating things.
|
| I worked at Google and then at Meta. Man, the amount of
| "nonsense" of the ACL system was insane. I write nonsense in
| quotes because for sure from a security point of view it all made
| a lot of sense. But there is exactly zero chance that such a
| system can be used in a less technical company. It took me 4
| years to understand how it worked...
|
| So I'll take this as another data point to create a startup that
| simplifies security... Seems a lot more complicated than AI
| deep_thinker26 wrote:
| It's so great that they allowed him to publish a technical blog
| post. I once discovered a big vulnerability in a listed consumer
| tech company -- exposing users' private messages and also
| allowing to impersonate any user. The company didn't allow me to
| write a public blogpost.
| qmr wrote:
| "Allow"?
|
| Go on write your blog post. Don't let your dreams be dreams.
| bigmadshoe wrote:
| Presumably they were paid for finding the bug and inn
| accepting relinquished their right to blog about it.
| hsbauauvhabzb wrote:
| No, you relinquish the right when you agree to their TOS
| irrespective of if they pay you.
| amackera wrote:
| TOS != law
|
| They will stop letting you use the service. That's the
| recourse for breaking the TOS.
| advisedwang wrote:
| Up until Van Buren v. United States in 2020, ToS
| violations were sometimes prosecuted as unauthorized
| access under the CFAA. I suspect there are other
| jurisdictions that still do the equivalent to that.
| hsbauauvhabzb wrote:
| I don't want to pay for a lawyer to argue that for me. !=
| law does not equate to 'won't come with a cost'.
|
| I say this as someone threatened by a billion dollar
| company for this very thing.
| gessha wrote:
| Why is the control of publication in their hands and not in
| yours? Shouldn't you be able to do whatever after disclosing it
| responsibly?
| CER10TY wrote:
| Presumably they'll threaten to sue you and/or file a criminal
| complaint, which can be pretty hard to deal with depending on
| the jurisdiction. At that point you'll probably start asking
| yourself if it's worth publishing a blog post for some
| internet points.
| trollbridge wrote:
| Yet another reason these disclosures should be anonymous (from
| the reporting side).
| hbarka wrote:
| > November 20, 2025: I followed up to confirm the patch was in
| place from my end, and informed them of my intention to write a
| technical blog post.
|
| Can that company tell you to cease and desist? How does the law
| work?
| me_again wrote:
| Lawyers can and will send cease and desist letters to people
| whether or not there is any legal basis for it. Often the
| threat of a lawsuit, even a meritless one, is enough to keep
| people quiet.
| dghlsakjg wrote:
| FYI, a "cease and desist" carries the same legal weight as me
| sending a one-liner saying "Knock it off".
|
| They are strongly worded requests from a legal point of view.
| The only real message they send is that the sender is serious
| enough about the issue to have involved a lawyer, unless of
| course you write it yourself, which is something that literally
| anyone can do.
|
| If you want to actually force an action, you need a court order
| of some type.
|
| NB for the actual lawyers: I'm oversimplifying, since they can
| be used in court to prove that you tried to get the other party
| to stop, and tried to resolve the issue outside of court.
| stanfordkid wrote:
| I mean... in what world would you send a customers private root
| key to a web browsing client. Like even if the user was
| authenticated why would they need this? This sort of secret
| shouldn't even be in an environment variable or database but
| stored with encryption at rest. There could easily have been a
| proxy service between client and box if the purpose is to search
| or download files. It's very bad, even for a prototype... this
| researcher deserves a bounty!
| bzmrgonz wrote:
| My thing is, even ingesting the BOK should have been done in
| phases, to avoid having all your virtual eggs in one basket or
| nest at any ONE time. Staggering tokens to these compartments
| would not have cost them anything at all . I always say, whatever
| convenience you enjoy yourself, will be highly appreciated by bad
| actors... WHEN, not if.. they get thru.
| badbird33 wrote:
| You'd think with a $1B valuation they could afford a pentest
| magnetowasright wrote:
| I am at a loss for words. This wasn't a sophisticated attack.
|
| I'd love to know who filevine uses for penetration testing (which
| they do, according to their website) because holy shit, how do
| you miss this? I mean, they list their bug bounty program under a
| pentesting heading, so I guess it's just nice internet people.
|
| It's inexcusable.
| rashidujang wrote:
| This was my impression after reading the article too. I have no
| doubt that the team at Filevine attempted to secure their
| systems and have probably thwarted other attackers, but got
| their foot stuck in what is an unsophisticated attack. It only
| takes one chain vulnerability to bring down the site.
|
| Security reminds me of the Anna Karenina principle: All happy
| families are alike; each unhappy family is unhappy in its own
| way.
| GJim wrote:
| > I am at a loss for words. This wasn't a sophisticated attack.
|
| To be fair, data security breaches seldom are.
| ethin wrote:
| I'll be honest... I'm not at all surprised that this happened.
| Purely because it seems like everyone who wants to implement AI
| just forgot all of the institutional knowledge that cybersecurity
| has acquired over the last 30-40 years. When you "forget" all of
| that because you want to rush out something really fast, well,
| you know what they say: play stupid games, win stupid prizes and
| all that.
| culanuchachamim wrote:
| -The Filevine team was responsive, professional, and took the
| findings seriously throughout the disclosure process. They
| acknowledged the severity, worked to remediate the issues,
| allowed responsible disclosure, and maintained clear
| communication. This is another great example of how organizations
| should handle security disclosures.
|
| In the same tenure I think that a professional etical hacker or a
| curious fellow that is poking around with no harm intent,
| shouldn't disclose the name of the company that had a security
| issue if they resolve it professionally.
|
| You can write the same blog post without mentioning that it was
| Filevine.
|
| If they didn't take care of the incident that's a different
| story...
| CBMPET2001 wrote:
| Eh, with something this horrendously egregious I think their
| customers have a right to know how carelessly their data was
| handled, regardless of the remediation steps taken after
| disclosure; that aside, who knows how many other AI SaaS
| vendors might stumble across this article and realize they've
| made a similarly boneheaded error, and save both themselves and
| their customers a huge amount of pain . . .
| evan_a_a wrote:
| This is a very standard part of responsible disclosure. Hacker
| finds bugs -> discloses them to the vendor -> (hopefully) the
| vendor communicates with them and remediates -> both sides
| publish the technical details. It also helps to demonstrate to
| the rest of the security world which companies will take
| reports seriously and which ones won't, which is very useful
| information to have.
| deelowe wrote:
| That's not how ethical disclosure works. Both parties should
| publish and we, the wider tech industry should see this as a
| good thing both for the hacker and the company that worked with
| them.
| manbash wrote:
| How else can you take responsibility if you don't make it
| public? You can't have integrity if you hide away your faults.
| theoldgreybeard wrote:
| The bigwigs at my company want to build out a document management
| suite. After talking to VP of technology about requirements I ask
| about security as well as what the regulatory requirements are
| and all I get is a blank stare.
|
| I used to think developers had to be supremely incompetent to end
| up with vulnerabilities like this.
|
| But now I understand it's not the developers who are
| incompetent...
| eru wrote:
| There's enough incompetence at all levels to go around.
| theoldgreybeard wrote:
| Maybe I have just been lucky, but I have not had the
| displeasure of working with people either tha incompetent or
| willfully ignorant yet.
| eru wrote:
| Oh, I should have been more careful in my formulation:
|
| There are organisations that are generally competent, and
| there are places that are less competent. It's not all that
| uncommon for the whole organisation to be generally
| incompetent.
|
| The saddest places (for me) are those where almost every
| individual you talk to seems generally competent, but
| judging by their output the company might as well be
| stuffed by idiots. Something in the way they are organised
| suppresses the competence. (I worked at one such company.)
|
| > Maybe I have just been lucky, but I have not had the
| displeasure of working with people either tha incompetent
| or willfully ignorant yet.
|
| It's very important before you start any new job to suss
| out how competent people and the organisation are. Ideally,
| you probably want to work for a competent company. But at
| least you want to know what you are getting into.
|
| There's a bit of luck involved, if you go in blindly, but
| you can also use skill and elbow-grease to investigate.
| chii wrote:
| > suss out how competent people and the organisation are.
|
| how does one do this, without first having the job and
| being embedded in there? From the outside, it's near
| impossible to see these details imho.
| eru wrote:
| Yes, it's hard, and I'm not sure there are general
| strategies that always work.
|
| It's fundamentally the same problem that the company is
| trying to solve when they interview you, just the other
| way 'round.
|
| Some ideas: observe and ask in the interviews and hiring
| process in general. See what you can find out about the
| company from friends, contacts and even strangers.
| Network! Do some online research, too.
|
| Btw, lots of the cliche interview questions ("What are
| your greatest weaknesses?" etc) actually make decent
| questions you can ask about the company and team you are
| about to join.
| SaltyBackendGuy wrote:
| Something I've found useful is just reaching out to past
| employees. Usually folks that don't work there anymore
| will be more transparent. Only challenge is getting
| someone to respond to you, but you'd be surprised how
| many folks will talk if you don't come off like you're
| trying to sell them something or a bot.
| vkou wrote:
| > Something in the way they are organised suppresses the
| competence.
|
| It's a natural outcome of authoritarian structures when
| the people at the top are idiots. When that happens, the
| whole organization rots.
| delaminator wrote:
| I'm governed by them
|
| Reeves orders Treasury inquiry over Budget leaks
|
| Chancellor's policies found their way to the press before
| she announced them to MPs
|
| https://www.telegraph.co.uk/news/2025/12/03/reeves-orders-
| tr...
| blitzar wrote:
| "Are We the Baddies?"
| samdung wrote:
| Incompetence compounds at an astonishing rate.
| Lord-Jobo wrote:
| Not only does the Peter principle generally show more
| incompetence the higher up a structure you move, but the
| outsized influence those positions have make for a very
| noticeably higher level of "things fucked up by incompetence"
| coming from the C suite compared to the rest of the
| structure.
|
| There's definitely plenty of incompetence regardless. But
| I've never seen a company where the incompetence was more
| noteworthy in the cog positions than "leadership".
| hahn-kev wrote:
| I've had the same. Ask them to come up with a ToS and they're
| like "we'll talk about that in an upcoming meeting" it's been a
| few years now with nothing.
| habosa wrote:
| They took a month to fix this? That's beyond inexcusable. I can't
| imagine how any customer could justify working with them going
| forward.
|
| Also ... shows you what a SOC 2 audit is worth:
| https://www.filevine.com/news/filevine-proves-industry-leade...
|
| Even the most basic pentest would have caught this.
| eru wrote:
| It looks like SOC 2 (and the other SOCs) where developed by
| accountants?
|
| I wouldn't expect them to find any computer problems either to
| be honest.
| jonny_eh wrote:
| Where did it say that they took a month to fix? The hacker just
| checked in 2 weeks later and it was fixed by that point.
| theodorejb wrote:
| According to the timeline it took more than a week just for
| Filevine to respond saying they would review and fix the
| vulnerability. It was 24 days after initial disclosure when
| he confirmed the fix was in place.
| OtherShrezzing wrote:
| Given that the author describes the company as prompt,
| communicative and professional, I think it's fair to assume
| there was more contact than the four events in the top of
| the article.
| technion wrote:
| Unless im missing something, they replied stating they would
| look into it and then its totally vague when they patched, with
| Alex apparently randomly testing later and telling them in a
| "follow up" that it was fixed.
|
| I dont at all get why there is a paragraph thanking their
| communication if that is the case.
| nick49488171 wrote:
| Probably given the alternative, being ghosted followed by a
| no-knock FBI raid
| rustystump wrote:
| Soc2 and most other certifications are akin to the tsa,
| security theater. After seeing the info sec security space from
| the inside i can only say that it blows my mind how abhorrent
| the security space is. Prod db creds in code? A ok. Not using
| some stupid vendors "pen testing" software on each mr,
| blasphemy?
| stingraycharles wrote:
| SOC2 is mainly to check boxes, and forces you to think about a
| few things. There's no real / actual audit, and in my
| experience the pen tests are very much a money grab. You're
| paying way too much money for some "pentesting" automated suite
| to run.
|
| The auditors themselves pretty much only care that you answered
| all questions, they don't really care what the answers are and
| absolutely aren't going to dig any deeper.
|
| (I'm responsible for the SOC2 audits at our firm)
| abustamam wrote:
| When I worked for a consulting firm some years back I
| randomly got put on a project that dealt with payment
| information. I had never had to deal with payment information
| before so I was a bit nervous about being compliant. I was
| pointed to SOC2 compliance which sounded scary. Much to my
| relief (and surprise), the SOC2 questionnaire was literally
| just what amounted to a survey monkey form. I answered as
| truthfully as I could and at the end it just said "congrats
| you're compliant!" or something to that effect.
|
| I asked my my manager if that's all that was required and he
| said yes, just make sure you do it again next year. I spent
| the rest of my time worrying that we missed something. I
| genuinely didn't believe him until your comment.
|
| Edit: missing sentence.
| chanux wrote:
| Once this type of issue gets publicized, does that in anyway
| affect the certification?
| aitchnyu wrote:
| Is there any stricter standard? Should one strive for PCI-DSS
| even if they are a regular SaaS?
| mrweasel wrote:
| The time to fix isn't really important, assuming that they took
| the system offline in the mean time... but we all know they
| didn't, because that would cost to much.
| satya71 wrote:
| I had a look at some of FileVine's output, and I can say I'm not
| surprised. This is not an organization that prizes engineering at
| all.
| BrenBarn wrote:
| I'm less and less sure that when a billion-dollar company screws
| up this bad, the right thing to do is privately disclose it and
| let them fix it. This kind of thing just allows companies to go
| on taking people's money without facing the consequences of their
| mistakes.
| barbazoo wrote:
| Does a disclosure like this absolve them of any responsibility?
| They still violated whatever user privacy act.
| mvkel wrote:
| It does. Most privacy laws are based on time-from-discovery.
| If they immediately sprung into action at the moment they
| were informed and remediated the issue, they're in
| compliance.
| mattmaroon wrote:
| So is that true if they find out when the public does too?
| It seems that disclosing it privately has some upside
| (protecting the users) and no downside.
| mvkel wrote:
| That depends more on what the Privacy Policy is of the
| service, which you agree to when you sign up and use it
| BrenBarn wrote:
| Right, that's the problem. There need to be standards that
| govern what can ever be released to customers/the public in
| the first place. When violations of those are discovered,
| the penalties should be based on time from release, so the
| longer it was out in the wild, the greater the penalty.
| mvkel wrote:
| But you can't remove something from the internet once
| it's there, so once it's released, it's expected that it
| always will be.
|
| It's also impossible to guarantee a 100% secure
| infrastructure, no matter how good your product team is.
|
| In the grey is a term of art: "best efforts."
|
| If data is leaking, and it wasn't because hackers
| bypassed a bunch of safeguards, if it can be shown that
| you didn't use Best Efforts to secure said data, there is
| liability.
| btbuildem wrote:
| What are you going to do, sue them? The place is literally
| teeming with lawyers.
| abustamam wrote:
| What would you suggest the right thing to do would be?
|
| Edit: I agree with you that we shouldnt let companies like this
| get away with what amounts to a slap on the wrist. But
| everything else seems irresponsible as well.
| BrenBarn wrote:
| I guess if I imagine the ideal world, it would be that you
| report it to the authorities and they impose penalties on the
| offender that are large enough that the company winds up
| significantly worse off than if they had just grown more
| slowly. In other words the punishment for moving fast and
| breaking things needs to be bad enough to outweigh the gains
| of doing so.
|
| In the current world, I dunno. I guess it depends on what the
| company is. If it's something like a hedge fund or a fossil
| fuel company I think I'd be fine with some kind of wikileaks-
| like avenue for exposing it in such a way that it results in
| the company being totally destroyed.
| 0xbadcafebee wrote:
| So, 1) a public service, 2) with no authentication, 3) and no
| encryption? (http only??), 4) sent every single response with a
| token, 5) giving _full admin access to every client 's legal
| documents_. This is like a law firm with an open back door, open
| back window, and all the confidential legal papers sprawled out
| on the floor.
|
| Imagine the potential impact. You're a single mother, fighting
| for custody of your kids. Your lawyer has some documentation of
| something that happened to you, that wasn't your fault, but would
| look bad if brought up in court. Suddenly you receive a phone
| call - it's a mysterious voice, demanding $10,000 or they will
| send the documents to the opposition. Neither of them knows each
| other; someone just found a trove of documents in an open back
| door and wanted to make a quick buck.
|
| This is exactly what a software building code would address (if
| we had one!). Just like you can't open a new storefront in a new
| building without it being inspected, you should not be able to
| process millions of sensitive files without having your
| software's building inspected. The safety and privacy of all of
| us shouldn't be optional.
| Fnoord wrote:
| http-only makes it also very easy to sniff for LE if they
| decide to. This allows them to get knowledge about cases. Like,
| they could be scanning it with their own AI tool for all we
| know. In a free country with proper LE, this would neither be
| legal nor happening. But I am not sure the USA is remaining
| one, given the leader is a convicted felon with _very_ dubious
| moral standards.
|
| The problem here however is that they get away with their
| sloppiness as long as the security researcher who found this is
| a whitehat, and the regular news won't pick it up. Once regular
| media pick this news up (and the local ones should), their name
| is tarnished and they may regret their sloppiness. Which is a
| good way to ensure they won't make the same mistake. After all,
| money talks.
| zwnow wrote:
| All the big tech companies are in the news every week.
| Everybody knows how bad they are. Their names are tarnished
| and yet everyone is still using their junk and they face zero
| repercussions when fucking up. I dont think things in the
| media would do any harm.
| Fnoord wrote:
| In the news, sure. But negatively? I consider myself
| included in 'everyone', and I am not using junk from all
| the big tech companies. More than once, I've successfully
| quit using certain ones, and Signal has become much more
| popular in my country ever since Trump II took office. Meta
| had to change the name of their company (Facebook) since it
| had such a bad name, and Zuck started a charm offensive.
| altmanaltman wrote:
| but google told me everyone can vibe code apps now and software
| engineers should count their days... it's almost as if there's
| more stuff we do than just write code...
| eru wrote:
| > it's almost as if there's more stuff we do than just write
| code..
|
| Yes, but adding these common sense considerations is actually
| something LLMs can already do reasonably well.
| darkwater wrote:
| In 90% of the cases. And if you don't know how to spot that
| other 10%, you are still screwed, cause someone else will
| found that (and you don't even need to be an elite black
| hat to find it).
| AbstractH24 wrote:
| What's to say a human would catch this 10% either?
| MangoToupe wrote:
| The salary you pay them, typically
| AbstractH24 wrote:
| Salaries make humans infallible?
| MangoToupe wrote:
| No, but it makes them motivated to be thorough. There is
| no way to motivate a chatbot (to do better or to any
| end).
| AbstractH24 wrote:
| But money is a way to motivate the people who created AI
| to create better AI. Because if it doesn't perform as
| expected, either people won't use it or they'll turn to a
| competitor next time they need to do something. And these
| companies need recurring revenue.
|
| If we're saying the way to ensure competency is to
| instill fear of not getting money tomorrow as a
| consequence of failure, then AI companies and humans are
| on equal footing.
| wesleywt wrote:
| Humans are pretty good at edge cases.
| LunaSea wrote:
| If you explicitly request it which means you need to know
| about it.
| degamad wrote:
| Or you need to guess that it exists, or you need to scan
| for places it exists.
| eru wrote:
| OpenAI can put that in the system prompt for their CTO-
| as-a-service once, and then forget about it.
| MangoToupe wrote:
| Clearly not
| blitzar wrote:
| humans used open s3 buckets stuffed with text files of
| usernames, passwords, addresses, credit card numbers etc long
| before vibe coding was a thing.
| dr_dshiv wrote:
| Prediction: Vibe coding systems will be better at security
| in 2 years than 90% of devs.
| input_sh wrote:
| Prediction: it won't.
|
| You can't fit every security consideration into the
| context window.
| dr_dshiv wrote:
| 90% of human devs are not aware of every security
| consideration.
| input_sh wrote:
| 90% of human devs can fit more than 3-5 files into their
| short-term memory.
|
| They also know not to, say, temporarily disable auth to
| be able to look at the changes they've made on a page
| hidden behind auth, which is what I observed Gemini 3 Pro
| doing just yesterday.
| dr_dshiv wrote:
| Ok, and that's your prediction for 2 years from now? It'd
| be quite remarkable if humans had a bigger short term
| memory than LLMs in 2 years. Or that the kind of dumb
| security mistakes LLMs make today don't trigger major,
| rapid improvements.
| input_sh wrote:
| Do you understand what the term "context window" means?
| Have you ever tried using an LLM to program anything even
| remotely complex? Have you observed how the quality of
| the output drastically reduces the longer the coversation
| gets?
|
| That's what makes it bad at security. It cannot
| comprehend more than a floppy drive worth of data before
| it reverts to absolute gibberish.
| eric-burel wrote:
| You may want to read about agentic AI, you can for
| instance call an LLM multiple times with different
| security consideration everytime.
| windexh8er wrote:
| And that buys you what, exactly? Your point is 100%
| correct and why LLMs are no where near able to manage /
| build _complete_ simple systems and surely not complex
| ones.
|
| Why? Context. LLMs, today, go off the rails fairly
| easily. As I've mentioned in prior comments I've been
| working a lot with different models and agentic coding
| systems. When a code base starts to approach 5k lines
| (building the entire codebase with an agent) things start
| to get very rough. First of all, the agent cannot wrap
| it's context (it has no brain) around the code in a
| complete way. Even when everything is very well
| documented as part of the build and outlined so the LLM
| has indicators of where to pull in code - it almost
| always cannot keep schemas, requirements, or patterns in
| line. I've had instances where APIs that were being
| developed were to follow a specific schema, should
| require specific tests and should abide by specific
| constraints for integration. Almost always, in that
| relatively small codebase, the agentic system gets
| something wrong - but because of sycophancy - it
| gleefully informs me all the work is done and everything
| is A-OK! The kicker here is that when you show it why /
| where it's wrong you're continuously in a loop of burning
| tokens trying to put that train back on the track. LLMs
| can't be efficient with new(ish) code bases because
| they're always having to go lookup new documentation and
| burning through more context beyond what it's targeting
| to build / update / refactor / etc.
|
| So, sure. You can "call an LLM multiple times". But this
| is hugely missing the point with how these systems work.
| Because when you actually start to use them you'll find
| these issues almost immediately.
| joshribakoff wrote:
| To add onto this, it is a characteristic of their design
| to statistically pick things that would be bad choices,
| because humans do too. It's not more reliable than just
| taking a random person off the street of SF and giving
| them instructions on what to copy paste without any
| context. They might also change unrelated things or get
| sidetracked when they encounter friction. My point is
| that when you try to compensate by prompting repeatedly,
| you are just adding more chances for entropy to leak in
| -- so I am agreeing with you.
| windexh8er wrote:
| > To add onto this, it is a characteristic of their
| design to statistically pick things that would be bad
| choices, because humans do too.
|
| Spot on. If we look at, historically, "AI" (pre-LLM) the
| data sets were much more curated, cleaned and labeled.
| Look at CV, for example. Computer Vision is a prime
| example of how AI can easily go off the rails with
| respect to 1) garbage input data 2) biased input data.
| LLMs have these two as inputs in spades _and_ in vast
| quantities. Has everyone forgotten about Google 's
| classification of African American people in images [0]?
| Or, more hilariously - the fix [1]? Most people I talk to
| who are using LLMs think that the data being strung into
| these models has been fine tuned, hand picked, etc. In
| some cases for small models that were explicitly curated,
| sure. But in the context (no pun) of all the popular
| frontier models: no way in hell.
|
| The one thing I'm really surprised nobody is talking
| about is the system prompt. Not in the manner of
| jailbreaking it or even extracting it. But I can't
| imagine that these system prompts aren't collecting mass
| tech debt at this point. I'm sure there's band aid after
| band aid of simple fixes to nudge the model in ever so
| different directions based on things that are,
| ultimately, out of the control of such a large
| culmination of random data. I can't wait to see how these
| long term issues crop and and duct taped for the quick
| fixes these tech behemoths are becoming known for.
|
| [0] https://www.bbc.com/news/technology-33347866 [1] http
| s://www.theguardian.com/technology/2018/jan/12/google-
| ra...
| input_sh wrote:
| There's about a dozen workarounds around context limits,
| agents being one of them, MCP servers being another one,
| AGENTS.md being the third one, but none of them actually
| solve the issue of a context window being so small that
| it's useless for anything even remotely complex.
|
| Let's imagine a codebase that can fit onto a
| revolutionary piece of technology known as a floppy
| drive. As we all know, a floppy drive can store <2
| megabytes of storage. But a 100k tokens is only about 400
| _kilobytes_. So, to process the whole codebase that can
| fit onto a floppy drive, you need 5 agents plus the sixth
| "parent process" that those 5 agents will report to.
|
| Those five agents can report "no security issues found"
| in their own little chunk of the codebase to the parent
| process, and that parent process will still be none the
| wiser about how those different chunks interact with each
| other.
| joshribakoff wrote:
| In one instance it could not even describe why a test is
| bad unit test (asserting true is equal to true), which
| doesn't even require context or multi file reasoning.
|
| Its almost as if it has additional problems beyond the
| context limits :)
| joshribakoff wrote:
| You may want to try using it, anecdotes often differ from
| theories, especially when they are being sold to you for
| profit. It takes maybe a few days to see a pattern of
| ignoring simple instructions even when context is clean.
| Or one prompt fixes one issue and causes new issues,
| rinse and repeat. It requires human guidance in practice.
| ethbr1 wrote:
| Strongman: LLMs aren't a tool, they're fuzzy automation.
|
| And what keeps security problems from making it into prod
| in the real world?
|
| Code review, testing, static and dynamic code scanning,
| and fuzzing.
|
| Why aren't these things done?
|
| Because there isn't enough people-time and expertise.
|
| So in order for LLMs to improve security, they need to be
| able to improve our ability to do one of: code review,
| testing, static and dynamic code scanning, and fuzzing.
|
| It seems very unlikely those forms of automation won't be
| improved in the near future by even the dumbest form of
| LLMs.
|
| And if you offered CISOs a "pay to scan" service that
| actually worked cross-language and -platform (in contrast
| to most "only supported languages" scanners), they'd jump
| at it.
| aduwah wrote:
| This will age badly
| dr_dshiv wrote:
| That's why we make concrete measurable predictions.
| MangoToupe wrote:
| Agreed, but "vibe coding will be better at security" is
| not one of them. Better by which metric, against which
| threat model, with which stakes? What security even means
| for greenfield projects is inherently different than for
| hardened systems. Vibe coding is sufficient for security
| today because it's not used for anything that matters.
| Cthulhu_ wrote:
| It'll play a role in both securing and security research
| I'm sure, but I'm not confident it'll be better.
|
| But also, you'd need to have some metrics - how good are
| developers at security already? What if the bar is on the
| floor and LLM code generators are already better?
| wizzledonker wrote:
| Only if they work in a fundamentally different manner. We
| can't solve that problem the way we are building LLMs
| now.
| Covenant0028 wrote:
| And those humans would be looking for a new job or face
| other consequences. An AI model can merrily do this with
| zero consequences because no meaningful consequences can be
| visited upon it.
|
| Just like if any human employee publicly sexually harassed
| his female CEO, he'd be out of a job and would find it very
| hard to find a new one. But Grok can do it and it's the
| _CEO_ who ends up quitting.
| compootr wrote:
| @grok all software engineers do is mindlessly turn
| specifications into code in one shot, right?!?
| agos wrote:
| See also: HN told me that regulation is bad and this is why
| the EU is behind!
| gbacon wrote:
| This is HN. We understood exactly what "exposed ...
| confidential files" meant before reading your overly dramatic
| scenario. As overdone as it is, it's not even realistic. A
| likely single mother is likely tiny potatoes in comparison to
| deep-pocketed legal firms or large corporations.
|
| The story _is an example of the market self-correcting_ , but
| out comes this "building code" hobby horse anyway. All a
| software "building code" will do is ossify _certain_ current
| practices, not even necessarily the best ones. It will tilt the
| playing field in favor of large existing players and to the
| disadvantage of innovative startups.
|
| The model fails to apply in multiple ways. Building physical
| buildings is a much simpler, much less complex process with
| many fewer degrees of freedom than building software. Local
| city workers inspecting by the local municipality's code at
| least has clear jurisdiction because of where the physical
| fixed location is. Who will write the "building code"? Who will
| be the inspectors?
|
| This is HN. Of all places, I'd expect to see this presented as
| an opportunity for new startups, not calls for slovenly
| bureaucracy and more coercion. The private market is perfectly
| capable of performing this function. E&O and professional
| liability insurers if they don't already will be soon motivated
| after seeing lawsuits to demand regular pentests.
|
| The reported incident is a great reminder of _caveat emptor_.
| objclxt wrote:
| > Building physical buildings is a much simpler, much less
| complex process with many fewer degrees of freedom than
| building software.
|
| I don't...think this is true? Google has no problems shipping
| complex software projects, their London HQ is years behind
| schedule and vastly over budget.
|
| Construction is really complex. These can be mega-projects
| with tens of thousands of people involved, where the
| consequences of failure are injury or even death. When
| software failure _does_ have those consequences - things like
| aviation control software, or medical device firmware -
| engineers are held to a considerably higher standard.
|
| > The private market is perfectly capable of performing this
| function
|
| But it's totally not! There are so many examples in the
| construction space of private markets being wholly unable to
| perform quality control because there are financial
| incentives not to.
|
| The reason building codes exist and are enforced by
| municipalities is because the private market is incapable of
| doing so.
| Sharlin wrote:
| Basically what happened in the Vastaamo case in Finland [1].
| Except of course it wasn't individual phone calls - it was mass
| extortion of 30,000 people at once via email.
|
| [1] https://en.wikipedia.org/wiki/Vastaamo_data_breach
| tryauuum wrote:
| if I remember correctly the attacker got caught in such a
| silly way
|
| he wanted to demonstrate that he indeed has the private data.
| But he fucked up the tar command and it ended up having his
| username in the directory names, a username he used in other
| places on the internet
| anshumankmr wrote:
| Can't even make the basic auth of the password is password123.
| 1vuio0pswjnm7 wrote:
| "(after looking through minified code, which SUCKS to do)"
|
| Would there be a "pretty printer" or some other "unminifier" for
| this task
|
| If not, then is minification effectively a form of obfuscation
| gu5 wrote:
| https://webcrack.netlify.app/ was made exactly for this, it's a
| really useful tool
| densone wrote:
| If I had a dollar for every time I saw something like this or for
| every time someone exposed a .git with a .env full of secrets
| keernan wrote:
| Attorneys are ethically obligated to follow very stringent rules
| to protect their client's confidential information. Having been a
| practicing litigator for 40+ years, I can confidently state I
| came across very few attorneys who truly understood their
| obligations.
|
| Things were easier when I first began practicing in the 1970s.
| There weren't too many ways confidential materials in our files
| could be compromised. Leaving my open file spread out on the
| conference room table while I went to lunch while attorneys
| arriving for a deposition on my partner's case were one by one
| seated into the conference room. That's the kind of thing we had
| to keep an eye on.
|
| But things soon got complicated. Computers. Digital copies of
| files that didn't disappear into an external site for storage
| like physical files. Then email. What were our obligations to
| know what could - and could not - be intercepted while email
| traveled the internet.
|
| Then most dangerous of all. Digital storage that was outside our
| physical domain. How could we now know if the cloud vendor had
| access to our confidential data? Where were the backups stored?
| How exactly was the data securely compartmentalized by a cloud
| vendor? Did we need our own IT experts to control the data
| located on the external cloud? What did the contracts with the
| cloud vendor say about the fact we were a law firm and that we,
| as the lawyers responsible for our clients confidential
| information, needed to know that they - the cloud vendor -
| understood the legal obligations and that they - the cloud vendor
| - would hire lawyers to oversee the manner in which the cloud
| vendor blocked all access to the legal data located on their own
| servers. And so on and so forth.
|
| I'm no longer in active practice but these issues were a big part
| of my practice my last few years at a Fortune 500 insurance
| company that used in-house attorneys nationwide to represent
| insureds in litigation - and the corporation was in engaged in
| signing onto a cloud service to hold all of the corporate data -
| including the legal departments across all 50 states. It was a
| nightmare. I'm confident it still is.
| tonyhart7 wrote:
| so the future is software engineer cosplaying an cybersecurity so
| we can implement _BASIC_ security????
| hansmayer wrote:
| For all the talk in the blog of how "super-professional" their
| team was (probably just a courtesy on the side of the author, I
| don't think he believes his own words either)... I have noticed
| using AI to produce some kind of API -OR- use a 3rd party point
| with integration into frontend, is almost guaranteed to produce
| code in which the frontend either exposes the API secrets
| directly in the frontend code (literally injecting it into a
| variable as string), or if you ask it for authentication, it will
| build some half-built lazy solution which makes no sense. So I
| imagine their "super-professional" team built this with AI,
| blindly trusting, probably even allowing it to commit and merge
| changes itself because if you are not merging 10K LoC a day with
| all this "great" technology, what are you even doing, right? It
| is not super-professional to work effectively with a blindfold
| on, I'd argue.
| MangoToupe wrote:
| > countless data protected by HIPAA
|
| People should really look this law up before they reference it
| xp84 wrote:
| Through that API, the frontend is handed a high-privilege API key
| for Box. Not only was that a huge blunder on the backend, but it
| reveals what passes for architecture these days. Should our
| application's backend speak to the super sensitive file store?
| No, we should hand over the keys to that to the React app,
| because _it literally did not occur to them that there 's
| anything that physically could be driven by the frontend that
| shouldn't be._
|
| My apologies to the frontend engineers out there who know what
| they're doing.
| nstj wrote:
| The post is a disclosure about a vuln which seem to have anything
| to do with AI.
| Fokamul wrote:
| "Coding is done", Adam Wolff, Anthropic
| aperture147 wrote:
| Hey I think I've just found a new marketing stunt for a new vibe-
| coding platform:
|
| "Worried your vibe-coded app is about to be broadcast on the
| internet's biggest billboard? Chill. ACME AI now wraps it in
| "NSA-grade" security armor."
|
| I've never thought that there will be multiple billion-dollar-AI-
| features that fixes all the monkey patching problems that no one
| saw them coming from the older billion-dollar-AI-features that
| fixes all the monkey patching problems that no one saw them
| coming from...
| canto wrote:
| How is following a http request and guessing some variables a
| "reverse" engineering now?
| testemailfordg2 wrote:
| When liability of a corporation and its owners is limited, does
| it benefit their business to be dilligent in every step or have a
| mentality of move fast and break things?
___________________________________________________________________
(page generated 2025-12-04 23:02 UTC)