[HN Gopher] Microsoft says bug causes Copilot to summarize confi...
___________________________________________________________________
Microsoft says bug causes Copilot to summarize confidential emails
Author : tablets
Score : 218 points
Date : 2026-02-18 12:16 UTC (10 hours ago)
(HTM) web link (www.bleepingcomputer.com)
(TXT) w3m dump (www.bleepingcomputer.com)
| tablets wrote:
| Initial date of issue 3rd Feb 2026
| dolphinscorpion wrote:
| A bug here and a bug there...
| p0w3n3d wrote:
| 100 nasty bugs in the code 100 bugs in the code
| Take one down Patch it around -127 nasty bugs in
| the code
| childofhedgehog wrote:
| > However, this ongoing incident has been tagged as an advisory,
| a flag commonly used to describe service issues typically
| involving limited scope or impact.
|
| How is having Copilot breach trust and privacy an "advisory"? Am
| I missing something?
| dijit wrote:
| Advisory doesn't have the same meaning in security research as
| it does in the english language.
|
| Unfortunately "Advisory" is a report written about a security
| incident, like an official statement about the bug, it's
| impact, and how to fix it -- which differs from the english
| meaning... it's not meant to mean to "advise" people or to
| "take something" under "advisory" (which, is a very soft
| statement typically).
| layer8 wrote:
| https://www.merriam-webster.com/dictionary/advise meaning 2: to
| give information or notice to : INFORM
|
| An advisory gives notice and/or warns about something, and
| _may_ give recommendations on possible actions (but doesn't
| have to).
| _verandaguy wrote:
| Words have multiple meanings depending on context, and here
| it's _at best_ ambiguous. In the context of security
| incidents, logging, auditing, etc., "advisory" is often used
| as a severity level (and one of the lower ones at that).
|
| So, yes, technically, it's de-facto advisory to publish this
| information, but assigning "advisory" as a severity tag here
| is questionable.
| lich_king wrote:
| The LLM that wrote this nearly content-free story doesn't know
| what it's talking about.
|
| The basic distinction in the infosec industry is that
| advisories are what you publish to tell customers that you had
| a bug in your product that _might_ have exposed them or their
| data to attacks and you want them to take some specific action
| (e.g., upgrade a package, review logs); while an incident
| report is what you publish when you know that the damage
| happened, it involved your infrastructure, and you want to
| share some details about happened and how you 're going to
| prevent it from happening again.
|
| Because the latter invites a lot more public attention and
| regulatory scrutiny, a company like Microsoft will go out of
| their way to stick to advisories whenever possible (or just
| keep incidents under wraps). It might have happened at some
| points in their history, but off the top of my head, I don't
| recall Microsoft ever publishing a first-party security
| incident report.
| bpodgursky wrote:
| If you inflate severity, people simply ignore incident
| warnings.
|
| What's the actual action needed here by a security team? None.
| You can hate it or not care but the end of the day there's no
| remediation or imminent harm, just a potential issue with DLP
| policies. Don't make it look like a 0-day that they actually
| have to deal with.
| indiekitai wrote:
| This highlights a fundamental challenge with AI assistants: they
| need broad access to be useful, but that access is hard to scope
| correctly.
|
| The bug is fixable, but the underlying tension--giving AI tools
| enough permissions to help while respecting confidentiality
| boundaries--will keep surfacing in different forms as these tools
| become more capable.
|
| We're essentially retrofitting permission models designed for
| human users onto AI agents that operate very differently.
| hippo22 wrote:
| How is this different than any other access control system?
| ses1984 wrote:
| When you frame it that way, it's really not that different.
| The issue isn't the access control system itself, more so
| that it's really asking too much of people who don't have the
| skills or understanding to manage it. Teams of trained
| professionals get it wrong when the scope is limited to a
| single application or suite of applications, and you think
| grandma is going to properly manage access control over her
| entire digital footprint?
| kakacik wrote:
| Well, its maintained by humans to start with, peer reviewed
| by humans. They fuck up from time to time in extremely
| limited scope, depending on how much given company is willing
| to invest into getting quality work, but nothing like this.
| Humans are clearly not the weak link to be automated away, in
| contrary.
|
| I work in one of the special legal jurisdictions, such fubar
| would normally mean banning such product from company for
| good. Its micro$oft so unfortunately not possible yet, but oh
| boy are they digging their grave with such public
| incompetence, with horrible handling of the situation on top
| of that. For many companies, this is top priority right
| behind assuring enough cash flow, not some marginal
| regulatory topic. Dumb greedy amateurs.
| jrjeksjd8d wrote:
| I think the fundamental tension is that AI produces a high
| volume of low quality output, and the human in the loop hates
| reviewing all the slop. So people want to just let the AI
| interface directly, but when you let slop into the real world
| there are consequences.
| pjc50 wrote:
| Crucially, this wouldn't be an issue if the AI ran locally, but
| "sending all your internal email in cleartext to the cloud" is
| a potentially serious problem for organizations with real
| confidentiality requirements.
| SignalStackDev wrote:
| The retrofitting problem is real, but there's a more specific
| design failure worth naming: the data flows in the wrong
| direction.
|
| In traditional access control, the pattern is: user requests
| data -> permissions checked -> data returned or denied. The
| model never sees unauthorized data.
|
| With Copilot and most LLM agents today, the pattern is: user
| asks question -> model retrieves broadly -> sensitivity label
| checked as a filter -> model generates answer. The label-
| checking happens after the data is already in the model's
| context.
|
| That's the bug waiting to happen, label system or not. You
| can't reliably instruct a model to 'ignore what you just read.'
|
| The pattern that actually works - and I've had to build this
| explicitly for agent pipelines - is pre-retrieval filtering.
| The model emits a structured query (what it needs), that query
| gets evaluated against a permission layer before anything comes
| back, and only permitted content enters the context window. The
| model architecturally can't see what it's not allowed to see.
|
| The DLP label approach is trying to solve a retrieval problem
| with a generation-time filter. It's a category error, and it'll
| keep producing bugs like this one regardless of how good the
| label detection gets.
| codeulike wrote:
| Reads to me like it is not accessing other users mailboxes, its
| just accessing the current user's mailbox (like its meant to) but
| its supposed to ignore current user's emails that have a
| 'confidential' flag and that bit had a bug
| layer8 wrote:
| I think the issue is that the confidential information is being
| sent to cloud AI, against DLP policies.
| tremon wrote:
| I think that Microsoft would rather not acknowledge that one.
| It's much easier to hide behind a simple "bug" than to admit
| to such a massive security breach.
| layer8 wrote:
| Not a bug, a "code issue".
| doodlebugging wrote:
| It's a feature now.
| SoftTalker wrote:
| I.e. LLM slop code that wasn't adequately tested.
| HeavyStorm wrote:
| Exactly.
| gortok wrote:
| There are two issues I see here (besides the obvious "Why do we
| even let this happen in the first place?"):
|
| 1. What happened to all the data Copilot trained on that was
| confidential? How is that data separated and deleted from the
| model's training? How can we be sure it's gone?
|
| 2. This issue was found; unfortunately without a much better
| security posture from Microsoft, we have no way of knowing what
| issues are currently lurking that are as bad as --- if not worse
| than --- what happened here.
|
| There's a serious fundamental flaw in the thinking and misguided
| incentives that led to "sprinkle AI everywhere", and instead of
| taking a step back and rethinking that approach, we're going to
| get pieced together fixes and still be left with the foundational
| problem that everyone's data is just one prompt injection away
| from being taken; whether it's labeled as "secure" or not.
| doctorpangloss wrote:
| All the vendors paraphrase user data, then use the paraphrased
| data for training. This is what their terms of service say.
|
| They have significant experience in this. Microsoft software
| since the 2014, for the most part, is also paraphrased from
| other people's code they find laying around online.
| benterix wrote:
| > All the vendors paraphrase user data, then use the
| paraphrased data for training. This is what their terms of
| service say.
|
| It depends. E.g. OpenAI says: "By default, we do not train on
| any inputs or outputs from our products for business users,
| including ChatGPT Team, ChatGPT Enterprise, and the API."[0]
|
| [0] https://openai.com/policies/how-your-data-is-used-to-
| improve...
| shakna wrote:
| "By default" is a fantastic escape catch in the language
| used there. So... What are the exceptions?
| moritzwarhier wrote:
| > Microsoft software since the 2014, for the most part, is
| also paraphrased from other people's code they find laying
| around online.
|
| That was pretty funny and explains a lot.
|
| I wish I could do more :(
|
| Instead I always break things when I paraphrase code without
| the GeniusParaphrasingTool
| nyrikki wrote:
| This is exactly why I moved to self hosted code in 2017.
|
| While I couldn't have predicted the future, even classic
| data mining posed a risk.
|
| It is just reality that if you give a third party access to
| your data, you should expect them to use it.
|
| It is just too tempting of a value stream and legislation
| just isn't there to avoid the EULA trap.
|
| I was targeting a market where fractions of a percentage
| advantage were important which did drive my what at the
| time was labeled paranoia
| simonw wrote:
| Why would they want to train on random garbage proprietary
| emails?
|
| If their models ever spit out obviously confidential
| information belonging to their paying customers they'll lose
| those paying customers to their competitors - and probably
| face significant legal costs as well.
|
| Your random confidential corporate email really isn't _that_
| valuable for training. I 'd argue it's more like toxic waste
| that should be avoided at all costs.
| carefulfungi wrote:
| > "The Microsoft 365 Copilot 'work tab' Chat is summarizing
| email messages even though these email messages have a
| sensitivity label applied and a DLP policy is configured."
|
| I'd add (3) - a DLP policy is apparently ineffective at its
| purpose: monitoring data sharing between machines.
| (https://learn.microsoft.com/en-us/purview/dlp-learn-about-
| dl...).
|
| Directly from the DLP feature page:
|
| > DLP, with collection policies, monitors and protects against
| oversharing to Unmanaged cloud apps by targeting data
| transmitted on your network and in Microsoft Edge for Business.
| Create policies that target Inline web traffic (preview) and
| Network activity (preview) to cover locations like:
|
| > OpenAI ChatGPT--for Edge for Business and Network options >
| Google Gemini--for Edge for Business and Network options >
| DeepSeek--for Edge for Business and Network options > Microsoft
| Copilot--for Edge for Business and Network options > Over
| 34,000 cloud apps in the Microsoft Defender for Cloud Apps
| cloud app catalog--Network option only
| caminante wrote:
| _> a DLP policy is apparently ineffective at its purpose_
|
| /Offtopic
|
| Yes, MSFT's DLP/software malfunctioned, but getting users to
| MANUALLY classify things as confidential is already an uphill
| battle. These are for the rare subset of people that are
| aware of and compliant with NDAs/Confidentiality Agreements!
| ImPostingOnHN wrote:
| Who can blame them, when in the end, it gets ignored
| anyways?
| samch wrote:
| Copilot is not a model, to my knowledge. When you're asking
| about the data that it was trained on, you are most likely
| referring to an OpenAI or, in some circumstances, an Anthropic
| model. Customer data is not used for training the models that
| run Copilot.
| 52-6F-62 wrote:
| None of this should surprise anyone by now. You are being lied
| to, continually.
|
| You guys need to read the actual manifestos these AI leaders have
| written. And if not them, then read the propagandist stories they
| have others write like The Overstory by Richard Powers which is
| an arrogant pile of trash that culminates in the moral:
|
| _humans are horrible and obsolete and all should die and leave
| the earth for our new AI child_
|
| Which is of course, horseshit. They just want _most_ people to
| die off, not all. And certainly not themselves.
|
| They don't care about your confidential information, or anything
| else about you.
| vinyl7 wrote:
| Everyone should go back and watch The Matrix again
| 52-6F-62 wrote:
| This is one of the things that boggles my mind the most.
|
| I guess everyone just ended up agreeing with Cypher, after
| all...
| josefritzishere wrote:
| AI is such garbage. There is considerable overlap between the
| security practices of AI and that of the slowest interns in the
| office.
| observationist wrote:
| Seems like every day there's another compelling reason to switch
| to Linux. Microsoft is doing truly incredible work this year!
| etchalon wrote:
| Apple not doing much better, but from the other end.
|
| Microsoft releasing overly ambitious features with disastrous
| consequences.
|
| Apple releasing features so unambitious it's hard to remember
| they're there.
| observationist wrote:
| Performance is also degrading on iphones as software bloats,
| and/or they're up to their old shenanigans and making older
| phones unbearable to force people to buy the newest ones.
|
| Big tech is reaping what they've sown in a very satisfying
| way.
| import wrote:
| We can safely assume that Apple will do much better compared
| to MS until they put AI to the Finder and Dock.
| varispeed wrote:
| Don't forget Apple handwaving serious security issues of
| their devices - users still cannot even check if their
| devices are compromised and only thing Apple can do here is
| "lockdown mode" - which again, after compromise is likely
| useless anyway.
| marcosdumay wrote:
| The problem with the Microsoft features is really not
| excessive ambition.
|
| Half of the time it's open user hostility and blatant
| incompetence. The other half it's just the incompetence.
| Ambition doesn't enter the picture at all.
| etchalon wrote:
| Eh. I think it is ambition. It's a lot product managers
| coming up with ideas, I think, and teams with a mandate to
| release those ideas.
| observationist wrote:
| Yes, and those ideas are user hostile and poorly
| conceived, badly executed, and incompetently built.
|
| A remote code execution exploit in _notepad_?! That 's
| not professional, or skillful, or well done. Unnecessary
| feature bloat and change for the sake of change, because
| some MBA dork wants to justify their department and
| continued employment by checking boxes on spreadsheets.
|
| There's no innovation or skillful, well built features.
| There's hardly any consideration of users at all, except
| as net continuing depositors of money into Microsoft
| coffers. Features and updates are nothing more than
| marketing slop and manipulation of enterprise into
| renewing subscriptions and purchasing the latest version
| of new hardware.
| esalman wrote:
| I recently switched my work laptop from a Dell to a MacBook. I
| found out that windows 11 has so much corporate bloat, than
| even MS apps like outlook, office and OneDrive functions better
| on a Mac than on Windows 11.
| kevincloudsec wrote:
| calling it a bug is generous. the whole point of these tools is
| to read everything you have access to. the 'bug' is that it
| worked exactly as designed but on the wrong emails
| pu_pe wrote:
| Microsoft somehow sees a future where LLMs have access to
| everything in your screen. In that dystopia, adding
| "confidential" tags or prompt instructions to ignore some types
| of content is never going to be enough. If you don't want LLMs to
| exfiltrate content then they cannot have access to it, period.
| autoexec wrote:
| Microsoft wants access to everything in your screen (as well as
| the contents of your personal files) and feeding that to an LLM
| just makes it easier for them to profit from that data
| steve1977 wrote:
| I'm shocked. Shocked!
| tartoran wrote:
| Oh, poor desperate Microsoft. No amount of bug fixing is going to
| fix Microsoft. Now that they've embarked on the LLM journey
| they're not going to know what's going to hit them next.
| asdefghyk wrote:
| Why was this bug not found in testing?
| nickdothutton wrote:
| "...including messages that carry confidentiality labels."
|
| Trusted operating system Mandatory Access Control where art thou?
| merb wrote:
| I more and more see a bug in my mouth that tries to encourage my
| boss to cancel Microsoft 365. I did not find the root cause yet
| ok123456 wrote:
| All these government contractors are forced to pay astronomical
| cloud bills to get "GCC-High" because it passes the right
| security-theater checklist, and then it totally ignores the DLP
| settings anyway!
| wartywhoa23 wrote:
| An exemplar BaaF corporation (Bug as a Feature).
| bronlund wrote:
| Microsoft deploying buggy software is hardly news.
| 8cvor6j844qw_d6 wrote:
| Is this a real bug or is it a "lets train on more emails" by
| being careless?
|
| I assume that whatever that is processed by AI service are
| generally retained for product improvements (training).
| allthetime wrote:
| This is one of many reasons we are taking all our current and
| future private repos off of GitHub.
___________________________________________________________________
(page generated 2026-02-18 23:01 UTC)