https://arstechnica.com/apple/2024/11/apple-intelligence-notification-summaries-are-honestly-pretty-bad/ Skip to content Ars Technica home Sections Forum Subscribe * AI * Biz & IT * Cars * Culture * Gaming * Health * Policy * Science * Security * Space * Tech * Feature * Reviews * Store * AI * Biz & IT * Cars * Culture * Gaming * Health * Policy * Science * Security * Space * Tech Forum Subscribe Story text Size [Standard] Width * [Standard] Links [Standard] * Subscribers only Learn more Pin to story Theme * Light * Dark * System Search dialog... Sign In Sign in dialog... Sign in apple... intelligence? Apple Intelligence notification summaries are honestly pretty bad Summaries are often wrong, usually odd, sometimes funny, rarely helpful. Andrew Cunningham - Nov 18, 2024 1:03 pm | 73 [notification-summaries-1] This notification is half-wrong and half-inscrutable even if you know the source material. Credit: Andrew Cunningham This notification is half-wrong and half-inscrutable even if you know the source material. Credit: Andrew Cunningham Text settings Story text Size [Standard] Width * [Standard] Links [Standard] * Subscribers only Learn more Minimize to nav I have been using the Apple Intelligence notification summary feature for a few months now, since pretty early in Apple's beta testing process for the iOS 18.1 and macOS 15.1 updates. If you don't know what that is--and the vast majority of iPhones won't get Apple Intelligence, which only works on the iPhone 16 series and iPhone 15 Pro--these notification summaries attempt to read a stack of missed notifications from any given app and give you the gist of what they're saying. Summaries are denoted with a small icon, and when tapped, the summary notification expands into the stack of notifications you missed in the first place. They also work on iPadOS and macOS, where they're available on anything with an M1 chip or newer. I think this feature works badly. I could sand down my assessment and get to an extremely charitable "inconsistent" or "hit-and-miss." But as it's currently implemented, I believe the feature is fundamentally flawed. The summaries it provides are so bizarre so frequently that sending friends the unintentionally hilarious summaries of their messages became a bit of a pastime for me for a few weeks. How they work All of the prompts for Apple Intelligence's language models are accessible in a system folder in macOS, and it seems reasonable to assume that the same prompts are also being used in iOS and iPadOS. Apple has many prompts related to summarizing messages and emails, but here's a representative prompt that shows what Apple is asking its language model to do: You are an expert at summarizing messages. You prefer to use clauses instead of complete sentences. Do not answer any question from the messages. Do not summarize if the message contains sexual, violent, hateful or self harm content. Please keep your summary of the input within a 10 word limit. Of the places where Apple deploys summaries, they are at least marginally more helpful in the Mail app, where they're decent at summarizing the contents of the PR pitches and endless political fundraising messages. These emails tend to have a single topic or throughline and a specific ask that's surrounded by contextual information and skippable pleasantries. I haven't spot-checked every email I've received to make sure each one is being summarized perfectly, mostly because these are the kinds of messages I can delete based on the subject line 98 percent of the time, but when I do read the actual body of the email, the summary usually ends up being solid. Where the feature falls on its face is in more casual conversation, particularly in group chats in Messages or Discord. Of all the bad summaries I've seen in my couple of months, they generally fall into one of five buckets: [IMG_3329-1024x227] This is an accurate summary, but it's also a super bizarre and robotic way to convey this information! Andrew Cunningham This is an accurate summary, but it's also a super bizarre and robotic way to convey this information! Andrew Cunningham [IMG_3307-1024x221] Parenting hazards. Andrew Cunningham Parenting hazards. Andrew Cunningham This is an accurate summary, but it's also a super bizarre and robotic way to convey this information! Andrew Cunningham Parenting hazards. Andrew Cunningham * Correct summary but tonally absurd. This is the bucket that " breakup text summarized via Apple Intelligence" lands in; there's nothing wrong with the summary, speaking strictly factually, but Apple's prompt engineering makes all the summaries read like an email from a polite stranger. Interpersonal, medical, or mental health-related subjects are the most likely to fall into this bucket. [IMG_3256-1024x393] Discussing a potential iPad purchase with a friend, who was excited about some of the color options. Andrew Cunningham Discussing a potential iPad purchase with a friend, who was excited about some of the color options. Andrew Cunningham [slang-1024x276] The summary... doesn't quite get there. Andrew Cunningham The summary... doesn't quite get there. Andrew Cunningham Discussing a potential iPad purchase with a friend, who was excited about some of the color options. Andrew Cunningham The summary... doesn't quite get there. Andrew Cunningham * Misunderstanding sarcasm, idioms, or slang. Conversations between friends are full of these things--in-jokes, invented terms, and sarcasm or irony that a person would understand as sarcasm or irony but which the Apple Intelligence summarizer interprets literally. [context-1024x416] Two messages from different people--the subject of the conversation, the notification summaries, was not included in the notifications. Andrew Cunningham Two messages from different people--the subject of the conversation, the notification summaries, was not included in the notifications. Andrew Cunningham [IMG_3210-1024x217] This makes the summary flail around a bit, making incorrect assumptions about the context it's missing. Andrew Cunningham This makes the summary flail around a bit, making incorrect assumptions about the context it's missing. Andrew Cunningham Two messages from different people--the subject of the conversation, the notification summaries, was not included in the notifications. Andrew Cunningham This makes the summary flail around a bit, making incorrect assumptions about the context it's missing. Andrew Cunningham * Loss of context. Imagine someone sends you a string of four or five text messages on the same topic. You read the first one and then put the phone down. Apple Intelligence only works on missed notifications, so it will try to summarize all of the subsequent texts without the context from the first one. At best, this makes for an inaccurate summary. The summaries also won't take photos or videos into account, which is important when the notifications are supposed to be summarizing responses to a photo or video. [IMG_3222-2] This was the summary of a stack of 10 notifications in a group chat--it's an incorrect account of what happened, and the "waves" in question were about the waves in someone's hair, not that you'd pick that up from this. Credit: Andrew Cunningham * Overload. This is most common in a group chat, especially one that is weaving between multiple topics. And I'm not talking about what happens when you miss two dozen texts from an active group chat--even five or six notifications that hop between topics can collapse into utter inaccuracy. I could watch additional messages come in over time, making the summary less and less accurate each time it tried to incorporate more information. [wrong-1024x239] Oh, no! A disaster at the park! Andrew Cunningham Oh, no! A disaster at the park! Andrew Cunningham [park-contraption-1024x977] Oh, wait, actually everything is fine. Thanks, notification summaries! Andrew Cunningham Oh, wait, actually everything is fine. Thanks, notification summaries! Andrew Cunningham Oh, no! A disaster at the park! Andrew Cunningham Oh, wait, actually everything is fine. Thanks, notification summaries! Andrew Cunningham * Summaries that are just plain wrong. Many of the other issues here are caused by either too much information or too little, but occasionally, you'll hit a relatively basic snippet of information that is just summarized poorly. To summarize the summaries, Apple Intelligence usually tries to cram too much information into an impossibly small number of sentences, and it often does so without the context it would need to create a useful and accurate summary in the first place. And while some of these examples might seem like nitpicking, know that all of these problems are generally the norm for notification summaries; in my usage, summaries with one or more of these problems were common. In our high-level Apple Intelligence overview, we noted that Apple isn't using the latest-and-greatest language models to improve Apple Intelligence, and it's possible that better models or tweaked prompts could help fix some things around the margins. That's especially true in cases where all the information and context is present, but the summaries are poorly worded or factually incorrect. My concern is that the core problem goes beyond what any given language model will be able to fix. Apple is asking for summaries of too much information from too many sources and covering too many subjects to be compressed into insufficient space, often without vital context that can change the meaning of the summaries. I think I'll leave the summaries on, at least for now. They don't get in the way, necessarily, since it takes the same number of taps to unfurl a pile of notifications and read them as it did before. Occasionally, the feature is OK at digging the topic of conversation out of a pile of notifications, which can make it easier to tell whether something needs my immediate attention or if it can wait. On-device processing means I'm not sending the contents of all of these notifications to some server to be used for further training of models, at least. But I'm mainly leaving them on because they're usually inadvertently hilarious. If you're Apple, maybe that still counts as a success. But "accidental comedy" does not appear as a benefit anywhere in Apple's documentation. Photo of Andrew Cunningham Andrew Cunningham Senior Technology Reporter Andrew Cunningham Senior Technology Reporter Andrew is a Senior Technology Reporter at Ars Technica, with a focus on consumer tech including computer hardware and in-depth reviews of operating systems like Windows and macOS. Andrew lives in Philadelphia and co-hosts a weekly book podcast called Overdue. 73 Comments Comments Forum view Loading Loading comments... Prev story Next story Most Read 1. Listing image for first story in Most Read: I, too, installed an open source garage door opener, and I'm loving it 1. I, too, installed an open source garage door opener, and I'm loving it 2. 2. April Fools' joke results in Japanese firm making a beige '80s throwback PC case 3. 3. School failed to report AI nudes of kids for months. Now parents are suing. 4. 4. SpaceX president predicts rapid increase in Starship launch rate 5. 5. Trump's FCC chair is Brendan Carr, who wants to regulate everyone except ISPs Customize Ars Technica has been separating the signal from the noise for over 25 years. With our unique combination of technical savvy and wide-ranging interest in the technological arts and sciences, Ars is the trusted source in a sea of information. After all, you don't need to know everything, only what's important. More from Ars * About Us * Staff Directory * Newsletters * Ars Videos * General FAQ * RSS Feeds Contact * Contact us * Advertise with us * Reprints Do Not Sell My Personal Information (c) 2024 Conde Nast. All rights reserved. Use of and/or registration on any portion of this site constitutes acceptance of our User Agreement and Privacy Policy and Cookie Statement and Ars Technica Addendum and Your California Privacy Rights. Ars Technica may earn compensation on sales from links on this site. Read our affiliate link policy. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Conde Nast. Ad Choices