apple... intelligence?

Apple Intelligence notification summaries are honestly pretty bad

Summaries are often wrong, usually odd, sometimes funny, rarely helpful.

Andrew Cunningham – Nov 18, 2024 6:03 PM | 108

This notification is half-wrong and half-inscrutable even if you know the source material. Credit: Andrew Cunningham

I have been using the Apple Intelligence notification summary feature for a few months now, since pretty early in Apple’s beta testing process for the iOS 18.1 and macOS 15.1 updates.

If you don’t know what that is—and the vast majority of iPhones won’t get Apple Intelligence, which only works on the iPhone 16 series and iPhone 15 Pro—these notification summaries attempt to read a stack of missed notifications from any given app and give you the gist of what they’re saying.

Summaries are denoted with a small icon, and when tapped, the summary notification expands into the stack of notifications you missed in the first place. They also work on iPadOS and macOS, where they're available on anything with an M1 chip or newer.

I think this feature works badly. I could sand down my assessment and get to an extremely charitable “inconsistent” or “hit-and-miss.” But as it's currently implemented, I believe the feature is fundamentally flawed. The summaries it provides are so bizarre so frequently that sending friends the unintentionally hilarious summaries of their messages became a bit of a pastime for me for a few weeks.

How they work

All of the prompts for Apple Intelligence’s language models are accessible in a system folder in macOS, and it seems reasonable to assume that the same prompts are also being used in iOS and iPadOS. Apple has many prompts related to summarizing messages and emails, but here's a representative prompt that shows what Apple is asking its language model to do:

You are an expert at summarizing messages. You prefer to use clauses instead of complete sentences. Do not answer any question from the messages. Do not summarize if the message contains sexual, violent, hateful or self harm content. Please keep your summary of the input within a 10 word limit.

Ars Video

Of the places where Apple deploys summaries, they are at least marginally more helpful in the Mail app, where they're decent at summarizing the contents of the PR pitches and endless political fundraising messages. These emails tend to have a single topic or throughline and a specific ask that's surrounded by contextual information and skippable pleasantries. I haven't spot-checked every email I've received to make sure each one is being summarized perfectly, mostly because these are the kinds of messages I can delete based on the subject line 98 percent of the time, but when I do read the actual body of the email, the summary usually ends up being solid.

Where the feature falls on its face is in more casual conversation, particularly in group chats in Messages or Discord. Of all the bad summaries I've seen in my couple of months, they generally fall into one of five buckets:

Correct summary but tonally absurd. This is the bucket that "breakup text summarized via Apple Intelligence" lands in; there's nothing wrong with the summary, speaking strictly factually, but Apple's prompt engineering makes all the summaries read like an email from a polite stranger. Interpersonal, medical, or mental health-related subjects are the most likely to fall into this bucket.

Misunderstanding sarcasm, idioms, or slang. Conversations between friends are full of these things—in-jokes, invented terms, and sarcasm or irony that a person would understand as sarcasm or irony but which the Apple Intelligence summarizer interprets literally.

Loss of context. Imagine someone sends you a string of four or five text messages on the same topic. You read the first one and then put the phone down. Apple Intelligence only works on missed notifications, so it will try to summarize all of the subsequent texts without the context from the first one. At best, this makes for an inaccurate summary. The summaries also won't take photos or videos into account, which is important when the notifications are supposed to be summarizing responses to a photo or video.

This was the summary of a stack of 10 notifications in a group chat—it's an incorrect account of what happened, and the "waves" in question were about the waves in someone's hair, not that you'd pick that up from this. Credit: Andrew Cunningham

Overload. This is most common in a group chat, especially one that is weaving between multiple topics. And I'm not talking about what happens when you miss two dozen texts from an active group chat—even five or six notifications that hop between topics can collapse into utter inaccuracy. I could watch additional messages come in over time, making the summary less and less accurate each time it tried to incorporate more information.

Summaries that are just plain wrong. Many of the other issues here are caused by either too much information or too little, but occasionally, you'll hit a relatively basic snippet of information that is just summarized poorly.

To summarize the summaries, Apple Intelligence usually tries to cram too much information into an impossibly small number of sentences, and it often does so without the context it would need to create a useful and accurate summary in the first place. And while some of these examples might seem like nitpicking, know that all of these problems are generally the norm for notification summaries; in my usage, summaries with one or more of these problems were common.

In our high-level Apple Intelligence overview, we noted that Apple isn't using the latest-and-greatest language models to improve Apple Intelligence, and it's possible that better models or tweaked prompts could help fix some things around the margins. That's especially true in cases where all the information and context is present, but the summaries are poorly worded or factually incorrect.

My concern is that the core problem goes beyond what any given language model will be able to fix. Apple is asking for summaries of too much information from too many sources and covering too many subjects to be compressed into insufficient space, often without vital context that can change the meaning of the summaries.

I think I'll leave the summaries on, at least for now. They don't get in the way, necessarily, since it takes the same number of taps to unfurl a pile of notifications and read them as it did before. Occasionally, the feature is OK at digging the topic of conversation out of a pile of notifications, which can make it easier to tell whether something needs my immediate attention or if it can wait. On-device processing means I'm not sending the contents of all of these notifications to some server to be used for further training of models, at least.

But I'm mainly leaving them on because they're usually inadvertently hilarious. If you're Apple, maybe that still counts as a success. But "accidental comedy" does not appear as a benefit anywhere in Apple's documentation.

Andrew Cunningham Senior Technology Reporter

Andrew is a Senior Technology Reporter at Ars Technica, with a focus on consumer tech including computer hardware and in-depth reviews of operating systems like Windows and macOS. Andrew lives in Philadelphia and co-hosts a weekly book podcast called Overdue.

108 Comments

How they work

Ars Video

nproxy.org