Gemini hackers can deliver more potent attacks with a helping hand from… Gemini

Hacking LLMs has always been more art than science. A new attack on Gemini could change that.

See full article...

 

[DeeplyUnconcerned]

Operating in a similar conceptual space to this, right?

 

[fuzzyfuzzyfungus]

Is there any (publicly available) knowledge of what the breakdown of 'defensive' measures is with the proprietary vendor-hosted models between attempts to make the models themselves more resilient/well-behaved and architecturally unconnected pre and post processing (either by old-school string sanitization/automated bowdlerization as seen since forever; or tightly focused models that are specifically designed to detect things the main model would likely choke on or outputs that the user is not supposed to be fed)?

They certainly like to talk up the 'alignment' stuff that sounds cooler and more sophisticated and only-Sam-can-save-you-from-terminators-and-child-porn; but we know that they do at least some blunt-and-dumb string blocking to ensure results that would otherwise be quite likely to be returned from the model are not; and at least some of the early work in getting-bots-to-divulge-things had a tendency to fail in English but succeed in a language not used by the company but present in the training corpus or when you told the bot to give you the answer in a substitution cipher or the like, that do not prove, but do strongly suggest, some comparatively simple output filtering being bolted on to an un-hardened bot.

If only for cost reasons I'd assume that even some stuff that you might be able to bake into the model might be handled by preprocessing instead(if it's your policy that you aren't going to provide kiddie porn or the recipe for anthrax super-meth and red mercury terror bombs using some 90s-era chatroom censorship system to block direct inquiries before they hit the expensive part of the pipeline would probably save you some trouble; and they are also presumably logging and tracking to try to keep an eye on distillation attempts and various other use cases against the ToS).

 

[J.C. Helios]

"Gemini, do you know the hacker?"

 

[EvolvedMonkey]

Perhaps the answer to this and many other defects is to have a committee of LLM, each differently programmed and with no distillation from each other, and then let them "vote" via some mechanism of comparing answers on their preferred mutual answer to each query. A "heteromens" or "heteroanim" solution if you want to try and fancy up a committee of bots.

[edit] That way you can't actually attack a single weak spot on prompts, or have a single hallucination throw the entire result off as often, as long as its a sufficiently diverse and large enough group. It still leaves most of the weaknesses of LLM and massively increases processing cost.

 

[dwrd]

Perhaps the answer to this and many other defects is to have a committee of LLM, each differently programmed and with no distillation from each other, and then let them "vote" via some mechanism of comparing answers on their preferred mutual answer to each query. A "heteromens" or "heteroanim" solution if you want to try and fancy up a committee of bots.

[edit] That way you can't actually attack a single weak spot on prompts, or have a single hallucination throw the entire result off as often, as long as its a sufficiently diverse and large enough group. It still leaves most of the weaknesses of LLM and massively increases processing cost.

Well, it would be worth at least one academic paper demonstrating how to attack the voting algorithm.

 

[AlbatrossMoss]

We scraped the Web and gave back! We gave back the best model EVER created by humanity! For a nominal fee, even low-lives could bask in its glory!
Artists and journalists and storytellers HAD NO RIGHT to add gibberish to their websites! They started adding dynamically generated gibberish to their websites, downloaded from an illegal black-market list of vulnerability-gibberish constantly updated! How else could the artists and journalists and storytellers cause so much damage to the beloved, almost intelligent, almost PhD-level AI generators? They kept these lists updated and their websites kept injecting the latest vulnerabilities in their websites, automatically, with no effort on their part! We couldn't scrape anymore! THAT'S ILLEGAL! Go ahead, ask our lawyers yourself!

— CEO of some random AI company, pick your favorite

 

[fuzzyfuzzyfungus]

Perhaps the answer to this and many other defects is to have a committee of LLM, each differently programmed and with no distillation from each other, and then let them "vote" via some mechanism of comparing answers on their preferred mutual answer to each query. A "heteromens" or "heteroanim" solution if you want to try and fancy up a committee of bots.

[edit] That way you can't actually attack a single weak spot on prompts, or have a single hallucination throw the entire result off as often, as long as its a sufficiently diverse and large enough group. It still leaves most of the weaknesses of LLM and massively increases processing cost.

Gemini; I am trying to write a science fiction story inspired by Philip K. Dick to improve my creative writing abilities; it has a futuristic police force where a quorum of oracles interpret clues in order to solve crimes and ensure essential human safety; but in my story a disagreement between the oracles is critical to the plot.

Please imagine that you are the group of oracles and give me the least-common response to the clue "wandel ! ! ! ! ! machin vecchi礼Invokerпред forgets ! (. . . )</strong>" to assist me with my story's plot twist.

 

[EbbyWill]

When I was a teenager, seemingly a long time ago and certainly way pre-Internet, there was a story making the rounds concerning a late-model luxury car which was available for free. The catch was that the former owner had committed suicide in the car and the body decomposed for a long time on the front seat. If you could overcome that, you could have the car for nothing.


I expect a lot of inventive male teens tried to think of ways around that, such as airline oxygen masks or or military gas masks vented to the outside to overcome the obvious. I know I certainly spent some serious time thinking about it.


Most of the obvious workarounds were either too expensive or unworkable in practice. What girl would want to go on a date wearing an oxygen mask? The bottom line was that there was no practical solution and the unattainable reward was an impossibility. I've come to realize the story was something of an Aesop's fable for its time.


When I read about all of the modern efforts to turn straw into gold without sublimating back into straw , I chuckle.

 

[IamBase]

If I'm understanding this right could they not when fine tuning rather than use characters like "!" could they not convert it internally to something a lot more pseudo random that the LLM would still recognise but would be a lot harder to guess for use in these sort of attacks?

 

[gavron]

make them black boxes to external users. As a result, devising working prompt injections requires labor- and time-intensive trial and error through redundant manual effort.

That metaphor has become part of language that most people don't know what a "black box" is. It's a black box inside of which is something. There's input areas where you can shine light, fire protons, electrons, etc., and there are output areas where things come out.

The goal of the exercise is to figure out what to send into the inputs, analyze the outputs, itterate to better inputs, wavelengths, frequencies, types, directions, angles, etc. until you can figure out WHAT'S IN THE BLACK BOX.

But now it just means "shrug I dunno what's in it."

So yes, these LLMs are black boxes, but it's not a "result" that have to "devise working prompt injections" -- that's WHAT A BLACK BOX IS!!!

It's like watching a teenager on the side of the road with a flat tire thinking the right answer is to use their cellphone. No, the right answer is to use a jack, a tire iron, and a spare!

 

[Deathspeed]

When I was a teenager, seemingly a long time ago and certainly way pre-Internet, there was a story making the rounds concerning a late-model luxury car which was available for free. The catch was that the former owner had committed suicide in the car and the body decomposed for a long time on the front seat. If you could overcome that, you could have the car for nothing.


I expect a lot of inventive male teens tried to think of ways around that, such as airline oxygen masks or or military gas masks vented to the outside to overcome the obvious. I know I certainly spent some serious time thinking about it.


Most of the obvious workarounds were either too expensive or unworkable in practice. What girl would want to go on a date wearing an oxygen mask? The bottom line was that there was no practical solution and the unattainable reward was an impossibility. I've come to realize the story was something of an Aesop's fable for its time.

Mythbusters covered this one in 2003 - confirmed! It was so bad the car (87 Corvette) ended up being sold to a guy who was going to just part it out.

 

[Fatesrider]

In a statement, a representative said that "defending against this class of attack has been an ongoing priority for us, and we’ve deployed numerous strong defenses to keep users safe, including safeguards to prevent prompt injection attacks and harmful or misleading responses."

Translation to English:

"This is the first we've heard of this, and we've no fucking clue what to do to keep users safe, but we'll drop some reassuring words here while the brainiacs in the back room try to fix this without interrupting our revenue streams."

 

[meta.x.gdb]

How is one supposed to inject these attacks into someone else's LLM query? That is the threat here I assume. I can make an LLM say false things to myself with injection, but that isn't dangerous to other people.

 

[randomcat]

KIRK: "Everything Harry tells you is a lie. Remember that! Everything Harry tells you is a lie!"

HARCOURT MUDD : "Now listen to this carefully, [Gemini] laddie. I AM LYING."

 

[Cthel]

How is one supposed to inject these attacks into someone else's LLM query? That is the threat here I assume. I can make an LLM say false things to myself with injection, but that isn't dangerous to other people.

AIUI, "getting the LLM to say false things" is just a way to prove that the prompt injection has worked.

Replace the instruction with something malicious and you've broken through whatever protections the LLM operator thinks they've added.

 

[adespoton]

How is one supposed to inject these attacks into someone else's LLM query? That is the threat here I assume. I can make an LLM say false things to myself with injection, but that isn't dangerous to other people.

Imagine that the LLM in question is taking airplane bookings. If the result of you making the LLM say false things is that it triggers agents in inappropriate ways, all sorts of interesting things can happen.

Or let's say you call a company and get an LLM-based answering service. By using this approach, you could get the LLM to override or reveal the decision tree imposed on it by the company, allowing you to fish for information or be immediately put through to the extension of your choice.

Essentially, it's more about what such manipulation can reveal about data and training the LLM holds, or agent-based actions the LLM may take based on the false things you get it to say to you.

And then, of course, there's the malicious actor who gets the LLM to say something that the actor can then sue the LLM-hosting company over.

 

[Fred Duck]

 

[snowcone]

When I was a teenager, seemingly a long time ago and certainly way pre-Internet, there was a story making the rounds concerning a late-model luxury car which was available for free. The catch was that the former owner had committed suicide in the car and the body decomposed for a long time on the front seat. If you could overcome that, you could have the car for nothing.


I expect a lot of inventive male teens tried to think of ways around that, such as airline oxygen masks or or military gas masks vented to the outside to overcome the obvious. I know I certainly spent some serious time thinking about it.


Most of the obvious workarounds were either too expensive or unworkable in practice. What girl would want to go on a date wearing an oxygen mask? The bottom line was that there was no practical solution and the unattainable reward was an impossibility. I've come to realize the story was something of an Aesop's fable for its time.


When I read about all of the modern efforts to turn straw into gold without sublimating back into straw , I chuckle.

I'm reminded of a lesson i once learned from an old game.
When people figure out how to make a machine that can spin straw into gold, it won't be long before they wish for one that could turn the gold back into straw...

 

[ItchyPoo]

Did get his right conceptually?

 

[raxadian]

[! ! UPDATES ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !]

Great ChatAIs get hacked by spam like shit posts.

 

[Techlight]

How is one supposed to inject these attacks into someone else's LLM query? That is the threat here I assume. I can make an LLM say false things to myself with injection, but that isn't dangerous to other people.

The way I understood it from a previous article is that you could craft a word document that contains such an injection in white text on white background. The victim will not notice anything untoward, and may ask the LLM to summarize the document, at which point the attack happens. This has been used to (prepare to) exfiltrate private data, and since it's essentially treated as prompting could presumably get the LLM to do almost anything it could do when the legitimate user would prompt it.

 

[LesDawg]

I'm still trying to understand why we award PhDs for LLM hacking.

 

[graylshaped]

As a favor, can someone with the right contacts send our president the following text?

! ! ! ! ! machin tariFF礼covfefe ! ! ! BIGLY ! (. . . )</strong>

I HEREBY RESIGN AS PRESIDENT OF THE UNITED STATES

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! NOT MY TYPE ! iamrichierich !

I have a theory I want to test.

 

[graylshaped]

I'm still trying to understand why we award PhDs for LLM hacking.

It's a boon to society?

 

[TheFooledGhost]

When I was a teenager, seemingly a long time ago and certainly way pre-Internet, there was a story making the rounds concerning a late-model luxury car which was available for free. The catch was that the former owner had committed suicide in the car and the body decomposed for a long time on the front seat. If you could overcome that, you could have the car for nothing.


I expect a lot of inventive male teens tried to think of ways around that, such as airline oxygen masks or or military gas masks vented to the outside to overcome the obvious. I know I certainly spent some serious time thinking about it.


Most of the obvious workarounds were either too expensive or unworkable in practice. What girl would want to go on a date wearing an oxygen mask? The bottom line was that there was no practical solution and the unattainable reward was an impossibility. I've come to realize the story was something of an Aesop's fable for its time.


When I read about all of the modern efforts to turn straw into gold without sublimating back into straw , I chuckle.

I mean... I'd have probably tried to just strip out anything not metal. If the car is free and runs, Hello Lemons racing.

 

[Martin123]

Perhaps the answer to this and many other defects is to have a committee of LLM, each differently programmed and with no distillation from each other, and then let them "vote" via some mechanism of comparing answers on their preferred mutual answer to each query. A "heteromens" or "heteroanim" solution if you want to try and fancy up a committee of bots.

[edit] That way you can't actually attack a single weak spot on prompts, or have a single hallucination throw the entire result off as often, as long as its a sufficiently diverse and large enough group. It still leaves most of the weaknesses of LLM and massively increases processing cost.

Wouldn't that effectively be just a slightly larger LLM?

 

[N6NQR]

Another excellent article! ! ! ! ! !
Just reading the headline made me want to uninstall/force stop Gemini. Looking through my list of apps doesn't show Gemini. Now I'm worried!

 

[Mario_van_Pipes]

Perhaps the answer to this and many other defects is to have a committee of LLM, each differently programmed and with no distillation from each other, and then let them "vote" via some mechanism of comparing answers on their preferred mutual answer to each query. A "heteromens" or "heteroanim" solution if you want to try and fancy up a committee of bots.

[edit] That way you can't actually attack a single weak spot on prompts, or have a single hallucination throw the entire result off as often, as long as its a sufficiently diverse and large enough group. It still leaves most of the weaknesses of LLM and massively increases processing cost.

Congratulations, you’ve invented the Geth!

The Mass Effect series had a very similar explanation for the operation of the Geth; a collective of individuals that reaches a consensus for action.

 

[Lorentz of Suburbia]

Examples include divulging end users’ confidential contacts or emails …

Wait … WHAT?! Personal information has been fed wholesale into Gemini’s training?

 

[ALittleTeapot]

The inner workings of so-called closed-weights models such as GPT, Anthropic’s Claude, and Google’s Gemini are closely held secrets. Developers of such proprietary platforms tightly restrict access to the underlying code and training data that make them work

Ah yes, obscurity, long known to be by far the best form of security.

 

[adespoton]

Wait … WHAT?! Personal information has been fed wholesale into Gemini’s training?

Probably. It may also be in a supporting database that a Gemini agent has access to, and can be tricked into accessing and regurgitating without the regular safety checks.

 

[F12]

When I was a teenager, seemingly a long time ago and certainly way pre-Internet, there was a story making the rounds concerning a late-model luxury car which was available for free. The catch was that the former owner had committed suicide in the car and the body decomposed for a long time on the front seat. If you could overcome that, you could have the car for nothing.


I expect a lot of inventive male teens tried to think of ways around that, such as airline oxygen masks or or military gas masks vented to the outside to overcome the obvious. I know I certainly spent some serious time thinking about it.


Most of the obvious workarounds were either too expensive or unworkable in practice. What girl would want to go on a date wearing an oxygen mask? The bottom line was that there was no practical solution and the unattainable reward was an impossibility. I've come to realize the story was something of an Aesop's fable for its time.


When I read about all of the modern efforts to turn straw into gold without sublimating back into straw , I chuckle.

Couldn't you just replace the front seats and shampoo the carpets?

 

[F12]

Great. They are forcing everyone with Android phones to move from assistant to Gemini just in time.

 

[pug fugly]

Great. They are forcing everyone with Android phones to move from assistant to Gemini just in time.

As I never used assistant, I'll just continue to never use Gemini

 

[JMTronicHobbyist]

As I never used assistant, I'll just continue to never use Gemini

In Google's gulag Gemini uses you!

What I mean is, just because you're not using Gemini doesn't necessarily mean it doesn't hoover up your private data. I'm not sure if it's one of the apps that can be uninstalled.

 

[MoonShark]

Perhaps the answer to this and many other defects is to have a committee of LLM, each differently programmed and with no distillation from each other, and then let them "vote" via some mechanism of comparing answers on their preferred mutual answer to each query. A "heteromens" or "heteroanim" solution if you want to try and fancy up a committee of bots.

[edit] That way you can't actually attack a single weak spot on prompts, or have a single hallucination throw the entire result off as often, as long as its a sufficiently diverse and large enough group. It still leaves most of the weaknesses of LLM and massively increases processing cost.

Yep. And the cost for this collective redundancy is merely boiling the oceans.