New o1 model can solve complex tasks due to a new processing step before answering.
See full article...
See full article...
OpenAI’s new “reasoning” AI models are here: o1-preview and o1-mini
- Thread starter JournalBot
- Start date
Machines can't think... same as bears can't wrestle.
Who wants to go in the ring?
Who wants to go in the ring?
Upvote
6
(13
/
-7)
The VCs don't believe the hype. They're looking at revenues and exit prices and the AGI stuff is irrelevant to that. If you can automate even 10% of commercial writing you're basically going to mint money, which means a hockey-stick revenue line that gets you the IPO in 2 years.
Altman talks AGI because that makes him more important, but I don't think the money people give a shit.
Upvote
15
(17
/
-2)
A machine told me that bears like false equivalencies.
Upvote
9
(10
/
-1)
As long as it keeps the rubes busily happy that they can outsmart the LLM and complacent in avoiding learning the new tech that's fine.
Give it just a few years and they'll be left behind exactly like those who couldn't use Google.
Give it just a few years and they'll be left behind exactly like those who couldn't use Google.
Upvote
-9
(1
/
-10)
Nah, it just tried a different but similar number, just like when you write "are you sure?".
Upvote
0
(0
/
0)
Programming is the thing I've tried again and again with ChatGPT, and I don't think I've ever got any correct piece of code... I guess it depends on the type of code.
Upvote
1
(2
/
-1)
I’m pretty confident that if OpenAI just paid some guy five bucks to answer each time a “how many letters” query came up, they’d have saved a significant amount of money by now.
Upvote
9
(10
/
-1)
I don’t know, it can’t do introspection, multi context or multidisciplinary analysis so why even bother.
Upvote
-1
(0
/
-1)
I had to try it to see how it tick'd.
GPT-4o1 resisted being "corrected", too.
Also I threw it a "simple" math question that all the other GPT's failed at because of step-bungling.
This time, it didn't bungle the steps, and apparently figured out a different way to do it than I would have done.
GPT-4o1 resisted being "corrected", too.
Also I threw it a "simple" math question that all the other GPT's failed at because of step-bungling.
This time, it didn't bungle the steps, and apparently figured out a different way to do it than I would have done.
Last edited:
Upvote
15
(15
/
0)
So, I just took this strawberry spelling anomaly on spec, but trying it out on just the boring regular gpt-4o-2024-08-06 (via the API) gives me the following:
I then pushed it further:
(You can check via:
As always, asking an LLM to step through it's reasoning massively increases the accuracy and processing ability of the LLM. I have no idea how it actually did this, btw -- I was fully expecting it to fail, or at least start to lose count and get confused once it got beyond a few repetitions.
Incidentally, for anyone still thinking that an LLM is just a random word generator with no additional processing ability -- please explain to me how it managed to do this? Because I'm pretty sure that sentence doesn't appear in any training data, let alone someone responding by counting up the "s"es one by one.
I then pushed it further:
<snip> I won't bore you with the iteration through every single letter of that sentence; suffice to say that it lists every letter in turn and counts them up correctly: </snip>
(You can check via:
perl -e '$_ = "Ars technica\'s posters who comment on LLM articles aren\'t always the most informed, and the discussion sometimes degrades into polemics"; print tr/s/s/,"\n"'. 14 is the correct answer.)As always, asking an LLM to step through it's reasoning massively increases the accuracy and processing ability of the LLM. I have no idea how it actually did this, btw -- I was fully expecting it to fail, or at least start to lose count and get confused once it got beyond a few repetitions.
Incidentally, for anyone still thinking that an LLM is just a random word generator with no additional processing ability -- please explain to me how it managed to do this? Because I'm pretty sure that sentence doesn't appear in any training data, let alone someone responding by counting up the "s"es one by one.
Upvote
-5
(7
/
-12)
The strawberry code name was months before anyone ran that problem that became well known. It is mere coincidence.
Tokenization not being character aware has long been known. Tokenization means the model doesn't see characters, it sees vectors representing meanings associated with the token. Tokenization is essentially translation to a different language that contains only semantic meaning and completely lacks knowledge of typographic form. Semantic meanings sometimes encode relationships between character tokenizations but it is rare.
The reason for tokenization is that there are relationship graphs ('attention') between tokens so the fewer tokens, the fewer relationships have to be tracked. The most naive graph is one to one, so memory grows with the square of the number of tokens. Also for common enough words, the vector representing meaning is usually useful to aggregate at the word level except for rare words.
Upvote
4
(6
/
-2)
Upvote
5
(6
/
-1)
According to Noam Brown (the researcher responsible for Claude the No-Limit Hold'em bot, he is one of the main researchers on this version of GPT)
https://x.com/polynoamial/status/1834312400419652079
Altman was doing Strawberry hints in early august. (August 7)
https://x.com/sama/status/1821207141635780938?lang=en
I can find a mention of the issue June 19th, so conceivably the idea is from the bug/meme,
https://community.openai.com/t/incorrect-count-of-r-characters-in-the-word-strawberry/829618
Last edited:
Upvote
4
(5
/
-1)
lol, in the "believe it or not" part, I definitely choose "not".
For people who don't click on tweets:
Upvote
8
(9
/
-1)
The issue with counting r’s is not that people are using LLMs to count r’s, it’s that it’s really hard to predict ahead of time what LLMs will get wrong.
It’s hard to determine what to use an LLM for with any confidence when the people making and selling LLMs aren’t clear about what they’re good at and bad at when they’re pitching them. Partly because it doesn’t look like they know either - they were swearing blind that it didn’t store original data until people started pulling it out in queries. Partly because they’re trying to have it both ways - LLMs will revolutionise all knowledge work! But double-check everything they say because they just make shit up a lot of the time.
Last edited:
Upvote
15
(17
/
-2)
Upvote
0
(0
/
0)
Artem S. Tashkinov
Ars Scholae Palatinae
"Not AI", "not thinking", "not general", etc. etc. etc. Denialism is strong yet, it still absolutely does something, it solves never before seen problems, it's not just regurgitating something it already knows. It's intelligent. How much? We don't know. Let's be a bit more humble.
Here's a nice overview from the team:
https://x.com/OpenAI/status/1834320155989664067
OpenAI o1 thinks before it answers and can produce a long internal chain-of-thought before responding to the user.
o1 ranks in the 89th percentile on competitive programming questions, places among the top 500 students in the US in a qualifier for the USA Math Olympiad, and exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems.
In other words o1 is smarter than 99% of people out there. "Still not intelligent", yeah.
Here's a nice overview from the team:
https://x.com/OpenAI/status/1834320155989664067
OpenAI o1 thinks before it answers and can produce a long internal chain-of-thought before responding to the user.
o1 ranks in the 89th percentile on competitive programming questions, places among the top 500 students in the US in a qualifier for the USA Math Olympiad, and exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems.
In other words o1 is smarter than 99% of people out there. "Still not intelligent", yeah.
Upvote
-12
(6
/
-18)
There are about a billion studies of LLMs out there. Have at it if you want statistics.
I'm comfortable they're already good enough to replace a fair amount of first draft commercial and professional writing, and they're a lot cheaper, a lot faster and a lot more reliable. Maybe that means 50%, maybe 10%. Either way it's going to be a big deal. Almost certainly already is, particularly in edu.
Upvote
-7
(5
/
-12)
Generally agreed, though they’d still need to include the caveat that it might get details wrong in the summary, and for THE LOVE OF GOD don’t rely on it for details of legal cases. Which reinforces your point if anything.
Last edited:
Upvote
2
(2
/
0)
I don't know anything about the Tik Tok video, never saw it. The screenshot I posted is from June.
https://community.openai.com/t/incorrect-count-of-r-characters-in-the-word-strawberry/829618
Upvote
7
(8
/
-1)
Yeah just found the bug report a little bit ago and updated as you were posting. So certainly conceivable that it was from the bug report/meme.
Upvote
0
(0
/
0)
Sorry didn't mean to ships in the night.
Anyways, no big deal, but I don't know how they can claim it's random, who would believe that. Just lean into it.
Upvote
3
(4
/
-1)
Hang on - the people who write papers on Babylonian history are almost entirely college students. And they aren’t paid to do it - they pay to do it.
The product here is automated cheating on college assignments, which seems a pretty dicey business case for the valuation. Leaving moral qualms aside, there’s the risk universities wise up and move to alternate assessment methods just to stop people using your product.
My broader issue with the LLM business case is that it’s based on an assumption that fixing dodgy output is a better process than doing things right in the first place. There’s a lot of work in the business process sphere that says the opposite, and it’s well established in computer security that security should be built in rather than bolted on.
Last edited:
Upvote
7
(10
/
-3)
Heh, I tend to post and then a few seconds later decide I need more details or clarification, or find a better source etc and update, so I open to edit, then ADHD down a rabbit hole and by the time I've editted more time than I realized has passed.
Upvote
3
(3
/
0)
Upvote
2
(2
/
0)
So many people concerned with "how many r's in strawberry" while you are thinking 2-d chess, openai-o1 is thinking 5-d chess
View: https://youtu.be/NbzdCLkFFSk?si=CNzHg8LXtyucLVdf&t=743
View: https://youtu.be/NbzdCLkFFSk?si=CNzHg8LXtyucLVdf&t=743
Upvote
-13
(1
/
-14)
I explicitly demanded analysis of the severity of the errors in the original post. There are loads of tasks for which AI already makes fewer, less consequential errors. And AI is likely to improve with further development.
Upvote
5
(6
/
-1)
Doing things right in the first place? Things are done as cheaply as possible in the first place and always will be. Up to now that's meant hiring entry-level employees to do first draft work and then having that edited by one or more layers of supervisors. No one's going to pay to get the first draft perfect or even very good. If LLMs can do it cheaper and even close to as good, they'll take over.
This isn't anything at all like computer security. Or nuclear weapons, or rocket science. This is business writing. Doing it badly isn't that big a deal.
As far as the cheating thing goes, so what? That's already a substantial business and the example was illustrative. I'm not suggesting actually building a business on Babylon papers.
Upvote
-10
(3
/
-13)
There's the very best kind of evidence: people choosing to use it instead of doing the work themselves. Surely if, for example, it made their lives harder rather than easier, they would not, so very very consistently, continue making this choice.
Upvote
-7
(4
/
-11)
Are they using it because it's quick and convenient or because the quality of the end result is better? We don't have that data.
Easier lines up with convenience more often than quality.
Now, doing things for convenience is perfectly fine, but then one needs to be clear about which dimension is being valued.
Upvote
11
(13
/
-2)
This will not fail:
https://github.com/mdegans/misanthropic/blob/dev/examples/strawberry.rs
strawberry.rs said:
Is isn't very hard at all to fix this issue y'all making a big deal about. A handful of lines of code. Informing the agent about their limitations and a tool.
The fact a language models doesn't see in letters but in tokens doesn't say anything whatsoever about intelligence. It's a different alphabet. That is all.
Words are made up of tokens, not letters but the agent doesn't necessarily know this. Fix that, provide a remedy and they get it right 100% of the time. Not 90 or 95 but 100%. And that's not even with a very intelligent model. Want more flexibility? Provide a Python interpreter.
Upvote
-6
(3
/
-9)
It is the dumbest possible benchmark. It speaks far more to the ignorance of those using it than about the relative intelligence of any given model.
Upvote
-3
(5
/
-8)
Or perhaps they're hiding the fact that they simply just copied someone else's idea and called it their own. Personally I find this statement to be highly suspicious, as I already published a proof of concept for a similar idea way back in February 2024, and this wouldn't be the first time I've witnessed others attempting to take credit for it.
https://thenewstack.io/limited-compute-resources-low-parameter-rag-can-help/
Does anyone from the Ars staff care to comment on this? I sent you guys a tip ages ago to indicate that I was working on this and nobody ever followed up.
Edit:
I literally wrote an app to do all the reading for people, and they still respond by not reading what I've written.
It seems you can lead a horse to water, but you can't force it to drink. Not even when you offer to help by directly submerging their noses in the trough.
Last edited:
Upvote
-12
(0
/
-12)
A lot of people had the idea of having some sort of inner voice. Westworld is prior art and they cribbed it from this guy:
https://en.m.wikipedia.org/wiki/The_Origin_of_Consciousness_in_the_Breakdown_of_the_Bicameral_Mind
Bing uses "inner_voice", Claude, "thinking" XML tags. It's demonstrated in the GPT-4 model card. Many more examples. You can use whatever you choose so long as it's descriptive.
Point is you didn't invent the idea. Whoever did probably is lost to history. The whole idea has been around probably as long as humans. Some inner voice you think in before speaking out loud.
Upvote
8
(8
/
0)
A lot of the proposed work for LLMs rely on perverse incentives within workplaces. “Forced to write documents no-one will care about? Code that you’ll never need to debug or understand? Use LLMs to automate your useless busywork and code that should have been a template!” It’s a market, sure, but it’s not exactly how this stuff is being sold.
Upvote
14
(15
/
-1)