[HN Gopher] First thoughts on o3 pro
___________________________________________________________________
First thoughts on o3 pro
Author : aratahikaru5
Score : 83 points
Date : 2025-06-10 20:42 UTC (2 days ago)
(HTM) web link (www.latent.space)
(TXT) w3m dump (www.latent.space)
| treetalker wrote:
| > We're in the era of task-specific models. On one hand, we have
| "normal" models like 3.5 Sonnet and 4o--the ones we talk to like
| friends, who help us with our writing ...
|
| ...
|
| > [M]odels today are so _good_ ...
|
| ...
|
| > o3 pro (left) clearly understanding the confines of it's
| environment way better.
|
| Miracle models that are so good at helping us with our writing,
| yet we still use _it 's_ as a possessive form.
| landl0rd wrote:
| I don't understand what you're suggesting. That they are
| sentient? That they have genders? That OpenAI models are male
| and Anthropic models are female? Or that you're simply not that
| impressed with them?
| DylanSp wrote:
| Not gender-related; the correct form for the possessive is
| "its" without an apostrophe, not "it's".
| jurgenaut23 wrote:
| lol, what an amazing meta exchange right there
| Groxx wrote:
| I'm on the fence with its/it's... but mostly I think I'd be
| fine with "its" disappearing. It's almost never ambiguous in
| writing or speaking given context, and it's a weird possessive-
| form inconsistency.
| david-gpu wrote:
| Which one is the most inconsistent? He ->
| His It -> Its She -> Her He ->
| He is -> He's It -> It is -> It's She ->
| She is -> She's
|
| The only one breaking the pattern is _Her_ , which instead
| should be _Shes_ if we wanted to keep things consistent.
| coffeecantcode wrote:
| Actually if we're keeping with the pattern it would be Shis
| spiderfarmer wrote:
| Or shits
| myk9001 wrote:
| Does this happen to be the one billionth HN comment?
| david-gpu wrote:
| Good catch! Thank you.
| tsimionescu wrote:
| All of the pronoun contractions are somewhat inconsistent
| with noun possessives. So "its", but "the dog's". There's
| no reason why you couldn't write "it's" for both "it is"
| and "its". You already write both "the dog's ball" and "the
| dog's tired", and both "it's" and "it's" are pronounced
| identically, which doesn't pose any difficulties in speech.
|
| His, her, and my are a different matter, since they are all
| pronounced differently from he's*, she's, and I'm.
|
| * at least in most accents, "his" uses the kit vowel, while
| "he's" uses the fleece vowel
| raincole wrote:
| Believe or not, "it's" as a possessive form came before
| "its."
|
| Source: https://www.merriam-webster.com/grammar/the-
| tangled-history-...
| janalsncm wrote:
| I have yet to encounter a case where words need an
| apostrophe to disambiguate. It's just an unnecessary
| exception. Writing is about communication and if everyone
| knows what you mean without the apostrophe then it adds no
| value.
|
| Same thing with book authors who have a good idea that can
| be expressed in 50 pages but for some reason publishers
| want 180+, so you end up with 130 pages of fluff, wasting
| everyone's time.
| layer8 wrote:
| zatsasliperislope
| hluska wrote:
| > It's just an unnecessary exception.
|
| Had you written this "it's just an unnecessary exception"
| you would be talking about a "just an unnecessary
| exception" that belongs to it. That's not clear in the
| slightest. Apostrophes exist for clarity and precision.
| hluska wrote:
| She->her (like everything in English) makes a lot more
| sense if you start in the roots of English. She was "hio"
| or "heo" and her was "hire". By the 13th century, the
| pronunciations of he and heo had converged and become
| synonyms so "she" was introduced.
|
| She shares a letter with seo, which was the root of both he
| and heo.
|
| Simple pattern matching really can't work in English - the
| roots are way too messy. And it's absolutely amazing that
| people can become reasonably fluent if they only start
| learning as adults.
| layer8 wrote:
| Maybe use of spell-checkers will finally catch on when they are
| AI-based. ;)
| simonw wrote:
| Something I like about this piece is how much in reinforces the
| idea that models like o3 Pro are _really hard_ to get good
| results out of.
|
| I don't have an intuition at all for when I would turn to o3 Pro
| yet. What kind of problems do I have where outsourcing to a huge
| model that crunches for several minutes are worthwhile?
|
| I'm enjoying regular o3 a lot right now, especially with the huge
| price drop from the other day. o3 Pro is a lot harder to get my
| head around.
| JSR_FDED wrote:
| Same here, I'd be very interested to learn how others decide
| what model to use for which tasks.
|
| I find these minutes-long iterations so painful that in
| practice I always go for the fast non-reasoning models.
| SOLAR_FIELDS wrote:
| Imagine a tricky distributed systems problem where you feed
| all of the context of your setup to the LLM and it uses the
| advanced reasoning to diagnose possible avenues. I did that
| recently with a frontier model to unwrap some very tricky
| istio related connection pooling issues causing syn/ack
| floods.
|
| For coding I usually use the fast frontier model like
| o4minihigh, but I bust out the fancy research models when I
| want things like general architecture and design feedbacks
| that require broader advanced reasoning
| CamperBob2 wrote:
| I don't often have LLMs write a lot of code for me, but when
| I do, I don't mind waiting a couple more minutes for a result
| that will waste less of my time in debugging when I try to
| use it.
|
| Also it's useful to have models review code that I wrote --
| in some cases years ago -- to uncover old bugs. Current
| models are generally far too eager to say "Yup! Looks good!
| You da _man_! " when there are actually serious flaws in the
| code they are reviewing. So again, this is a task that
| justifies use of the most powerful models currently
| available, and that doesn't have to run in real time.
| dist-epoch wrote:
| Random thought: dump your knowledge base into it (Obsidian,
| ...) and ask it to reorganize it, delete duplication, obsolete
| stuff, optimize it.
|
| Or tell it what you know about non-programming subject X, and
| ask it to explain it to you such that you understand it better.
|
| Or for coding: ask it to look at your code, and suggest large
| scale architecture changes.
|
| For these kinds of tasks, the models are still lacking.
| tunesmith wrote:
| Would o3 pro be the first one that can _reliably_ understand a
| gigantic congressional bill, to the point where it could
| analyze and warn of side effects?
| simonw wrote:
| Oh that's a _really_ interesting test case for it.
| nxobject wrote:
| Something that comes to mind - I code with for platform that
| doesn't have a lot of source code or documentation simply
| available online for training; I have to provide a lot of
| context. A lot more inference lets it combine its general
| knowledge about systems programming to this really niche
| domain, with a lot less hallucination and a lot more systematic
| reasoning.
| gleb wrote:
| o3 pro seems to be good with meta-prompting. Meaning, when you
| ask it to create a prompt for you. In particular it seems to be
| more concise than o3 when doing this.
|
| Has anybody else noticed this?
| bananapub wrote:
| > On the other, we have gigantic, slow, expensive, IQ-maxxing
| reasoning models that we go to for deep analysis (they're great
| at criticism), one-shotting complex problems, and pushing the
| edge of pure intelligence.
|
| I quite enjoy having an LLM write much of my tedious code these
| days, but comments like this are just bizarre to me. Can someone
| share a text question that I can ask an expensive slow LLM that
| will demonstrate "deep analysis" or "iq-maxxing" on any topic?
| Whenever I ask them factual or discussion questions I usually get
| something riddled with factual errors or just tedious, like
| reading an essay someone wrote for school.
| semi-extrinsic wrote:
| I don't have any good idea of what are "good" prompts for
| demonstrating such models. But what I would ask such a model is
| the following. I have no idea if it would fall on it's face or
| not.
|
| Can you write a version of Chorin's projection method for the
| Navier-Stokes equations that is both explicit and second order
| in time?
|
| Ideally the model should not need a more detailed prompt than
| this. A first-year grad student in numerical analysis certainly
| would not.
| varjag wrote:
| The one I asked o3-pro yesterday was "Research the annual
| smoking tobacco production in Soviet Union 1939-1958 and plot
| it in a graph versus male population size"
| riku_iki wrote:
| And how was result? Did you verify that it found reliable
| source of data?
| golol wrote:
| I use o3 for my PhD math research. When I am facing a specific
| problem and I am out of ideas I oass it to o3. It will usually
| say something with a fair number if errors and eventually claim
| to have solved my problem in a standard manner, which it almost
| never does. But that does not mean it is not useful to me. My
| attention is light a flashlight illuminating a tiny spot in the
| possibly vast field of methods I could try. Right now my head
| is full of dispersive PDEs so I will not think of using
| parabolic regularization. But o3 is more of a dim background
| light. I am in the end better at using any particular technique
| that is familiar to me than o3, but in this very moment I can
| only think of a few options. Sometimes my specific problem is
| actually naturally tackled by a method I have not considered,
| and o3 suggests it. If you consider that iq-maxxing or not, in
| this moment for me it is, because it helps me.
| aeonik wrote:
| I ran into a weird joystick bug the other week, and I wanted
| ChatGPT to figure out the exact code flow of how a specific
| paramter is set.
|
| I had it analyze different related libraries, and it zeroed in
| on SDL and Wine codebases, and it found the exact lines of code
| related to the logic error in Winebus.
|
| It really helps me dig deep for certain hard to track bugs.
| serjester wrote:
| I found o3 pro to need a paradigm shift, where the latency makes
| it impossible to use in anything but in async manner.
|
| You have a broad question, likely somewhat vague, and you pass it
| off to o3 with a ton of context. Then maybe 20 minutes later,
| you're going to have a decently good answer. Definitely stronger
| than any other models - it genuinely has taste.
|
| Yet, the scary thing here is that increasingly I'm starting to
| feel like the bottleneck. A human can only think about so many
| tasks in parallel and it seems like my contributions are getting
| less and less important with every model upgrade.
|
| Every now and then I question why I'm paying $200 for the max
| plan, but then something like this comes out and makes it a no
| brainer.
| dist-epoch wrote:
| Do the coherent extrapolated volition thing: "o3-pro, please
| reinterpret the prompt and do what I would have actually
| requested instead if I was 50 IQ points smarter."
| serjester wrote:
| I'd be curious to see benchmarks but this kind of query
| rewriting seems almost guaranteed to already be baked into
| the model.
| swyx wrote:
| is there documented/writeup on how this improves thinking?
| atleastoptimal wrote:
| This is gonna keep happening with every AI advance until humans
| are an absolute bottleneck in every domain. May take a bit of
| time for some professions, but the writing is on the wall. This
| will be the greatest shift in human history, and I think a lot
| of people will have trouble grappling with it because its not
| fun to think about being made irrelevant.
|
| The only thing that will slow AI down is massive universal
| international regulation. Human intelligence really isn't the
| be all end all to intelligence in general, it's just a stepping
| stone. I feel many on this site don't want to accept this
| because their intelligence has been such a valuable tool and
| source of personal pride/identity for them for so long.
| timhigins wrote:
| > unilateral international regulation
|
| is an oxymoron/contradictory
| atleastoptimal wrote:
| sorry I meant "universal" or "omnilateral"
| layer8 wrote:
| Did you mean global regulation?
| atleastoptimal wrote:
| yeah
| matthewkayin wrote:
| What is all of this for if the result is that human beings
| are "made irrelevant"? If these LLMs truly become as game
| changing as so many say they will be, then can we agree that
| it's time to stop thinking that a person's worth equals their
| economic output?
| __MatrixMan__ wrote:
| It is definitely past time to start thinking outside of the
| economy.
|
| Although must we deal in "worth" at all at that point? If
| two people have conflicting visions, it shouldn't be the
| one who is "worth" more that gets their way, it should be
| the one whose vision is most appealing to the rest of us.
| Onawa wrote:
| I agree with you, the problem currently is that the balance
| of power has shifted so far in favor of the 0.1%. And those
| people will not want to give up the power that they already
| have.
|
| I fear for a future where the technocrats win out and we
| end up in an "Altered Carbon" scenario. We are on the
| precipice of AI and robotics equalizing the playing field
| for everyone, but only if the power is held by the people
| and not the few at the top with the most resources.
|
| Not sure how to steer the ship in that direction, but I do
| have a few ideas...
| metalliqaz wrote:
| What good is intelligence if there is nobody with the money
| to pay for it? We run our brains on a few thousand calories a
| day. Who is going to pay to provide the billions of calories
| it takes to run/cool GPUs all day long if there are no humans
| with marketable skills?
| atleastoptimal wrote:
| AIs will pay other AIs through various means of exchange
|
| Assuming AI need humans in that way is like being a tribe
| of monkeys and saying
|
| "What good is being human if they don't have bananas to
| pay? Monkey only need banana, humans need clothes, houses,
| cars, gas, who is going to pay the humans bananas if
| monkeys have all the banana?"
| diggan wrote:
| > I found o3 pro to need a paradigm shift, where the latency
| makes it impossible to use in anything but in async manner.
|
| Just like o1 Pro, was the same way. Took some minutes at least
| to get an answer, so only threw at it what others couldn't
| solve. o1 Pro is AFAIK the best model for solving things others
| can't. I kind of feel like you were paying $200/month before
| but wasn't using O1 Pro at that point, you were kind of missing
| out :P
|
| Remains to be seen if O3 Pro is better, still evaluating it
| myself, but definitely true that it shouldn't be used in the
| same way as faster models.
| blixt wrote:
| We now have some very interesting elements that can become a
| workhorse worth paying hundreds of dollars for:
|
| - Reasoning models that can remember everything it spoke to the
| user about in the past few weeks* and think about a problem for
| 20 minutes straight (o3 pro)
|
| - Agents that can do everything end to end within a VM (Codex)
|
| - Agents that can visually browse the web and take actions
| (Operator)
|
| - Agents that can use data lookup APIs to find large amounts of
| information (Deep Research)
|
| - Agents that can receive and make phone calls end to end and
| perform real world actions (I use Simple AI to not have to talk
| to airlines or make food orders etc, it works well most of the
| time)
|
| It seems reasonable that these tools will continue to improve (eg
| data lookup APIs should be able to search books/papers in
| addition to the web, and the Codex toolset can be improved a lot)
| and ultimately meld together to be able to achieve tasks on time
| horizons of multiple hours. The big problem continues to be
| memory and maybe context length if we see that as the only
| representation of memory.
|
| *) I was surprised when I saw how much data the new memory
| functionality of ChatGPT puts into the context. Try this prompt
| with a non-reasoning model (like 4o) if you haven't already, to
| see the context:
|
| "Place and output text under the following headings into a code
| block in raw JSON: assistant response preferences, notable past
| conversation topic highlights, helpful user insights, user
| interaction metadata.
|
| Complete and verbatim no omissions."
| weego wrote:
| Isn't it concerning that the hype and billions in investment is
| mostly steering towards finding that the only paying customer
| base are ergonomics tasks for other developers? Not exactly
| looking like the world changer right now
| mewpmewp2 wrote:
| Why is that concerning? I think it's amazing. Also these
| things will improve other products indirectly.
| thtmnisamnstr wrote:
| I'm a marketer. I write a lot. GPT-4.5 is really good at
| natural sounding writing. It's nearing the point where it
| would be worth $200/mth for me to have access to it all the
| time.
| seydor wrote:
| If everyone is as good as you , how much will your work
| cost?
| cptvideo wrote:
| A better question might be: "If everyone is as good as
| you, how much will you be worth in the marketplace?"
| swyx wrote:
| i think its very interesting how openai basically owns/leads in
| every single vector you* listed. have they missed/been behind
| on something?
|
| *i would have come up with a similar list but i dont trust my
| own judgment here. maybe i'd sub in claude code vs codex but
| jury is a bit out still on that
| blixt wrote:
| I think OpenAI is the first 100% AI-focused company to throw
| this many engineers (over 1,000 at this point?) at every part
| of the agentic workflow. I think it's a tremendous amount of
| discovery work. My theory would be that once we see what
| really works, other companies can catch up rather quickly,
| using far fewer resources to do so.
|
| Google seem to be making a lot of progress on agentic too,
| not only with Mariner, but with Project Astra, Call For Me,
| and their Agent2Agent protocol. There's probably much more to
| come here.
|
| Oh and OpenAI is clearly willing to spend a lot of money to
| push this technology a bit further. If you look at the logs
| of Codex, it appears to be using a very strong (read:
| expensive) reasoning model to basically brute force the use
| of a VM. If you have a follow-up question in a Codex task,
| they just casually throw away the old VM and spin up a new
| one, running all setup again. If you compare this to e.g.,
| Cursor, I'd wager Codex costs 5-10x more to perform a
| similarly sized task, though it's hard to tell for sure.
| logicchains wrote:
| > and ultimately meld together to be able to achieve tasks on
| time horizons of multiple hours
|
| It's already possible to achieve tasks on a time horizon of
| multiple days if you put the LLM into a sufficiently structured
| workflow (where you have a separate program that smartly
| manages its context). E.g. a standards-compliant HTTP 2.0
| server where the code is 100% written by Gemini Pro (over 40k
| lines of code total, including unit tests, in around 120 hours
| of API time):
| https://open.substack.com/pub/outervationai/p/building-a-100...
| blixt wrote:
| This is very interesting, and nice learnings in there too,
| thank you for sharing! It seems the author monitored the LLM,
| stopped it from going off-track a few times, fixed some unit
| test code manually, etc. Plus this is strictly re-
| implementing a very well-specced library that already exists
| in the same programming language. So I think it's still a bit
| hard to say we can let an LLM work for multiple days, if we
| imply that this work should be domain-specific to a
| particular company. But it's very promising to see this was
| possible with very little interaction!
| wahnfrieden wrote:
| Xcode and ChatGPT.app are in severe need of better ways to run
| multiple queries in parallel, operating on the same project
| (Xcode or whatever other dev tools)
| janalsncm wrote:
| > Trying out o3 Pro made me realize that models today are so good
| in isolation, we're running out of simple tests.
|
| Are Towers of Hanoi not a simple test? Or chess? A recursive
| algorithm that runs on my phone can outclass enormous models that
| cost billions to train.
|
| A reasoning model should be able to reason about things. I am
| glad models are better and more useful than before but for an
| author to say they can't even evaluate o3 makes me question their
| credibility.
|
| https://machinelearning.apple.com/research/illusion-of-think...
|
| AGI means the system can reason through any problem logically,
| even if it's less efficient than other methods.
| antiphon wrote:
| It single-shots the towers of Hanoi
| https://chatgpt.com/share/6848fff7-0080-8013-a032-e18c999dc3...
| samtheprogram wrote:
| This isn't reasoning at all. It's applying a well known
| algorithm to a problem. It literally says "classic" in its
| response.
| janalsncm wrote:
| It is "reasoning" in the same way that a calculator or
| compiler is reasoning. But I checked the solution, it's
| actually wrong so it's a moot point.
| CamperBob2 wrote:
| What will really bake your noodle is when you realize
| that just because the model's answer is wrong doesn't
| mean it didn't use reasoning to reach it.
|
| Is _your_ reasoning always perfect? No? Ever get partial
| credit on a test question in school? Yes? Well, maybe don
| 't expect perfection from a model that didn't exist 5
| years ago, that was considered impossible 10 years ago,
| and that would have gotten you burned as a witch 15 years
| ago.
|
| Nobody claims that o3-pro is AGI, or even that it is
| going to lead up to AGI.
| janalsncm wrote:
| It's not correct.
|
| In move 95 the disks are
|
| Tower 1: 10, 9, 8, 5, 4, 3, 2, 1
|
| Tower 2: 7
|
| Tower 3: 6
|
| It attempts to move disk 6 from tower 2 to tower 3, but disk
| 6 is already at tower 3, and moving 7 on top of 6 would be
| illegal.
|
| In fact this demonstrates that o3 is unable to implement a
| simple recursive algorithm.
| foundry27 wrote:
| I find it amusingly ironic how one comment under yours is
| pointing out that there's a mistake in the model output, and
| the other comment under yours trusts that it's correct but
| says that it isn't "real reasoning" anyways because it knows
| the algorithm. There's probably something about moving
| goalposts to be said here
| janalsncm wrote:
| If both criterion A and B need to be satisfied for
| something to be true, it's not moving the goalposts for one
| person to point out A is not true, and another person to
| point out that B is not true.
| bcoates wrote:
| The towers of Hanoi one is kind of weird, the prompt asks for a
| complete move by move solution and the 15 or 20 disk version
| (where reasoning models fail) means the result is unreasonably
| long and very repetitive. Likely as not it's just running into
| some training or sampler quirk discouraging the model to just
| dump huge amounts of low-entropy text.
|
| I don't have a Claude in front of me -- if you just give it the
| algorithm to produce the answer and ask it to give you the huge
| output for n=20, will it even do that?
| janalsncm wrote:
| If I have to give it the algorithm as well as the problem,
| we're no longer even pretending to be in the AGI world. If it
| falls down interpreting an algorithm it is worse than even a
| python interpreter.
|
| Towers of Hanoi is a well-known toy problem. The algorithm is
| definitely in any LLM's training data. So it doesn't even
| need to come up with a new algorithm.
|
| There may be some technical reason it's failing but the more
| fundamental reason is that an autoregressive statistical
| token generator isn't suited to solving problems with
| symbolic solutions.
| hluska wrote:
| You are the only person suggesting that o3 is AGI or even an
| approach to AGI. They're different beasts entirely.
| MagicMoonlight wrote:
| >The plan o3 gave us was plausible, reasonable; but the plan o3
| Pro gave us was specific and rooted enough that it actually
| changed how we are thinking about our future. >This is hard to
| capture in an eval.
|
| ChatGPT wrote this article
| zone411 wrote:
| I benchmarked it on four of my benchmarks so far. Got first place
| in two of them:
|
| https://github.com/lechmazur/confabulations
|
| https://github.com/lechmazur/nyt-connections
|
| https://github.com/lechmazur/generalization
|
| https://github.com/lechmazur/writing
| nxobject wrote:
| Re context and overthinking:
|
| > One thing I noticed from early access: if you don't give it
| enough context, it does tend to overthink.
|
| I agree with this - that being said, I find that simply asking at
| the end of a prompt "Do you need any clarifications before you
| continue?" does a pretty good job at helping AI pin down details
| as well.
| joshvm wrote:
| The first thing I asked was along the lines of "Find me an
| implementation of X that does not depend on framework Y,
| because it's not maintained". o3 thought for a minute and then
| returns a repository that depends on Y. In its defense, 4.1
| also found the same repo and commented that it "has a _light_
| dependency ", which was cute, if unhelpful.
|
| Adding a "think about whether your answer actually depends on
| Y" fixed it, and the response was pretty well thought out. My
| only gripe is the current persona seems to be tech-bro-
| informal. It opened with "TL;DR" and included a lot of
| developer slang that reads like it's trying to be cool in a way
| that masks overconfidence.
___________________________________________________________________
(page generated 2025-06-12 23:00 UTC)