[HN Gopher] An analysis of DeepSeek's R1-Zero and R1
___________________________________________________________________
An analysis of DeepSeek's R1-Zero and R1
Author : meetpateltech
Score : 315 points
Date : 2025-01-29 17:44 UTC (5 hours ago)
(HTM) web link (arcprize.org)
(TXT) w3m dump (arcprize.org)
| dagelf wrote:
| Well o3 scored 75% on AGI-1, R1 and o1 only 25%.... watch this
| space though....
| mohsen1 wrote:
| with 57 million(!!) tokens
| sheepdestroyer wrote:
| From the article :
|
| o3 (low) 75.7% 335K $20
|
| o3 (high) 87.5% 57M $3.4K
| jl6 wrote:
| $3.4K is about what you might pay a magic circle lawyer for
| an opinion on a matter. Not saying o3 is an efficient use
| of resources, just saying that it's not outlandish that a
| sufficiently good AI could be worth that kind of money.
| ant6n wrote:
| What's the liability insurance of the AI like
| baq wrote:
| Refer to IBM's 1979 slide for details on that
| victorbjorklund wrote:
| You pay that price to a law firm to get good service and
| to get a "guarantee" of correctness. You get neither from
| an LLM. Not saying it is not worth anything but you cant
| compare it to a top law firm.
| mrandish wrote:
| When I saw these numbers back in the initial o3-ARC post, I
| immediately converted them into "$ per ARC-AGI-1 %" and
| concluded we may be at a point where each increased
| increment of 'real human-like novel reasoning' gets
| exponentially more compute costly.
|
| If Mike Knoop is correct, maybe R1 is pointing the way
| toward more efficient approaches. That would certainly be a
| good thing. This whole DeepSeek release and the reactions
| have shown by limiting the export to China of high-end
| GPUs, the US incentivized China to figure out how to make
| low-end GPUs work _really_ well. The more subtle meta-
| lesson here is that the massive flood of investment capital
| being shoved toward leading edge AI companies has fostered
| a drag race mentality which prioritized winning top-line
| performance far above efficiency, costs, etc.
| Davidzheng wrote:
| I view it as a positive that the methodology can take in more
| compute (bitter lesson style)
| optimalsolver wrote:
| But can o3 write a symphony?
|
| Seriously though, I'd like to hear suggestions on how to
| automatically evaluate an AI model's creativity, no humans in
| the loop.
| fragmede wrote:
| we'd have to create a numerical scale for creativity, from
| boring to Dali, with milliEschers and MegaGeigers somewhere
| in there as well
| rpastuszak wrote:
| It's essential that we quantify everything so that we can
| put a price on it. I'd go with Kahlograms though.
| johnfn wrote:
| Have you tried suno.ai?
| baq wrote:
| LLMs have read everything humans made so just ask one if
| there's anything truly new in that freshly confabulated slop-
| phony.
| gsam wrote:
| In my view there's two modes of creativity:
|
| 1. That two distant topics or ideas are actually much more
| closely related. The creative sees one example of an idea and
| applies it to a discipline that nobody expects. In theory,
| reduction of the maximally distant can probably be measured
| with a tangible metric.
|
| 2. Discovery of ideas that are even more maximally distant.
| Pushing the edge, and this can be done by pure search and
| randomness actually. But it's no good if it's garbage. The
| trick is, what is garbage? That is very context dependent.
|
| (Also, a creative might be measured on the efficiency of
| these metrics rather than absolute output)
| levocardia wrote:
| What's interesting is that you can already see the "AI race"
| dynamics in play -- OpenAI must be under immense market
| pressure to push o3 out to the public to reclaim "king of the
| hill" status.
| spoaceman7777 wrote:
| I suppose they're under some pressure to release o3-mini,
| since r1 is roughly a peer for that, but r1 itself is still
| quite rough. The o1 series had seen significantly more QA
| time to smooth out the rough edges, and idiosyncracies what a
| "production" model should be optimized for, vs. just a top
| scorer on benchmarks.
|
| We'll likely only see o3 once there is a true polished peer
| for it. It's a race, and companies are keeping their best
| models close to their chest, as they're used internally to
| train smaller models.
|
| e.g., Claude 3.5 Opus has been around for quite a while, but
| it's unreleased. Instead, it was just used to refine Claude
| Sonnet 3.5 into Claude Sonnet 3.6 (3.6 is for lack of a
| better name, since it's still called 3.5).
|
| We also might see a new GPT-4o refresh trained up using
| GPT-o3 via deepseek's distillation technique and other
| tricks.
|
| There are a lot of new directions to go in now for OpenAI,
| but unfortunately, we won't likely see them until their API
| dominance comes under threat.
| anothermathbozo wrote:
| The claim is that this removes the human bottleneck (aka SFT or
| supervised fine tuning) on domains with a verifiable reward.
| Critically, this verifiable reward is extremely hard to pin down
| in nearly all domains besides mathematics and computer science.
| Onavo wrote:
| By verifiable do they mean it in the complexity theory P/NP
| sense of the word?
| ks2048 wrote:
| I think it just means that you can objectively score an
| answer as being correct or not. (e.g. if the generated
| program passes some tests; a discovered proof is valid, etc).
| pertymcpert wrote:
| They mean that the solutions can be verified to be correct in
| a binary sense. E.g. a coding solution passes all the unit
| tests vs writing poetry.
| logicchains wrote:
| As in there's an objective truth that can be determined by a
| computer. E.g. whether code compiles, whether a unit test
| passes, whether the answer given to a mathematical question
| like 3+5 is correct. Many other fields have no objective
| truth (like art or creative writing), or objective truth
| requires measurement of the physical world (although if the
| world can be simulated accurately enough for the problem
| class at hand, then sufficient training data can still be
| generated by a computer).
| bpfrh wrote:
| Isn't "code compiles" an insufficient criteria?
|
| e.g you would need to prove that for all inputs the code
| produces the correct output which would in turn make the
| problem way more complex
| pressbuttons wrote:
| Not if the problem as written is "does this code
| compile", which is still a useful stepping stone for some
| workflows. Yours is certainly a more useful query in most
| cases but repositioning or re-scoping the original
| question can still lead to a net win.
| logicchains wrote:
| It's not a sufficient criteria by itself, but where no
| better criteria is possible it would still produce better
| results in reinforcement learning than if the model has
| no reward for producing correctly compiling code vs code
| that failed to compile.
| drdeca wrote:
| The other replies have said what was meant, but I don't think
| they've explicitly addressed whether or not that is the sense
| used in the idea of NP.
|
| I would say... it is at least somewhat similar.
|
| A problem in NP might be of the form "For this value of X,
| does there exist a Y such that q(X,Y)?" for some predicate q
| and value X, and where when the answer is "yes", the answer
| of "yes" can be verified by being given a value Y, and
| evaluating q(X,Y). (Specifically in the case of 3SAT, X would
| be a 3CNF formula, Y would be an assignment of values to the
| variables in the formula, and q(X,Y) would be "the formula X
| when evaluated with variable assignments Y, results in
| 'true'.".)
|
| This is sort of like the task of "Given requirements X that
| can be checked automatically, produce code Y which satisfies
| those requirements", except that in this case the question is
| specifically asking for Y, not just asking whether such a Y
| exists, but.. well, often in practice when one wants a
| solution to a problem in NP, one actually wants the witness,
| not just whether there exists such a Y, right?
|
| So, I would say there is a substantial similarity, but also a
| difference.
| sgt101 wrote:
| There's a big difference. The membership of these classes is
| determined in the worst case - so if there is no polynomial
| time solution in the worst case then it's NP.
|
| For this problem we don't care if it's possible that
| sometimes there are things that aren't verifiable, or the
| answers aren't exact, we just need training signal.
| calebkaiser wrote:
| In the case of DeepSeek-R1, they used a series of heuristic
| reward functions that were built for different data types.
| The paper mentions the use of sandboxed environments to
| execute generated code against a suite of tests, for example,
| to evaluate it for correctness. The reward functions also
| evaluated syntax and formatting.
|
| In general, the use of externally verifiable sources of truth
| (like simulators) is referred to as "grounding" and there has
| been quite a bit of research around it over the years, if
| you're interested in digging deeper. I've always found it
| super compelling as a research direction.
| HarHarVeryFunny wrote:
| For some reasoning data (e.g. you talking out loud as you
| figure something out, mistakes and all) to be useful for RL
| training, the conclusion to your reasoning needs to be
| correct/verified, else that's not the kind of reasoning you
| want to learn!
|
| Some types of reasoning output, such as solving a math
| problem or writing a computer program can be automatically
| verified (e.g. respectively by a symbolic solver, or by
| compiling and running the program), but in the general case
| it's hard for a computer to verify whether a chain of
| reasoning is correct and arrived at a valid answer or not,
| although LLM-as-judge should work some of the time.
| gadtfly wrote:
| Reasoning transfers across domains.
| Philpax wrote:
| See https://www.interconnects.ai/p/why-reasoning-models-will-
| gen... for more information.
| nextos wrote:
| IMHO, there are strategies that could extend this approach to
| many other domains.
|
| I was discussing this idea (along with a small prototype) with
| a prominent symbolic AI researcher who also agrees, and thinks
| that with the emergence of RL as a viable training method for
| LLMs, it might be possible to pursue neuro-symbolic learning at
| a large scale.
|
| Current systems are impressive, but reasoning is too fragile to
| trust them. They fall into obvious logical and statistical
| fallacies that are evident to a layperson.
| aithrowawaycomm wrote:
| It's also extremely hard to nail down in much of mathematics or
| computer science!
|
| - is such-and-such theorem deep or shallow?
|
| - is this definition/axiom useful? (there's a big difference
| between doing compass-straightedge proofs vs. wondering about
| the parallel postulate)
|
| - more generally, _discovering_ theorems is generally not
| amenable to verifiable rewards, except in domains where simpler
| deterministic tools exist (in which case LLMs can likely help
| reduce the amount of brute forcing)
|
| - is this a good mathematical / software model of a given real-
| world system?
|
| - is the flexibility of dynamic/gradual typing worth the risk
| of type errors? is static typing more or less confusing for
| developers?
|
| - what features should be part of a programming language's
| syntax? should we opt for lean-and-extensible or batteries-
| included?
|
| - are we prematurely optimizing this function?
|
| - will this program's memory needs play nicely with Rust's
| memory model? What architectural decisions do we need to make
| now to avoid headaches 6 months down the line?
| Davidzheng wrote:
| Not clear to me that theorem discovery is not amenable to
| verifiable rewards. I think most important theorems probably
| are recovered automatically by asking AI systems to proof
| increasing complicated human conjectures. Along the way I
| expect emergent behaviors of creating conjectures and
| recognizing important self-breakthroughs. Much like regret
| emergence
| youoy wrote:
| Theorems discovery is amenable to verifiable rewards. But
| is meaningful theorems discovery too? Is the ability to
| discern between meaningful theorems and bad ones an
| emergent behaviour? You can check for yourself examples of
| automatic proofs, and the huge amount of intermediate
| theorems that they can generate which are not very
| meaningful.
| artninja1988 wrote:
| >Ultimately, R1-Zero demonstrates the prototype of a potential
| scaling regime with zero human bottlenecks - even in the training
| data acquisition itself.
|
| I would like this to be true, but doesn't the way they're doing
| RL also require tons of human data?
| Davidzheng wrote:
| I think yes. But hopefully in math with compute advances we can
| lower the human data input by increasing the gap that is
| bridged by raw model capabilities vs search augmentation
| (either with tree search or full rollouts)
| polishdude20 wrote:
| I predict that the future of LLM's when it comes to coding and
| software creation is in "custom individually tailored apps".
| Imagine telling an AI agent what app you want, the requirements
| and all that and it just builds everything needed from backend to
| frontend, asks for your input on how things should work,
| clarifying questions etc.
|
| It tests the software by compiling and running it reading errors
| and failed tests and fixing the code.
|
| Then, it deploys the software in production for you. It compiles
| your app to an APK file and publishes it on the Google play store
| for example.
|
| Sure an LLM now may still not be able to get everything perfect
| as far as it's outputs go. But surely there's already systems and
| workflows in place that will auto run your code, compile it, feed
| errors back to the LLM, some api to interact with cloud providers
| for hosting etc?
| dboreham wrote:
| It doesn't need to write tests: it can just use the application
| and figure out if it works.
| logicchains wrote:
| That's going to be much slower and more expensive than
| writing tests because image/video processing is slower and
| more expensive than writing tests. And because of lag in
| using the UI (and re-building the whole application from
| scratch after every change to test again).
| ClumsyPilot wrote:
| But it's actually correct from a usability perspective
| drdeca wrote:
| Hm, what if instead of using video of the application...
|
| Ok, so if one can have one program snoop on all the
| rendering calls made by another program, maybe there could
| be a way of training a common representation of "an image
| of an application" and "the rendering calls that are made
| when producing a frame of the display for the application"?
| Hopefully in a way that would be significantly smaller than
| the full image data.
|
| If so, maybe rather than feeding in the video of the
| application, said representation could be applied to the
| rendering calls the application makes each frame, and this
| representation would be given as input as the model
| interacts with the application, rather than giving it the
| actual graphics?
|
| But maybe this idea wouldn't work at all, idk.
|
| Like, I guess the rendering calls often involve image data
| in their arguments, and, you wouldn't want to include the
| same images many time as the input to the encoding thing,
| as that would probably (or, I imagine) make it slower than
| just using the overall image of the application. I guess
| the calls are probably more pointing to the images in
| memory though, not putting an entire image on the stack.
|
| I don't know enough about low-level graphics programming to
| know if this idea of mine makes any sense.
| jumploops wrote:
| The future is bespoke software.
|
| In some sense, this is how computers were always supposed to
| work!
| prmph wrote:
| This will almost certainly never materialize, and the reasons
| are not just technical
| ClumsyPilot wrote:
| > Imagine telling an AI agent ... requirements... asks for your
| input on how things should work, clarifying questions etc.
|
| That's hard work. I watch people do that every day, and always
| get something wrong.
|
| Also what about deploying the application, paying for database
| or cloud resource that will run it, etc?
| aprilthird2021 wrote:
| > auto run your code, compile it, feed errors back to the LLM,
|
| Can't wait for companies to juice profits by having the LLM run
| excessive cycles or get stuck in a loop and run up my bill
| genewitch wrote:
| aider jams the backend on my PC, i have to kill the tcp
| connection or python to stop it running a GPU on the backend,
| from time to time. I can't imagine paying for tokens and not
| knowing if it's working or wasting money.
| girvo wrote:
| The loops and constant useless changes drive me nuts haha
| jacobsenscott wrote:
| What's it called when you describe an app with sufficient
| detail that a computer can carry out the processes you want?
| Where will the record of those clarifying questions and updates
| be kept? What if one developer asks the AI to surreptitiously
| round off pennies and put those pennies into their bank
| account? Where will that change be recorded, will humans be
| able to recognize it? What if two developers give it
| conflicting instructions? Who's reviewing this stream of
| instructions to the LLM?
|
| "AI" driven programming has a long way to go before it is just
| a better code completion.
| repelsteeltje wrote:
| That.
|
| Plus coding (producing a working program that fits some
| requirement) is the least interesting part of software
| development. It adds complexity, bugs and maintenance.
| fragmede wrote:
| I mean, we're halfway there, with aider and open-interpreter,
| just give it a couple of years
| kristjansson wrote:
| Most people really do not know what they want at any level of
| detail.
| travoc wrote:
| It's ok, they'll know it when they see it. Keep trying.
| jrsdav wrote:
| I have been trying to imagine something similar, but without
| all the middleware/distribution layer. You need to do a thing?
| The LLM just does it and presents the user with the desired
| experience. Kind of upending the notion that we need "apps" in
| the first place. It's all materialized, just-in-time style.
| acchow wrote:
| Have you tried https://bolt.diy ?
|
| It does what you describe
| IAmGraydon wrote:
| It _claims_ to do what he describes.
| IAmGraydon wrote:
| Most software is useful because a large number of people can
| interact with it or with each other over it. I'm not so certain
| that one-off software would be very useful for anyone beyond
| very simple functionality.
| mohsen1 wrote:
| The idea that a lot of compute is moving towards inference has a
| huge consequence for the current "AI investments". This is bad
| news for NVDA particularly. The inference focused solutions have
| better economics than paying NVDA those huge margins (e.g. Grog)
| talldayo wrote:
| Nvidia can actually charge _larger_ margins if inference
| compute goes down. It would enable them to manufacture more
| units of smaller GPUs using inferior and cheaper silicon, all
| of which would increase the profits per unit sold as well as
| the number of units they can manufacture.
|
| The industry has to find a way to separate itself from Nvidia's
| GPGPU technology if they want to stop being gouged. The issue
| is that nobody, not Apple, not AMD, not Intel, has been
| treating Nvidia's hardware as a serious threat.
| logicchains wrote:
| >The issue is that nobody, not Apple, not AMD, not Intel, has
| been treating Nvidia's hardware as a serious threat
|
| Google has and they've built a much more cost efficient (for
| them) system: the TPU. They even rent them out, and in terms
| of cost per unit compute TPUs are significantly cheaper than
| renting GPUs from the big cloud providers. Amazon's also
| tried to do something similar with Trainium chips, however
| they're usefulness is more limited due to software issues
| (Amazon's much weaker at compiler development than Google, so
| Trainium software is quite slow and buggy).
| vidarh wrote:
| For inference Nvidia has more significant competition than
| for training. See Groq, Google's TPU's etc.
| pants2 wrote:
| People talk about Groq and Cerberus as competitors but it
| seems to me their manufacturing process makes the
| availability of those chips extremely limited. You can call
| up Nvidia and order $10B worth of GPUs and have them
| delivered the next week. Can't say the same for these
| specialty competitors.
| moralestapia wrote:
| You mean Cerebras.
|
| >call up Nvidia and order $10B worth of GPUs
|
| Doubt it.
|
| No idea about Groq, but Cerebras might give you a similar
| timeline than nVidia. Each of their wafers are 50x-100x
| H100s so they need to make less of them, in absolute
| units.
|
| But cooling, power, etc... nVidia might have an advantage
| as their ecosystem is huge and more "liquid" in a sense.
| vidarh wrote:
| > You can call up Nvidia and order $10B worth of GPUs and
| have them delivered the next week
|
| Nvidia sold $14.5 billion of datacenter hardware in the
| third quarter of their fiscal 2024 and that led to severe
| supply constraints, with estimate lead times for H100's
| up to 52 weeks some places, so no you can't, as that
| $14.5 billion was clearly capped by their ability to
| supply, not demand.
|
| You're right, though, that Groq etc. can't deliver
| anywhere near the same volume _now_ , but there's little
| reason to believe that will continue. There's no need for
| full GPU's for inference only workloads, so competitors
| can enter the space with a tiny proportion of
| functionality.
| panabee wrote:
| Nvidia generates revenue with hardware, but digs moats with
| software.
|
| The CUDA moat is widely unappreciated and misunderstood.
| Dethroning Nvidia demands more than SOTA hardware.
|
| Without diving into the gory details, the simple proof is
| that billions were spent on inference last year by some of
| the most sophisticated technology companies in the world.
|
| They had the talent and the incentive to migrate, but
| didn't.
|
| OpenAI, Meta, AWS, AMD, and others have long attempted to
| eliminate the Nvidia tax, yet failed.
|
| People are desperate to quit their NVDA-tine addiction, but
| they can't, at least for now.
| vidarh wrote:
| The CUDA moat is largely irrelevant for inference. The
| code needed for inference is small enough that there are
| e.g. bare-metal CPU only implementations. That isn't
| what's limiting people from moving fully off Nvidia for
| inference. And you'll note almost "everyone" in this game
| are in the process of developing their own chips.
| onlyrealcuzzo wrote:
| > OpenAI, Meta, AWS, AMD, and others have long attempted
| to eliminate the Nvidia tax, yet failed.
|
| Gemini / Google runs and trains on TPUs.
|
| You have no incentive to infer on AMD if you need to buy
| a massive Nvidia cluster to train.
| panabee wrote:
| Google was omitted because they own the hardware and the
| models, but in retrospect, they represent a proof point
| nearly as compelling as OpenAI. Thanks for the comment.
|
| Google has leading models operating on leading hardware,
| backed by sophisticated tech talent who could facilitate
| migrations, yet Google still cannot leap over the CUDA
| moat and capture meaningful inference market share.
|
| Yes, training plays a crucial role, since this is where
| companies get shoehorned into the CUDA ecosystem, but if
| CUDA were not so intertwined with performance and
| reliability, customers could theoretically switch after
| training.
| buyucu wrote:
| My company recently switched from A100s to MI300s. I can
| confidently say that in my line of work, there is no CUDA
| moat. Onboarding took about month, but afterwards
| everything was fine.
| ClumsyPilot wrote:
| I think future of inference is on the client side
|
| You can do inference on almost any hardware, I do not see any
| edge for NVIDIA here
|
| I can download DeepSeek 30b model and run inference at good
| speed on AMD GPU ms and even on CPU. Apple silicon works fine
| too. I get >50 tokens/s on PS300 AMD GPUs.
|
| The main bottleneck appears to be memory, not processing
| power.
| mrbungie wrote:
| In fact, I'm not sure how the "we will need tons of
| centralized inference infrastructure" argument works when
| Apple with +50% smartphone market share in the USA has a
| totally opposite strategy focused on privacy: on-device
| inference.
| gajjanag wrote:
| This is much more nuanced now. See Apple "Private Cloud
| Compute": https://security.apple.com/blog/private-cloud-
| compute/ ; they run a lot of the larger models on their
| own servers.
|
| Fundamentally it is more efficient to process a batch of
| tokens from multiple users/requests than processing them
| from a single user's request on device.
| talldayo wrote:
| Apple's strategy already failed. Their big bet on NPU
| hardware did not pay off at all, and right now it's
| effectively wasted silicon on every iDevice while the GPU
| does all the heavy inference work. Now they partner with
| OpenAI to handle their inference (and even that's not
| good enough in many cases[0]). The "centralized compute"
| lobby is being paid by Apple to do the work their devices
| cannot.
|
| Until Apple or AMD unifies their GPU architectures and
| implements complex streaming multiprocessors, Nvidia will
| remain in a class of their own. Apple used to lead the
| charge on the foremost CUDA alternative too, but then
| they abandoned it to focus on proprietary standards
| instead. It's pretty easy to argue that Apple shot
| themselves in the foot with every opportunity they had to
| compete on good faith. And make no mistake: Apple _could_
| have competed with Nvidia if they weren 't so stubborn
| about Linux support and putting smartphone GPUs in
| laptops and desktops.
|
| [0] https://apnews.com/article/apple-ai-news-
| hallucinations-ipho...
| pertymcpert wrote:
| So far it's moving towards test time compute true, but
| reasoning models are still far too large to be done on the
| edge.
| rybosome wrote:
| Fascinating. R1 really punches above its weight with respect to
| cost-per-token.
|
| As the article alluded to at the end, my thoughts immediately go
| to using R1 as a data generator for complex problems, since we
| have many examples of successful distillation into smaller models
| on well-defined tasks.
| mikejulietbravo wrote:
| Mike from Baseten here
|
| We're super proud to support this work. If you're thinking of
| running deepseek in production, give us a shout!
| fxttr wrote:
| We currently evaluate DeepSeek-R1 for our production system. We
| aren't done yet, but I think it's a match.
| mikejulietbravo wrote:
| Awesome - we'd love to have our CEO/CTO chat with you and
| your team if you're interested. Shoot me a note at
| mike.bilodeau @ baseten.co and I'll make it happen!
| littlestymaar wrote:
| Earlier today I read a reddit comment[1] about a guy who tried
| running the quantized version from unsloth[2] on 4xH100 and the
| results was underwhelming (it ended up costing $137 per 1
| million tokens).
|
| Any idea of what they're doing wrong?
|
| [1]:
| https://www.reddit.com/r/LocalLLaMA/comments/1icphqa/how_to_...
|
| [2]: https://unsloth.ai/blog/deepseekr1-dynamic
| philipkiely wrote:
| They're using Llama.cpp which is an amazing tool for local
| inference but doesn't match fast inference frameworks like
| TensorRT-LLM/SGLang for production speeds and throughputs on
| Hopper GPUs.
|
| The Unsloth quantizations are really cool, but if you want to
| experiment with the R1 models in a smaller form factor the R1
| Distills like Llama 70B are great and should run a lot faster
| as they take advantage of existing optimizations around
| inferencing llama-architecture models.
| coder543 wrote:
| I'm not an expert on at-scale inference, but they surely
| can't have been running at a batch size of more than 1 if
| they were getting performance that bad on 4xH100... and I'm
| not even sure how they were getting performance that low even
| at batch size 1. Batching is essential to serving large token
| volumes at scale.
|
| As the comments on reddit said, those numbers don't make
| sense.
| WhitneyLand wrote:
| Can you share at a high level how you run this model?
|
| We know it's 671B params with each MOE node at 37B...
|
| If the GPUs have say, 140GB for an H200, then do you just load
| up as many nodes as will fit into a GPU?
|
| How much do interconnects hurt performance vs being able to
| load the model into a single GPU?
| philipkiely wrote:
| Yeah so MoE doesn't really come into play for production
| serving -- once you are batching your requests you hit every
| expert at a large enough batch size so you have to think
| about running the models as a whole.
|
| There are two ways we can run it:
|
| - 8xH200 GPU == 8x141GB == 1128 GB VRAM
|
| - 16xH100 GPU == 8x80GB == 1280 GB VRAM
|
| Within a single node (up to 8 GPUs) you don't see any
| meaningful hit from GPU-to-GPU communication.
|
| More than that (e.g. 16xH100) requires multi-node inference
| which very few places have solved at a production-ready
| level, but it's massive because there are way more H100s out
| there than H200s.
| Stevvo wrote:
| "The o3 system demonstrates the first practical, general
| implementation of a computer adapting to novel unseen problems"
|
| Yet, they said when it was announced:
|
| "OpenAI shared they trained the o3 we tested on 75% of the Public
| Training set. They have not shared more details. We have not yet
| tested the ARC-untrained model to understand how much of the
| performance is due to ARC-AGI data."
|
| These two statements are completely opposed. I can't take
| seriously anything this article says about o3.
| usaar333 wrote:
| No they aren't. Every arc problem is novel - that's why it
| resisted deep learning for so long (and still does to a
| degree).
|
| We just don't know how much the model seeing what an arc
| problem is on the first place boosts its ability to solve them
| - that limited statement is all the author is making.
| 7thpower wrote:
| They are testing with a different dataset. The authors saying
| that they have not tested on the version of o3 that has not
| seen the _training_ set.
| spyckie2 wrote:
| > But now with reasoning systems and verifiers, we can create
| brand new legitimate data to train on. This can either be done
| offline where the developer pays to create the data or at
| inference time where the end user pays!
|
| > This is a fascinating shift in economics and suggests there
| could be a runaway power concentrating moment for AI system
| developers who have the largest number of paying customers. Those
| customers are footing the bill to create new high quality data
| ... which improves the model ... which becomes better and more
| preferred by users ... you get the idea.
|
| While I think this is an interesting hypothesis, I'm skeptical.
| You might be lowering the cost of your training corpus by a few
| million dollars, but I highly doubt you are getting novel, high
| quality data.
|
| We are currently in a world where SOTA base model seems to be
| capped at around GPT4o levels. I have no doubt that in 2-3 years
| our base models will compete with o1 or even o3... just it
| remains to be seen what innovations/optimizations get us there.
|
| The most promising idea is to use reasoning models to generate
| data, and then train our non-reasoning models with the reasoning-
| embedded data. But... it remains to be seen how much of the chain
| of thought reasoning you can really capture into model weights.
| I'm guessing some, but I wonder if there is a cap to multi-head
| attention architecture. If reasoning can be transferred from
| reasoning models to base models, OpenAI should have already
| trained a new model with o3 training data, right?
|
| Another thought is maybe we don't need to improve our base models
| much. It's sufficient to have them be generalists, and to improve
| reasoning models (lowering price, improving quality) going
| forward.
| mohsen1 wrote:
| every time you respond to an AI model "no, you got that wrong,
| do it this way" you provide a very valuable piece of data to
| train on. With reasoning tokens there is just a lot more of
| that data to train on now
| jeffbee wrote:
| If such labels are collected and used to retrain the model
| then yes. But these models are not learning online.
| jvanderbot wrote:
| Really? Isn't that the point of RL used in the way R1 did?
|
| Provide a cost function (vs labels) and have it argue
| itself to greatness as measured by that cost function?
|
| I believe that's what GP meant by "respond", not telling
| GPT they were wrong.
| daveguy wrote:
| That is still inference. It is using a model generated
| from the RL process. The RL process is what used the cost
| function to add another model layer. Any online/continual
| learning would have to be performed by a different
| algorithm than classical LLM or RL. You can think of RL
| as a revision, but it still happens offline.
| Online/continual learning is still a very difficult
| problem in ML.
| jvanderbot wrote:
| Yes, that makes sense. We're both talking about offline
| learning.
| aprilthird2021 wrote:
| So if I just pay OpenAI $200/mo, and randomly tell the AI, no
| that's wrong.
|
| I can stop the AI takeover?
| Exoristos wrote:
| You can have our thank-you cards forwarded to your cell at
| Guantanamo Bay.
| dr_kiszonka wrote:
| You would need a lot of pro accounts! I would be surprised
| if they didn't use any algorithms for detecting well
| poisoning.
| nine_k wrote:
| This assumes that you give honest feedback.
|
| Efforts to feed deployed AI models various epistemic poisons
| abound in the wild.
| hammock wrote:
| The AI models to begin with assume that a significant
| majority of the training material is honest/in good faith.
| So that is not new?
| BorisMelnik wrote:
| I am not in this space, question: are there "bad actors"
| that are known to feed AI models with poisonous
| information?
| tsunamifury wrote:
| If the AI already has a larger knowledge domain space
| than the user then all users are bad actors. They are
| just too stupid to know it.
| nine_k wrote:
| Bad or not, depends on your POV. But certainly there are
| efforts to feed junk to AI web scrapers, including
| specialized tools: https://zadzmo.org/code/nepenthes/
| mrandish wrote:
| I'm not in the space either but I think the answer is an
| emphatic yes. Three categories come to mind:
|
| 1. Online trolls and pranksters (who already taught
| several different AIs to be racist in a matter of hours -
| just for the LOLs).
|
| 2. Nation states like China who already require models to
| conform to state narratives.
|
| 3. More broadly, when training on "the internet" as a
| whole there is a huge amount of wrong, confused
| information mixed in.
|
| There's also a meta-point to make here. On a lot of
| culture war topics, one person's "poisonous information"
| is another person's "reasonable conclusion."
| theendisney wrote:
| The part where people disagree seems fun.
|
| Im looking forwards to protoscience/unconventional
| science and perhaps even that what is worthy of the
| fringe or pseudoscience labels. The debunking there
| usually fails to adress the topic as it is incredibly
| hard to spend even a single day reading about something
| you "know" to be nonsense. Who has time for that?
|
| If you take a hundred thousand such topics the odds they
| should all be dismissed without looking arent very good.
| immibis wrote:
| Creators who use Nightshade on their published works.
| blibble wrote:
| yes, example: me
|
| I more often than not use the thumbs up on bad Google AI
| answers
|
| (but not always! can't find me that easily!)
| kristofferR wrote:
| Great arsticle from today: https://arstechnica.com/tech-
| policy/2025/01/ai-haters-build-...
| Uehreka wrote:
| This assumes that the companies gathering the data don't
| have silent ways of detecting bad actors and discarding
| their responses. If you're trying to poison an AI, are you
| making all of your queries from the same IP? Via a VPN
| whose IP block is known? Are you using a tool to generate
| this bad data, which might have detectable word frequency
| patterns that can be detected with something cheap like tf-
| idf?
|
| There's a lot of incentive to figure this out. And they
| have so much data coming in that they can likely afford to
| toss out some good data to ensure that they're tossing out
| all of the bad.
| aprilthird2021 wrote:
| > If you're trying to poison an AI, are you making all of
| your queries from the same IP? Via a VPN whose IP block
| is known?
|
| We can use the same tactics they are using to crawl the
| web and scrape pages and bypass anti-scraping mechanisms.
| Uehreka wrote:
| Not necessarily, not all tactics can be used
| symmetrically like that. Many of the sites they scrape
| feel the need to support search engine crawlers and RSS
| crawlers, but OpenAI feels no such need to grant
| automated anonymous access to ChatGPT users.
|
| And at the end of the daty, they can always look at the
| responses coming in and make decisions like "95% of users
| said these responses were wrong, 5% said these responses
| were right, let's go with the 95%". As long as the vast
| majority of their data is good (and it will be) they have
| a lot of statistical tools they can use to weed out the
| poison.
| whilenot-dev wrote:
| > As long as the vast majority of their data is good (and
| it will be)
|
| So expert answers are out of scope? Nice, looking forward
| to those quality data!
| scarmig wrote:
| Probably it's something like "give feedback that's on
| average slightly more correct than incorrect," though you'd
| get more signal from perfect feedback.
|
| That said, I suspect the signal is very weak even today and
| probably not too useful except for learning about human
| stylistic preferences.
| stetrain wrote:
| > What is today's date?
|
| >> Today's date is Tuesday, January 28, 2025.
|
| > No, you're wrong, today's date is actually Wednesday the
| 29th.
|
| >> My mistake. Yes, today's date is Wednesday, January 29th,
| 2025.
|
| Three months later in April when this tagged data is used to
| train the next iteration, the AI can successfully learn that
| today's date is actually January 29th.
| mr-wendel wrote:
| But thats exactly what you get when you ask questions that
| require shifting, specific contextual knowledge. The model
| weights, by their nature, cannot encode that information.
|
| At best, you can only try to layer in contextual info like
| this as metadata during inference, akin to how other
| prompting layers exist.
|
| Even then, what up-to-date information should present for
| every round-trip is a matter of opinion and use-case.
| genewitch wrote:
| the date is in the "system prompt", so the cron job that
| updates the prompts to the current date may be in a
| different time zone than you. 7f5dbb71f54322f271c4d3fc3aaa4
| d3282a1af5541d82b2cbc5aa10c1420b6bc
| echelon wrote:
| > you provide a very valuable piece of data to train on
|
| We've been saying this "we get valuable data" thing since the
| 2010s [1].
|
| When will our collective Netflix thumbs ups give us
| artificial super-intelligence?
|
| [1] _Especially_ to investors. They love that line.
| genewitch wrote:
| our collective netflix thumbs up indicators gave investors
| and netflix the confidence to deploy a series of adam
| sandler movies that cost 60 to 80 million US dollars to
| "make". So depending on who you are, the system might be
| working great.
| beAbU wrote:
| Through analytics Netflix should know exactly when people
| stop watching a series, or even when in a movie they exit
| out. They no doubt know this by user.
|
| They know exactly what makes you stay, and what makes you
| leave.
|
| I would not be surprised if in the near future movies and
| series are modifed _on the fly_ to ensure users stay
| glued to their screens.
|
| In the distant future this might be done on a per user
| level.
| deegles wrote:
| not being snarky, but what is the point of using the model if
| you already know enough to correct it into giving the right
| answer?
|
| an example that just occurred to me - if you asked it to
| generate an image of a mushroom that is safe to eat in your
| area, how would you tell it it was wrong? "oh, they never got
| back to me, I'll generate this image for others as well!"
| vincentperes wrote:
| You constantly have to correct an AI when using it because
| it either didn't get the question right or you guide him
| towards a more narrowed answer. There is only more to
| learn.
| dematz wrote:
| A common use of these models is asking for code, and maybe
| you don't know the answer or would take a while to figure
| it out. For example, here's some html, make it blue and
| centered. You could give the model feedback on if its
| answer worked or not, without knowing the correct answer
| yourself ahead of time.
| Levitz wrote:
| >not being snarky, but what is the point of using the model
| if you already know enough to correct it into giving the
| right answer?
|
| For your example, what if you want to show what such a
| mushroom looks like to a friend? What if you want to use it
| on a website?
| amluto wrote:
| Does it?
|
| If I say "no, you hallucinated basically the entire content
| of the response", then maybe a newer training set derived
| from that could train on the specific fact that that specific
| hallucinated response is hallucinated. This seems to be of
| dubious value in a training set.
| ClumsyPilot wrote:
| > The most promising idea is to use reasoning models to
| generate data, and then train our non-reasoning models with the
| reasoning-embedded data.
|
| Why is it promising, aren't you potentially amplifying AI
| biases and errors?
| jcims wrote:
| >You might be lowering the cost of your training corpus by a
| few million dollars, but I highly doubt you are getting novel,
| high quality data.
|
| The large foundational models don't really need more empirical
| data about the world. ChatGPT already 'knows' way more than I
| do, probably by many orders of magnitude. Yet it's still
| spewing nonsense at me regularly because it doesn't know how to
| think like a human or interact with me in a human-like way. To
| that end, the ability for a company like OpenAI to collect
| novel data from interacting with real humans is a material
| advantage over their competition.
| fizx wrote:
| You're not getting new high-quality textual data for pre-
| training from your chat service. But you are potentially
| getting a lot of RL feedback on ambiguous problems.
| baq wrote:
| It doesn't need much. 1 good lucky answer in a 1000 or maybe
| 10k queries gives you the little exponential kick you need to
| improve. This is how the hockey stick take off looks like and
| we're already here - OpenAI has it, now deepseek has it, too.
| You can be sure others also have it; Anthropic at the very
| least, they just never announced it officially, but go read
| what their CEO has been speaking and writing about.
| sheepscreek wrote:
| > The most promising idea is to use reasoning models to
| generate data, and then train our non-reasoning models with the
| reasoning-embedded data.
|
| DeepSeek did precisely this with their LLama fine-tunes. You
| can try the 70B one here (might have to sign up):
| https://groq.com/groqcloud-makes-deepseek-r1-distill-llama-7...
| vagabund wrote:
| > I highly doubt you are getting novel, high quality data.
|
| Why wouldn't you? Presumably the end user would try their use
| case on the existing model, and if it performs well, wouldn't
| bother with the expense of setting up an RL environment
| specific to their task.
|
| If it doesn't perform well, they do bother, and they have all
| the incentive in the world to get the verifier right -- which
| is not an extraordinarily sophisticated task if you're only
| using rules-based outcome rewards (as R1 and R1-Zero do)
| dr_dshiv wrote:
| > The R1-Zero training process is capable of creating its own
| internal domain specific language ("DSL") in token space via RL
| optimization.
|
| Um, what's that now? Really?
| svdr wrote:
| Yeah, if I understand correctly AI will create it's own
| internal reasoning language through RL. In R1-Zero it was
| already a strange mix of languages. They corrected that for R1
| to make the thinking useful for humans.
| Developerx wrote:
| Make America s*k again
| Developerx wrote:
| Make America s again
___________________________________________________________________
(page generated 2025-01-29 23:00 UTC)