Post AkMpTDWDBwqCDPlnW4 by pretergeek@mstdn.social
(DIR) More posts by pretergeek@mstdn.social
(DIR) Post #AkMJ5zVZuzVhZOM3oe by ZachWeinersmith@mastodon.social
2024-07-27T14:37:01Z
1 likes, 1 repeats
Are there any LLMs yet that are able to kick questions over to a physics model? Like, it seems that at least for some questions, the way we get an answer isn't by thinking about what we've seen or learned or said before, but literally imagining the world. For kids, this seems to include things like finger counting for addition.
(DIR) Post #AkMJM3X6iqXUgYjLTk by ZachWeinersmith@mastodon.social
2024-07-27T14:39:53Z
0 likes, 0 repeats
Like, GPT-3 failed questions of the form "I'm in the basement and look at the sky. What do I see?" GPT-4 fixed this by having humans correct its mistakes. I imagine if I were a kid getting this question for the first time, especially in a place where there aren't typically basements, what I'd do is probably imagine being in a basement.
(DIR) Post #AkMJO15KnKoVfkft7w by ZachWeinersmith@mastodon.social
2024-07-27T14:40:17Z
0 likes, 0 repeats
And the model I use could be fairly stupid. Just a sort of underground box. No need for deep physics or even an understanding of what the point of a basement is.
(DIR) Post #AkMJplG7EHQohGwmjA by jenzi@mastodon.social
2024-07-27T14:45:13Z
0 likes, 0 repeats
@ZachWeinersmith This is an interesting thought. Without the physics model itself you're asking the model to move about in a virtual 3D space. The human mind can hold a ball and rotate it, picturing the other side it's not yet seen - but can an LLM be trained to think of it's position in 3D space or the 3D space of the topic of discussion? Maybe there is hints to this already in how the model can create and modify CSS layouts based on feedback - it has consumed and learned that type of space.
(DIR) Post #AkMJtOjeufkq03V3Vw by mistersql@mastodon.social
2024-07-27T14:45:54Z
0 likes, 0 repeats
@ZachWeinersmith Sort of- they're coming at it from a different angle. They're putting bots (neural networks) into games with physics engine to generate tons of training data. Then when they set the robot loose in the real world, it performs really well. I haven't read about multimodal physics-language-vision-audio models, but if they can do language-vision-audio why not also physics. It all just really large matrices.
(DIR) Post #AkMK2mwuFyAJhgAqxs by fivey@mastodo.neoliber.al
2024-07-27T14:47:35Z
0 likes, 0 repeats
@ZachWeinersmith There’s a wolfram plugin for chatGPT which should help it to do actual calculations rather than just hallucinating numbers. I haven’t tried it. Saying “use python” also works sometimes.https://gpt.wolfram.com/
(DIR) Post #AkMKFImniHARv3Pjwu by BoydStephenSmithJr@hachyderm.io
2024-07-27T14:49:48Z
0 likes, 0 repeats
@ZachWeinersmith I think this is what research is into "embodied" AI. Rather than baking in a physics model, the AI should "learn" "physics" by it's "bodily" interactions with the real world.But also, it would be nice to have a non-sentient robot that could do my laundry from pile/bag via washer/dryer to closet/drawers. Chat-GPT and variants can't do that due to (at least) lack of physical mechanisms.
(DIR) Post #AkMKWbRdasnSi8lUQq by potpie@mastodon.social
2024-07-27T14:52:59Z
0 likes, 0 repeats
@ZachWeinersmith My understanding is that LLMs are really only one component of an AI. All they do is approximate human language. If you ask a question, they will provide something that greatly resembles an answer, but it's based on language examples, not logic or understanding. Imagine, though, an LLM that only acts as a mouthpiece for a logic engine on, say, a spaceship with many systems and diagnostics to analyze. 1/
(DIR) Post #AkML9qpsIwpXjrohUm by RGBes@mastodon.social
2024-07-27T15:00:02Z
0 likes, 0 repeats
@ZachWeinersmith as I understand it, LLMs interpolate while humans have the ability to extrapolate. In order to extrapolate from a limited dataset, you need a model of reality, and LLMs don't have one, they just look for the most likely token to drop next to the previous one. They may (just may) be able to find interesting connections between data points, but they cannot go outside their training box.
(DIR) Post #AkMLCbFchkRCGxAfEO by Phosphenes@glasgow.social
2024-07-27T15:00:24Z
0 likes, 0 repeats
@ZachWeinersmith The visual neural networks seem to be smarter this way. I saw a study demonstrating that, given only a lot of 2D photos, the network reinvented a third dimension and a Z buffer in order to generate realistic fake photos. (Makes me wonder about 4D!) That said, if you've never seen a sky, and the prompt says you're looking at one, wouldn't it be reasonable to assume there is a sky?
(DIR) Post #AkMLWmHJPo6wGSaB2u by claudiacaesaris@ohai.social
2024-07-27T15:04:06Z
0 likes, 0 repeats
@ZachWeinersmith LLMs just look at text and make a statistical analysis of what word is most likely to come next. There's no understanding of meaning. Without that I think your idea is fundamentally impossible.
(DIR) Post #AkMLtI4prvbrG1TvPc by Phosphenes@glasgow.social
2024-07-27T15:08:18Z
0 likes, 0 repeats
@ZachWeinersmith Yeah why can't we just give a neural network a calculator, an encyclopedia, and a video game engine to get its facts straight? That leaves the network as just an interface/ambassador between humans and more classical machinery.
(DIR) Post #AkMMj4ypo7jHhGMyMC by AlexWe@genomic.social
2024-07-27T15:17:39Z
0 likes, 0 repeats
@ZachWeinersmith Not exactly the same, but see the recent success of AlphaGeometry to write formal proofs. So while I don't think it exists yet I would totally expect people are working on DL models that can model physics.https://www.technologyreview.com/2024/07/25/1095315/google-deepminds-ai-systems-can-now-solve-complex-math-problems/
(DIR) Post #AkMRYxR1VPJpAcY0Vk by rotopenguin@mastodon.social
2024-07-27T16:11:52Z
0 likes, 0 repeats
@ZachWeinersmith imagine you are in Plato's cave and look at the sky. What reality do you see?
(DIR) Post #AkMTGYYVT2PwhjOvcu by TheHolyPenguin@mastodon.online
2024-07-27T16:30:56Z
0 likes, 0 repeats
@ZachWeinersmith Google were working on something exactly like this a couple of years ago ("Mind's Eye"; https://research.google/pubs/minds-eye-grounded-language-model-reasoning-through-simulation/) but I haven't kept up to date with it so no idea whether or not their research continued beyond this paper.
(DIR) Post #AkMWCh83wuMrzysllo by johnbierce@wandering.shop
2024-07-27T17:03:47Z
0 likes, 0 repeats
@ZachWeinersmith how would it know when to kick a question over to the physics model?(The correct answer is, of course, it couldn't using machine learning algorithms. It would need some intermediary technology to know when to activate the physics model, which... Not gonna be other machine learning algorithms.)
(DIR) Post #AkMpTDWDBwqCDPlnW4 by pretergeek@mstdn.social
2024-07-27T20:39:43Z
0 likes, 0 repeats
@ZachWeinersmith That would be Grounding. Grounding an AI to a model of reality. Usually done for visual models that are supposed to navigate the world, but LLMs can do it to some extent. It has been shown that LLMs hold 3D representations of places based on descriptions. Think of all the books they have been trained on that had well described environments. There are also software for grounding AIs (most for visual models but LLMs too), some are like text adventure games.
(DIR) Post #AkMurgv227G07Trvii by zeborah@mastodon.nz
2024-07-27T21:40:06Z
0 likes, 0 repeats
@ZachWeinersmith If any LLM could kick questions over to some other kind of algorithm then it wouldn't be an LLM anymore.There are systems that use "retrieval augmented generation" to try and ensure that the facts come from actual science and the citations are real citations. But because the answer passes back through the LLM it's still bullshit (in the technical sense described in https://doi.org/10.1007/s10676-024-09775-5 )Some other model might get better at GenAI, but it won't be an LLM.
(DIR) Post #AkO2BnHcyQGyUHvkum by natrhein@c.im
2024-07-28T10:36:59Z
0 likes, 0 repeats
@ZachWeinersmith What are you referring with "GPT-4 [had] humans fix its mistakes"?
(DIR) Post #AkOBMoGMZSQSIZfOOe by 2tussock@mastodon.nz
2024-07-28T12:19:47Z
0 likes, 0 repeats
@ZachWeinersmith Why would a response completion program know what words mean?Like, at all. Obviously it can quote the dictionary if you ask it to, but it's not a child with limited understanding of the world, it's just an enormously large if-then-else statement that makes familiar-seeming responses to text.I'm pretty sure cockroaches are far more complex than GPT, they can respond appropriately to several types of stimulus.And like, physics models don't understand physics either.