[HN Gopher] The leading AI models are now good historians
___________________________________________________________________
The leading AI models are now good historians
Author : benbreen
Score : 54 points
Date : 2025-01-22 23:29 UTC (3 days ago)
(HTM) web link (resobscura.substack.com)
(TXT) w3m dump (resobscura.substack.com)
| BeefWellington wrote:
| Good read on what someone in a specific field considers to have
| been achieved (rightly or wrongly). It does lead me to wonder how
| many of these old manuscripts and their translations are in the
| training set. That may limit its abilities against any random
| sample that isn't included.
|
| Then again, maybe not; OCR is one of the most worked on problems,
| so the quality of parsing characters into text maybe shouldn't be
| as surprising.
|
| Off topic: it's wild to me that in 2025 sites like substack don't
| apply `prefers-color-scheme` logic to all their blogs.
| satisfice wrote:
| The intractable problem, here, is that "LLMs are good historians"
| is a nearly useless heuristic.
|
| I'm not a historian. I don't speak old spanish. I am not a domain
| expert at all. I can't do what the author of this post can do:
| expertly review the work of an LLM in his field.
|
| My expertise is in software testing, and I can report that LLMs
| sometimes have reasonable testing ideas-- but that doesn't mean
| they are safe and effective when used for that purpose by an
| amateur.
|
| Despite what the author writes, I cannot use an LLM to get good
| information about history.
| amelius wrote:
| You __can__ get good information from an LLM, however you just
| have to backtrack every once in a while because the information
| turned out to be false.
| userbinator wrote:
| _however you just have to backtrack every once in a while
| because the information turned out to be false._
|
| The problem is, how do you know? I've seen developers go
| completely off-course just from bad search engine results and
| one did admit he felt something wasn't right but kept going
| because he didn't know better; now imagine he's being told by
| a very confident but incorrect LLM, and you can see how
| hazardous that'll be.
|
| "You don't know what you don't know."
| sebmellen wrote:
| Unless you have a very good understanding of the system
| you're working on or the tools you're using, it's very
| possible to get knee deep in crap without knowing it.
| That's one of the biggest risks of using LLMs as
| assistants.
| simonw wrote:
| You need to develop skills like critical thinking,
| metacognition, analytical reasoning - being able to get to
| a robust mental model from a bunch of different inputs,
| some of which may even contradict each other.
| amelius wrote:
| In the case of software, you can ask the LLM to write a
| test.
|
| In other domains, I'm sure there are ways to verify
| information.
| mvdtnz wrote:
| And therein lies the problem - if you're not already an
| expert there's no way to tell when is the right moment to
| backtrack.
| nithril wrote:
| The exact definition of a useful heuristic, "good enough"
| simonw wrote:
| This is similar to the problem with some of the things people
| have been doing with o1 and o3. I've seen people share "PhD
| level" results from them... but if I don't have a PhD myself in
| that subject it's almost impossible for me to evaluate their
| output and spot if it makes sense or not.
|
| I get a ton of value out of LLMs as a programmer partly because
| I have 20+ years of programming experience, so it's trivial for
| me to spot when they are doing "good" work as opposed to making
| dumb mistakes.
|
| I can't credibly evaluate their higher level output in other
| disciplines at all.
| britch wrote:
| Interesting perspective. I appreciate that it tests the models at
| different "layers" of understanding.
|
| I have always felt that LLMs would fall apart beyond the
| summarization. Maybe they would be able to regurgitate someone
| else's analysis. The author seems to think there's some level of
| intelligent creativity at play
|
| I'm hopeful that the author is right. That truly creative
| thinking may be beyond the abilities of LLMs and be decades away.
|
| I think the author doesn't consider the implications of broad use
| of LLM societally. Will people be willing to fund human historian
| grad students when they can get a LLM for a fraction of the
| price? Will prospective historians have gained the training
| necessary if they've used an LLM through all of school?
|
| I believe the education system could figure it out over time. I'm
| more worried that LLMs like this will be used as further
| justification to defund or halt humanities research. Who needs a
| history department when I can get 80% for the cost of a few
| chatGPT queries?
| simonw wrote:
| I'd love to read way more stuff like this. There are plenty of
| people writing about LLMs from a computer science point of view,
| but I'm much more interested in hearing from people in fields
| like this one (academic history) who are vigorously exploring
| applications of these tools.
| quantadev wrote:
| American-made AIs are probably mostly good historians (other than
| having Silicon Valley Left-wing progressive wokeness built in),
| but DeepSeek, or anything from China is going to _not_ be a good
| historian, but more of a propaganda tool, to further China 's
| interests.
| cyrillite wrote:
| Now the question is how can I, someone without a PhD in history
| but currently a PhD candidate in another discipline, use these
| tools to reliably interrogate topics of interest and produce at
| least a graduate level understanding of them?
|
| I know this is possible, but the further away I get from my core
| domains, the harder it is for me to use these tools in a way that
| doesn't feel like too much blind faith (even if it works!)
| kozikow wrote:
| > the harder it is for me to use these tools in a way that
| doesn't feel like too much blind faith (even if it works!)
|
| I tend to ask multiple models and if they all give me roughly
| the same answer, then it's probably right.
| aquafox wrote:
| > if they all give me roughly the same answer, then it's
| probably right.
|
| ... or they had a lot of overlapping training data in that
| area.
| energy123 wrote:
| Also keeping context short. Virtually all my cases of bad
| hallucinations with o1 have been when I've provided too much
| context or the conversation has been going on for too long.
| Starting a new chat fixes it.
|
| You can see this effect in the ARC-AGI evals, too much
| context impacts even o3(high).
| aquafox wrote:
| You ask them for references and check yourself. They are good
| exploratory and hypothesis generating tools, but not more.
| Getting a sensible sounding answer should not be an excuse for
| you to confirm. Often, the devil is in the details.
| simonw wrote:
| I think the trick here is to treat everything these models tell
| you as part of a larger information diet.
|
| Like if you have a friend who's very well-read and talkative
| but is also extremely confident and loves the sound of their
| own voice. You quickly learn to treat them as a source of
| probably-correct information, but only part of they way you
| learn any given topic.
|
| I do this with LLMs all the time: I'm constantly asking them
| clarifying questions about things, but I always assume that
| they might be making mistakes or feeding me convincing sounding
| half-truths or even full hallucinations.
|
| Being good at mixing together information from a variety of
| sources - of different levels of accuracy - is key to learning
| anything well.
| option wrote:
| Yeah, especially the ones from China /s
| abathur wrote:
| Someone who knows a lot of history is a history buff.
|
| A historian works with (and may even seek out in musty rooms)
| primary and secondary sources to produce novel research and
| interpretation.
|
| An AI is at best limited to ~reading sources that human
| historians/archivists/librarians have already identified and
| digitized.
|
| Certainly value to be had here wrt to finding needles in and
| making sense of already-digitized historical records, but that's
| more like a research assistant.
___________________________________________________________________
(page generated 2025-01-26 23:00 UTC)