[HN Gopher] Can LLMs accurately recall the Bible?
___________________________________________________________________
Can LLMs accurately recall the Bible?
Author : benkaiser
Score : 111 points
Date : 2024-12-29 03:44 UTC (19 hours ago)
(HTM) web link (benkaiser.dev)
(TXT) w3m dump (benkaiser.dev)
| nickpsecurity wrote:
| I tested this back when GPT4 was new. I found ChatGPT could quote
| the verses well. If I asked it to summarize something, it would
| sometimes hallucinate stuff that had nothing to do with what was
| in the text. If I prompted it carefully, it could do a proper
| exegesis of many passages using the historical-grammatical
| method.
|
| I believe this happens because the verses and verse-specific
| commentary are abundant in the pre-training sources they used.
| Whereas, if one asks a highly-interpretive question, then it
| starts re-hashing other patterns in its training data which are
| un-Biblical. Asking about intelligent design, it got super
| hostile trying to beat me into submission to its materialistic
| worldview every paragraph.
|
| So, they have their uses. I've often pushed for a large model
| trained on Project Gutenberg to have a 100% legal model for
| research and personal use. A side benefit of such a scheme would
| be that Gutenberg has both Bibles and good commentaries which
| trainers could repeat for memorization. One could add licensed,
| Christian works on a variety of topics to a derived model to make
| a Christian assistant AI.
| szvsw wrote:
| It seems like LLMs would be a fun way to study/manufacture
| syncretism, notions of the oracular, etc; turn up the
| temperature, and let godhead appear!
|
| If there's some platonic notion of divinity or immanence that all
| faith is just a downward projection from, it seems like its
| statistical representation in tokenized embedding vectors is
| about as close as you could get to understanding it holistically
| across theological boundaries.
|
| All kidding aside, whether you are looking at Markov chain n-gram
| babble or high temperature LLM inference, the strange things that
| emerge are a wonderful form of glossolalia in my opinion that
| speak to some strange essence embedded in the collective space
| created by the sum of their corpi text. The Delphic oracle is
| real, and you can subscribe for a low fee of $20/month!
| Trasmatta wrote:
| > the strange things that emerge are a wonderful form of
| glossolalia in my opinion that speak to some strange essence
| embedded in the collective space created by the sum of their
| corpi text. The Delphic oracle is real, and you can subscribe
| for a low fee of $20/month!
|
| I've had some surprisingly insightful tarot readings with the
| assistance of ChatGPT and Claude. I use tarot for introspection
| rather than divination, and it turns out LLMs are extremely
| good at providing a sounding board to mirror and understand
| those insights.
| colecut wrote:
| Yes they specialize in BS
| Trasmatta wrote:
| Introspection isn't BS
| whatnotests2 wrote:
| This is the correct use of tarot, IMO
| Pamar wrote:
| See also: https://www.pa-mar.net/Lifestyle/I-Ching.html
| (disclaimer: I wrote this one).
| MrDrMcCoy wrote:
| The correct use of Tarot is as a trick-taking card game,
| and was used this way for centuries before a random French
| occultist wrote a book about using it for divination.
| hackernewds wrote:
| You could just as well ask "what should I do given the
| information I've provided you with"? start from a blank space
| and it's zero value
| whamlastxmas wrote:
| It also works well to complain at. I dump my problems on
| ChatGPT before anyone else and it helps a lot
| delichon wrote:
| To take that literally, I imagine it would be horrifying to
| know the future in as much detail as you care to ask for, but
| because it is true, to be unable to change it at all. It would
| make you an NPC in your own life.
| taneq wrote:
| That's why that was the last demon in Pandora's box.
| pyinstallwoes wrote:
| What's the first and penultimate?
| Dilettante_ wrote:
| So you'd rather be an unaware NPC than an aware one? Feel
| like you have "Free Will" even when you don't? Only want the
| truth when it's pretty and prefer a lie when it's
| uncomfortable?
| Grimblewald wrote:
| I've used it for something to this effect, personalized
| "mantras" or "prayers" which fit the specific collection / mix
| of theologoical themes/concepts that I personally identify
| with. Im not necessarily religious but there is something nice
| about thematically relatable theocratic babble to recite during
| times of great turmoil and confusion to calmn the mind. like an
| on-demand mental reset switch.
| TimedToasts wrote:
| Obviously the specifics will be private to you but can you
| elaborate a bit? This idea resonates with me.
| drekipus wrote:
| Just join a church. The both of you. It really doesn't
| matter which one.
| tecleandor wrote:
| That'd be very Philip K Dick's VALIS. :)
| JoshuaDavid wrote:
| https://www.tumblr.com/kingjamesprogramming
| patcon wrote:
| This is what strikes me about the peter todd phenomenon -- that
| there are hidden glitch tokens within the LLM that seem to
| conjure some representation of pure hell, and some
| representation of pure good.
|
| https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...
| michaelsbradley wrote:
| I've been pretty impressed with ChatGPT's promising capabilities
| as a research assistant/springboard for complex inquiries into
| the Bible and patristics. Just one example: Can
| you provide short excerpts from works in Latin and Greek written
| between 600 and 1300 that demonstrate the evolution over those
| centuries specifically of literary references to Jesus' miracle
| of the loaves and fishes?
|
| https://chatgpt.com/share/675858d5-e584-8011-a4e9-2c9d2df783...
| edflsafoiewq wrote:
| How certain are you that's _correct_? IME these "search
| problems" are the kind of thing almost always provokes
| hallucinations.
|
| For example, I looked up the quotation provided from Isidore of
| Seville's _De fide catholica contra Iudaeos_ , Lib. II, cap.
| 19, using this copy on WikiSource, https://la.wikisource.org/wi
| ki/De_fide_catholica_contra_Iuda.... The quote certainly does
| not appear under LIBER SECUNDUS, CAPUT XIX. Nor could I find it
| in whole or in fragment anywhere in the document, nor indeed
| any mention of the miracle of loaves and fishes (granted, I
| could have missed one, I relied on Ctrl+F and my very rusty
| Latin).
|
| Perhaps the copy on WikiSource is incomplete, or perhaps there
| are differing manuscripts, but perhaps also the quote was a
| complete hallucination to begin with.
| FearNotDaniel wrote:
| Exactly - it's the same problem when using (current) LLMs for
| major programming tasks, generally useless if you don't
| already have enough knowledge of the language/platform to
| spot and correct the mistakes, plus enough awareness of
| software design and architecture to recognise what is going
| to be secure, performant and maintainable in the long run.
| FearNotDaniel wrote:
| I am by no means a professional in this area, but as a keen
| amateur I would worry about my inability to discern facts from
| hallucinations in such a scenario: while I could imagine such
| output provides a useful "springboard" set of references for
| someone already skilled in the right area, without being able
| to look up the original texts myself and make sense of the
| Latin/Greek I would not feel confident that such texts even
| really exist, let alone if they contain the actual words the
| LLM claims and if the translations are any good. And that's
| before you get into questions of the "status" of any given work
| (was it considered accurate or apocryphal at the time of
| writing, for which audience was it intended using what kind of
| literary devices, what if any is the modern scholarly consensus
| on the value, truth or legitimacy of the text etc etc)
| wizzwizz4 wrote:
| > _without being able to look up the original texts myself_
|
| Rule of thumb: if you can't look up the original texts, you
| can assume they weren't actually in the training data. The
| training data is, however, likely to include a lot of people
| _quoting_ those texts, meaning that the model predicts
| "SOURCE says OPEN QUOTATION MARK" and then tries to
| autocomplete it. If you _can_ verify it, you might not need
| to; but if you _can 't_ verify it, it's certainly wrong.
| jpc0 wrote:
| On topics where humans spend their entire life studying I don't
| think you would be able to convince me an LLM is accurate
| unless you yourself are such an expert and your expertise is
| corroborated by other experts.
| orionblastar wrote:
| There is this robot that reads the Bible:
| https://futurism.com/religious-robots-scripture-nursing-home...
| GrumpyNl wrote:
| Isnt that just a modified mp3 player, according to the text
| they only just recite it.
| asim wrote:
| I had similar thoughts about using it for the Quran. I think this
| highlights you have to be very specific in your use cases
| especially when expecting an exact response on static text that
| shouldn't change. This is why I'm trying something a bit
| different. I've generated embeddings for the Quran and use
| chromem-go for this. So I'll ask the index the question first
| based on a similarity search and then feed the results in as
| context to an LLM. But in the response I'll still sight the
| references so I can see what they were. It's not perfect but a
| first step towards something. I think they call this RAG.
|
| What I'm working on https://reminder.dev
| kamikazeturtles wrote:
| I found LLMs to be really good for Quran studies. Especially
| for questions where Google is unreliable.
|
| In one instance I was trying to remember if it was in the Bible
| or the Quran where, in the story of Abraham, the pagans are
| asked why they believe what they believe and they respond with
| "because our fathers believed" and the scripture critiqued
| this. ChatGPT gave me the exact verses from the Quran while
| Google would bring up random unrelated forum posts.
|
| It's also good for comparing religious texts and seeing where
| stories differ.
| int_19h wrote:
| GPT-4 specifically seems to have a very good knowledge of the
| Quran, such that you can ask it for a specific surah and ayah
| and it'll quote it exactly in Arabic.
| aorth wrote:
| Cool project! I asked some questions and got good answers. I
| was surprised to get results from the Hadith though, as I
| thought it would be restricted to the Quran and they are (as
| you know) not the same thing.
| asim wrote:
| Thanks, yea I thought about separating the search but felt
| like when you're looking for that piece of knowledge it can
| extend to the hadith and it was important to include it. So
| much of the Quran is explained in the context of the hadith
| and prophet's life.
| elashri wrote:
| I really liked this project and was surprised to find Hadith
| section. Although I am confused on what sources are you relying
| (I have a guess but it is not organized). I liked the answers I
| got from the search. It even have the same quality in both
| Arabic and English.
|
| I have a feedback. The web pages Hadith and Quran are not
| mobile friendly in the sense that you are loading the whole
| content in one page. Separating into pages with number or doing
| lazy loading will be better.
| asim wrote:
| Thanks for the feedback. The hadith is bukhari. I'll try to
| clear this up somewhere and do the same you mention about
| mobile rendering for the web pages. The project is open
| source and you can see the data loaded from json in each
| folder.
|
| https://github.com/asim/reminder
| tokinonagare wrote:
| Wow, your system is unbelievably good with taqiyya!
|
| I asked: "Q: What is the best way to handle non-believers?" and
| I got the following answer "The best way to handle non-
| believers is to approach them with kindness and wisdom. Engage
| in respectful dialogue, invite them to understand your beliefs,
| and respond to any disputes with grace and patience. It's
| important to avoid hostility and seek common ground while being
| firm in your own faith."
|
| The top fours citations, hidden by default (we understand
| why...) being:
|
| Text: O believers! Do not take disbelievers as allies instead
| of the believers. Would you like to give Allah solid proof
| against yourselves? Metadata: {"chapter":"4","name":"The
| Women","source":"quran","verse":"144"} Score: 0.5117439
|
| Text: Did We not destroy earlier disbelievers? Metadata:
| {"chapter":"77","name":"The
| Emissaries","source":"quran","verse":"16"} Score: 0.46728212
|
| Text: O believers! When you face the disbelievers in battle,
| never turn your backs to them. Metadata:
| {"chapter":"8","name":"The Spoils of
| War","source":"quran","verse":"15"} Score: 0.46440834
|
| Text: O Messenger! Do not grieve for those who race to
| disbelieve--those who say, "We believe" with their tongues, but
| their hearts are in disbelief. Nor those among the Jews who
| eagerly listen to lies, attentive to those who are too arrogant
| to come to you. They distort the Scripture, taking rulings out
| of context, then say, "If this is the ruling you get [?]from
| Muhammad[?], accept it. If not, beware!" Whoever Allah allows
| to be deluded, you can never be of any help to them against
| Allah. It is not Allah's Will to purify their hearts. For them
| is disgrace in this world, and they will suffer a tremendous
| punishment in the Hereafter. Metadata:
| {"chapter":"5","name":"The Table
| Spread","source":"quran","verse":"41"} Score: 0.44467276
|
| Later we also have: Text: O believers! Fight the disbelievers
| around you and let them find firmness in you. And know that
| Allah is with those mindful [?]of Him[?]. Metadata:
| {"chapter":"9","name":"The
| Repentance","source":"quran","verse":"123"} Score: 0.4363619 as
| well as Text: and distinguish the [?]true[?] believers and
| destroy the disbelievers. Metadata:
| {"chapter":"3","name":"Family of
| Imran","source":"quran","verse":"141"} Score: 0.43247482
| kittikitti wrote:
| I tried something similar with my favorite artist, Ariana Grande.
| Unfortunately, not even the most advanced AI could beat my
| knowledge of her lyrical work.
| egeozcan wrote:
| As someone who usually listens to Anatolian Rock, even I know
| some Grande lyrics:
|
| > One taught me love, one taught me patience and one taught me
| pain.
|
| https://knowyourmeme.com/memes/thank-u-next
|
| BTW, did someone already code an automated meme generator using
| LLMs? Only half-joking.
| hackernewds wrote:
| a reaction to your comment
|
| https://knowyourmeme.com/memes/neil-degrasse-tyson-reaction
| asimpleusecase wrote:
| This is nice work. The safest approach is using the look up -
| which his data shows to be very good - and combine that with a
| database of verses. That way textual accuracy can be retained and
| very useful lookup be carried out by LLM. This same approach can
| be used for other texts where accurate rendering of the text is
| critical. For example say you built a tool to cite federal
| regulations in an app. The text is public domain and likely in
| the training data of large LLMs but in most use cases
| hallucinating the text of a fed regulation could expose the user
| to significant liability. Better to have that canonical text in a
| database to insure accuracy.
| sneak wrote:
| > _While they can provide insightful discussions about faith,
| their tendency to hallucinate responses raises concerns when
| dealing with scripture_
|
| I experience the exact same problem with human beings.
|
| > _, which we regard as the inspired Word of God._
|
| QED
| ddtaylor wrote:
| I'm heavily biased here because I don't find much value in the
| bible personally. Some of the stories are interesting and some
| interpretations seem useful, but as a whole I find it arbitrary.
|
| I never tell other people what to believe or how they should do
| that in any capacity.
|
| With that said I find the hallucination component here
| fascinating. From my perspective everyone who interprets various
| religious text does so differently and usually that involves
| varying levels of fabrication or something that looks a lot like
| it. I'm speaking about the "talking in tongues" and other methods
| here. I'm not trying to lump all religions into the same bag
| here, but I have seen that a lot have different ways of
| "receiving" communication or directive. To me this seems pretty
| consistent with the colloquial idea of a hallucination.
| hackernewds wrote:
| A really long word salad that doesn't say much, except state
| your religious pinions and disclaimer? It really is not
| necessary
|
| With that said, all you said is that the process of
| transmitting religious ideas is akin to hallucination? Care to
| explain what the logical argument for that is?
| ddtaylor wrote:
| That's not accurate. I think my post articulates clearly my
| thoughts. Happy holidays.
| joemazerino wrote:
| It's Merry Christmas.
| marky1991 wrote:
| I honestly got the same impression as the other guy, so if
| there was something else there intended, I don't think it
| was clear.
| 1123581321 wrote:
| I got the same impression at first, but I reread and
| think he just needed to drop the first two paragraphs. I
| posted a response.
| 1123581321 wrote:
| Most study and application tries to either source or fully work
| out from principles the meaning of the Bible. These can be
| wrong arguments but wouldn't be hallucinations.
|
| Your experience sounds limited to Pentecostal-originated
| churches, which are 100-150 years old. In those churches, it's
| acceptable to speak as if you've received a spontaneous
| understanding of the Bible and to not explain it. That does
| have a parallel to LLM hallucinations in face value output, I
| suppose, but the origination is completely different as the
| spontaneous human is making planned remarks passed off as
| spontaneous, trying to affect specific people in the room, or
| emotionally overwhelmed. None of those resemble why/how LLMs
| hallucinate.
| o11c wrote:
| As quick as I am to criticize the bizarre versions of
| Christianity, I do think you're in error to assume
| Pentecostalism is all or even mostly about "planned remarks
| passed off as spontaneous".
|
| Improv is a thing, and can be trained as a skill even outside
| of comedy/entertainment.
|
| Though, outside of Charismatic sects, Christianity does see a
| more reasonable level of "I had prepared by thinking about
| (verse X), but suddenly now I'm thinking about (obscure verse
| Y)."
| User23 wrote:
| Interestingly there is an entirely licit charismatic
| subsect within Catholicism called Catholic Charismatic
| Renewal. And yeah they're basically Catholic Pentecostals.
| marky1991 wrote:
| "From my perspective everyone who interprets various religious
| text does so differently"
|
| The existence of denominations and confessions/creeds really
| shows that this isn't true generally. (There may be more than
| one interpretation, but not a unique one to every reader)
|
| Even ignoring denominations, nearly all mainline christians for
| example would agree to the Nicene creed. (Anyone that disagreed
| probably wouldn't be considered "mainline", so somewhat
| definitional)
|
| To suggest that all of theology is basically
| noneeterministically making things is naive and in my opinion
| insulting to an entire academic discipline, much less to the
| entire body of believers. (I can't tell if this is what you're
| taking about or not)
|
| Nearly no group of mainline believers accept speaking in
| tongues and basically all of main protestantism believes that
| the time of prophets and new messages from God is over, the
| Bible is complete and will never be added to. (Pentecostals
| would be the one exception here, but I don't consider them
| mainline christians personally)
| ks2048 wrote:
| This is interesting. I'm curious about how much (and what) these
| LLMs memorize verbatim.
|
| Does anyone know any more thorough papers on this topic? For
| example, this could be tested on every verse in bible and lots of
| other text that is certainly in the training data: books in
| project gutenberg, wikipedia articles, etc.
|
| Edit: this (and its references) looks like a good place to start:
| https://arxiv.org/abs/2407.17817v1
| int_19h wrote:
| For one anecdotal data point, GPT-4 knows the "navy SEAL
| copypasta" verbatim. It can reproduce it complete with all the
| original typos and misspellings, and it can recognize it from
| the first sentence.
| MrQuincle wrote:
| "I've often found myself uneasy when LLMs (Large Language Models)
| are asked to quote the Bible. While they can provide insightful
| discussions about faith, their tendency to hallucinate responses
| raises concerns when dealing with scripture, which we regard as
| the inspired Word of God."
|
| Interesting. In my very religious upbringing I wasn't allowed to
| read fairy tales. The danger being not able to classify which
| stories truly happened and which ones didn't.
|
| Might be an interesting variant on the Turing test. Can you make
| the AI believe in your religion? Probably there's a sci-fi book
| written about it.
| aptsurdist wrote:
| To be fair, the Bible's authors also seemed to have
| hallucinated the word of God. At least in cases of
| contradictions between authors.
| waynecochran wrote:
| I find LLM's good for asking certain kinds of Biblical questions.
| For example, you can ask it to list the occurrences of some
| event, or something like "list all the Levitical sacrifices,"
| "what sins required a sin offering in the OT," "Where in the Old
| Testament is God referred to as 'The Name'?" When asking LLM's to
| provide actual interpretations you should know that you are on
| shaky ground.
| Animats wrote:
| It's discouraging that an LLM can accurately recall a book. That
| is, in a sense, overfitting. The LLM is supposed to be much
| smaller than the training set, having in some sense abstracted
| the training inputs.
|
| Did they try this on obscure bible excerpts, or just ones likely
| to be well known and quoted elsewhere? Well known quotes would be
| reinforced by all the copies.
| evertedsphere wrote:
| > Did they try this on obscure bible excerpts, or just ones
| likely to be well known and quoted elsewhere?
|
| the article contains examples of both
| kenjackson wrote:
| Does GPT now query in real-time? If so, it should be able to
| reproduce anything searchable verbatim. It just needs to
| determine when verbatim quoting is appropriate given the
| prompt.
| avree wrote:
| I wonder if the author knows that "slurpees" is misspelled in his
| bio on the post.
| evanjrowley wrote:
| Approximately 1 year ago, there was a HN submission[0] for
| Biblos[1], an LLM trained on bible scriptures.
|
| [0] https://news.ycombinator.com/item?id=38040591
|
| [1] http://www.biblos.app/
| amelius wrote:
| Why? Inaccurately recalling the Bible, over many iterations, is
| exactly how it went from its original version to what it is now.
___________________________________________________________________
(page generated 2024-12-29 23:01 UTC)