[HN Gopher] Task-free intelligence testing of LLMs
       ___________________________________________________________________
        
       Task-free intelligence testing of LLMs
        
       Author : amarble
       Score  : 26 points
       Date   : 2026-01-08 19:51 UTC (3 hours ago)
        
 (HTM) web link (www.marble.onl)
 (TXT) w3m dump (www.marble.onl)
        
       | vitaelabitur wrote:
       | Aren't LLMs just super-powerful pattern matchers? And guessing
       | "taps" a pattern recognition task? I am struggling to understand
       | how your experiment relates to intelligence in any way.
       | 
       | Also, commercial LLMs generally have system instructions baked on
       | top of the core models, which intrinsically prompt them to look
       | for purpose even in random user prompts.
        
         | crooked-v wrote:
         | There's definitely more than "just" pattern matching in there -
         | for example, current SOTA models 'plan ahead' to simultaneously
         | process both rough outlines of an answer and specific subject
         | details to then combine internally for the final result
         | (https://www.anthropic.com/research/tracing-thoughts-
         | language...).
        
           | wood_spirit wrote:
           | Eh that is still encompassed by the term "pattern matching"
           | in this context. Sure it's complicated, but it's still just a
           | glorified spell checker.
        
       | sdenton4 wrote:
       | I like the high level idea! (how do we test intelligence in a non
       | functional way?)
       | 
       | I'm effect, the different response types are measuring how the
       | models respond to a context-free novel environment. I imagine
       | humans would also respond on a variety of ways to this test, none
       | of which are necessarily incorrect from the perspective of
       | intelligence testing .
       | 
       | Many tests of human behavior (eg, n behavioral economics) create
       | some pretense context to avoid boarding the response that is
       | actually being measured. For example, we may invite a participant
       | to a study of color preference, but actually measure how fast
       | they complete the task when the scientist has/hasn't bathed in a
       | week (or whatever).
       | 
       | Likewise, for llm intelligence testing, you could create pretext
       | tasks and context, and perhaps measure what the model considered
       | along the way, instead of the actual task outcome.
        
       | nestorD wrote:
       | On alternative ways to measure LLM intelligence, we had good
       | success with this: https://arxiv.org/abs/2509.23510
       | 
       | In short: start with a dataset of question and answer pairs,
       | where each question has been answered by two different LLMs. Ask
       | the model you want to evaluate to choose the better answer for
       | each pair. Then measure how consistently it selects winners. Does
       | it reliably favor some models over the questions, or does it
       | behave close to randomly? This consistency is a strong proxy for
       | the model's intelligence.
       | 
       | It is not subject to dataset leaks, lets you measure intelligence
       | in many fields where you might not have golden answers, and
       | converges pretty fast making it really cheap to measure.
        
       | forty wrote:
       | tap tap tap
        
       ___________________________________________________________________
       (page generated 2026-01-08 23:00 UTC)