[HN Gopher] A statistical approach to model evaluations
       ___________________________________________________________________
        
       A statistical approach to model evaluations
        
       Author : RobinHirst11
       Score  : 29 points
       Date   : 2024-11-23 12:37 UTC (6 days ago)
        
 (HTM) web link (www.anthropic.com)
 (TXT) w3m dump (www.anthropic.com)
        
       | fnordpiglet wrote:
       | This does feel a bit like under grad introduction to statistical
       | analysis and surprising anyone felt the need to explain these
       | things. But I also suspect most AI people out there now a days
       | have limited math skills so maybe it's helpful?
        
         | fsndz wrote:
         | AI engineers just use "vibes" currently haha
        
         | godelski wrote:
         | As an ML researcher who started in physics (this seems common
         | among physics/math turned ML people. Which Evan is included), I
         | cannot tell you how bad is it... One year at CVPR when
         | diffusion models hit the scenes I was asking what people's
         | covariance was (I had overestimated the model complexity), and
         | the most common answer I got was "how do I calculate that?"
         | People do not understand things like what "pdf" means. People
         | at top schools! I've been told I'm "gatekeeping" for saying
         | that you should learn math (I say "you don't need math to build
         | good models, but you do to understand why they're wrong"). Not
         | that you need to, but should. (I guess this explains why
         | Mission Impossible Language Models won best paper...)
         | 
         | I swear, the big reason models are black boxes are because we
         | _want_ them to be. There's clear anti-sentiment mentality
         | against people doing theory and the result of this shows. I
         | remember not too long ago Yi Tay (under @agihippo but main is
         | @YiTayML) said "fuck theorists". I guess it's not a surprise
         | Deep Mind recently hired him after that "get good" stuff.
         | 
         | Also, I'd like to point out, the author uses "we" but the paper
         | only has one author on it. So may I suggest adding their cat as
         | a coauthor? [0]
         | 
         | [0] https://en.wikipedia.org/wiki/F._D._C._Willard
        
           | canjobear wrote:
           | What's your objection to Mission Impossible Language Models?
        
         | runeblaze wrote:
         | Maths also mean different things. Your average number theorist
         | or algebraic geometer will most likely not touch statistical
         | techniques day-to-day. Reading this Anthropic article was
         | helpful because I am constantly catching up on my statistical
         | background
        
       ___________________________________________________________________
       (page generated 2024-11-29 23:00 UTC)