[HN Gopher] Evals: a framework for evaluating OpenAI models and ...
       ___________________________________________________________________
        
       Evals: a framework for evaluating OpenAI models and a registry of
       benchmarks
        
       Author : tosh
       Score  : 104 points
       Date   : 2023-03-14 17:01 UTC (6 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | piqi wrote:
       | > Evals is a framework for evaluating OpenAI models and an open-
       | source registry of benchmarks.
       | 
       | Is the purpose to know which of the models OpenAI offers is most
       | suitable for your workload/app? Could I use this to know if the
       | cheaper model is sufficient for a particular use-case?
        
       | aliljet wrote:
       | I think this is a clever way for OpenAI to trade access to GPT-4
       | and simultaneously decrease their own intellectual burden for
       | their for-profit models. Here's the offending parts of the
       | README.md:
       | 
       | "For a limited time, we will be granting GPT-4 access to those
       | who contribute high quality evals. Please follow the instructions
       | mentioned above and note that spam or low quality submissions
       | will be ignored!"
        
         | nabakin wrote:
         | I'm surprised they would give it out when millions, if not
         | billions are potentially at stake and following the LLaMA leak
        
           | astrange wrote:
           | There wasn't a leak. You don't need to make it sound like
           | we're in a cyberpunk RPG.
        
             | nabakin wrote:
             | what are you talking about? The LLaMA weights leaked from
             | someone on 4chan.
        
               | smoldesu wrote:
               | Facebook is still directly hosting the model, if they
               | wanted to stop people from downloading it then it
               | wouldn't be on their public CDN.
        
               | astrange wrote:
               | Facebook released them with essentially zero effort to
               | stop anyone from resharing them. I think they let anyone
               | with a .edu email get them. It wasn't hacked or anything.
        
           | Centigonal wrote:
           | Access to the inference API, not model code or weights.
        
             | nabakin wrote:
             | Are you sure? API access is already given to everyone I
             | think
             | 
             | Edit: We have access through ChatGPT+, not the API. Got
             | confused there.
        
               | monkeydust wrote:
               | No, its on waitlist.
        
               | nabakin wrote:
               | Ty
        
             | not-chatgpt wrote:
             | to be fair, with alpaca, access to API is even dangerous
        
       ___________________________________________________________________
       (page generated 2023-03-14 23:02 UTC)