[HN Gopher] InternLM2
       ___________________________________________________________________
        
       InternLM2
        
       Author : milliondreams
       Score  : 127 points
       Date   : 2024-03-31 23:51 UTC (23 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | milliondreams wrote:
       | TLDR; 1. InternLM2 is an open-source Large Language Model that
       | has shown improvements over previous models, particularly in
       | long-context modeling. 2. The model uses a unique approach,
       | combining traditional training with Supervised Fine-Tuning and
       | Conditional Online Reinforcement Learning from Human Feedback. 3.
       | It offers a variety of model sizes and training stages to the
       | community, demonstrating significant advancements in AI research
       | and application.
        
         | jerpint wrote:
         | Excited to see how it will perform on the lmsys leaderboard
        
       | ilaksh wrote:
       | Does anyone know how the free commercial license works? Do they
       | usually grant it? https://wj.qq.com/s2/12727483/5dba/ looks like
       | a form there.
       | 
       | Apache 2 code, free commercial license with application form for
       | weights.
        
       | esha_manideep wrote:
       | Pretty amazing to see training data being discussed more openly
        
         | WiSaGaN wrote:
         | Indeed. I think part of the reason when they are not discussed
         | openly may be that much of the data used is copyrighted, which
         | introduces some legal ambiguities.
        
           | YetAnotherNick wrote:
           | IANAL but hiding something doesn't make someone legally
           | immune. Any company could sue LLM companies and they can't
           | hide it during the case. e.g. there is already a similar case
           | on OpenAI.
        
             | fl0id wrote:
             | Yes, but it at the very least delays any findings while you
             | rake in the cash and try to create a favorable environment.
             | OpenAI even stated that think using copyrighted texts is
             | necessary and should be covered by fair use.
        
       | dannyw wrote:
       | How good is the base (non-instruction-tuned) model? Everyone is
       | trying to make chat bots, but for my use cases, I find base
       | models more suitable.
        
         | fragmede wrote:
         | Interesting. What are some of those use cases?
        
       | viraptor wrote:
       | The repo is here: https://github.com/InternLM/InternLM
        
       | zone411 wrote:
       | We really need better long context benchmarks than needle-in-a-
       | haystack. There is LV-Eval (https://arxiv.org/abs/2402.05136)
       | with multi-hop QA that's better but still pretty basic.
        
         | andersa wrote:
         | Yes, I don't understand why they are using a _search_ benchmark
         | for these... it would be much better to have something like
         | giving it a story up to the context length (from a book? how to
         | find one that 's not in the training data?) and have it write a
         | new chapter/ending that is consistent with _all_ prior text and
         | introduces zero inconsistencies.
         | 
         | But how can you automatically evaluate whether it did this?
        
           | ricw wrote:
           | Because that's how people use llms. You go to ChatGPT to ask
           | a question and get an answer, rather than searching on
           | Google, revising your search because you didn't know a term,
           | and then look at 3-5 different links to find the answer to
           | what you were searching for
        
             | loa_in_ wrote:
             | That's how people use llms because they (llms) don't seem
             | to be good at the more sophisticated thing
        
               | sp332 wrote:
               | But also because search engines have gotten even worse at
               | answering questions.
        
             | zone411 wrote:
             | It seems you might be mixing up different types of
             | "context" in LLM benchmarking. In this case, it refers to
             | the input text directly provided to the model during
             | evaluation by the user (as in in-context learning). This is
             | separate from the text an LLM is trained on or can access
             | via RAG methods.
        
       | barsonme wrote:
       | Is it normal for papers to have that many authors?
        
         | esafak wrote:
         | Only in fields like foundation models and high-energy physics,
         | where immense resources are required. Look at GPT-4's credits:
         | https://openai.com/contributions/gpt-4
        
         | barkingcat wrote:
         | It's not abnormal in many fields. A lot of biology or physics
         | papers have more than that number.
         | 
         | In academic labs, so you're a postgrad working the overnight
         | shift watching some petri dish to make sure the bacteria
         | doesn't die, etc. It's super boring grunt work but you do it so
         | you get on the paper's author list.
        
         | exe34 wrote:
         | Particle physics papers usually have more pages for the names
         | than for the work.
        
         | m3kw9 wrote:
         | Anyone that even sniffed it can get in on the action
        
       | pilotneko wrote:
       | I experimented with this model and vLLM around a month ago. The
       | long context length is attractive, but it was incredibly slow on
       | a g5.12xlarge (4 NVIDIA A10G GPUs). I actually could not get it
       | to respond for single examples longer than 50K tokens.
        
       | Kwpolska wrote:
       | The name suggests this is interns posing as a chatbot, especially
       | considering today's date.
        
       ___________________________________________________________________
       (page generated 2024-04-01 23:02 UTC)