[HN Gopher] HyperAttention: Long-Context Attention in Near-Linea...
       ___________________________________________________________________
        
       HyperAttention: Long-Context Attention in Near-Linear Time[pdf]
        
       Author : kelseyfrog
       Score  : 51 points
       Date   : 2023-10-10 14:31 UTC (8 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | akomtu wrote:
       | ML researchers are playing scientists: tweak a few parameters in
       | an LLM, re-train it on a largish dataset (need access to $$$
       | GPUs), find metrics on which the tweaked LLM makes a barely
       | noticeable improvement and make the other metrics where it
       | actually gets worse look insignificant, write a paper, upload to
       | arxiv, and update your resume.
        
         | marcinzm wrote:
         | So true, if they were real scientists then they'd never publish
         | negative results and only pass them along in their network to
         | most efficiently gate keep the field!
        
         | sebzim4500 wrote:
         | I genuinely can't tell if this is supposed to be a criticism.
         | 
         | They tried something, it improved some things and made other
         | things worse.
        
         | timkam wrote:
         | This paper presents formal results, apparently. Also, in the
         | case of formal results, peer review by experts in the exact
         | sub-field makes sense/increases trust, sure, but why not share
         | on archive instead of waiting for a year (or however long the
         | process takes in the particular instance)?
        
         | Tostino wrote:
         | I'd rather the knowledge be out there, it helps you not go down
         | dead end paths that you would otherwise have explored.
        
       | lappa wrote:
       | "For example, HyperAttention makes the inference time of ChatGLM2
       | 50% faster on 32k context length while perplexity increases from
       | 5.6 to 6.3."
       | 
       | "when half of all attention layers are patched (i.e., 14 layers),
       | we verify that most of the tasks do not degrade more than 13%."
       | 
       | According to the paper, for most tasks it reduces benchmark
       | scores substantially. Perhaps to the point where a smaller model
       | would yield better inference time and higher benchmarks.
       | 
       | However, summarization benchmarks see almost no degredation,
       | great!
        
       ___________________________________________________________________
       (page generated 2023-10-10 23:01 UTC)