[HN Gopher] StreamingLLM: Efficient streaming technique enable i...
       ___________________________________________________________________
        
       StreamingLLM: Efficient streaming technique enable infinite
       sequence lengths
        
       Author : TheJCDenton
       Score  : 13 points
       Date   : 2023-10-03 20:10 UTC (2 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | TheJCDenton wrote:
       | The demo [3] seems very promising.
       | 
       | "Their method cleverly exploits the LLMs' tendency to use initial
       | tokens as "attention sinks" to anchor the distribution of
       | attention scores. By caching initial tokens alongside recent
       | ones, StreamingLLM restored perplexity and achieved up to 22x
       | faster decoding than prior techniques." [1]
       | 
       | "We show that StreamingLLM can enable Llama-2, MPT, Falcon, and
       | Pythia to perform stable and efficient language modeling with up
       | to 4 million tokens and more." [2]
       | 
       | "we discover that adding a placeholder token as a dedicated
       | attention sink during pre-training can further improve streaming
       | deployment." [2]
       | 
       | "StreamingLLM achieves an impressive speedup, reaching up to
       | 22.2x per token. Despite its reduced latency, StreamingLLM
       | sustains a memory footprint consistent with the re-computation
       | baseline." [2]
       | 
       | [1] https://notes.aimodels.fyi/llm-infinite-context-window-
       | strea...
       | 
       | [2] https://arxiv.org/pdf/2309.17453.pdf
       | 
       | [3] https://github.com/mit-han-lab/streaming-llm
        
       | firebaze wrote:
       | See also https://arxiv.org/abs/2309.17453, or
       | https://news.ycombinator.com/item?id=37740932
        
       ___________________________________________________________________
       (page generated 2023-10-03 23:01 UTC)