[HN Gopher] Outlier Weighed Layerwise Sparsity: A Missing Secret...
       ___________________________________________________________________
        
       Outlier Weighed Layerwise Sparsity: A Missing Secret Sauce for
       Pruning LLMs
        
       Author : amilios
       Score  : 4 points
       Date   : 2023-10-10 20:04 UTC (2 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | amilios wrote:
       | Paper abstract: Large Language Models (LLMs), renowned for their
       | remarkable performance across diverse domains, present a
       | challenge due to their colossal model size when it comes to
       | practical deployment. In response to this challenge, efforts have
       | been directed toward the application of traditional network
       | pruning techniques to LLMs, uncovering a massive number of
       | parameters can be pruned in one-shot without hurting performance.
       | Building upon insights gained from pre-LLM models, particularly
       | BERT-level language models, prevailing LLM pruning strategies
       | have consistently adhered to the practice of uniformly pruning
       | all layers at equivalent sparsity levels, resulting in robust
       | performance. However, this observation stands in contrast to the
       | prevailing trends observed in the field of vision models, where
       | non-uniform layerwise sparsity typically yields substantially
       | improved results. To elucidate the underlying reasons for this
       | disparity, we conduct a comprehensive analysis of the
       | distribution of token features within LLMs. In doing so, we
       | discover a strong correlation with the emergence of outliers,
       | defined as features exhibiting significantly greater magnitudes
       | compared to their counterparts in feature dimen- sions. Inspired
       | by this finding, we introduce a novel LLM pruning methodology
       | that incorporates a tailored set of non-uniform layerwise
       | sparsity ratios specif- ically designed for LLM pruning, termed
       | as Outlier Weighed Layerwise sparsity (OWL). The sparsity ratio
       | of OWL is directly proportional to the outlier ratio observed
       | within each layer, facilitating a more effective alignment
       | between layer- wise weight sparsity and outlier ratios. Our
       | empirical evaluation, conducted across the LLaMA-V1 family and
       | OPT, spanning various benchmarks, demonstrates the distinct
       | advantages offered by OWL over previous methods. For instance,
       | our approach exhibits a remarkable performance gain, surpassing
       | the state-of-the-art Wanda and SparseGPT by 61.22 and 6.80
       | perplexity at a high sparsity level of 70%, respectively. Codes
       | are available at https://github.com/luuyin/OWL.
        
       ___________________________________________________________________
       (page generated 2023-10-10 23:02 UTC)