[HN Gopher] Two sparsities are better than one: Performance of s...
       ___________________________________________________________________
        
       Two sparsities are better than one: Performance of sparse-sparse
       networks
        
       Author : Anon84
       Score  : 42 points
       Date   : 2022-02-11 15:25 UTC (7 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | PaulHoule wrote:
       | Sparsity could really be a good thing but so many times I've
       | tried to use it and walked away disappointed in terms of
       | accuracy.
        
         | Voloskaya wrote:
         | It is already a good thing, but it currently requires a lot of
         | engineering effort to actually get it to work with acceptable
         | quality. It's not something that works out of the box like half
         | precision, or for some models, int8. And to your point, for
         | many production scenarios the ratio of engineering work vs
         | performance gains is maybe not worth it. But for models that
         | are going to handle massive load in inference it is worth it in
         | my experience.
         | 
         | I expect that this will be made much easier in future with
         | better hardware support and smarter sparsification libraries.
        
           | synthos wrote:
           | It's the power savings that are the real goal. If you can
           | create and (re)train a network for incredible sparsity you
           | can fit inference in some pretty low power envelopes. I think
           | to that end the work required to get to adequate perform is
           | justified
        
       | md2020 wrote:
       | Always cool to see research from Numenta. They don't get as much
       | love as DeepMind, Google Brain, and OpenAI because their results
       | aren't as flashy, but I do feel like they've got a principled
       | approach to engineering intelligent systems distinct from that of
       | the big players.
        
       | Chinjut wrote:
       | An abundance of sparsities.
        
       ___________________________________________________________________
       (page generated 2022-02-11 23:01 UTC)