[HN Gopher] Experiments with Bitnet 1.5 (Ngmi)
       ___________________________________________________________________
        
       Experiments with Bitnet 1.5 (Ngmi)
        
       Author : loser777
       Score  : 7 points
       Date   : 2024-03-23 20:00 UTC (3 hours ago)
        
 (HTM) web link (huggingface.co)
 (TXT) w3m dump (huggingface.co)
        
       | xiphias2 wrote:
       | NVIDIA was confident enough to come out with FP4 cards for
       | training, so 16 bit floats are not the best that can be done.
       | 
       | As most hardware speedups over the years came from decreasing
       | precision, I'm sure NVIDIA tries everything to make FP4 work (and
       | then get rid of FP8 multiplies if it can get away with it).
        
       | kken wrote:
       | Nvidia went from FP8 to FP4 in their charts to embellish the
       | gains in Flops between the previous generation and their B100.
       | They will certainly not be able to resist another sweet
       | performance gain from a new number format.
       | 
       | However, one has to keep in mind that the 1.5bit models still
       | need high resolution weights during training. (In the paper they
       | used int8).
       | 
       | "ngmi" (Not gonna make it) may be a bit premature...
        
       ___________________________________________________________________
       (page generated 2024-03-23 23:01 UTC)