[HN Gopher] 4-bit floating point FP4
___________________________________________________________________
4-bit floating point FP4
Author : chmaynard
Score : 31 points
Date : 2026-04-18 17:21 UTC (5 hours ago)
(HTM) web link (www.johndcook.com)
(TXT) w3m dump (www.johndcook.com)
| chrisjj wrote:
| > Programmers were grateful for the move from 32-bit floats to
| 64-bit floats. It doesn't hurt to have more precision
|
| Someome didn't try it on GPU...
| kimixa wrote:
| Even the latest CPUs have a 2:1 fp64:fp32 performance ratio -
| plus the effects of 2x the data size in cache and bandwidth use
| mean you can often get greater than a 2x difference.
|
| If you're in a numeric heavy use case that's a massive
| difference. It's not some outdated "Ancient Lore" that causes
| languages that care about performance to default to fp32 :P
| adgjlsfhk1 wrote:
| > languages that care about performance to default to fp32
|
| What do you mean by this? In C 1.0 is a double.
| pixelesque wrote:
| > Even the latest CPUs have a 2:1 fp64:fp32 performance ratio
|
| Not completely - for basic operations (and ignoring byte size
| for things like cache hit ratios and memory bandwidth) if you
| look at (say Agner Fog's optimisation PDFs of instruction
| latency) the basic SSE/AVX latency for basic add/sub/mult/div
| (yes, even divides these days), the latency between float and
| double is almost always the same on the most recent AMD/Intel
| CPUs (and normally execution ports can do both now).
|
| Where it differs is gather/scatter and some shuffle
| instructions (larger size to work on), and maths routines
| like transcendentals - sqrt(), sin(), etc, where the backing
| algorithms (whether on the processor in some cases or in libm
| or equivalent) obviously have to do more work (often more
| iterations of refinement) to calculate the value to greater
| precision for f64.
| Sharlin wrote:
| Yeah, and even on CPU using doubles is almost unheard of in
| many fields.
| ant6n wrote:
| > In ancient times, floating point numbers were stored in 32
| bits.
|
| I thought in ancient times, floating point numbers used to be 80
| bit. They lived in a funky mini stack on the coprocessor (x87).
| Then one day, somebody came along and standardized those 32 and
| 64 bit floats we still have today.
| _trampeltier wrote:
| 80 bits is just in the processor. Thats why you might a little
| bit different result, depending how you calculated first and
| maybe stored something in the RAM
| convolvatron wrote:
| I was going to reply that just because intel did something
| funny doesn't mean that it was the beginning of the story. but
| it turns out that the release of the 8087 predates the
| ratification of IEEE floats by 2 years. in addition, the
| primary numeric designer for the 8087 was apparently Kahan,
| which means that they were both part of the same design
| process. of course there were other formats predating both of
| these
| indolering wrote:
| The floating point "standard" was basically codifying
| multiple different vendor implementations of the same idea.
| Hence the mess that floating point is not consistent across
| implementations.
| jcranmer wrote:
| IEEE 754 basically had three major proposals that were
| considered for standardization. There was the "KCS draft"
| (Kahan, Coonen, Stone), which was the draft implemented for
| the x87 coprocessor. There was DEC's counter proposal (aka
| the PS draft, for Payne and Strecker), and HP's counter
| proposal (aka, the FW draft for Fraley and Walther).
| Ultimately, it was the KCS draft that won out and become
| what we now know as IEEE 754.
|
| One of the striking things, though, is just how radically
| _different_ KCS was. By the time IEEE 754 forms, there is a
| basic commonality of how floating-point numbers work. Most
| systems have a single-precision and double-precision form,
| and many have an additional extended-precision form. These
| formats are usually radix-2, with a sign bit, a biased
| exponent, and an integer mantissa, and several
| implementations had hit on the implicit integer bit
| representation. (See
| http://www.quadibloc.com/comp/cp0201.htm for a tour of
| several pre-IEEE 754 floating-point formats). What KCS did
| that was _really_ new was add denormals, and this was very
| controversial. I also think that support for infinities was
| introduced with KCS, although there were more precedents
| for the existence of NaN-like values. I 'm also pretty sure
| that sticky bits as opposed to trapping for exceptions was
| considered innovative. (See, e.g., https://ethw-
| images.s3.us-east-va.perf.cloud.ovh.us/ieee/f/f... for a
| discussion of the differences between the early drafts.)
|
| Now, once IEEE 754 came out, pretty much every subsequent
| implementation of floating-point has started from the IEEE
| 754 standard. But it was definitely not a codification of
| existing behavior when it came out, given the number of
| innovations that it had!
| Sharlin wrote:
| x87 always had a choice of 32/64/80-bit user-facing floats. It
| just operated internally on 80 bits.
| burnt-resistor wrote:
| FP4 1:2:0:1 (other examples: binary32 1:8:0:23, 8087 ep
| 1:15:1:63)
|
| S:E:l:M
|
| S = sign bit present (or magnitude-only absolute value)
|
| E = exponent bits (typically biased by 2^(E-1) - 1)
|
| l = explicit leading integer present (almost always 0 because the
| leading digit is always 1 for normals, 0 for denormals, and not
| very useful for special values)
|
| M = mantissa (fraction) bits
|
| The limitations of FP4 are that it lacks infinities, [sq]NaNs,
| and denormals that make it very limited to special purposes only.
| There's no denying that it might be extremely efficient for very
| particular problems.
|
| If a more even distribution were needed, a simpler _fixed point_
| format like 1:2:1 (sign:integer:fraction bits) is possible.
| conaclos wrote:
| There is a relevant Wikipedia page about minifloats [0]
|
| > The smallest possible float size that follows all IEEE
| principles, including normalized numbers, subnormal numbers,
| signed zero, signed infinity, and multiple NaN values, is a 4-bit
| float with 1-bit sign, 2-bit exponent, and 1-bit mantissa.
|
| [0] https://en.wikipedia.org/wiki/Minifloat
| karmakaze wrote:
| There's an "Update:" note for a next post on NF4 format. As far
| as I can tell this is neither NVFP4 nor MXFP4 which are commonly
| used with LLM model files. The thing with these formats is that
| common information is separated in batches so not a singular
| format but a format for groups of values. I'd like to know more
| about these _(but not enough to go research them myself)_.
| sc0ttyd wrote:
| 9 years ago, I shared this as an April Fools joke here on HN.
|
| It seems that life is imitating art.
|
| https://github.com/sdd/ieee754-rrp
| mysterydip wrote:
| I especially like your HQQ precision
| sc0ttyd wrote:
| I think it is only a matter of time before HQQ / 1FP takes
| over. It's the logical conclusion. I hope to be using my
| 96-blade razor by then too
___________________________________________________________________
(page generated 2026-04-18 23:00 UTC)