https://clemenswinter.com/2024/04/07/the-simple-beauty-of-xor-floating-point-compression/ Clemens' Blog Menu Skip to content * Home * Contact [image-7] The Simple Beauty of XOR Floating Point Compression 5 Replies I recently implemented a small program to visualize the inner workings of a scheme that compresses floating point timeseries by XORing subsequent values. The resulting visualizations are quite neat and made it much easier for me to understand this beautiful algorithm than any of the explanations that I had previously encountered. The algorithm The algorithm^1This particular version of XOR floating point compression was first described in "Gorilla: A Fast, Scalable, In-Memory Time Series Database" and is often referred to as "Gorilla compression. is simple. We start by writing out the first floating point number in full. All subsequent numbers are XORed with the previous number and then encoded in one of three ways: 1. We write a single 0[2] bit. This indicates that the number is identical to the previous number. 2. We write the two bit sequence 11[2]. We write a 5 bit integer that indicates the number of leading zeros in the XORed value. We write a 6 bit integer that indicates the number of bits in the subsequence that starts at the first 1 bit and includes and ends at the last 1 bit. We then write out this subsequence, omitting all leading and trailing zeros. 3. We write the two bit sequence 10[2]. We then immediately write out the subsequence of the XORed value that has the same starting index and length as the last subsequence we wrote. How do we choose between (2) and (3)? The advantage of (3) is that we omit the 5+6=11 bits we would otherwise require for storing the number of leading zeros and length of the sequence, but we can only use (3) if our nonzero subsequence fits within the window defined by the previous offset and length values. The Gorilla paper proposes a simple heuristic: if the previously used subsequence of bits includes all nonzero bits, we go with (3) and omit writing out the offset and length bits. If not, we go with (2) and write out a new offset and length. Note that this heuristic minimizes the amount times we have to write out the 11 bits of offset/length values, but doesn't necessarily give optimal compression since we might be writing much longer subsequences on subsequent values than necessary (more on that later). While the algorithm is simple, its not immediately obvious from its description why it should lead to good compression. Case (1) is easy, repeated values are compressed to a single bit. But why does slicing off leading and trailing zeros on the XOR of subsequent floating point values work well? Let's apply the algorithm to some real data and see what it looks like! expandable image [image-3] scrollable text DECIMAL BITS IN XOR BITS OUT 0.007859230041503906 001111111000 0000000110001000000000000000000000000000000000000000 001111111000 0000000110001000000000000000000000000000000000000000 0.0076792240142822266 001111110111 1111011101000100000000000000000000000000000000000000 00000000 11111111011011001100000000000000000000000000000000000000 1101000 010001111111110110110011 0.008411884307861328 001111111000 0001001110100100000000000000000000000000000000000000 00000000 11111110010011100000000000000000000000000000000000000000 10 111111100100111000 0.00831747055053711 001111111000 0001000010001100000000000000000000000000000000000000 0000000000000000001100101000000000000000000000000000000000000000 10 000000000011001010 0.008234977722167969 001111111000 0000110111011000000000000000000000000000000000000000 000000000000000 1110101010100000000000000000000000000000000000000 10 000000011101010101 0.008572578430175781 001111111000 0001100011101000000000000000000000000000000000000000 000000000000000 1010100110000000000000000000000000000000000000000 10 000000010101001100 0.009616851806640625 001111111000 0011101100100000000000000000000000000000000000000000 00000000000000 10001111001000000000000000000000000000000000000000 10 000000100011110010 0.009622573852539063 001111111000 0011101101010000000000000000000000000000000000000000 0000000000000000000001110000000000000000000000000000000000000000 10 000000000000011100 0.00926518440246582 001111111000 0010111110011010000000000000000000000000000000000000 000000000000000 1010011001010000000000000000000000000000000000000 1101111001011 101001100101 0.009131669998168945 001111111000 0010101100111010000000000000000000000000000000000000 0000000000000000010010100000000000000000000000000000000000000000 10 001001010000 0.008625507354736328 001111111000 0001101010100100000000000000000000000000000000000000 00000000000000 11000110011110000000000000000000000000000000000000 1101110001100 1100011001111 0.008272171020507813 001111111000 0000111100010000000000000000000000000000000000000000 000000000000000 1010110110100000000000000000000000000000000000000 100101011011010 0.008666276931762695 001111111000 0001101111111010000000000000000000000000000000000000 000000000000000 1010011101010000000000000000000000000000000000000 100101001110101 0.009252548217773438 001111111000 0010111100110000000000000000000000000000000000000000 00000000000000 11010011001010000000000000000000000000000000000000 101101001100101 0.009163856506347656 001111111000 0010110001001000000000000000000000000000000000000000 0000000000000000001101111000000000000000000000000000000000000000 10 0000110111100 0.008917093276977539 001111111000 0010010000110010000000000000000000000000000000000000 0000000000000000 100001111010000000000000000000000000000000000000 100010000111101 0.009442806243896484 001111111000 0011010101101100000000000000000000000000000000000000 000000000000000 1000101011110000000000000000000000000000000000000 100100010101111 0.009443044662475586 001111111000 0011010101101110000000000000000000000000000000000000 0000000000000000000000000010000000000000000000000000000000000000 10 0000000000001 0.009201765060424805 001111111000 0010110110000110000000000000000000000000000000000000 000000000000000 1100011101000000000000000000000000000000000000000 100110001110100 0.009229660034179688 001111111000 0010111001110000000000000000000000000000000000000000 0000000000000000001111110110000000000000000000000000000000000000 10 0000111111011 0.008151531219482422 001111111000 0000101100011100000000000000000000000000000000000000 00000000000000 10010101101100000000000000000000000000000000000000 101001010110110 0.008004188537597656 001111111000 0000011001001000000000000000000000000000000000000000 0000000000000000 110101010100000000000000000000000000000000000000 100011010101010 0.007958173751831055 001111111000 0000010011000110000000000000000000000000000000000000 0000000000000000001010001110000000000000000000000000000000000000 10 0000101000111 0.0069658756256103516 001111110111 1100100010000100000000000000000000000000000000000000 00000000 11111100110001000010000000000000000000000000000000000000 1101000 0100101111110011000100001 0.0044634342193603516 001111110111 0010010010000100000000000000000000000000000000000000 000000000000 1110110000000000000000000000000000000000000000000000 10 0000111011000000000 0.004178524017333984 001111110111 0001000111011000000000000000000000000000000000000000 00000000000000 11010101011100000000000000000000000000000000000000 10 0000001101010101110 0.00484156608581543 001111110111 0011110101001100000000000000000000000000000000000000 00000000000000 10110010010100000000000000000000000000000000000000 10 0000001011001001010 0.003814220428466797 001111110110 1111001111110000000000000000000000000000000000000000 00000000000 11100111010111100000000000000000000000000000000000000 10 0001110011101011110 chart [image-4] The first column shows the decimal representation of the timeseries we are compressing. This particular timeseries happens to measure the time spent in a particular code section of enn-trainer on each iteration of a training run. The second column is the 64-bit floating point representation of the same numbers with sign bit in red, exponent in green, and mantissa in blue. Note that the exponent bits are identical for almost all of the numbers in the series, which is expected when values vary by a lot less than 2x. Note also that the numbers have a lot of trailing zero bits, which is not something that's obvious from the decimal representation. It looks like these timings were recorded at microsecond resolution, which means that values smaller than 0.02 all fit into just 15bits. The third column shows the XOR of each value which the previous value, with the region of non-zero bits highlighted in yellow. The XORed values have many leading zeros (because the sign bit and exponent don't change on subsequent values) and many trailing zeros (because the original values already shared a lot of trailing zeros). These regular long runs of leading and trailing zero bits are what makes the compression work! The fourth column shows the bits output by the algorithm. On the first row, we simply have the full bit representation of the first number without any compression. On the second row, we first write the 11 control bits, the number of leading zeros (8[10] = 01000[2]), the length of the nonzero region - 1 (17[10] = 010001[2])^2We subtract 1 from the length beause otherwise we might overflow the 6 bits allocated for the length when all bits are different and the length is 64. This is sound because the length is always at least 1., and the nonzero bits 111111110110110011. On the third row, the region of nonzero bits falls into the same window, so we instead write the 10 control bits and the nonzero subsequence 111111100100111000. Note that this subsequence has three trailing 0s, but these 3 bits still take up less space than the 11 bits we would need to write out a new offset and length. We continue to omit the offset and length until the ninth row, where we reset to a new offset and length since the region of nonzero bits in the XOR does not fit within the previously used window. This entire series (including more values not shown above) is compressed by a factor of 3.03. Even if we compare to just using 32-bit single precision floating point values to represent these numbers, we're still compressing by a factor of 1.51. See here for a simple implementation of both the encoding and decoding algorithm in less than 100 lines of Rust (not including the BitReader/BitWriter structs). It is adapted from https://github.com/ jeromefroe/tsz-rs which has somewhat more complicated code but much better comments. My implementation contains two additional options: lossy compression that masks out some portion of the least significant bits in the mantissa, and a simple tweak described in the next section that can significantly improve compression for some time series. Tweaking the algorithm As we noted in the previous section, the Gorilla compression algorithm uses a particular heuristic for resetting the window size. It greedily minimizes the number of window resets, which works well in practice most of the time, but is not guaranteed to yield maximum compression. In particular, there is one pathological failure mode: If a series contains an outlier value that requires a very large window to encode, and all subsequent values have nonzero values only in a much smaller subwindow, the inefficient larger window will get locked in because we never hit the condition that would trigger a reset of the window size. Here's an example of this happening: image [image-5] text DECIMAL BITS IN XOR BITS OUT 0 0000000000000000000000000000000000000000000000000000000000000000 0 000000000000000000000000000000000000000000000000000000000000000 26.33333396911621 010000000011 1010010101010101010101100000000000000000000000000000 0 100000000111010010101010101010101100000000000000000000000000000 11 000011000011000000001110100101010101010101011 42.69767379760742 010000000100 0101010110010100110101100000000000000000000000000000 000000000 1111111000011000001100000000000000000000000000000000000 10 0000000011111110000110000011000000 56.29090881347656 010000000100 1100001001010011110010000000000000000000000000000000 000000000000 1001011111000111000111100000000000000000000000000000 10 0000000000010010111110001110001111 70.9795913696289 010000000101 0001101111101011000110100000000000000000000000000000 00000000000 11101100110111000110100100000000000000000000000000000 10 0000000000111011001101110001101001 77.60713958740234 010000000101 0011011001101101101101100000000000000000000000000000 00000000000000 10110110000110101011000000000000000000000000000000 10 0000000000000101101100001101010110 73.88888549804688 010000000101 0010011110001110001110000000000000000000000000000000 000000000000000 1000111100011100011100000000000000000000000000000 10 0000000000000010001111000111000111 82.1276626586914 010000000101 0100100010000010101110100000000000000000000000000000 0000000000000 110111100001100100000100000000000000000000000000000 10 0000000000001101111000011001000001 82.46154022216797 010000000101 0100100111011000100111100000000000000000000000000000 0000000000000000000101011010001001000000000000000000000000000000 10 0000000000000000001010110100010010 78.86000061035156 010000000101 0011101101110000101001000000000000000000000000000000 0000000000000 111001010101000001110100000000000000000000000000000 10 0000000000001110010101010000011101 85.75609588623047 010000000101 0101011100000110001111100000000000000000000000000000 0000000000000 110110001110110100110100000000000000000000000000000 10 0000000000001101100011101101001101 80.33999633789063 010000000101 0100000101011100001010000000000000000000000000000000 000000000000000 1011001011010000101100000000000000000000000000000 10 0000000000000010110010110100001011 77.66666412353516 010000000101 0011011010101010101010100000000000000000000000000000 0000000000000 111011111110110100000100000000000000000000000000000 10 0000000000001110111111101101000001 87.82353210449219 010000000101 0101111101001011010011000000000000000000000000000000 0000000000000 110100111100001111001100000000000000000000000000000 10 0000000000001101001111000011110011 70.93478393554688 010000000101 0001101110111101001110000000000000000000000000000000 0000000000000 100010011110110011101000000000000000000000000000000 10 0000000000001000100111101100111010 87.77083587646484 010000000101 0101111100010101010101100000000000000000000000000000 0000000000000 100010010101000011011100000000000000000000000000000 10 0000000000001000100101010000110111 80.75 010000000101 0100001100000000000000000000000000000000000000000000 000000000000000 1110000010101010101100000000000000000000000000000 10 0000000000000011100000101010101011 76.04255676269531 010000000101 0011000000101011100101000000000000000000000000000000 0000000000000 111001100101011100101000000000000000000000000000000 10 0000000000001110011001010111001010 80.05000305175781 010000000101 0100000000110011001101000000000000000000000000000000 0000000000000 111000000011000101000000000000000000000000000000000 10 0000000000001110000000110001010000 90.20454406738281 010000000101 0110100011010001011101000000000000000000000000000000 00000000000000 10100011100010010000000000000000000000000000000000 10 0000000000000101000111000100100000 82.25 010000000101 0100100100000000000000000000000000000000000000000000 00000000000000 10000111010001011101000000000000000000000000000000 10 0000000000000100001110100010111010 72.01786041259766 010000000101 0010000000010010010010100000000000000000000000000000 0000000000000 110100100010010010010100000000000000000000000000000 10 0000000000001101001000100100100101 76 0100000001010011000000000000000000000000000000000000000000000000 0000000000000001000000010010010010100000000000000000000000000000 10 0000000000000010000000100100100101 80.44000244140625 010000000101 0100000111000010100100000000000000000000000000000000 0000000000000 111000111000010100100000000000000000000000000000000 10 0000000000001110001110000101001000 85.78845977783203 010000000101 0101011100100111011000100000000000000000000000000000 000000000000000 1011011100101111100100000000000000000000000000000 10 0000000000000010110111001011111001 90.89473724365234 010000000101 0110101110010100001101100000000000000000000000000000 00000000000000 11110010110011010101000000000000000000000000000000 10 0000000000000111100101100110101010 91.60465240478516 010000000101 0110111001101011001010100000000000000000000000000000 0000000000000000010111111111000111000000000000000000000000000000 10 0000000000000000101111111110001110 99.51851654052734 010000000101 1000111000010010111101100000000000000000000000000000 000000000000 1110000001111001110111000000000000000000000000000000 10 0000000000011100000011110011101110 chart [image-6] As a simple fix, I added a "regret" counter that keeps track of how many leading and trailing zeros we have encountered in the subsequences the algorithm outputs. If this this sum exceeds some threshold, we force a reset of the window. A "max regret" threshold of 100 seems to work quite well in practice, substantially improving compression for some series while only slightly worsening compression on others. Here is what compression of the same timeseries looks like with "max regret" of 100. image [image-7] text DECIMAL BITS IN XOR BITS OUT 0 0000000000000000000000000000000000000000000000000000000000000000 0 000000000000000000000000000000000000000000000000000000000000000 26.33333396911621 010000000011 1010010101010101010101100000000000000000000000000000 0 100000000111010010101010101010101100000000000000000000000000000 11 000011000011000000001110100101010101010101011 42.69767379760742 010000000100 0101010110010100110101100000000000000000000000000000 000000000 1111111000011000001100000000000000000000000000000000000 10 0000000011111110000110000011000000 56.29090881347656 010000000100 1100001001010011110010000000000000000000000000000000 000000000000 1001011111000111000111100000000000000000000000000000 10 0000000000010010111110001110001111 70.9795913696289 010000000101 0001101111101011000110100000000000000000000000000000 00000000000 11101100110111000110100100000000000000000000000000000 10 0000000000111011001101110001101001 77.60713958740234 010000000101 0011011001101101101101100000000000000000000000000000 00000000000000 10110110000110101011000000000000000000000000000000 10 0000000000000101101100001101010110 73.88888549804688 010000000101 0010011110001110001110000000000000000000000000000000 000000000000000 1000111100011100011100000000000000000000000000000 10 0000000000000010001111000111000111 82.1276626586914 010000000101 0100100010000010101110100000000000000000000000000000 0000000000000 110111100001100100000100000000000000000000000000000 10 0000000000001101111000011001000001 82.46154022216797 010000000101 0100100111011000100111100000000000000000000000000000 0000000000000000000101011010001001000000000000000000000000000000 10 0000000000000000001010110100010010 78.86000061035156 010000000101 0011101101110000101001000000000000000000000000000000 0000000000000 111001010101000001110100000000000000000000000000000 10 0000000000001110010101010000011101 85.75609588623047 010000000101 0101011100000110001111100000000000000000000000000000 0000000000000 110110001110110100110100000000000000000000000000000 1101101010101 1101100011101101001101 80.33999633789063 010000000101 0100000101011100001010000000000000000000000000000000 000000000000000 1011001011010000101100000000000000000000000000000 10 0010110010110100001011 77.66666412353516 010000000101 0011011010101010101010100000000000000000000000000000 0000000000000 111011111110110100000100000000000000000000000000000 10 1110111111101101000001 87.82353210449219 010000000101 0101111101001011010011000000000000000000000000000000 0000000000000 110100111100001111001100000000000000000000000000000 10 1101001111000011110011 70.93478393554688 010000000101 0001101110111101001110000000000000000000000000000000 0000000000000 100010011110110011101000000000000000000000000000000 10 1000100111101100111010 87.77083587646484 010000000101 0101111100010101010101100000000000000000000000000000 0000000000000 100010010101000011011100000000000000000000000000000 10 1000100101010000110111 80.75 010000000101 0100001100000000000000000000000000000000000000000000 000000000000000 1110000010101010101100000000000000000000000000000 10 0011100000101010101011 76.04255676269531 010000000101 0011000000101011100101000000000000000000000000000000 0000000000000 111001100101011100101000000000000000000000000000000 10 1110011001010111001010 80.05000305175781 010000000101 0100000000110011001101000000000000000000000000000000 0000000000000 111000000011000101000000000000000000000000000000000 10 1110000000110001010000 90.20454406738281 010000000101 0110100011010001011101000000000000000000000000000000 00000000000000 10100011100010010000000000000000000000000000000000 10 0101000111000100100000 82.25 010000000101 0100100100000000000000000000000000000000000000000000 00000000000000 10000111010001011101000000000000000000000000000000 10 0100001110100010111010 72.01786041259766 010000000101 0010000000010010010010100000000000000000000000000000 0000000000000 110100100010010010010100000000000000000000000000000 10 1101001000100100100101 76 0100000001010011000000000000000000000000000000000000000000000000 0000000000000001000000010010010010100000000000000000000000000000 10 0010000000100100100101 80.44000244140625 010000000101 0100000111000010100100000000000000000000000000000000 0000000000000 111000111000010100100000000000000000000000000000000 10 1110001110000101001000 85.78845977783203 010000000101 0101011100100111011000100000000000000000000000000000 000000000000000 1011011100101111100100000000000000000000000000000 10 0010110111001011111001 90.89473724365234 010000000101 0110101110010100001101100000000000000000000000000000 00000000000000 11110010110011010101000000000000000000000000000000 10 0111100101100110101010 91.60465240478516 010000000101 0110111001101011001010100000000000000000000000000000 0000000000000000010111111111000111000000000000000000000000000000 10 0000101111111110001110 99.51851654052734 010000000101 1000111000010010111101100000000000000000000000000000 000000000000 1110000001111001110111000000000000000000000000000000 1101100010101 1110000001111001110111 86.34782409667969 010000000101 0101100101100100001011000000000000000000000000000000 000000000000 1101011101110110110110100000000000000000000000000000 1101100010110 11010111011101101101101 81.20833587646484 010000000101 0100010011010101010101100000000000000000000000000000 000000000000000 1110110110001011110100000000000000000000000000000 10 00011101101100010111101 88.29729461669922 010000000101 0110000100110000011011100000000000000000000000000000 00000000000000 10010111100101001110000000000000000000000000000000 10 00100101111001010011100 92.60416412353516 010000000101 0111001001101010101010100000000000000000000000000000 000000000000000 1001101011010110001000000000000000000000000000000 10 00010011010110101100010 chart [image-6] I haven't given this too much thought so there's a good chance there exist better and more principled heuristics (EDIT: Hacker News commenter EdSchouten has a neat approach). It would also be interesting to figure out what the most efficient algorithm is for finding the optimal choices for when to reset the window and to what range when shown the entire timeseries. This would be useful in cases where you care much more about minimizing compressed size rather than compression throughput (decompression could still use the same algorithm and would have similar throughput as with the heuristic compression). Performance My naive implementation achieves a throughput of more than 100MiB/s for encoding and 200MiB/s for decoding random floats on an i7-10875H (more compressible data achieves more than 200MiB/s encode, 350MiB/s decode). Not amazing, but pretty decent for such little code, and still a lot faster than most network connections (my main use case for this was reducing the size of network transfers). There's a SIMD Rust implementation that achieves multiple GB/s, though it seems to use unstable features that need some updating to make it compatible the most recent version of Rust. Profiling the naive implementation reveals that it spends most if its time in the BitWriter/BitReader methods. [image-25] Swapping these out for bitbuffer and adding some other small optimizations gets us to 370MiB/s encode, 1GiB/s decode. I tried bitter as well, an early (broken) prototype that didn't correctly handle reads with more than 56 bits got to > 2GiB/s decode even in auto mode and a (probably correct) version was still something like 30% faster than my initial bitbuffer implementation at > 1GiB/s. More examples Finally, here's a collection of different timeseries for your viewing pleasure. Compression for all of these uses the default "max regret" threshold of 100. You can reproduce these by running cargo run --example gorilla_time -- --verbose at 4cfbda210. All the timeseries here contain 246 values in total, only the first few are shown. Low-bit floating point Values in this time series are mean values calculated by dividing a small integer with a power of two, which results in very few significant bits (5.68x compression). image [image-8] text DECIMAL BITS IN XOR BITS OUT 0.061279296875 001111111010 1111011000000000000000000000000000000000000000000000 001111111010 1111011000000000000000000000000000000000000000000000 0.060546875 001111111010 1111000000000000000000000000000000000000000000000000 0000000000000000011000000000000000000000000000000000000000000000 11 1000100000111 0.072265625 001111111011 0010100000000000000000000000000000000000000000000000 00000000000 11101100000000000000000000000000000000000000000000000 1101011000101 111011 0.060791015625 001111111010 1111001000000000000000000000000000000000000000000000 00000000000 11101101000000000000000000000000000000000000000000000 1101011000111 11101101 0.06103515625 001111111010 1111010000000000000000000000000000000000000000000000 0000000000000000011000000000000000000000000000000000000000000000 10 00000011 0.066650390625 001111111011 0001000100000000000000000000000000000000000000000000 00000000000 11110010100000000000000000000000000000000000000000000 1101011001000 111100101 0.056884765625 001111111010 1101001000000000000000000000000000000000000000000000 00000000000 11100001100000000000000000000000000000000000000000000 10111000011 0.053955078125 001111111010 1011101000000000000000000000000000000000000000000000 0000000000000 110100000000000000000000000000000000000000000000000 10001101000 0.051513671875 001111111010 1010011000000000000000000000000000000000000000000000 000000000000000 1110000000000000000000000000000000000000000000000 10000011100 0.062744140625 001111111011 0000000100000000000000000000000000000000000000000000 00000000000 11010011100000000000000000000000000000000000000000000 10110100111 0.0556640625 001111111010 1100100000000000000000000000000000000000000000000000 00000000000 11100100100000000000000000000000000000000000000000000 10111001001 0.063232421875 001111111011 0000001100000000000000000000000000000000000000000000 00000000000 11100101100000000000000000000000000000000000000000000 10111001011 0.07373046875 001111111011 0010111000000000000000000000000000000000000000000000 00000000000000 10110100000000000000000000000000000000000000000000 10000101101 0.060791015625 001111111010 1111001000000000000000000000000000000000000000000000 00000000000 11101110000000000000000000000000000000000000000000000 10111011100 0.062744140625 001111111011 0000000100000000000000000000000000000000000000000000 00000000000 11111001100000000000000000000000000000000000000000000 10111110011 0.06298828125 001111111011 0000001000000000000000000000000000000000000000000000 0000000000000000001100000000000000000000000000000000000000000000 10 000000011 0.082275390625 001111111011 0101000100000000000000000000000000000000000000000000 0000000000000 101001100000000000000000000000000000000000000000000 10001010011 0.069580078125 001111111011 0001110100000000000000000000000000000000000000000000 0000000000000 100110000000000000000000000000000000000000000000000 10001001100 0.070556640625 001111111011 0010000100000000000000000000000000000000000000000000 00000000000000 11110000000000000000000000000000000000000000000000 10000111100 0.064208984375 001111111011 0000011100000000000000000000000000000000000000000000 00000000000000 10011000000000000000000000000000000000000000000000 10000100110 0.059814453125 001111111010 1110101000000000000000000000000000000000000000000000 00000000000 11110110100000000000000000000000000000000000000000000 10111101101 0.052490234375 001111111010 1010111000000000000000000000000000000000000000000000 0000000000000 100010000000000000000000000000000000000000000000000 10001000100 0.04345703125 001111111010 0110010000000000000000000000000000000000000000000000 000000000000 1100101000000000000000000000000000000000000000000000 10011001010 0.052978515625 001111111010 1011001000000000000000000000000000000000000000000000 000000000000 1101011000000000000000000000000000000000000000000000 10011010110 0.0546875 001111111010 1100000000000000000000000000000000000000000000000000 0000000000000 111001000000000000000000000000000000000000000000000 10001110010 chart [image-9] Highly redundant integers Values in this series report the average throughput since program start rounded to an integer, which causes the timeseries to mostly converge to a single value as time goes on (13.5x compression). image [image-10] text DECIMAL BITS IN XOR BITS OUT 1682 0100000010011010010010000000000000000000000000000000000000000000 0100000010011010010010000000000000000000000000000000000000000000 1617 0100000010011001010001000000000000000000000000000000000000000000 0000000000000011000011000000000000000000000000000000000000000000 11 0111000011111000011 1580 0100000010011000101100000000000000000000000000000000000000000000 0000000000000001111101000000000000000000000000000000000000000000 10 01111101 1546 0100000010011000001010000000000000000000000000000000000000000000 0000000000000000100110000000000000000000000000000000000000000000 10 00100110 1518 0100000010010111101110000000000000000000000000000000000000000000 0000000000001111100100000000000000000000000000000000000000000000 11 0110000011111111001 1494 0100000010010111010110000000000000000000000000000000000000000000 0000000000000000111000000000000000000000000000000000000000000000 10 00001110 1487 0100000010010111001111000000000000000000000000000000000000000000 0000000000000000011001000000000000000000000000000000000000000000 11 1000100010011001 1488 0100000010010111010000000000000000000000000000000000000000000000 0000000000000000011111000000000000000000000000000000000000000000 10 11111 1478 0100000010010111000110000000000000000000000000000000000000000000 0000000000000000010110000000000000000000000000000000000000000000 10 10110 1470 0100000010010110111110000000000000000000000000000000000000000000 0000000000000001111000000000000000000000000000000000000000000000 11 011110000111111 1471 0100000010010110111111000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 11 101010000001 1469 0100000010010110111101000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 11 101000000001 1464 0100000010010110111000000000000000000000000000000000000000000000 0000000000000000000101000000000000000000000000000000000000000000 11 10011000010101 1461 0100000010010110110101000000000000000000000000000000000000000000 0000000000000000001101000000000000000000000000000000000000000000 11 100100000111101 1456 0100000010010110110000000000000000000000000000000000000000000000 0000000000000000000101000000000000000000000000000000000000000000 10 0101 1447 0100000010010110100111000000000000000000000000000000000000000000 0000000000000000010111000000000000000000000000000000000000000000 11 1000100010010111 1447 0100000010010110100111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1442 0100000010010110100010000000000000000000000000000000000000000000 0000000000000000000101000000000000000000000000000000000000000000 10 00101 1440 0100000010010110100000000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 10 00010 1437 0100000010010110011101000000000000000000000000000000000000000000 0000000000000000111101000000000000000000000000000000000000000000 11 10000000101111101 1380 0100000010010101100100000000000000000000000000000000000000000000 0000000000000011111001000000000000000000000000000000000000000000 11 0111000011111111001 1382 0100000010010101100110000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 10 00000010 1384 0100000010010101101000000000000000000000000000000000000000000000 0000000000000000001110000000000000000000000000000000000000000000 10 00001110 1385 0100000010010101101001000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00000001 1385 0100000010010101101001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1384 0100000010010101101000000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00000001 1387 0100000010010101101011000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00000011 1390 0100000010010101101110000000000000000000000000000000000000000000 0000000000000000000101000000000000000000000000000000000000000000 10 00000101 1395 0100000010010101110011000000000000000000000000000000000000000000 0000000000000000011101000000000000000000000000000000000000000000 10 00011101 1398 0100000010010101110110000000000000000000000000000000000000000000 0000000000000000000101000000000000000000000000000000000000000000 10 00000101 1401 0100000010010101111001000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 00001111 1402 0100000010010101111010000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00000011 1405 0100000010010101111101000000000000000000000000000000000000000000 0000000000000000000111000000000000000000000000000000000000000000 10 00000111 1408 0100000010010110000000000000000000000000000000000000000000000000 0000000000000011111101000000000000000000000000000000000000000000 10 11111101 1411 0100000010010110000011000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00000011 1412 0100000010010110000100000000000000000000000000000000000000000000 0000000000000000000111000000000000000000000000000000000000000000 10 00000111 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 10 00000010 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00000001 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000001110000000000000000000000000000000000000000000 10 00001110 1420 0100000010010110001100000000000000000000000000000000000000000000 0000000000000000000101000000000000000000000000000000000000000000 10 00000101 1421 0100000010010110001101000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00000001 1421 0100000010010110001101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1421 0100000010010110001101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1423 0100000010010110001111000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 11 101000000001 1425 0100000010010110010001000000000000000000000000000000000000000000 0000000000000000011110000000000000000000000000000000000000000000 11 100010000111111 1425 0100000010010110010001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1427 0100000010010110010011000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 10 0001 1427 0100000010010110010011000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000111000000000000000000000000000000000000000000 11 10011000010111 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 001 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 011 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 011 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 011 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 011 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 001 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 001 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 011 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 001 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 001 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 001 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 11 100100000111111 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1433 0100000010010110011001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1432 0100000010010110011000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 1111 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1431 0100000010010110010111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1430 0100000010010110010110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 10 0010 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1429 0100000010010110010101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1428 0100000010010110010100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1427 0100000010010110010011000000000000000000000000000000000000000000 0000000000000000000111000000000000000000000000000000000000000000 10 0111 1427 0100000010010110010011000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1427 0100000010010110010011000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1426 0100000010010110010010000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1426 0100000010010110010010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1425 0100000010010110010001000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 0011 1424 0100000010010110010000000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 0001 1423 0100000010010110001111000000000000000000000000000000000000000000 0000000000000000011111000000000000000000000000000000000000000000 11 1000100010011111 1423 0100000010010110001111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1423 0100000010010110001111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1421 0100000010010110001101000000000000000000000000000000000000000000 0000000000000000000010000000000000000000000000000000000000000000 10 00010 1421 0100000010010110001101000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1420 0100000010010110001100000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1420 0100000010010110001100000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1419 0100000010010110001011000000000000000000000000000000000000000000 0000000000000000000111000000000000000000000000000000000000000000 10 00111 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1414 0100000010010110000110000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1418 0100000010010110001010000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000011000000000000000000000000000000000000000000 10 00011 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1417 0100000010010110001001000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000001000000000000000000000000000000000000000000 10 00001 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1416 0100000010010110001000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000001111000000000000000000000000000000000000000000 10 01111 1415 0100000010010110000111000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 chart [image-11] Incrementing integers Integer that increments by 1 at every timestep. Treating this as an integer rather than float and using double delta compression would be much more effective, but the XOR algorithm still manages to reduce this to just over one byte per data point (6.98x compression). image [image-12] text DECIMAL BITS IN XOR BITS OUT 1 0011111111110000000000000000000000000000000000000000000000000000 0 011111111110000000000000000000000000000000000000000000000000000 2 0100000000000000000000000000000000000000000000000000000000000000 0 111111111110000000000000000000000000000000000000000000000000000 11 0000100101011111111111 3 0100000000001000000000000000000000000000000000000000000000000000 0000000000001000000000000000000000000000000000000000000000000000 11 011000000001 4 0100000000010000000000000000000000000000000000000000000000000000 0000000000011000000000000000000000000000000000000000000000000000 11 0101100000111 5 0100000000010100000000000000000000000000000000000000000000000000 0000000000000100000000000000000000000000000000000000000000000000 11 011010000001 6 0100000000011000000000000000000000000000000000000000000000000000 0000000000001100000000000000000000000000000000000000000000000000 11 0110000000111 7 0100000000011100000000000000000000000000000000000000000000000000 0000000000000100000000000000000000000000000000000000000000000000 1001 8 0100000000100000000000000000000000000000000000000000000000000000 0000000000111100000000000000000000000000000000000000000000000000 11 010100000111111 9 0100000000100010000000000000000000000000000000000000000000000000 0000000000000010000000000000000000000000000000000000000000000000 11 011100000001 10 0100000000100100000000000000000000000000000000000000000000000000 0000000000000110000000000000000000000000000000000000000000000000 11 0110100000111 11 0100000000100110000000000000000000000000000000000000000000000000 0000000000000010000000000000000000000000000000000000000000000000 1001 12 0100000000101000000000000000000000000000000000000000000000000000 0000000000001110000000000000000000000000000000000000000000000000 11 01100000010111 13 0100000000101010000000000000000000000000000000000000000000000000 0000000000000010000000000000000000000000000000000000000000000000 10 001 14 0100000000101100000000000000000000000000000000000000000000000000 0000000000000110000000000000000000000000000000000000000000000000 10 011 15 0100000000101110000000000000000000000000000000000000000000000000 0000000000000010000000000000000000000000000000000000000000000000 10 001 16 0100000000110000000000000000000000000000000000000000000000000000 0000000000011110000000000000000000000000000000000000000000000000 11 010110000111111 17 0100000000110001000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 11 011110000001 18 0100000000110010000000000000000000000000000000000000000000000000 0000000000000011000000000000000000000000000000000000000000000000 11 0111000000111 19 0100000000110011000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 1001 20 0100000000110100000000000000000000000000000000000000000000000000 0000000000000111000000000000000000000000000000000000000000000000 11 01101000010111 21 0100000000110101000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 10 001 22 0100000000110110000000000000000000000000000000000000000000000000 0000000000000011000000000000000000000000000000000000000000000000 10 011 23 0100000000110111000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 10 001 24 0100000000111000000000000000000000000000000000000000000000000000 0000000000001111000000000000000000000000000000000000000000000000 11 011000000111111 25 0100000000111001000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 10 0001 26 0100000000111010000000000000000000000000000000000000000000000000 0000000000000011000000000000000000000000000000000000000000000000 10 0011 27 0100000000111011000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 10 0001 28 0100000000111100000000000000000000000000000000000000000000000000 0000000000000111000000000000000000000000000000000000000000000000 10 0111 29 0100000000111101000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 10 0001 30 0100000000111110000000000000000000000000000000000000000000000000 0000000000000011000000000000000000000000000000000000000000000000 10 0011 31 0100000000111111000000000000000000000000000000000000000000000000 0000000000000001000000000000000000000000000000000000000000000000 10 0001 32 0100000001000000000000000000000000000000000000000000000000000000 0000000001111111000000000000000000000000000000000000000000000000 11 010010001101111111 chart [image-13] Unix timestamps This series is a sequence of unix timestamps with ~microsecond precision. Our choice of using just 5 bits to store the number of leading 0s fails us here, since this limits us to at most 2^5 - 1 = 31 leading zeros. This series would also benefit from being stored as an integer and double delta compression (1.80x compression). image [image-14] text DECIMAL BITS IN XOR BITS OUT 1652566453.6112177 010000011101 1000101000000000101001101101011001110001111000110001 010000011101 1000101000000000101001101101011001110001111000110001 1652566455.1489727 010000011101 1000101000000000101001101101110010011000100011000101 0000000000000000000000000000000000000000101011101001011011110100 11 111110111100000000001010111010010110111101 1652566456.750367 010000011101 1000101000000000101001101110001100000000011000000011 0000000000000000000000000000000000000011111110011000111011000110 11 1111101111100000001111111001100011101100011 1652566458.3119035 010000011101 1000101000000000101001101110100100111111011000111010 0000000000000000000000000000000000000000101000111111000000111001 11 11111100000000000000101000111111000000111001 1652566459.9219267 010000011101 1000101000000000101001101110111110110000000011011001 0000000000000000000000000000000000000000011010001111011011100011 10 000000000011010001111011011100011 1652566461.561778 010000011101 1000101000000000101001101111011000111111010000101100 0000000000000000000000000000000000000001100110001111010011110101 10 000000001100110001111010011110101 1652566463.257791 010000011101 1000101000000000101001101111110100000111111110100110 0000000000000000000000000000000000000000101100111000101110001010 10 000000000101100111000101110001010 1652566464.8399875 010000011101 1000101000000000101001110000001101011100001001011011 0000000000000000000000000000000000011111111001011011110111111101 10 000011111111001011011110111111101 1652566466.3860452 010000011101 1000101000000000101001110000100110001011010011110111 0000000000000000000000000000000000000000101011010111011010101100 10 000000000101011010111011010101100 1652566467.9558356 010000011101 1000101000000000101001110000111111010010110001101001 0000000000000000000000000000000000000000011001011001100010011110 10 000000000011001011001100010011110 1652566469.5445094 010000011101 1000101000000000101001110001011000101101100100111110 0000000000000000000000000000000000000001100111111111010101010111 10 000000001100111111111010101010111 1652566471.164418 010000011101 1000101000000000101001110001110010101000010111010011 0000000000000000000000000000000000000000101010000101110011101101 10 000000000101010000101110011101101 1652566472.7336688 010000011101 1000101000000000101001110010001011101111010001101110 0000000000000000000000000000000000000011111001000111000110111101 10 000000011111001000111000110111101 1652566474.2879102 010000011101 1000101000000000101001110010100100100110110100011111 0000000000000000000000000000000000000000101111001001100101110001 10 000000000101111001001100101110001 1652566475.8257375 010000011101 1000101000000000101001110010111101001101100011100010 0000000000000000000000000000000000000000011001101011010111111101 10 000000000011001101011010111111101 1652566477.406342 010000011101 1000101000000000101001110011010110100000000110000010 0000000000000000000000000000000000000001101011101101100101100000 10 000000001101011101101100101100000 1652566478.9654105 010000011101 1000101000000000101001110011101111011100100101001001 0000000000000000000000000000000000000000111001111100100011001011 10 000000000111001111100100011001011 1652566480.4778848 010000011101 1000101000000000101001110100000111101001010110101010 0000000000000000000000000000000000000111101000110101110011100011 10 000000111101000110101110011100011 1652566482.0515509 010000011101 1000101000000000101001110100100000110100110010011100 0000000000000000000000000000000000000000100111011101100100110110 10 000000000100111011101100100110110 1652566483.541527 010000011101 1000101000000000101001110100111000101010100001100001 0000000000000000000000000000000000000000011000011110010011111101 10 000000000011000011110010011111101 1652566485.121764 010000011101 1000101000000000101001110101010001111100101011111011 0000000000000000000000000000000000000001101001010110001010011010 10 000000001101001010110001010011010 1652566486.6397657 010000011101 1000101000000000101001110101101010001111000111101100 0000000000000000000000000000000000000000111011110011101100010111 10 000000000111011110011101100010111 1652566488.2642183 010000011101 1000101000000000101001110110000100001110100011110100 0000000000000000000000000000000000000011101110000001100100011000 10 000000011101110000001100100011000 1652566489.7945278 010000011101 1000101000000000101001110110011100101101100110001011 0000000000000000000000000000000000000000011000100011000101111111 10 000000000011000100011000101111111 1652566491.4354746 010000011101 1000101000000000101001110110110110111101111011010001 0000000000000000000000000000000000000000101010010000011101011010 10 000000000101010010000011101011010 1652566493.0550997 010000011101 1000101000000000101001110111010000111000011011000001 0000000000000000000000000000000000000001100110000101100000010000 10 000000001100110000101100000010000 1652566494.6093755 010000011101 1000101000000000101001110111101001110000000000000010 0000000000000000000000000000000000000000111001001000011011000011 10 000000000111001001000011011000011 chart [image-16] Noisy single precision float This is a fairly typical floating point time series which records the gradient norm at each iteration of a machine learning run. Most of the significant bits are incompressible noise. We still get some compression because the exponent bits don't change and the number only uses single precision and does not use the least significant 29 bits in the mantissa (2.37x compression). image [image-18] text DECIMAL BITS IN XOR BITS OUT 0.012796709313988686 001111111000 1010001101010010100101000000000000000000000000000000 001111111000 1010001101010010100101000000000000000000000000000000 0.3509874641895294 001111111101 0110011101101001010000100000000000000000000000000000 000000000 1011100010000111011110101100000000000000000000000000000 1101001011001 10111000100001110111101011 0.37030330300331116 001111111101 0111101100110000110010100000000000000000000000000000 000000000000000 1110001011001100010000000000000000000000000000000 10 00000011100010110011000100 0.3147137761116028 001111111101 0100001001000100010101000000000000000000000000000000 00000000000000 11100101110100100111100000000000000000000000000000 10 00000111001011101001001111 0.3929761052131653 001111111101 1001001001101000010101000000000000000000000000000000 000000000000 1101000000101100000000000000000000000000000000000000 10 00011010000001011000000000 0.40006211400032043 001111111101 1001100110101001111000100000000000000000000000000000 0000000000000000 101111000001101101100000000000000000000000000000 10 00000001011110000011011011 0.5028038024902344 001111111110 0000000101101111100000000000000000000000000000000000 0000000000 111001100011000110011000100000000000000000000000000000 10 01110011000110001100110001 0.4919178783893585 001111111101 1111011110111001010100100000000000000000000000000000 0000000000 111111011011010110110100100000000000000000000000000000 10 01111110110110101101101001 0.4074249267578125 001111111101 1010000100110100000000000000000000000000000000000000 0000000000000 101011010001101010100100000000000000000000000000000 10 00001010110100011010101001 0.45758020877838135 001111111101 1101010010001111111010000000000000000000000000000000 0000000000000 111010110111011111010000000000000000000000000000000 10 00001110101101110111110100 0.40965843200683594 001111111101 1010001101111101100000000000000000000000000000000000 0000000000000 111011111110010011010000000000000000000000000000000 10 00001110111111100100110100 0.4113210439682007 001111111101 1010010100110001010110000000000000000000000000000000 0000000000000000011001001100110110000000000000000000000000000000 10 00000000110010011001101100 0.3712143003940582 001111111101 0111110000011111100110100000000000000000000000000000 000000000000 1101100100101110110000100000000000000000000000000000 10 00011011001001011101100001 0.4784809947013855 001111111101 1110100111110110111011000000000000000000000000000000 000000000000 1001010111101001011101100000000000000000000000000000 10 00010010101111010010111011 0.49545547366142273 001111111101 1111101101011000101011100000000000000000000000000000 000000000000000 1001010101110010000100000000000000000000000000000 10 00000010010101011100100001 0.4399729073047638 001111111101 1100001010001000010000100000000000000000000000000000 00000000000000 11100111010000111011000000000000000000000000000000 10 00000111001110100001110110 0.45343348383903503 001111111101 1101000001010000110111100000000000000000000000000000 000000000000000 1001011011000100111000000000000000000000000000000 10 00000010010110110001001110 0.5084419846534729 001111111110 0000010001010010100000100000000000000000000000000000 0000000000 111101010000000010010111000000000000000000000000000000 10 01111010100000000100101110 0.6009485125541687 001111111110 0011001110101111100001100000000000000000000000000000 00000000000000 11011111111101000001000000000000000000000000000000 10 00000110111111111010000010 0.48541179299354553 001111111101 1111000100001111110010100000000000000000000000000000 0000000000 111100001010100000010011000000000000000000000000000000 10 01111000010101000000100110 0.4059547781944275 001111111101 1001111110110010100111000000000000000000000000000000 0000000000000 110111010111101010101100000000000000000000000000000 10 00001101110101111010101011 0.4256344139575958 001111111101 1011001111011001100000100000000000000000000000000000 00000000000000 10110001101011000111100000000000000000000000000000 10 00000101100011010110001111 0.4449596107006073 001111111101 1100011110100011011111100000000000000000000000000000 0000000000000 111010001111010111111000000000000000000000000000000 1101101010100 111010001111010111111 0.5190045833587646 001111111110 0000100110111010111110000000000000000000000000000000 0000000000 111100111000011001100001100000000000000000000000000000 1101010011000 1111001110000110011000011 0.6285673379898071 001111111110 0100000111010011100101000000000000000000000000000000 0000000000000 100100001101001011011000000000000000000000000000000 10 0001001000011010010110110 0.5742775797843933 001111111110 0010011000000111101101100000000000000000000000000000 0000000000000 110011111010100001000100000000000000000000000000000 10 0001100111110101000010001 0.5105176568031311 001111111110 0000010101100010100100100000000000000000000000000000 00000000000000 10001101100101001001000000000000000000000000000000 10 0000100011011001010010010 0.6416354179382324 001111111110 0100100010000100011100000000000000000000000000000000 0000000000000 100110111100110111000100000000000000000000000000000 10 0001001101111001101110001 chart [image-17] We can save a tiny amount of extra space if we treat it as a single precision floating point number to begin with since in that case we need one less bit to store the length of the nonzero subsequence. This makes very little difference in this case because the length field is stored rarely (2.41x compression). image [image-19] text DECIMAL BITS IN XOR BITS OUT 0.012796709 00111100010100011010100101001010 001111000 10100011010100101001010 0.35098746 00111110101100111011010010100001 000000 10111000100001110111101011 11001101100110111000100001110111101011 0.3703033 00111110101111011001100001100101 000000000000 11100010110011000100 1000000011100010110011000100 0.31471378 00111110101000010010001000101010 00000000000 111001011101001001111 1000000111001011101001001111 0.3929761 00111110110010010011010000101010 00000000011010000001011 000000000 1000011010000001011000000000 0.4000621 00111110110011001101010011110001 0000000000000 1011110000011011011 1000000001011110000011011011 0.5028038 00111111000000001011011111000000 0000000 1110011000110001100110001 1001110011000110001100110001 0.49191788 00111110111110111101110010101001 0000000 1111110110110101101101001 1001111110110110101101101001 0.40742493 00111110110100001001101000000000 0000000000 1010110100011010101001 1000001010110100011010101001 0.4575802 00111110111010100100011111110100 0000000000 1110101101110111110100 1000001110101101110111110100 0.40965843 00111110110100011011111011000000 0000000000 1110111111100100110100 1000001110111111100100110100 0.41132104 00111110110100101001100010101100 00000000000000 110010011001101100 1000000000110010011001101100 0.3712143 00111110101111100000111111001101 000000000 11011001001011101100001 1000011011001001011101100001 0.478481 00111110111101001111101101110110 000000000 10010101111010010111011 1000010010101111010010111011 0.49545547 00111110111111011010110001010111 000000000000 10010101011100100001 1000000010010101011100100001 0.4399729 00111110111000010100010000100001 00000000000 111001110100001110110 1000000111001110100001110110 0.45343348 00111110111010000010100001101111 000000000000 10010110110001001110 1000000010010110110001001110 0.508442 00111111000000100010100101000001 0000000 1111010100000000100101110 1001111010100000000100101110 0.6009485 00111111000110011101011111000011 00000000000 110111111111010000010 1000000110111111111010000010 0.4854118 00111110111110001000011111100101 0000000 1111000010101000000100110 1001111000010101000000100110 0.40595478 00111110110011111101100101001110 0000000000 1101110101111010101011 1000001101110101111010101011 0.4256344 00111110110110011110110011000001 00000000000 101100011010110001111 1000000101100011010110001111 0.4449596 00111110111000111101000110111111 0000000000 1110100011110101111110 110101010100111010001111010111111 0.5190046 00111111000001001101110101111100 0000000 1111001110000110011000011 1100111110001111001110000110011000011 0.62856734 00111111001000001110100111001010 0000000000 1001000011010010110110 100001001000011010010110110 0.5742776 00111111000100110000001111011011 0000000000 1100111110101000010001 100001100111110101000010001 0.51051766 00111111000000101011000101001001 00000000000 100011011001010010010 100000100011011001010010010 0.6416354 00111111001001000100001000111000 0000000000 1001101111001101110001 100001001101111001101110001 0.51030874 00111111000000101010001110011000 0000000000 1001101110000110100000 100001001101110000110100000 0.64693856 00111111001001011001110111000100 0000000000 1001110011111001011100 100001001110011111001011100 0.51288015 00111111000000110100110000011101 0000000000 1001101101000111011001 100001001101101000111011001 0.6287092 00111111001000001111001100010110 0000000000 1000111011111100001011 100001000111011111100001011 0.65078825 00111111001001101001101000001111 0000000000000 1100110100100011001 100000001100110100100011001 0.5141415 00111111000000111001111011000111 0000000000 1001010000010011001000 100001001010000010011001000 chart [image-17] If we're willing to lossily compress, we can get significantly better compression by reducing the size of the mantissa even further to 4bits (8.48x compression). image [image-24] text MASK: 1111111111111111000000000000000000000000000000000000000000000000 DECIMAL BITS IN XOR BITS OUT 0.012796709313988686 001111111000 1010001101010010100101000000000000000000000000000000 001111111000 1010001101010010100101000000000000000000000000000000 0.3509874641895294 001111111101 0110011101101001010000100000000000000000000000000000 00000000010111 00000000000000000000000000000000000000000000000000 110100100010010111 0.37030330300331116 001111111101 0111101100110000110010100000000000000000000000000000 0000000000000001 000000000000000000000000000000000000000000000000 11011110000001 0.3147137761116028 001111111101 0100001001000100010101000000000000000000000000000000 0000000000000011 000000000000000000000000000000000000000000000000 110111000000111 0.3929761052131653 001111111101 1001001001101000010101000000000000000000000000000000 0000000000001101 000000000000000000000000000000000000000000000000 11011000000111101 0.40006211400032043 001111111101 1001100110101001111000100000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 0.5028038024902344 001111111110 0000000101101111100000000000000000000000000000000000 0000000000111001 000000000000000000000000000000000000000000000000 1101010000101111001 0.4919178783893585 001111111101 1111011110111001010100100000000000000000000000000000 0000000000111111 000000000000000000000000000000000000000000000000 10111111 0.4074249267578125 001111111101 1010000100110100000000000000000000000000000000000000 0000000000000101 000000000000000000000000000000000000000000000000 10000101 0.45758020877838135 001111111101 1101010010001111111010000000000000000000000000000000 0000000000000111 000000000000000000000000000000000000000000000000 10000111 0.40965843200683594 001111111101 1010001101111101100000000000000000000000000000000000 0000000000000111 000000000000000000000000000000000000000000000000 10000111 0.4113210439682007 001111111101 1010010100110001010110000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 0.3712143003940582 001111111101 0111110000011111100110100000000000000000000000000000 0000000000001101 000000000000000000000000000000000000000000000000 10001101 0.4784809947013855 001111111101 1110100111110110111011000000000000000000000000000000 0000000000001001 000000000000000000000000000000000000000000000000 10001001 0.49545547366142273 001111111101 1111101101011000101011100000000000000000000000000000 0000000000000001 000000000000000000000000000000000000000000000000 10000001 0.4399729073047638 001111111101 1100001010001000010000100000000000000000000000000000 0000000000000011 000000000000000000000000000000000000000000000000 10000011 0.45343348383903503 001111111101 1101000001010000110111100000000000000000000000000000 0000000000000001 000000000000000000000000000000000000000000000000 10000001 0.5084419846534729 001111111110 0000010001010010100000100000000000000000000000000000 0000000000111101 000000000000000000000000000000000000000000000000 10111101 0.6009485125541687 001111111110 0011001110101111100001100000000000000000000000000000 0000000000000011 000000000000000000000000000000000000000000000000 10000011 0.48541179299354553 001111111101 1111000100001111110010100000000000000000000000000000 00000000001111 00000000000000000000000000000000000000000000000000 10111100 0.4059547781944275 001111111101 1001111110110010100111000000000000000000000000000000 000000000000011 0000000000000000000000000000000000000000000000000 10000110 0.4256344139575958 001111111101 1011001111011001100000100000000000000000000000000000 000000000000001 0000000000000000000000000000000000000000000000000 10000010 0.4449596107006073 001111111101 1100011110100011011111100000000000000000000000000000 0000000000000111 000000000000000000000000000000000000000000000000 10000111 0.5190045833587646 001111111110 0000100110111010111110000000000000000000000000000000 00000000001111 00000000000000000000000000000000000000000000000000 10111100 0.6285673379898071 001111111110 0100000111010011100101000000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.5742775797843933 001111111110 0010011000000111101101100000000000000000000000000000 000000000000011 0000000000000000000000000000000000000000000000000 10000110 0.5105176568031311 001111111110 0000010101100010100100100000000000000000000000000000 000000000000001 0000000000000000000000000000000000000000000000000 10000010 0.6416354179382324 001111111110 0100100010000100011100000000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.5103087425231934 001111111110 0000010101000111001100000000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.6469385623931885 001111111110 0100101100111011100010000000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.5128801465034485 001111111110 0000011010011000001110100000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.6287091970443726 001111111110 0100000111100110001011000000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.6507882475852966 001111111110 0100110100110100000111100000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 0.5141414999961853 001111111110 0000011100111101100011100000000000000000000000000000 00000000000001 00000000000000000000000000000000000000000000000000 10000100 0.5344889163970947 001111111110 0001000110101000100010000000000000000000000000000000 0000000000000001 000000000000000000000000000000000000000000000000 10000001 0.5667627453804016 001111111110 0010001000101110101110100000000000000000000000000000 0000000000000011 000000000000000000000000000000000000000000000000 10000011 0.6604780554771423 001111111110 0101001000101010001011100000000000000000000000000000 0000000000000111 000000000000000000000000000000000000000000000000 1101101000010111 0.6362493634223938 001111111110 0100010111000010011110100000000000000000000000000000 0000000000000001 000000000000000000000000000000000000000000000000 10001 0.47262275218963623 001111111101 1110001111110111001110000000000000000000000000000000 000000000011101 0000000000000000000000000000000000000000000000000 110101000010011101 0.5955216288566589 001111111110 0011000011101000001101100000000000000000000000000000 0000000000111101 000000000000000000000000000000000000000000000000 1101010000101111101 0.6833667159080505 001111111110 0101110111100010001111100000000000000000000000000000 000000000000011 0000000000000000000000000000000000000000000000000 10000110 0.4804770052433014 001111111101 1110110000000010001010100000000000000000000000000000 0000000000111011 000000000000000000000000000000000000000000000000 10111011 0.476748526096344 001111111101 1110100000110000110001000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000 0 chart (original) [image-22] chart (4 bit mantissa) [image-23] This entry was posted in Algorithms, LocustDB on April 7, 2024 by clemenswinter. Post navigation - Entity-Based Reinforcement Learning 5 thoughts on "The Simple Beauty of XOR Floating Point Compression" 1. [280a4]June April 11, 2024 at 1:36 pm i really hope I can one day be 1/10 as skilled of an engineer as you are. This is just so fascinating. Reply | 2. [0c401]Pete April 11, 2024 at 5:20 pm Cool idea. Have you tried using standard compression algorithms, like gzip or xz, on the pattern after you apply your XORs (center column in above screenshots)? Many compressors are already adept at reducing long strings of 0-bits mixed with short bursts of higher entropy. Reply | 1. [b7afd]clemenswinter Post authorApril 11, 2024 at 6:44 pm Yeah that would be a good baseline, haven't tried! Reply | 3. [ef8f2]Martin April 12, 2024 at 7:10 am Have you tried pcodec? It gets better compression than xor schemes and typically compresses around 200-500MB/s, decompresses 1-4GB/s out of the box. https://github.com/mwlon/pcodec/ Disclaimer: am primary author Reply | 1. [b7afd]clemenswinter Post authorApril 12, 2024 at 7:47 am Oh very cool, thanks for the tip! I gave it a quick try, I'm getting around 150MiB/s encode,1.5GiB/s decode for random f64 for pco (my xor implementation is around 370MiB/s encode, 1GiB/s decode). Compression ratios are often similar, but in some cases a lot better for pco. Will have to take a closer look later! Code: https://github.com/cswinter/LocustDB/commit/ 46edbce88f2aceb3f5c12da2fcc123486390ff58 Compression ratios: https://gist.github.com/cswinter/ 853e53f58c5745bc84db3ddcbe6abf35 Reply | Leave a ReplyCancel reply This site uses Akismet to reduce spam. Learn how your comment data is processed. Top Posts & Pages * The Simple Beauty of XOR Floating Point Compression * How to Analyze Billions of Records per Second on a Single Desktop PC * How to Read 100s of Millions of Records per Second from a Single Disk * Contact * Entity-Based Reinforcement Learning * Machine Learning and the Future of Video Games * Removing Algae from my Water Loop * My Reinforcement Learning Learnings * New roadmap after finishing multiplayer * Mastering Real-Time Strategy Games with Deep Reinforcement Learning: Mere Mortal Edition Categories * AI * Algorithms * Builds * CodeCraft * LocustDB * ML * RL Social Media * Twitter * LinkedIn Powered by WordPress.com. Discover more from Clemens' Blog Subscribe now to keep reading and get access to the full archive. Type your email... [ ] Subscribe Continue reading Loading Comments... Write a Comment... [ ] Email (Required) [ ] Name (Required) [ ] Website [ ] [Post Comment]