[HN Gopher] SPEC says Intel's compiler used unfair optimizations...
       ___________________________________________________________________
        
       SPEC says Intel's compiler used unfair optimizations to boost
       performance
        
       Author : ksec
       Score  : 24 points
       Date   : 2024-02-18 06:26 UTC (16 hours ago)
        
 (HTM) web link (www.tomshardware.com)
 (TXT) w3m dump (www.tomshardware.com)
        
       | almatabata wrote:
       | "SPEC has ruled that the compiler used for this result was
       | performing a compilation that specifically improves the
       | performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks
       | using a priori knowledge of the SPEC code and dataset to perform
       | a transformation that has narrow applicability."
       | 
       | Does this mean that if I use their compiler in production I might
       | get worse performance because they apply this optimization to my
       | own code as well?
        
         | riscy wrote:
         | It's probably designed to recognize a very specific code
         | pattern in SPEC, so it's very unlikely to affect any other
         | code. If it wasn't so hyper targeted to only kick in for those
         | two benchmarks, then there'd be no issue: it'd just be a common
         | (mis)optimization that happens to help SPEC.
        
         | bastawhiz wrote:
         | My read is that they do what Volkswagen did with the emissions
         | scandal: they detect when they're compiling code for the
         | benchmark, and compile it in a way that makes the benchmark
         | code (specifically) faster. That optimization wouldn't
         | generalize to "real" code because the optimization relies on
         | the specific code and data ("data" being fixed non-code inputs
         | processed at runtime) being processed by the benchmark to have
         | an effect. If you know based on prior knowledge that, for
         | instance, code that uses a signed integer will always only
         | handle positive integers, you might cheat by optimizing it to
         | be unsigned integers. But a compiler couldn't possibly know
         | that in advance because it can't see the data the code
         | processes at compile time.
         | 
         | For your code to benefit, it would have to do the same things
         | the benchmark is computing and have the same shape of data as
         | the data included with the benchmark suite.
        
       | KTibow wrote:
       | Is there a way to see the optimization?
        
       | 0cf8612b2e1e wrote:
       | According to Phoronix, the optimization could boost performance
       | in SPECint by 9% overall. The publication also notes that
       | versions 2022.0 to 2023.0 of the Intel oneAPI Compiler are
       | impacted, meaning most of the now-invalidated results were run in
       | 2022, largely on Sapphire Rapids CPUs.
       | 
       | 9% bump for free.
        
       | musicale wrote:
       | Cheating on SPEC benchmarks? Surely that has never happened
       | before...
       | 
       | "Issue: Special library for a specific benchmark... Issue:
       | Pattern matching for the inner loop of a specific benchmark...
       | Issue: Compilers optimize too much for the SPEC CPU benchmarks...
       | Issue: The compiler optimization parameters (flags) used are
       | unrealistic..."[1]
       | 
       | See also: Goodhart's Law. [2]
       | 
       | [1] R. Weicker, "SPEC OSG Benchmarks: History, Use and Current
       | Issues", in "Performance Evaluation and Benchmarking with
       | Realistic Applications", R. Eigenmann (Ed.), 2001
       | 
       | [2] https://en.wikipedia.org/wiki/Goodhart%27s_law
        
       ___________________________________________________________________
       (page generated 2024-02-18 23:01 UTC)