https://www.slideshare.net/howarddgreen/2002-microbenchmarks HomeExplore [ ]Submit Search UploadLoginSignup Advertisement How NOT to Write a Microbenchmark Report Azul Systems Inc. Azul Systems Inc.Follow Azul Systems Inc. Nov. 29, 2013*0 likes 2 likes x Be the first to like this Show More *1,729 views views x Total views 0 On Slideshare 0 From embeds 0 Number of embeds 0 How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark How NOT to Write a Microbenchmark Check these out next Webinar: Zing Vision: Answering your toughest production Java performance que... Webinar: Zing Vision: Answering your toughest production Java performance que... Azul Systems Inc. Experiences with Debugging Data Races Experiences with Debugging Data Races Azul Systems Inc. Speculative Locking: Breaking the Scale Barrier (JAOO 2005) Speculative Locking: Breaking the Scale Barrier (JAOO 2005) Azul Systems Inc. What's New in the JVM in Java 8? What's New in the JVM in Java 8? Azul Systems Inc. ObjectLayout: Closing the (last?) inherent C vs. Java speed gap ObjectLayout: Closing the (last?) inherent C vs. Java speed gap Azul Systems Inc. Towards a Scalable Non-Blocking Coding Style Towards a Scalable Non-Blocking Coding Style Azul Systems Inc. Priming Java for Speed at Market Open Priming Java for Speed at Market Open Azul Systems Inc. The Java Evolution Mismatch - Why You Need a Better JVM The Java Evolution Mismatch - Why You Need a Better JVM Azul Systems Inc. 1 of 41 thumbTop clipped slide How NOT to Write a Microbenchmark Nov. 29, 2013*0 likes 2 likes x Be the first to like this Show More *1,729 views views x Total views 0 On Slideshare 0 From embeds 0 Number of embeds 0 Download NowDownload to read offline Report Technology Microbenchmarks are like a microscope. Magnification is high, but what the heck are you looking at? Like "truths, half-truths, and statistics", microbenchmarks can be very misleading. In this presentation, learn how to tell when a benchmark lies, recognize what a benchmark can really tell you and understand how some popular benchmarks are flawed. Azul Systems Inc. Azul Systems Inc.Follow Azul Systems Inc. Advertisement Advertisement Recommended Contiki os timer tutorialContiki os timer tutorial Contiki os timer tutorialSalah Amean7.2K views*27 slides Nmap 9 truth "Nothing to say any more"Nmap 9 truth "Nothing to say any more" Nmap 9 truth "Nothing to say any more"abend_cve_9999_00011.2K views* 90 slides Tech Days 2015: Dynamic AnalysisTech Days 2015: Dynamic Analysis Tech Days 2015: Dynamic AnalysisAdaCore837 views*50 slides Full solution to bounded bufferFull solution to bounded buffer Full solution to bounded bufferSyed Zaid Irshad4K views*25 slides Lesson 5Lesson 5 Lesson 5Fernando Loizides264 views*73 slides NmapNmap NmapFat-Thing Gabriel-Culley5K views*14 slides More Related Content Viewers also liked(15) Webinar: Zing Vision: Answering your toughest production Java performance que...Webinar: Zing Vision: Answering your toughest production Java performance que... Webinar: Zing Vision: Answering your toughest production Java performance que... Azul Systems Inc.*1.4K views Experiences with Debugging Data RacesExperiences with Debugging Data Races Experiences with Debugging Data Races Azul Systems Inc.*556 views Speculative Locking: Breaking the Scale Barrier (JAOO 2005) Speculative Locking: Breaking the Scale Barrier (JAOO 2005) Speculative Locking: Breaking the Scale Barrier (JAOO 2005) Azul Systems Inc.*2.1K views What's New in the JVM in Java 8?What's New in the JVM in Java 8? What's New in the JVM in Java 8? Azul Systems Inc.*1.5K views ObjectLayout: Closing the (last?) inherent C vs. Java speed gap ObjectLayout: Closing the (last?) inherent C vs. Java speed gap ObjectLayout: Closing the (last?) inherent C vs. Java speed gap Azul Systems Inc.*971 views Towards a Scalable Non-Blocking Coding StyleTowards a Scalable Non-Blocking Coding Style Towards a Scalable Non-Blocking Coding Style Azul Systems Inc.*874 views Priming Java for Speed at Market OpenPriming Java for Speed at Market Open Priming Java for Speed at Market Open Azul Systems Inc.*1.3K views The Java Evolution Mismatch - Why You Need a Better JVMThe Java Evolution Mismatch - Why You Need a Better JVM The Java Evolution Mismatch - Why You Need a Better JVM Azul Systems Inc.*1.7K views Start Fast and Stay Fast - Priming Java for Market Open with ReadyNow!Start Fast and Stay Fast - Priming Java for Market Open with ReadyNow! Start Fast and Stay Fast - Priming Java for Market Open with ReadyNow! Azul Systems Inc.*596 views Lock-Free, Wait-Free Hash TableLock-Free, Wait-Free Hash Table Lock-Free, Wait-Free Hash Table Azul Systems Inc.*1.4K views DotCMS Bootcamp: Enabling Java in Latency Sensitivie Environments DotCMS Bootcamp: Enabling Java in Latency Sensitivie Environments DotCMS Bootcamp: Enabling Java in Latency Sensitivie Environments Azul Systems Inc.*759 views What's Inside a JVM?What's Inside a JVM? What's Inside a JVM? Azul Systems Inc.*1.5K views Zulu Embedded Java IntroductionZulu Embedded Java Introduction Zulu Embedded Java Introduction Azul Systems Inc.*3.5K views Java vs. C/C++Java vs. C/C++ Java vs. C/C++ Azul Systems Inc.*5.6K views C++vs javaC++vs java C++vs java Pradeep wolf king*2.3K views Similar to How NOT to Write a Microbenchmark(20) Algorithm-RepetitionSentinellNestedLoop_Solution.pptx Algorithm-RepetitionSentinellNestedLoop_Solution.pptx Algorithm-RepetitionSentinellNestedLoop_Solution.pptx AliaaAqilah3*55 views TCLSH and Macro Ping Test on Cisco Routers and SwitchesTCLSH and Macro Ping Test on Cisco Routers and Switches TCLSH and Macro Ping Test on Cisco Routers and Switches NetProtocol Xpert*1K views SPARKNaCl: A verified, fast cryptographic librarySPARKNaCl: A verified, fast cryptographic library SPARKNaCl: A verified, fast cryptographic library AdaCore*30 views Verilog Lecture3 hust 2014Verilog Lecture3 hust 2014 Verilog Lecture3 hust 2014 Beo Tu*2.3K views DSA 103 Object Oriented Programming :: Week 3DSA 103 Object Oriented Programming :: Week 3 DSA 103 Object Oriented Programming :: Week 3 Ferdin Joe John Joseph PhD*344 views 06.Loops06.Loops 06.Loops Intro C# Book*23.9K views Analyzing and Interpreting AWRAnalyzing and Interpreting AWR Analyzing and Interpreting AWR pasalapudi*64K views 02 performance02 performance 02 performance marangburu42*284 views Operators loops conditional and statementsOperators loops conditional and statements Operators loops conditional and statements Vladislav Hadzhiyski*200 views Timing Diagram.pptxTiming Diagram.pptx Timing Diagram.pptx ISMT College*157 views Jvm memory modelJvm memory model Jvm memory model Yoav Avrahami*1.2K views Actor ConcurrencyActor Concurrency Actor Concurrency Alex Miller*3K views Week2 ch4 part1edited 2020Week2 ch4 part1edited 2020 Week2 ch4 part1edited 2020 Osama Ghandour Geris*26 views Week2 ch4 part1edited 2020Week2 ch4 part1edited 2020 Week2 ch4 part1edited 2020 Osama Ghandour Geris*19 views Java Programming: LoopsJava Programming: Loops Java Programming: Loops Karwan Mustafa Kareem*2K views FPGA based 10G Performance Tester for HW OpenFlow SwitchFPGA based 10G Performance Tester for HW OpenFlow Switch FPGA based 10G Performance Tester for HW OpenFlow Switch Yutaka Yasuda*528 views Gordon morrison temporalengineering-delphi-v3Gordon morrison temporalengineering-delphi-v3 Gordon morrison temporalengineering-delphi-v3 Gordon Morrison*442 views JVM Memory Model - Yoav Abrahami, WixJVM Memory Model - Yoav Abrahami, Wix JVM Memory Model - Yoav Abrahami, Wix Codemotion Tel Aviv*1.4K views 04 performance04 performance 04 performance marangburu42*892 views IOT Firmware: Best PraticesIOT Firmware: Best Pratices IOT Firmware: Best Pratices farmckon*1.2K views Advertisement More from Azul Systems Inc.(16) Advancements ingc andc4overview_linkedin_oct2017Advancements ingc andc4overview_linkedin_oct2017 Advancements ingc andc4overview_linkedin_oct2017 Azul Systems Inc.*302 views Understanding GC, JavaOne 2017Understanding GC, JavaOne 2017 Understanding GC, JavaOne 2017 Azul Systems Inc.*847 views Azul Systems open source guideAzul Systems open source guide Azul Systems open source guide Azul Systems Inc.*498 views Intelligent Trading Summit NY 2014: Understanding Latency: Key Lessons and ToolsIntelligent Trading Summit NY 2014: Understanding Latency: Key Lessons and Tools Intelligent Trading Summit NY 2014: Understanding Latency: Key Lessons and Tools Azul Systems Inc.*1K views Understanding Java Garbage CollectionUnderstanding Java Garbage Collection Understanding Java Garbage Collection Azul Systems Inc.*4K views The evolution of OpenJDK: From Java's beginnings to 2014The evolution of OpenJDK: From Java's beginnings to 2014 The evolution of OpenJDK: From Java's beginnings to 2014 Azul Systems Inc.*600 views Push Technology's latest data distribution benchmark with Solarflare and ZingPush Technology's latest data distribution benchmark with Solarflare and Zing Push Technology's latest data distribution benchmark with Solarflare and Zing Azul Systems Inc.*902 views The Art of Java BenchmarkingThe Art of Java Benchmarking The Art of Java Benchmarking Azul Systems Inc.*1.1K views Azul Zulu on Azure Overview -- OpenTech CEE Workshop, Warsaw, Poland Azul Zulu on Azure Overview -- OpenTech CEE Workshop, Warsaw, Poland Azul Zulu on Azure Overview -- OpenTech CEE Workshop, Warsaw, Poland Azul Systems Inc.*1.7K views Understanding Application Hiccups - and What You Can Do About Them Understanding Application Hiccups - and What You Can Do About Them Understanding Application Hiccups - and What You Can Do About Them Azul Systems Inc.*1.4K views JVM Memory Management DetailsJVM Memory Management Details JVM Memory Management Details Azul Systems Inc.*3.4K views JVM Mechanics: A Peek Under the HoodJVM Mechanics: A Peek Under the Hood JVM Mechanics: A Peek Under the Hood Azul Systems Inc.*1.6K views Enterprise Search Summit - Speeding Up SearchEnterprise Search Summit - Speeding Up Search Enterprise Search Summit - Speeding Up Search Azul Systems Inc.*935 views Understanding Java Garbage Collection - And What You Can Do About It Understanding Java Garbage Collection - And What You Can Do About It Understanding Java Garbage Collection - And What You Can Do About It Azul Systems Inc.*3.3K views Enabling Java in Latency-Sensitive ApplicationsEnabling Java in Latency-Sensitive Applications Enabling Java in Latency-Sensitive Applications Azul Systems Inc.*1.3K views Java at Scale, Dallas JUG, October 2013Java at Scale, Dallas JUG, October 2013 Java at Scale, Dallas JUG, October 2013 Azul Systems Inc.*749 views Recently uploaded(20) AndroidMakers2023.pdfAndroidMakers2023.pdf AndroidMakers2023.pdf Confetti3*0 views pcmaintenance_ppt_ch02_modified.ppt pcmaintenance_ppt_ch02_modified.ppt pcmaintenance_ppt_ch02_modified.ppt asaijohn*6 views ING Data Services hosted on ICHP DoK Amsterdam 2023ING Data Services hosted on ICHP DoK Amsterdam 2023 ING Data Services hosted on ICHP DoK Amsterdam 2023 DoKC*6 views Advantages of cloud based ERP.pdfAdvantages of cloud based ERP.pdf Advantages of cloud based ERP.pdf SimranKhan287430*5 views ITSM-Process-Description doc.docxITSM-Process-Description doc.docx ITSM-Process-Description doc.docx Asad Abbas*0 views Empowering Insurance Industry with winning Total Experience Strategy Empowering Insurance Industry with winning Total Experience Strategy Empowering Insurance Industry with winning Total Experience Strategy Espire infolabs*0 views Johns Brothers SecurityJohns Brothers Security Johns Brothers Security jbsva001*0 views Information and Communication Technology in dissemination of Agricultural Tec...Information and Communication Technology in dissemination of Agricultural Tec... Information and Communication Technology in dissemination of Agricultural Tec... Lokesh Waran*10 views Microsoft (A Tech Giant)Microsoft (A Tech Giant) Microsoft (A Tech Giant) DebadityaMandal3*5 views ACTIVE-DIRECTORY.pptACTIVE-DIRECTORY.ppt ACTIVE-DIRECTORY.ppt mwti2*0 views DecisionTreeClassification (1).pptxDecisionTreeClassification (1).pptx DecisionTreeClassification (1).pptx BasirKhan20*3 views Cybersecurity Statistics.pdfCybersecurity Statistics.pdf Cybersecurity Statistics.pdf SimranKhan287430*8 views MICROSOFT_LOOP.pptxMICROSOFT_LOOP.pptx MICROSOFT_LOOP.pptx flamboyantgirl*5 views Computer_Fundamentals_Lectures_1_to_5 (1).ppt Computer_Fundamentals_Lectures_1_to_5 (1).ppt Computer_Fundamentals_Lectures_1_to_5 (1).ppt asaijohn*4 views Best Version Of FTM For Yourself?Best Version Of FTM For Yourself? Best Version Of FTM For Yourself? kavin smith*4 views An Intrusion Detection System, a Risk Analysis System , a Secured Expert Med...An Intrusion Detection System, a Risk Analysis System , a Secured Expert Med... An Intrusion Detection System, a Risk Analysis System , a Secured Expert Med... Nathanael Asaam*6 views Welcome to React.pptxWelcome to React.pptx Welcome to React.pptx PraveenKumar680401*0 views Presentation overview.pptxPresentation overview.pptx Presentation overview.pptx Kindalem2*1 view Implementing data and databases on K8s within the Dutch government Implementing data and databases on K8s within the Dutch government Implementing data and databases on K8s within the Dutch government DoKC*9 views trellix-dlp-buyers-guide.pdftrellix-dlp-buyers-guide.pdf trellix-dlp-buyers-guide.pdf LaLaBlaGhvgT*0 views Advertisement How NOT to Write a Microbenchmark 1. How NOT To Write A Microbenchmark Dr. Cliff Click Senior Staff Engineer Sun Microsystems Session # 2. How NOT To Write A Microbenchmark or Lies, Damn Lies, and Microbenchmarks Session # 3. Microbenchmarks are a sharp knife Microbenchmarks are like a microscope Magnification is high, but what the heck are you looking at? Like "truths, half-truths, and statistics", microbenchmarks can be very misleading Beginning 3 Session # 1816 4. Learning Objectives * As a result of this presentation, you will be able to: - Recognize when a benchmark lies - Recognize what a benchmark can tell you (as opposed to what it purports to tell you) - Understand how some popular benchmarks are flawed - Write a microbenchmark that doesn't lie (to you) Beginning 4 Session # 1816 5. Speaker's Qualifications * Dr. Click is a Senior Staff Engineer at Sun Microsystems * Dr. Click wrote his first compiler at age 16 and has been writing: - runtime compilers for 15 years, and - optimizing compilers for 10 years * Dr. Click architected the HotSpot(tm) Server Compiler Beginning 5 Session # 1816 6. Your JVM is Lousy Because it Doesn't Do... "X" I routinely get handed a microbenchmark and told "HotSpot doesn't do X" 49% chance it doesn't matter, 49% chance they got fooled, 1% chance HotSpot didn't do "Y" but should 1% chance HotSpot didn't do "X" but should Beginning 6 Session # 1816 7. Agenda * * * * Beginning 7 Recent real-word benchmark disaster Popular microbenchmarks and their flaws When to disbelieve a benchmark How to write your own Session # 1816 8. What is a Microbenchmark? * Small program - Datasets may be large * All time spent in a few lines of code * Performance depends on how those few lines are compiled * Goal: Discover some particular fact * Remove all other variables Middle 8 Session # 1816 9. Why Run Microbenchmarks? * Discover some targeted fact - Such as the cost of 'turned off' Asserts, or - Will this inline? - Will another layer of abstraction hurt performance? * Fun - My JIT is faster than your JIT - My Java is faster than your C * But dangerous! Middle 9 Session # 1816 10. How HotSpot Works * HotSpot is a mixed-mode system * Code first runs interpreted - Profiles gathered * Hot code gets compiled * Same code "just runs faster" after awhile * Bail out to interpreter for rare events - Never taken before code - Class loading, initializing Middle 10 Session # 1816 11. Example from Magazine Editor * * * * * * * Middle 11 What do (turned off) Asserts cost? Tiny 5-line function r/w's global variable Run in a loop 100,000,000 times With explicit check - 5 sec With Assert (off) - 0.2 sec With no test at all - 5 sec What did he really measure? Session # 1816 12. Assert Example static int sval; // Global variable static int test_assert(int val) { assert (val >= 0) : "should be positive"; sval = val * 6; sval += 3; // Read/write global sval /= 2; return val+2; // } static void main(String args[]) { int REPEAT = Integer.parseInt(args[0]); int v=0; for( int i=0; i= 0) : "should be positive"; sval = val * 6; sval += 3; // Read/write global sval /= 2; return val+2; } static void main(String args[]) { int REPEAT = Integer.parseInt(args[0]); int v=0; for( int i=0; i= 0) : "should be positive"; sval = val * 6; sval += 3; // Read/write global sval /= 2; return val+2; // } static void main(String args[]) { int REPEAT = Integer.parseInt(args[0]); int v=0; for( int i=0; i