https://testing.googleblog.com/2024/10/smurf-beyond-test-pyramid.html * [googlelogo] Testing Blog SMURF: Beyond the Test Pyramid Tuesday, October 15, 2024 This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. By Adam Bender The test pyramid is the canonical heuristic for guiding test suite evolution. It conveys a simple message - prefer more unit tests than integration tests, and prefer more integration tests than end-to-end tests. A diagram of the test pyramid While useful, the test pyramid lacks the details you need as your test suite grows and you face challenging trade-offs. To scale your test suite, go beyond the test pyramid. The SMURF mnemonic is an easy way to remember the tradeoffs to consider when balancing your test suite: * Speed: Unit tests are faster than other test types and can be run more often--you'll catch problems sooner. * Maintainability: The aggregated cost of debugging and maintaining tests (of all types) adds up quickly. A larger system under test has more code, and thus greater exposure to dependency churn and requirement drift which, in turn, creates more maintenance work. * Utilization: Tests that use fewer resources (memory, disk, CPU) cost less to run. A good test suite optimizes resource utilization so that it does not grow super-linearly with the number of tests. Unit tests usually have better utilization characteristics, often because they use test doubles or only involve limited parts of a system. * Reliability: Reliable tests only fail when an actual problem has been discovered. Sorting through flaky tests for problems wastes developer time and costs resources in rerunning the tests. As the size of a system and its corresponding tests grow, non-determinism (and thus, flakiness) creeps in, and your test suite is more likely to become unreliable. * Fidelity: High-fidelity tests come closer to approximating real operating conditions (e.g., real databases or traffic loads) and better predict the behavior of our production systems. Integration and end-to-end tests can better reflect realistic conditions, while unit tests have to simulate the environment, which can lead to drift between test expectations and reality. A radar chart depicting the relationship between SMURF attributes as applied to unit, integration, and end-to-end tests. Unit tests perform best on all attributes except fidelity, where they are the worst. Integration tests are mid-way performers on all aspects. End-to-end tests are worst on all aspects, except fidelity where they are the best. A radar chart of Test Type vs. Test Property (i.e. SMURF). Farther from center is better. In many cases, the relationships between the SMURF dimensions are in tension: improving one dimension can affect the others. However, if you can improve one or more dimensions of a test without harming the others, then you should do so. When thinking about the types of your tests (unit, integration, end-to-end), your choices have meaningful implications for your test suite's cost and the value it provides. Share on Twitter Share on Facebook Google Labels: Adam Bender , TotT [ ] [ncccFICB87] Labels * TotT 102 * GTAC 61 * James Whittaker 42 * Misko Hevery 32 * Code Health 30 * Anthony Vallone 27 * Patrick Copeland 23 * Jobs 18 * Andrew Trenk 12 * C++ 11 * Patrik Hoglund 8 * JavaScript 7 * Allen Hutchison 6 * George Pirocanac 6 * Zhanyong Wan 6 * Harry Robinson 5 * Java 5 * Julian Harty 5 * Adam Bender 4 * Alberto Savoia 4 * Ben Yu 4 * Erik Kuefler 4 * Philip Zembrod 4 * Shyam Seshadri 4 * Chrome 3 * Dillon Bly 3 * John Thomas 3 * Lesley Katzen 3 * Marc Kaplan 3 * Markus Clermont 3 * Max Kanat-Alexander 3 * Sonal Shah 3 * APIs 2 * Abhishek Arya 2 * Alan Myrvold 2 * Alek Icev 2 * Android 2 * April Fools 2 * Chaitali Narla 2 * Chris Lewis 2 * Chrome OS 2 * Diego Salas 2 * Dori Reuveni 2 * Jason Arbon 2 * Jochen Wuttke 2 * Kostya Serebryany 2 * Marc Eaddy 2 * Marko Ivankovic 2 * Mobile 2 * Oliver Chang 2 * Simon Stewart 2 * Stefan Kennedy 2 * Test Flakiness 2 * Titus Winters 2 * Tony Voellm 2 * WebRTC 2 * Yiming Sun 2 * Yvette Nameth 2 * Zuri Kemp 2 * Aaron Jacobs 1 * Adam Porter 1 * Adam Raider 1 * Adel Saoud 1 * Alan Faulkner 1 * Alex Eagle 1 * Amy Fu 1 * Anantha Keesara 1 * Antoine Picard 1 * App Engine 1 * Ari Shamash 1 * Arif Sukoco 1 * Benjamin Pick 1 * Bob Nystrom 1 * Bruce Leban 1 * Carlos Arguelles 1 * Carlos Israel Ortiz Garcia 1 * Cathal Weakliam 1 * Christopher Semturs 1 * Clay Murphy 1 * Dagang Wei 1 * Dan Maksimovich 1 * Dan Shi 1 * Dan Willemsen 1 * Dave Chen 1 * Dave Gladfelter 1 * David Bendory 1 * David Mandelberg 1 * Derek Snyder 1 * Diego Cavalcanti 1 * Dmitry Vyukov 1 * Eduardo Bravo Ortiz 1 * Ekaterina Kamenskaya 1 * Elliott Karpilovsky 1 * Elliotte Rusty Harold 1 * Espresso 1 * Felipe Sodre 1 * Francois Aube 1 * Gene Volovich 1 * Google+ 1 * Goran Petrovic 1 * Goranka Bjedov 1 * Hank Duan 1 * Havard Rast Blok 1 * Hongfei Ding 1 * Jason Elbaum 1 * Jason Huggins 1 * Jay Han 1 * Jeff Hoy 1 * Jeff Listfield 1 * Jessica Tomechak 1 * Jim Reardon 1 * Joe Allan Muharsky 1 * Joel Hynoski 1 * John Micco 1 * John Penix 1 * Jonathan Rockway 1 * Jonathan Velasquez 1 * Josh Armour 1 * Julie Ralph 1 * Kai Kent 1 * Karin Lundberg 1 * Kaue Silveira 1 * Kevin Bourrillion 1 * Kevin Graney 1 * Kirkland 1 * Kurt Alfred Kluever 1 * Manjusha Parvathaneni 1 * Marek Kiszkis 1 * Marius Latinis 1 * Mark Ivey 1 * Mark Manley 1 * Mark Striebeck 1 * Matt Lowrie 1 * Meredith Whittaker 1 * Michael Bachman 1 * Michael Klepikov 1 * Mike Aizatsky 1 * Mike Wacker 1 * Mona El Mahdy 1 * Noel Yap 1 * Palak Bansal 1 * Patricia Legaspi 1 * Per Jacobsson 1 * Peter Arrenbrecht 1 * Peter Spragins 1 * Phil Norman 1 * Phil Rollet 1 * Pooja Gupta 1 * Project Showcase 1 * Radoslav Vasilev 1 * Rajat Dewan 1 * Rajat Jain 1 * Rich Martin 1 * Richard Bustamante 1 * Roshan Sembacuttiaratchy 1 * Ruslan Khamitov 1 * Sam Lee 1 * Sean Jordan 1 * Sharon Zhou 1 * Shiva Garg 1 * Siddartha Janga 1 * Simran Basi 1 * Stan Chan 1 * Stephen Ng 1 * Tejas Shah 1 * Test Analytics 1 * Test Engineer 1 * Tim Lyakhovetskiy 1 * Tom O'Neill 1 * Vojta Jina 1 * automation 1 * dead code 1 * iOS 1 * mutation testing 1 Archive * V 2024 (12) + V Oct (1) o SMURF: Beyond the Test Pyramid + > Sep (1) + > Aug (1) + > Jul (1) + > May (3) + > Apr (3) + > Mar (1) + > Feb (1) * > 2023 (14) + > Dec (2) + > Nov (2) + > Oct (5) + > Sep (3) + > Aug (1) + > Apr (1) * > 2022 (2) + > Feb (2) * > 2021 (3) + > Jun (1) + > Apr (1) + > Mar (1) * > 2020 (8) + > Dec (2) + > Nov (1) + > Oct (1) + > Aug (2) + > Jul (1) + > May (1) * > 2019 (4) + > Dec (1) + > Nov (1) + > Jul (1) + > Jan (1) * > 2018 (7) + > Nov (1) + > Sep (1) + > Jul (1) + > Jun (2) + > May (1) + > Feb (1) * > 2017 (17) + > Dec (1) + > Nov (1) + > Oct (1) + > Sep (1) + > Aug (1) + > Jul (2) + > Jun (2) + > May (3) + > Apr (2) + > Feb (1) + > Jan (2) * > 2016 (15) + > Dec (1) + > Nov (2) + > Oct (1) + > Sep (2) + > Aug (1) + > Jun (2) + > May (3) + > Apr (1) + > Mar (1) + > Feb (1) * > 2015 (14) + > Dec (1) + > Nov (1) + > Oct (2) + > Aug (1) + > Jun (1) + > May (2) + > Apr (2) + > Mar (1) + > Feb (1) + > Jan (2) * > 2014 (24) + > Dec (2) + > Nov (1) + > Oct (2) + > Sep (2) + > Aug (2) + > Jul (3) + > Jun (3) + > May (2) + > Apr (2) + > Mar (2) + > Feb (1) + > Jan (2) * > 2013 (16) + > Dec (1) + > Nov (1) + > Oct (1) + > Aug (2) + > Jul (1) + > Jun (2) + > May (2) + > Apr (2) + > Mar (2) + > Jan (2) * > 2012 (11) + > Dec (1) + > Nov (2) + > Oct (3) + > Sep (1) + > Aug (4) * > 2011 (39) + > Nov (2) + > Oct (5) + > Sep (2) + > Aug (4) + > Jul (2) + > Jun (5) + > May (4) + > Apr (3) + > Mar (4) + > Feb (5) + > Jan (3) * > 2010 (37) + > Dec (3) + > Nov (3) + > Oct (4) + > Sep (8) + > Aug (3) + > Jul (3) + > Jun (2) + > May (2) + > Apr (3) + > Mar (3) + > Feb (2) + > Jan (1) * > 2009 (54) + > Dec (3) + > Nov (2) + > Oct (3) + > Sep (5) + > Aug (4) + > Jul (15) + > Jun (8) + > May (3) + > Apr (2) + > Feb (5) + > Jan (4) * > 2008 (75) + > Dec (6) + > Nov (8) + > Oct (9) + > Sep (8) + > Aug (9) + > Jul (9) + > Jun (6) + > May (6) + > Apr (4) + > Mar (4) + > Feb (4) + > Jan (2) * > 2007 (41) + > Oct (6) + > Sep (5) + > Aug (3) + > Jul (2) + > Jun (2) + > May (2) + > Apr (7) + > Mar (5) + > Feb (5) + > Jan (4) [8LnOxYWFhZ] Feed follow us in feedly Follow @googletesting [P2DAYDAaDwWA] * Google * Privacy * Terms