https://www.cobaltrobotics.com/premortems-and-postmortems/ Schedule a Demo * Email* [ ] * First Name* [ ] * Last Name [ ] * Company/Organization [ ] * Job Title [ ] * Phone [ ] * Additional Information * Comments [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] * CAPTCHA * We're committed to your privacy. Cobalt Robotics uses the information you provide to us to contact you about our relevant content, products, and services. You may unsubscribe from these communications at any time. For more information, check out our Privacy Policy. [Submit] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] * Why Cobalt + Overview + Highly Capable Robots + Expert Support + Detailed Reporting * Application + Service o Overview o Security o Facility Management o Environmental Health o Revenue Generation + Value Benefits o Detailed Reporting o Gaps & Behavioral Change o Cost & Risk Reduction * Impact + Return on Investment o ROI Estimator o Risk Analysis o Business Outcomes + Implementation o Process Overview o Demo + Site Assesment o Installation + Deployment o Support + Updates + Cyber Security o Compliance o Data Privacy o People * Company + Commitment o Overview o Mission + History o Certifications and Accolades + Leadership o Executive Team o Operation Team o Engineering Team o Investors * Resources + News + Media Center + Resource Center + Blog + Culture + Careers * Contact Cobalt Robotics * Why Cobalt + Why Cobalt o Overview o Highly Capable Robots o Expert Support o Detailed Reporting * Applications + Service o Overview o Security o Facility Management o Environmental Health o Revenue Generation + Value Benefits o Detailed Reporting o Gaps & Behavioral Change o Cost & Risk Reduction * Impact + Return on Investment o ROI Estimator o Risk Analysis o Business Outcomes + Implementation o Process Overview o Demo + Site Assesment o Installation + Deployment o Support + Updates + Cybersecurity + Privacy o Compliance o Data Privacy o People * Company + Commitment o Overview (Experts+Tech) o Mission + History o Certifications and Accolades + Leadership o Executive Team o Operation Team o Engineering Team o Investors * Resources + News + Media Center + Resource Center + Blog + Culture + Careers * Contact + Contact + Demo Customer Login SCHEDULE A DEMO Premortems and Postmortems October 12, 2021 | How it works, Robot design, Robotics [Screenshot-from-2021-10-12-14-48-48-1-e1634076989107] Premortems and Postmortems Two crucial tools for writing code when the stakes are high, like managing our fleet of over 100 robots at Cobalt Robotics. This is part 2 of a 2 part series on High Stakes Code. See Part 1. - Erik Schluntz, Cofounder & CTO Premortems Premortems are remarkably quick, and have saved my team from making bad mistakes many times. What's a premortem? Before deploying new code, ask yourself and at least one other person "If this is going to cause a major issue, what would it be?" Then, ask yourselves what you could do to protect yourself better. That could include doing additional testing or adding safeguards before deploying the code. No one intentionally releases code that they think is going to cause major problems, so the author of code about to go out will think that it's safe. The benefit of the premortem is to stretch your mind outside of the mental box you've created where your code is safe, and then look around from that perspective. We've found premortems particularly good at discovering "unknown unknowns" and issues that occur when moving from a test environment to a production environment. Will that database upgrade be just as quick when the database is under load? Will that new safety feature have disruptive false positives? Is every robot starting from the same state for this update? Part of the magic of a premortem is it gets the team to discuss worst case scenarios, despite any over-optimism or the planning fallacy about the feature itself. It creates a safe space to bring up issues, without being seen as negative, obstructionist, or not committed to the goal. The Princess Bride meme Fig 1. Vizzini never did premortems and look what happened to him Postmortems Postmortems are the sadder, but even more useful, version of premortems. The idea here is that when you do have a failure, you want to learn absolutely as much from it as you can. Think about how much you'll need to learn about your situation to truly write a failure proof system - if you learn twice as much from each failure, then you'll need half as many failures to get there! Huge amounts have been written about good postmortems, most notably Google's SRE book. I won't go through the details of how to write a postmortem, but instead focus on what makes a good postmortem. BLAMELESS Humans are going to make mistakes. At the end of the day, what matters is setting up a process to catch and prevent those errors, not which human made the mistake. This takes a strong culture to do well, but there are little tactical things you can do that help. For instance, never name names in postmortems. Instead of "at 1am Erik rebooted the database" write "at 1am an engineer rebooted the database". If you don't make your postmortems blameless, people will hide details and try to cover up their mistakes in the future. If people cover up mistakes, your team isn't going to learn from them and someone is going to make the same mistake again. ACTION ITEMS Action items (AIs) are the silver lining of your incident--and the entire point of the postmortem--so make sure you mine as many valuable ones out of the wreckage of your code while they're fresh. Ask "why" 5 times for any problem to go down into not just the immediate problem, but how it came to exist in the root cause. The deeper you mine, the more valuable the action items become. For instance, the top level AI might be "fix this bug on line 237", but 4 levels down the AI might be "we could have caught this by adding a stress test to our CI". Any one "problem" can help you uncover a number of valuable action items for your team to work on. Level "Why?" to row above Action Item 1 System ran out of memory Add memory monitor 2 Bug on line 237 Fix memory leak bug 3 Easy to make this kind of bug Refactor library 4 Difficult to review for this kind of Add stress tests to CI bug 5 Little known topic Lunch 'n Learn on memory Fig 2. Example of "5 Whys" digging into the root cause of a problem LOOK BROADLY Don't just brainstorm around the specific bug that hurt you - look more broadly for "things that went poorly" and "things that went well". Even if something wasn't directly responsible for the incident, such as "it took us too long to find the logs for this issue" you can still find valuable action items there. Noting "things that went well" can help you identify strengths that you can double down on as a team. Miners Fig 3. Engineers mining for action items in the aftermath of an incident Premortems and Postmortems at Cobalt Robotics At Cobalt, we build autonomous indoor security guard robots that patrol through office buildings and warehouses looking for anything out of the ordinary. We write code that controls a 120lb robot navigating around people - basically an indoor self driving car, and our customers are relying on us to keep their most sensitive areas secure and protected. We're committed to keeping a great engineering culture, moving fast, and NOT breaking things. To do that we need great engineers like you! [WHSC8] Fig 4. A Cobalt Robot patrolling an office space * Email* [ ] [SUBSCRIBE] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Cobalt Robotics * About * Careers * Media * * * * * (c) Copyright 2021 Cobalt Robotics. All rights reserved. Terms of Use | Privacy Policy | Responsible Disclosure This website uses cookies to improve your experience. By using our website, you agree to our use of cookies in accordance with our Privacy Policy. Cookie settingsACCEPT Privacy & Cookies Policy Close Privacy Overview This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience. Necessary [*] Necessary Always Enabled Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information. Non-necessary [*] Non-necessary Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website. SAVE & ACCEPT