https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ Skip to content Dan Slimmon SRE at Hashicorp Menu Search * About Search for: [ ] Search Do-nothing scripting: the key to gradual automation On 2019/07/15 By Dan SlimmonIn Uncategorized Every ops team has some manual procedures that they haven't gotten around to automating yet. Toil can never be totally eliminated. Very often, the biggest toil center for a team at a growing company will be its procedure for modifying infrastructure or its procedure for provisioning user accounts. Partial instructions for the latter might look like this: 1. Create an SSH key pair for the user. 2. Commit the public key to Git and push to master. 3. Wait for the build job to finish. 4. Find the user's email address in the employee directory. 5. Send the user their private key via 1Password. This is a relatively short example. Sometimes there are 20 steps in the process. Sometimes there are branches and special cases to keep track of as you go. Over time, these procedures can become unmanageably large and complex. Procedures like this are frustrating because they're focus-intensive yet require very little thought. They demand our full attention, but our attention isn't rewarded with interesting problems or satisfying solutions - just another checkbox checked. I have a word for a procedure like this: a slog. We know that this procedure is ripe for automation. We can easily see how to automate any given step. And we know that a computer could carry out the instructions with far greater speed and accuracy than we can, and with less tendency toward practical drift. However, automating slogs sometimes feels like an all-or-nothing proposition. Sure, we could write a script to handle step 2, or step 5. But that wouldn't really make the procedure any less cumbersome. It would lead to a proliferation of single-purpose scripts with different conventions and expectations, and you'd still have to follow a documented multi-step procedure for using those scripts. This perception of futility is the problem we really need to solve in order to escape from these manual slogs. I've found an approach that works pretty reliably: do-nothing scripting. Do-nothing scripting Almost any slog can be turned into a do-nothing script. A do-nothing script is a script that encodes the instructions of a slog, encapsulating each step in a function. For the example procedure above, we could write the following do-nothing script: import sys def wait_for_enter(): raw_input("Press Enter to continue: ") class CreateSSHKeypairStep(object): def run(self, context): print("Run:") print(" ssh-keygen -t rsa -f ~/{0}".format(context["username"])) wait_for_enter() class GitCommitStep(object): def run(self, context): print("Copy ~/new_key.pub into the `user_keys` Git repository, then run:") print(" git commit {0}".format(context["username"])) print(" git push") wait_for_enter() class WaitForBuildStep(object): build_url = "http://example.com/builds/user_keys" def run(self, context): print("Wait for the build job at {0} to finish".format(self.build_url)) wait_for_enter() class RetrieveUserEmailStep(object): dir_url = "http://example.com/directory" def run(self, context): print("Go to {0}".format(self.dir_url)) print("Find the email address for user `{0}`".format(context["username"])) context["email"] = raw_input("Paste the email address and press enter: ") class SendPrivateKeyStep(object): def run(self, context): print("Go to 1Password") print("Paste the contents of ~/new_key into a new document") print("Share the document with {0}".format(context["email"])) wait_for_enter() if __name__ == "__main__": context = {"username": sys.argv[1]} procedure = [ CreateSSHKeypairStep(), GitCommitStep(), WaitForBuildStep(), RetrieveUserEmailStep(), SendPrivateKeyStep(), ] for step in procedure: step.run(context) print("Done.") This script doesn't actually do any of the steps of the procedure. That's why it's called a do-nothing script. It feeds the user a step at a time and waits for them to complete each step manually. At first glance, it might not be obvious that this script provides value. Maybe it looks like all we've done is make the instructions harder to read. But the value of a do-nothing script is immense: * It's now much less likely that you'll lose your place and skip a step. This makes it easier to maintain focus and power through the slog. * Each step of the procedure is now encapsulated in a function, which makes it possible to replace the text in any given step with code that performs the action automatically. * Over time, you'll develop a library of useful steps, which will make future automation tasks more efficient. A do-nothing script doesn't save your team any manual effort. It lowers the activation energy for automating tasks, which allows the team to eliminate toil over time. Share this: * Twitter * Facebook * Like this: Like Loading... Post navigation Previous An Incident Command Training Handbook Next Stop apologizing for bugs 57 thoughts on "Do-nothing scripting: the key to gradual automation" 1. Pingback: New top story on Hacker News: Do-Nothing Scripts - Outside The Know 2. Pingback: New top story on Hacker News: Do-Nothing Scripts - protipsss 3. Pingback: New top story on Hacker News: Do-Nothing Scripts - Golden News 4. Pingback: New top story on Hacker News: Do-Nothing Scripts - World Best News 5. Pingback: New top story on Hacker News: Do-Nothing Scripts - News about world 6. Pingback: New top story on Hacker News: Do-Nothing Scripts - Latest news 7. Pingback: New top story on Hacker News: Do-Nothing Scripts - Hckr News 8. [picture] Wojtek Bazant I run a complicated release process. It has to be as complicated as it is, because involves long computationally intensive analyses on the primary data. Each cycle takes a few months. I've never used your approach but I like the idea. I've been a fan of really good instructions that I copy from the instruction page and execute at the right time. I pay attention to keep them all working, to keep them readable, and to make them mention important places - e.g. a dump script takes the dump location as a parameter, probably involving an environmental variable for the release number. 2019/07/22 at 09:46 Reply 9. Pingback: Do-Nothing Scripts - INDIA NEWS 10. [9bf7448] Sydney Deals We use this hybrid process effectively in a script. I never thought of it as a design pattern, so thanks for that. Design patterns help with communication of an idea within a team, and it'll be interesting to explore where else this can be applied. 2019/07/22 at 11:07 Reply 11. Pingback: Do-Nothing Scripts - Hacker News Robot 12. Pingback: Do-nothing scripting: the key to gradual automation - Latest News,Jobs,Notifications.Gk,Telugu News,Results,MCQs,Science,Tips and Tricks and More. 13. [caligra] Eric Dean Wade (@nandesu) Love this article. However, since it's 2019 it's really time to stop using RSA ssh keys. View at Medium.com Keep up the interesting articles! 2019/07/22 at 12:11 Reply 1. [bb447f0] Dan Slimmon Unfortunately for those of us who use Amazon AWS, ECDSA and Ed25519 SSH keys are kind of a no-go. Maybe some day they'll add support. 2019/07/22 at 13:06 Reply 1. [b8f7008] Erik Anderson I send an email to my AWS Account Rep quarterly to inquire about ed25519 support. I can't imagine what amount of cruft they have sitting around that's making it take so long for them to adopt support. (Great post, by the way! The checklist aspect of it reminded me a lot of Atul Gawande's "Checklist Manifesto". I have several processes here that could benefit from this type of thing.) 2019/07/22 at 15:18 Reply 1. [bb447f0] Dan Slimmon Keep fighting the good fight. I love _Checklist Manifesto_. I hadn't thought about it, but that book probably inspired me to have this idea in the first place. 2019/07/22 at 17:02 Reply 14. Pingback: Do-nothing scripting: the key to gradual automation - RAM NETWORK 15. Pingback: Do-nothing scripting: the key to gradual automation - LazyProgrammer 16. [132f2d3] Stephen Dowdy Nice! I do something similar, w/o the prompting bit. The script (mostly POSIX shell), doesn't do something (because i think there may be a failure mode i didn't account for, or just to ensure a real person is sanity checking what'll happen), by dumping out informational/warning text interspersed with actions tagged: : CUTANDPASTE :; command-to-exec args... (making it easy to identify the steps that need to be cut-pasted, and also allowing triple-click line-select on it (POSIX Shell effectively ignores the CUTANDPASTE and execs the command after ';' ) ) Use of variables are tested by POSIX emtpy/null check: "${var:? You Forgot To Set var}" There's also the CYA of : " *My Script* didn't bork the system, *You* blindly taking my script's advice borked the system ;)" 2019/07/22 at 14:20 Reply 17. Pingback: Wordpress Web Site Disaster Recovery On The Cheap - Developer Coach 18. Pingback: Do-nothing scripting: the key to gradual automation | Pratikar 19. Pingback: New best story on Hacker News: Do-nothing scripting: the key to gradual automation - Golden News 20. Pingback: Do-nothing scripting: the key to gradual automation - Infinity News 21. Pingback: New best story on Hacker News: Do-nothing scripting: the key to gradual automation - protipsss 22. Pingback: Do-nothing scripting: the key to gradual automation 23. Pingback: Do-nothing scripting: the key to gradual automation | toppertrick 24. [ykh6WSp] Tecdom (@tecdom) Esse script e em Python ? 2019/07/23 at 18:09 Reply 1. [bb447f0] Dan Slimmon Sim! 2019/07/23 at 21:58 Reply 25. [000817e] Jorg Strebel Congratulations on re-inventing a (simple) workflow management system. You could easily add logging and auditing for legal/ safety reasons, so the do-nothing script would at least add some value. 2019/07/24 at 07:27 Reply 1. [bb447f0] Dan Slimmon ok 2019/07/24 at 10:30 Reply 26. [a60d796] Ironman Your approach is really appreciated, but how about the logic used like this: import sys def wait_for_enter(): raw_input("Press Enter to continue: ") class CreateSSHKeypairStep(object): def run(self, context): print("Run:") print(" ssh-keygen -t rsa -f ~/{0}".format(context["username"])) class GitCommitStep(object): def run(self, context): print("Copy ~/new_key.pub into the `user_keys` Git repository, then run:") print(" git commit {0}".format(context["username"])) print(" git push") class WaitForBuildStep(object): build_url = "http://example.com/builds/user_keys" def run(self, context): print("Wait for the build job at {0} to finish".format (self.build_url)) class RetrieveUserEmailStep(object): dir_url = "http://example.com/directory" def run(self, context): print("Go to {0}".format(self.dir_url)) print("Find the email address for user `{0}`".format(context ["username"])) context["email"] = raw_input("Paste the email address and press enter: ") class SendPrivateKeyStep(object): def run(self, context): print("Go to 1Password") print("Paste the contents of ~/new_key into a new document") print("Share the document with {0}".format(context["email"])) class DoSequanceTask(task_number i) { def run(self, context): if task_number==1: CreateSSHKeypairStep() elif task_number==2: GitCommitStep() elif task_number==3: WaitForBuildStep() elif task_number==4: RetrieveUserEmailStep() elif task_number==5: SendPrivateKeyStep() if task_number<5: wait_for_enter() DoSequanceTask(task_number+1) else: print("Done.") } if __name__ == "__main__": context = {"username": sys.argv[1]} DoSequanceTask(1) 2019/07/24 at 09:08 Reply 27. [4823ab0] Patrick Blesi Awesome post! I created a Ruby DSL specifically designed for this use case. It also incorporates a number of other helpful features for working with your do-nothing script such as generating a representation in a yaml format and resuming your script at the same step if it fails. Check it out at: https://github.com/braintree/runbook 2019/07/24 at 16:28 Reply 1. [bb447f0] Dan Slimmon Patrick, this kicks ass. Thank you for writing it. 2019/08/02 at 20:09 Reply 28. [897a21a] maxluebbe My srecon talk was about this, strong believer in this approach! https://www.usenix.org/conference/srecon19americas/presentation/ luebbe 2019/07/24 at 21:05 Reply 29. Pingback: === popurls.com === popular today 30. Pingback: Quick Update | SwimGeek 31. Pingback: Collective #536 | Web Design News from Dubado 32. Pingback: Collective #536 - Unsorted 33. Pingback: Collective #536 - Find Right Software 34. Pingback: Collective #536 - Slacker News 35. Pingback: Collective #536 - News 20h 36. Pingback: 2019_07_31 News of Note? - DC3IT Helpful Links 37. Pingback: Android-VPN mit Wireguard - Mitch's Manga Blog 38. [OxzpLhA] Seth Livingston (@sethattheoffice) Absolutely love this. Thanks, Dan. https://github.com/ sethlivingston/do-nothing-scripts 2019/08/03 at 14:02 Reply 39. [113bcc8] Will Sewell Great post. Do you know of any frameworks that help with writing these "do nothing scripts"? It would be nice to factor out the common parts. 2019/08/05 at 12:35 Reply 1. [4823ab0] pblesi I created a framework that provides a lot of the boilerplate for do-nothing scripts and a handful of other features for graduallly automating a process: https://github.com/braintree /runbook 2019/08/07 at 12:32 Reply 2. [9d5487d] Ben I will throw my hat into the ring with a lightweight approach to turning these concepts into a Python library: https:// github.com/UnquietCode/runbook.py 2019/08/14 at 02:43 Reply 40. [9d5487d] Benjamin Fagin Hi Dan. Thanks for this post! Like several other folks, I was inspired to turn this into a minimal framework for running and authoring simple repeatable processes. The idea sticks, and I'll never look at a 20 page "runbook" document the same again. https://github.com/UnquietCode/runbook.py 2019/08/14 at 02:42 Reply 41. [b91a192] j2 Some of these steps could be executed by the script - why not do that? (Some can't of course). Also, a Jenkins pipeline might be an alternative way to achieve this with some accountability of who executed which step plus it can be replayed from a middle step if it fails (well if it can fail). 2019/08/21 at 12:06 Reply 42. Pingback: Links: 9-3-2019 | Aaron Johnson 43. Pingback: Get started with automation by writing scripts that do nothing - logandonley.com 44. Pingback: Notes: Do-nothing scripting: the key to gradual automation 45. Pingback: Gradual automation in Ruby - Slacker News 46. Pingback: Gradual automation in Ruby - Tech News 47. Pingback: Do-nothing scripting: the key to gradual automation 48. Pingback: Do-nothing scripting: the key to gradual automation - The web development company 49. [a1db351] Jay Kint I really like this idea. Implementing it, I would probably put in some sort of audit trail for each step, such as a log into a database or something, that assured that the step was done. There might not be a way to verify that the step was actually done, but it would serve as some sort of proof that the script was run and that the step was alleged to have been done. 2021/11/02 at 19:13 Reply Leave a Reply Cancel reply Enter your comment here... [ ] Fill in your details below or click an icon to log in: * * * * Gravatar Email (required) (Address never made public) [ ] Name (required) [ ] Website [ ] WordPress.com Logo You are commenting using your WordPress.com account. ( Log Out / Change ) Google photo You are commenting using your Google account. ( Log Out / Change ) Twitter picture You are commenting using your Twitter account. ( Log Out / Change ) Facebook photo You are commenting using your Facebook account. ( Log Out / Change ) Cancel Connecting to %s [ ] Notify me of new comments via email. [ ] Notify me of new posts via email. [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Search for: [ ] Search Top Posts * Do-nothing scripting: the key to gradual automation * An Incident Command Training Handbook * Stop apologizing for bugs * What makes a good alert? * The Latency/Throughput Tradeoff: Why Fast Services Are Slow And Vice Versa Archives * March 2021 * August 2019 * July 2019 * June 2019 * May 2019 * March 2019 * February 2019 * October 2017 * July 2017 * June 2017 * April 2017 * September 2016 * August 2016 * June 2016 * March 2016 * December 2015 * October 2015 * July 2015 * June 2015 * September 2014 * August 2014 * July 2014 * May 2014 * October 2013 * September 2013 * June 2013 * May 2013 * April 2013 * March 2013 * December 2012 * November 2012 * August 2012 * July 2012 Categories * Monitoring * Problem-Solving * Statistics and probability * Uncategorized * Workflow Meta * Register * Log in * Entries feed * Comments feed * WordPress.com Blog at WordPress.com. %d bloggers like this: [b]