[HN Gopher] Atuin Desktop: Runbooks That Run
___________________________________________________________________
Atuin Desktop: Runbooks That Run
Author : freetonik
Score : 536 points
Date : 2025-04-22 20:54 UTC (1 days ago)
(HTM) web link (blog.atuin.sh)
(TXT) w3m dump (blog.atuin.sh)
| gyrovagueGeist wrote:
| All the problems of reproducibility in Python notebooks
| (https://arxiv.org/abs/2308.07333,
| https://leomurta.github.io/papers/pimentel2019a.pdf) with the
| power of a terminal.
| milkshakes wrote:
| what are the problems you're talking about? your references
| seem to refer to reproducing scientific publications,
| dependency issues, and cell execution ordering.
|
| this project appears to be intended for operational
| documentation / living runbooks. it doesn't really seem like
| the same use case.
| shadowgovt wrote:
| Agreed. The problem with reproducing Jupyter runbooks in
| academia is that someone thought a Jupyter runbook is a way
| to convey information from one person to another. Those are
| an awful model for that.
|
| As an on-the-fly debugging tool, they're great: you get a
| REPL that isn't actively painful to use, a history (roughly,
| since the state is live and every cell is not run every time)
| of commands run, and visualization at key points in the
| program to check as you go your assumptions are sound.
| rtpg wrote:
| I mean it feels pretty obvious to me that cell execution
| order is a pretty real issue for a runbook with a bunch of
| steps if you're not careful.
|
| I do think that given the fragile nature of shell scripts
| people tend to write their operation workflows in a pretty
| idempotent way, though...
| ellieh wrote:
| agreed - we actually have a dependency system in the works
| too!
|
| you can define + declare ordering with dependency
| specification on the edges of the graph (ie A must run
| before B, but B can run as often as you'd like within 10
| mins of A)
| nine_k wrote:
| There of course should be a way to override the
| dependency, by explicitly pressing a big scary "[I know
| what I'm doing]" button.
|
| Another thing is that you'll need branches. As in:
| - Run `foo bar baz` - If it succeeds, run `foo
| quux`, Else run `rm -rf ./foo/bar` and rerun the
| previous command with `--force` option. - `ls
| ./foo/bar/buur` and make certain it exists.
|
| Different branches can be separated visually; one can be
| collapsed if another is taken.
|
| Writing _robust_ runbooks is not that easy. But I _love_
| the idea of mixing the explanatory text and various types
| of commands together.
| noodletheworld wrote:
| I mean, is it worse than having it:
|
| - in excel
|
| - in a confluence document
|
| - in a text file on your desktop
|
| The use case this addresses is 'adhoc activites must be
| performed without being totally chaotic'.
|
| Obviously a nice one-click/trigger based CI/CD deployment
| pipeline is lovely, but uh, this is the real world. There
| are plenty of cases where that's simply either not
| possible, or not worth the effort to setup.
|
| I think this is great; if I have one suggestion it would
| just be integrated logging so there's an immutable shared
| record of what was _actually done_ as well. I would _love_
| to be able to see that Bob started the 'recover user
| profile because db sync error' runbook but _didn 't finish
| running it_, and _exactly_ when that happened.
|
| If you think it's a terrible idea, then uh, what's your
| suggestion?
|
| I'm pretty tired of copy-pasting commands from confluence.
| I think that's, I dunno, unambiguously terrible, and
| depressingly common.
|
| One time scripts that are executed in a privileged remote
| container also works, but at the end of that day, those
| script tend to be specific and have to be _invoked_ with
| custom arguments, which, guess what, usually turn up as a
| sequence of operations in a runbook; query db for user id
| (copy-paste SQL) - > run script with id (copy paste to
| terminal) -> query db to check it worked (copy paste SQL)
| -> trigger notification workflow with user id if it did
| (login to X and click on button Y), etc.
| rtpg wrote:
| I'm not against this notebook style, I have runbooks in
| Jupyter notebooks.
|
| I just think it's pretty easy to do things like start a
| flow back up halfway through the book and not fix some
| underlying ordering issues.
|
| With scripts that you tend to have to run top to bottom
| you end up having to be more diligent with making sure
| the initial steps are still OK because on every test you
| tend to run everything. Notebook style environments favor
| running things piecemeal. Also very helpful! It
| introduces a much smaller problem in the process of
| solving the larger issue of making it easier to do this
| kind of work in the first place.
| packetlost wrote:
| This is more like literate programming (but for shells) than
| jupyter notebooks.
| mananaysiempre wrote:
| Literate programming really needs the ability to reorder,
| otherwise it's just sparkling notebooks. (Except for Haskell,
| which is order-independent enough as it is that the
| distinction rarely matters.)
| theLiminator wrote:
| Give marimo a try, it's much better for reproducibility.
| mdaniel wrote:
| linky https://github.com/marimo-
| team/marimo#:~:text=all%20in%20a%2... _(Apache 2)_
| ugiox wrote:
| From their repo:
|
| "A reactive notebook for Python -- run reproducible
| experiments, query with SQL, execute as a script, deploy as
| an app, and version with git. *All in a modern, AI-native
| editor.*
|
| Why does it need to be in a "modern, AI-native editor"?
|
| (Closing tab, flashing marimo out of brain)
| dang wrote:
| " _Please don 't post shallow dismissals, especially of other
| people's work. A good critical comment teaches us something._"
|
| https://news.ycombinator.com/newsguidelines.html
| freedomben wrote:
| Will this be open source like Atuin CLI and the sync server are?
| Is this going to be productized?
| gniting wrote:
| Most likely not free. Regardless, happy to see this be
| announced!
| iamwil wrote:
| Are you worried about getting rug pulled by the platform?
| freedomben wrote:
| Yes, enshittification potential is top of mind for me now
| when considering adopting any product. If it's open source
| then I worry less.
| jamietanna wrote:
| It'll be Open Source'd:
| https://news.ycombinator.com/item?id=43766200#43766584
| brunoqc wrote:
| Probably not the server though.
| dheerkt wrote:
| How is this different from a local Jupyter notebook? Can we not
| do this with ! or % in a .ipynb?
|
| Genuine question. Not familiar with this company or the CLI
| product.
| RestartKernel wrote:
| Jupyter Notebooks have always felt a bit hacky for terminal
| purposes to me, so I'm excited to give this a shot.
| BOOSTERHIDROGEN wrote:
| How about marimo?
| hashstring wrote:
| 100% same question.
|
| Usually, I feel like Jupyter gives both worlds--- flexible
| scripting and support for os commands (either through !/% or
| even os.system()
| berkes wrote:
| The main thing that keeps me from using Jupyter notebooks for
| anything that's not entirely Python, is Python.
|
| For me, pipenv/pyenv/conda/poetry/uv/dependencies.txt and the
| invitable "I need to upgrade Python to run this notebook, ugh,
| well, ok -- two weeks later - g####m that upgrade broke that
| unrelated and old ansible and now I cannot fix these fifteen
| barely held up servers" is pure hell.
|
| I try to stay away from Python for foundational stuff, as any
| Python project that I work on1 will break at least yearly on
| some dependency or other runtime woe. That goes for Ansible,
| Build Pipelines, deploy.py or any such thing. I would certainly
| not use Jupyter notebooks for such crucial and foundational
| automation, as the giant tree of dependencies and requirements
| it comes with, makes this far worse.
|
| 1 Granted, my job makes me work on an excessive amount of
| codebases, At least six different Python projects last two
| months, some requiring python 2.7, some requiring deprecated
| versions of lib-something.h some cutting edge, some very strict
| in practice but not documented (It works on the machine of the
| one dev that works on it as long as he never updates
| anything?). And Puppet or Chef - being Ruby, are just as bad,
| suffering from the exact same issues, only that Ruby has had
| one (and only one!) package management system for decades now.
| stevelacy wrote:
| Have been following along with the development, glad to see it
| announced!
| nu11ptr wrote:
| Looks neat. What tech stack is used for this? Is it open source
| by chance?
| iamwil wrote:
| Do you want it to be open source because of the price or
| because you're afraid of being rug pulled by the platform or
| you want to contribute?
| nu11ptr wrote:
| If I use something I like the idea that I can fix bugs should
| the need arise.
| ellieh wrote:
| Thanks! We're using Tauri (https://v2.tauri.app/) on the
| client, and Elixir + Phoenix (with a little bit of Rust via
| Rustler) on the server
|
| Tauri means we can reuse a lot of the Rust we already have,
| easily do the systems stuff we need, and have something light +
| fast. Elixir has been awesome and makes a realtime sync backend
| easier
|
| Not currently open source while it's under heavy early
| development, we will be opening up the desktop app later on
| tymscar wrote:
| Amazing. Im very happy this is not yet another electron app
| benatkin wrote:
| This is one place where it would be more likely to make
| sense to have an electron app, because with user code,
| you'd already have a lot of variables out of your control,
| and having a standard browser engine would help. Also
| unlike other apps, you hopefully wouldn't have 5 code
| notebook apps running.
| Philpax wrote:
| Tauri wraps around the system's web view, so it's
| semantically equivalent to Electron.
|
| (nb: system web views are very inconsistent, so they're
| considering adding a Chromium renderer, which will bring
| everything full circle)
| ellieh wrote:
| > nb: system web views are very inconsistent
|
| we've found they're generally ok between mac/windows,
| with some issues on Linux. Nothing insurmountable,
| however.
|
| Anything super complex (terminals, charts) we can render
| with canvas or webgl anyway
| benatkin wrote:
| > we will be opening up the desktop app later on
|
| This leaves room for stuff like the Functional Software
| License.
| sneak wrote:
| It is bothersome to see people who obviously don't believe in
| free software ideology and software freedoms (otherwise you
| would never produce nonfree software) (ab)using the open
| source community in this way.
|
| Software freedoms exist as a concept for a reason, not just a
| bullet point to get people to click a download link that
| doesn't even include source anyway.
|
| I call such projects "open source cosplay". It's an outfit
| you put on for conferences, then take off when back at the
| office working on the nonfree valuable parts.
| danenania wrote:
| Atuin's CLI for shell history is open source, has been free
| for years, and is a very useful tool. If the author now
| wants to build a product on top so she can make a living,
| that's a win for everyone: the author, the open source
| users (since the project will keep being maintained), and
| people who get value out of the new product she's building.
|
| The irony of this purist mindset is that it's actually very
| corporatist, big-tech, and proprietary in its implications.
| If open source devs are discouraged by the culture from
| building products and making a living independently, it
| means that the only people who can devote significant time
| to open source are employees of established companies (who
| themselves often sell closed source proprietary products)
| and people who are wealthy enough to work for free. Is that
| the world you want?
| sneak wrote:
| This only logically follows if you believe in the
| mistaken premise and false dichotomy that it is
| impossible to make money with foss software.
| shawabawa3 wrote:
| This kind of attitude is why less and less people are open
| sourcing software
|
| Why would I waste my time releasing any of my projects for
| free when people will attack me and call me a poser anyway
|
| Might as well charge people money, who by the way will
| actually be grateful to do so, that try to keep up with the
| open source community's purity treadmill
| sneak wrote:
| There's no treadmill. Belief in software freedoms has
| always been belief in software freedoms.
|
| It's ok if you don't believe in software freedoms, but
| you shouldn't pretend to be someone who does by releasing
| _some_ software that respects users' software freedoms.
| It's deceptive.
|
| Either you care about software freedoms, or you don't. If
| you don't, why are you releasing any software under free
| software licenses? If you do, why are you releasing any
| nonfree software?
|
| Also, do you have a single bit of backing data to suggest
| that your first sentence is true? I don't believe that it
| is. It seems to me there is more free software than ever
| before.
| freedomben wrote:
| > _It's ok if you don't believe in software freedoms, but
| you shouldn't pretend to be someone who does by releasing
| some software that respects users' software freedoms.
| It's deceptive._
|
| As long as the different parts are clearly
| marked/indicated as such, why would you impose such a
| ridiculous standard? In your world, if a company makes
| 99% of their software GPL, and then releases some
| proprietary tool, they're suddenly being deceptive? Would
| you prefer to just lose the 99%?
| freedomben wrote:
| I agree GP's attitude is ridiculous, but if a very small
| number of purists on the internet is the reason less
| people are open sourcing software, then those people are
| just as guilty of bad reasoning/judgment as the purists.
|
| IMHO the real reason is that the threat of
| hostile/competing forks has gone up. It used to be gauche
| at best, evil at worst, to take somebody's open source
| code and compete with them, but increasingly the
| landscape is changing. I think that's the real problem,
| and IMHO the answer to that is the AGPL, not to go
| proprietary.
| paradox460 wrote:
| Are there any plans to add an integration to something like
| Phoenix LiveBook?
| moonlion_eth wrote:
| the waitlist social media jump the list mechanic is kinda sus,
| regardless joined the waitlist
| 0xbadcafebee wrote:
| If it's local-first then it's already subject to rot. Unless
| they're running it all in containers? In which case local doesn't
| matter.
|
| If you want to record a runbook, then record a runbook. You can
| do that a million ways. Text file, confluence doc, screen
| recording, shell script, etc. People already don't do that;
| they're not gonna suddenly start doing it more because your UI is
| fancier.
|
| Personally, I don't want to sit around all day writing code (or
| docs) to try to get the system to be like X state. I want to
| manually make it have X state, and then run a tool to dump the
| state, and later re-run the tool to create (or enforce) that
| state again. I do not want to write code to try to tell the
| computer how to get to that state. Nor do I want to write
| "declarative configuration", which is just more code with a
| different name. I want to do the thing manually, then snapshot
| it, then replay it. And I want this to work on any system,
| anywhere, without dependence on monitoring a Bash shell for
| commands or something. Just dump state and later reapply state.
| milkshakes wrote:
| what happens when you want to tweak something you did in the
| middle of this process? do you have to go through the whole
| flow again manually to make a single change?
| 0xbadcafebee wrote:
| I imagine you could either A) just modify the dumped state,
| B) paramaterize it, C) have the program split up the state
| into transactions and modify those. The program will probably
| have to take more than one step, in order, in order to
| accomplish everything. If it fails, you'd want it to try to
| undo it, ala transactions. And since it can do all that, it
| can stop, start, or resume at specific steps.
|
| Like, Terraform has always sucked because there was no way to
| dump existing resources as new code. So a team at Google made
| a tool to do it (Terraform- _er_ ). If Terraform had already
| had that feature, and if it didn't rely on having pre-
| existing state to manage resources, that would be like 90% of
| the way to what I'd want. Just dump resources as code, then
| let me re-run the code, and if I want I can modify the code
| to ask me for inputs or change things. (People think of
| Terraform as only working on Cloud resources, but you could
| (for example) make an _Ubuntu Linux provider_ that just
| configures Ubuntu for you, if you wanted)
| kiitos wrote:
| Any notion of state that satisfies requirements like
|
| > Just dump state and later reapply state
|
| is necessarily declarative.
|
| > Just dump resources as code,
|
| What is the code for this resource? VM
| foo1 Memory 16GiB Network mynet1
|
| It depends on the current state of the system where the
| resource is applied. If VM foo1 already exists, with 16GiB
| of memory, and connected to network mynet1, then the code
| is a no-op, no code at all. Right? Anything else would be a
| mistake. For example if the code would delete any matching
| VM and re-create it, that would be disastrous to continuity
| and availability, clearly a non-starter. Or, if VM foo1
| exists, with 16GiB of memory, but connected to anothernet3,
| then the code should just change the network for that VM
| from anothernet3 to mynet1, and should definitely not
| destroy and re-create the VM entirely. And so on.
| 0xbadcafebee wrote:
| It depends what you're talking about; Terraform
| specifically has a flawed model where it assumes nothing
| in the world exists that it didn't create itself. Other
| configuration management tools don't assume that; they
| assume that you just want _an item to exist_ ; if it does
| exist, great, if it doesn't exist, you create it. But for
| a moment I'll assume you're talking about the _other_
| problem with configuration management tools, which is "
| _which_ of the existing resources do I actually want to
| exist or modify? "
|
| That's a solved problem. Anything that you use on a
| computer that controls a resource, can uniquely identify
| said resource, through either a key or composite key.
| This has to be the case, otherwise you could create
| things that you could never find again :) (Even if you
| created an array of things with no name, since it exists
| as an item in a list, the list index is its unique
| identifier)
|
| Taking Terraform as example again, the provider has code
| in it that specifies what the unique identifier is, per-
| resource. It might be a single key (like 'id', 'ASN',
| 'Name', etc) or a composite key ( {'id' + 'VPC' +
| 'Region'} ).
|
| If the code you've dumped does not have the unique
| identifier for some reason, then the provider has to make
| a decision: either try to look up existing resources that
| match what you've provided and assume the closest one is
| the right one, or error out that the unique identifier is
| missing. Usually the unique identifier is not hard to
| look up in the first place (yours has a composite
| identifier: {VM:"foo1", Network:"mynet1"}). But it's also
| (usually) not fool-proof.
|
| Imagine a filesystem. You actually have two unique
| identifiers: the fully-qualified file path, and the inode
| number. The inode number is the actual unique identifier
| in the filesystem, but we don't tend to reference it, as
| 1) it's not that easy to remember/recognize an inode
| number, 2) it can be recycled for another file, 3) it'll
| change across filesystems. We instead reference the file
| path. But file paths are subtly complex: we have sym-
| links, hard-links and bind-mounts, so two different paths
| can actually lead to the same file, or different files!
| On top of that, you can remove the file and then create
| an identically- _named_ file. Even if the file had
| identical contents, removing it and creating a new one is
| technically a whole new resource, and has impact on the
| system (permissions may be different, open filehandles to
| deleted files are a thing, etc).
|
| So what all of us do, all day, every day, is lie to
| ourselves. We pretend we can recognize files, that we
| have a unique identifier for them. But actually we don't.
| What we do is use a composite index and guess. We say, _"
| well it looks like the right file, because it's in the
| right file path, with the right size, and right name, and
| right permissions, and (maybe) has the right inode"_. But
| actually there's no way to know for sure it's the same
| file we expect. We just hope it is. If it looks good
| enough, we go with it.
|
| So that's how you automate managing resources. For each
| type of resource, you use whatever you can as a unique
| (or composite) identifier, guesstimate, and prompt the
| user if it's impossible to get a good enough guess.
| Because that's how humans do it anyway.
| kiitos wrote:
| > Terraform specifically has a flawed model where it
| assumes nothing in the world exists that it didn't create
| itself.
|
| I don't think this is accurate. Terraform operates
| against a state snapshot, which is usually local but can
| also be remote. But it has several mechanisms to update
| that state, based on the current status of any/all
| defined resources, see e.g. `terraform refresh` (https://
| developer.hashicorp.com/terraform/cli/commands/refre...)
| -- and there are other, similar, commands.
|
| > But for a moment I'll assume you're talking about the
| other problem with configuration management tools, which
| is "which of the existing resources do I actually want to
| exist or modify?"
|
| I'm not really talking about that specific thing, no.
| That problem is one of uncountably many other similar
| sub-problems that configuration management tools are
| designed to address. And, for what it's worth, it's not a
| particularly interesting or difficult problem to solve,
| among all problems in the space.
|
| If you have a desired state X, and an actual state Y,
| then you just diff X and Y to figure out the operations
| you need to apply to Y in order to make it end up like X.
| Terraform does this in `terraform plan` via a 3-way
| reconciliation merge/diff. Pretty straightforward.
|
| > you just want an item to exist; if it does exist,
| great, if it doesn't exist, you create it
|
| It's not as simple as whether or not an item should
| exist. Being able to uniquely identify a resource is step
| one for sure. But a single resource, with a stable
| identifier, can have different properties. The entire
| resource definition -- identifier, properties, and
| everything else -- is what you type and save and commit
| and push and ultimately declare as the thing you want to
| be true (X). That's not code, it's state (definitions).
| Code is what's executed to diff that declarative state
| (X) against actual state (Y) to produce a set of delta
| operations. Or, it's those delta operations themselves.
|
| > If the code you've dumped does not have the unique
| identifier for some reason, then the provider has to make
| a decision: either try to look up existing resources that
| match what you've provided and assume the closest one is
| the right one...
|
| First, you "dump" state, not code. More importantly, no
| configuration management system would ever take one
| identifier and "guesstimate" that it should match a
| different identifier, because it's "close", whatever that
| means.
|
| > or error out that the unique identifier is missing.
| Usually the unique identifier is not hard to look up in
| the first place (yours has a composite identifier:
| {VM:"foo1", Network:"mynet1"}). But it's also (usually)
| not fool-proof.
|
| I really don't understand what you mean, here, nor do I
| understand your mental model of these systems. It's
| certainly not the case that my example VM has the
| composite identifier {vm:foo1 network:mynet1}. The
| identifier is, intuitively, just foo1. Even if we were to
| say the identifier were an object, the object you propose
| is missing the memory size. But more importantly,
| changing the foo1 VM from network:mynet1 to
| network:othernet2 probably should not have the effect of
| destroying the existing VM, and re-provisioning a brand
| new VM with the new network. Sometimes configuration
| changes require this kind of full teardown/spinup, but
| these conditions are generally rare, and all modern
| configuration management tools avoid this kind of
| destructive work whenever possible and most of the time.
|
| > So that's how you automate managing resources. For each
| type of resource, you use whatever you can as a unique
| (or composite) identifier, guesstimate, and prompt the
| user if it's impossible to get a good enough guess.
| Because that's how humans do it anyway.
|
| Just to reiterate, I'm not aware of any configuration
| management tool that "guesstimates" when making changes
| in this way. For good reason.
| 0xbadcafebee wrote:
| `terraform refresh` (which is now `terraform apply
| -refresh-only`) is an exception to the rule. Terraform
| doesn't know what's going on in the outside world. If you
| write configuration to create a Security Group named
| "foobar", and do a `terraform plan`, it will say it's
| about to create "foobar". When you go to apply, it will
| error out, saying "foobar already exists".
|
| If Terraform wasn't completely idiotic, it could have
| just checked if it existed in the planning stage. If
| Terraform was even mildly helpful, it would have
| suggested to the user at either plan or apply time that
| the security group already exists, and do you want to
| manage that with your code? But it doesn't do those
| things, because it's a completely dumb-ass design.
|
| > I'm not aware of any configuration management tool that
| "guesstimates" when making changes. Thank God.
|
| Many of them do. Ansible does, Puppet does, Terraform
| does. They have to, for the same reason as my filesystem
| example: it's often impossible to know that a resource is
| unique, because there aren't actually unique identifiers.
| My definition of "Guesstimation" is specifically "using
| the identifiers you have available to select an entry
| from a list of potential options with the closest match".
| Ansible does this all the time. Puppet and Terraform do
| this for every provider that doesn't have a totally
| unique identifier (there basically are no totally unique
| identifiers, as I pointed out in my filesystem example)
| kiitos wrote:
| Wow you really hate Terraform!
|
| It seems to me that your frustration with Terraform being
| "completely idiotic" is ultimately frustration with the
| underlying design model.
|
| > If you write configuration to create a Security Group
| named "foobar",
|
| That configuration is a declaration: a security group
| named "foobar" should exist, with the declared
| properties.
|
| > and do a `terraform plan`, it will say it's about to
| create "foobar".
|
| That plan would be based on the most recent snapshot of
| the target "outside world" resources, which, if you
| haven't synced them recently (or at all) would probably
| be empty, resulting in `terraform plan` proposing to
| create foobar afresh.
|
| > When you go to apply, it will error out, saying "foobar
| already exists"
|
| Sure, which should hopefully make sense. You've declared
| a resource locally, and asked Terraform to "make it so"
| basically. But that resource is in conflict with an
| identical remote resource. You can `terraform refresh` or
| sync or whatever, to pull down the current relevant
| remote resource state locally, and then operate from
| there. Or you can manually blow away the remote foobar
| and retry. Or etc.
|
| But this kind of situation is not common. Terraform
| assumes and expects that the declarations (and state) it
| has access to locally is an authoritative source of truth
| for what the target remote system(s) should be. The
| config files define what should be running in AWS, not
| the other way around.
|
| It's fine if this isn't a fit for your use cases, but I
| don't think that means the entire tool is stupid or
| whatever. It just means it's not for you.
| jimbokun wrote:
| That was the Docker manifesto.
| x-complexity wrote:
| > That was the Docker manifesto.
|
| It essentially still is.
|
| Unless the Dockerfiles are kept secret, any container can be
| replicated from the given Dockerfile. Barring extreme
| (distro/system/hardware)-level quirks, a Docker container
| should be able to run anywhere that Linux can.
| spott wrote:
| A container can very rarely be reproduced by a dockerfile.
|
| I imagine with a lot of discipline (no apt update, no
| "latest" tag, no internet access) you can make a
| reproducible docker file.... But it is far from normal.
| taberiand wrote:
| Well sure, making a 100% reproducible build is hard - but
| Docker makes it easier, not harder. If 100% reproducible
| is the goal, what's easier than docker?
| zelphirkalt wrote:
| Tools that have been designed with reproducibility in
| mind. Like Guix.
|
| Beware, I am definitely not claiming those are easy to
| use in general. Just that you can get to reproducibility
| using them more reliably and maybe easier than with
| docker.
| otabdeveloper4 wrote:
| > but Docker makes it easier, not harder
|
| Incorrect. Step one of reproducibility is "disable
| unconstrained downloading from the internet". Docker does
| the opposite.
| taberiand wrote:
| Presumably if your goal is a reproducible build you just
| wouldn't do any unconstrained downloading in the process
| of designing the dockerfile and building the image.
| Making a choice to use a tool poorly for you requirements
| isn't a problem with the tool.
| soraminazuki wrote:
| The claim the parent was addressing was that Docker helps
| with reproducibility. It doesn't. Docker does nothing at
| all in this regard.
|
| If you want a reproducible Docker image, you're on your
| own. For example, the most common problem is that many
| build scripts out in the wild download stuff willy nilly
| without verifying anything. I've seen NPM package post
| install scripts do the craziest things. Catching all of
| that is harder than most would give credit for at first
| glance, considering that tons of build scripts are
| written in Turing complete languages. Help from tooling
| is essential.
|
| When you have to fight the tool to achieve
| reproducibility, not choosing to do so isn't "using a
| tool poorly." It's simply using the tool as is.
| Especially when the vast majority of Dockerfiles out
| there happily run something along the lines of `apt
| install foo`, again, without verifying anything.
| metabagel wrote:
| Vagrant creates reproducible VMs. Not quite the same
| thing of course.
|
| https://developer.hashicorp.com/vagrant
| jpgvm wrote:
| Full reproducibility isn't easy, there is a cost to it.
|
| However the payoff is rather significant so if you can
| temper that cost a bit and make it less inconvenient to
| achieve then you have a winning solution.
|
| I have cooked this up based on Bazel, rules_oci and
| rules_distroless:
| https://github.com/josephglanville/images Specifically
| this file is a busybox based image with some utilities
| included from a Debian snapshot: https://github.com/josep
| hglanville/images/blob/master/toolbo...
|
| More difficult than Dockerfile? Sure. However better in
| pretty much every way otherwise including actual
| simplicity.
| soraminazuki wrote:
| A Dockerfile is essentially a shell script with access to
| the outside world. It has unconstrained network access.
| It can access local hardware and filesystem if instructed
| to. However, it doesn't verify that whatever stuff it
| took from the outside remains the same across builds.
| Docker doesn't care if the same Dockerfile builds Apache
| httpd in one build and Nginx in another. It literally
| can't get more irreproducible than that.
|
| But mysteriously, people say that Docker is reproducible
| because, uh, you can download gigabyte-sized binary blobs
| from the Docker registry. I wonder, what's _not_
| reproducible by that metric?
|
| Docker images may be _portable_ compared to binaries
| targeting traditional FHS distros. But it 's not
| _reproducible_ whatsoever.
| angra_mainyu wrote:
| People rarely mean 100% build reproducibility, but simply
| within a reasonable limit, Dockerfiles are mostly "run
| stable" and provide the same OS abstraction and process
| encapsulation.
| manquer wrote:
| You are mixing build time reproduction with run time ones.
|
| Docker images (not files) help with the run time
| consistency .
|
| Docker (files) barely scratch the surface of build
| reproducibility. Most applications depend on the
| distribution package manager ( apt, apk etc) and language
| package manager (npm, cargo, etc), both sets of them have
| various challenges in consistent dependency resolution.
|
| In addition build steps might have ordering challenges RPC
| calls to remote services no longer running and so on.
|
| Anyone trying to to build a docker image from 10 years back
| experiences this problem
| wink wrote:
| You're right in the absolute form, but I've yet to see a
| Dockerfile where (with a little thinking and elbow
| grease) I couldn't "easily" port it or update it, even
| after years.
|
| It's basically the best and easiest "I am documenting how
| it works now" thing without any arcane "works on my
| machine" quirks I have yet found.
|
| So I'm still agreeing here that it's a very good
| approximation of this idea.
|
| Real reproducability is miles better, but usually cannot
| be formulated in a ~20 line single file "recipe". (and
| before anyone mentions Nix.. no, there's so much inherent
| complexity involved, that doesn't count like "apt-get
| install docker && docker build ."
| LinXitoW wrote:
| So you then have binary blobs of state without any
| documentation of how or why it is the way it is? That doesn't
| seem maintainable.
|
| Dockerfiles are basically this, but with a file documenting the
| different steps you took to get to that state.
| dewey wrote:
| That's not what they are saying. They are saying that the
| system where you have to declare everything manually is
| annoying (which it is), ideally it would record the changes
| while you make changes and then deduplicate them, remove
| unnecessary ones to arrive at the final playbook that can be
| replayed if needed.
| naikrovek wrote:
| yes it _would_ be nice to have a computer that could read
| your mind flawlessly.
| 0xbadcafebee wrote:
| Then I should explain why "a blob" is, in a weird way,
| actually superior to a Dockerfile.
|
| Step 1. You write a Dockerfile. You build it. You test with
| it. Hey, it works! You push it to production.
|
| Step 2. Years later, you need to patch that container in
| production. So you change the Dockerfile, rebuild, and re-
| test.
|
| Step 3. Uh-oh! The tests don't work anymore! What's going on?
| I changed one line in the Dockerfile but now the app isn't
| working (or perhaps the build isn't). What's going on?
|
| What's going on is a _reproducibility failure_. Just having
| the instructions (or what someone thought was the
| instructions, or what they were years ago) isn 't enough to
| ensure you get the same results a second time. A million
| little things you didn't think of may change between builds
| (and the more time in-between, the more things change), and
| they may end up breaking things. Without a truly reproducible
| build, you are likely to have problems trying to rebuild with
| just the instructions.
|
| That's why with Docker containers, we rely on _build
| artifacts_. The container you had two years ago? However that
| was built, with whatever weird combination of arguments and
| files-on-the-internet-being-pulled-at-build-time and
| everything else, it was built successfully. If you want to be
| really sure you patch it correctly, you pull the old
| container _image_ (not Dockerfile), modify _that_ , and push
| that to production as a new container. No rebuilding, just
| patching. This avoids reproducibility failures.
|
| That same idea is why you'd want to just download a blob and
| later re-apply it.
|
| The blob was the state of things when it was working. If you
| tried to just write down the instructions to replicate it,
| it's likely you'd either 1) get it wrong (it was only working
| because of some unrelated changes somebody else made and
| forgot about) or 2) you'd get a reproducibility error.
|
| So "the blob" I'm talking about doesn't have to be a literal
| binary blob. It could be whatever, we're talking about a
| theoretical idea here. It could be layers like a container,
| or metadata in a JSON file, or configuration/code that gets
| auto-generated, etc. I don't care what it is. It just has to
| _describe the state as it was when X was working_. It 's then
| up to the tool to figure out _how to get back to that state_.
|
| People already write this as "declarative code" for
| configuration management tools to do the same thing. I'm
| saying, I don't want to have to write the code. Just dump it
| out for me.
| pram wrote:
| Sounds like you want autoexpect!
|
| https://linux.die.net/man/1/autoexpect
| 0xbadcafebee wrote:
| Actually no, that's still just monitoring a series of steps
| to eventually lead to a thing. I don't want to record a
| series of steps, I just want to dump existing state of a
| thing.
| retrochameleon wrote:
| It kind of sounds like you're describing Ansible. You use
| modules for common tasks like ensuring a package is installed,
| a file is present or has certain content, etc. It's declarative
| and imdempotent.
| bobthecowboy wrote:
| I've written some fairly complex stuff in Ansible. It is
| mostly declarative but you should be careful with assumptions
| about its idempotency, especially if you reach out for
| community modules.
| 0xbadcafebee wrote:
| No, I don't want to write Ansible configuration. I don't want
| to write _any_ configuration. I just want to dump state and
| restore it. (To put it another way: I want to auto-generate
| ansible configuration based on an existing system, and then
| apply that configuration again later)
| zelphirkalt wrote:
| Such a process is rarely portable though, and will need to be
| repeated for each different system, at which point it would be
| great to already have a declarative description, that can
| automatically be translated into those steps required to get to
| state X.
| sorrythanks wrote:
| > If it's local-first then it's already subject to rot.
|
| Can you expand on this?
| 0xbadcafebee wrote:
| When most people say "local" what they mean is "i'm running
| _something_ on my laptop ". That is to say, it's a random
| operating system, with a randomly set-up environment, with
| randomly installed tools. Could be different tools that have
| the same name but incompatible options. Could be any version
| (version 3 might be incompatible with version 4, but it's the
| same name for the command). And it will definitely change
| over time.
|
| This will lead whatever steps you've recorded to 1) stop
| working on the existing "local" machine, and 2) be
| incompatible with other people's "local" machines. So the
| instructions have "rotted".
|
| To avoid this you could write Ansible/Puppet to install and
| configure all the same tools, but that will break too over
| time too, and be a maintenance hassle. The only reliable
| solution is to use containers to run it all in; that
| guarantees the same version of everything. But at that point
| it's not really "running locally" anymore, it's running in
| the container, which is sort of its own bag of issues.
|
| At that point you might as well have a SaaS tool to run your
| runbook in a cloud environment in containers or something, as
| that's way easier to set up and manage than either
| ansible/puppet, or Docker on everyone's machine (there's
| still a million tech "engineers" out there who don't
| understand containers).
| IOT_Apprentice wrote:
| Pre Cloud at eBay we managed 30k windows servers via a tool
| called site controller that referenced ALL servers as
| objects with properties of configuration state, build
| version and operational state.
|
| ALL databases had their startup configuration parameters
| defined per instance across datacenters.
|
| Furthermore SRE had tools to rate limit connections or
| restart of a database so that it was not overwhelmed by
| incoming connection requests. We also built tools to do
| fine grained definition of load balancers and what services
| were behind them to be able to redirect traffic on the fly
| and then reset to the original mappings once connectivity
| of the LBs was resolved.
|
| These tools weee centralized and available to both SRE and
| senior system administrators.
|
| These things evolved to new tools which accommodated a
| private cloud, then docker then Kubernetes. I left prior to
| kubernetes implementation.
| account-5 wrote:
| I can't say I see the point in this. Can someone explain what I'm
| missing? Why would I use this over a simple shell script?
| rc00 wrote:
| Because it's written in Rust and this is Hacker News.
| account-5 wrote:
| I was going to talk about using powershell but just for the
| rust I also really like Nushell. I personally would take
| either one over this...
| joh6nn wrote:
| My experience with runbooks has been:
|
| - I am on a team that oversees a bunch of stuff, some of which
| I am very hands-on with and comfortable with, and some of which
| I am vaguely aware exists, but rarely touch
|
| - X, a member of the latter category, breaks
|
| - Everyone who actually knows about X is on vacation/dead/in a
| meeting
|
| - Fortunately, there is a document that explains what to do in
| this situation
|
| - It is somehow both obsolete and wrong, a true miracle of bad
| info
|
| So that is the problem this is trying to solve.
|
| Having discussed this with the creator some[1], the intent here
| (as I understand it) is to build something like a cross between
| Jupyter Notebooks and Ansible Tower: documentation, scripts,
| and metrics that all live next to each other in a way that
| makes it easier to know what's wrong, how to fix it, and if the
| fix worked
|
| [1]Disclosure: I help mod the atuin Discord
| mmooss wrote:
| > It is somehow both obsolete and wrong, a true miracle of
| bad info
|
| How does Atuin solve that problem? It seems to me that
| inaccurate and obsolete information can be in an Atuin
| document as easily as in a text document, wiki, etc., but
| possibly I'm not seeing something?
| joh6nn wrote:
| I'm just a community mod, not a dev on the project, so take
| this with a grain of salt:
|
| I believe the intent is that you get bidirectional
| selective sync between your terminal and the docs, so that
| if what's in the docs is out of date or wrong, then
| whatever you did to actually fix things can be synced back
| to the docs to reduce the friction of keeping the docs
| updated.
| metabagel wrote:
| Thanks for this explanation. This makes sense.
| roblh wrote:
| To me, it seems like it's because the thing you're fixing
| is actually the "runbook" that's being run. Instead of
| separating the documentation from the code, they're married
| together so it's easier to keep them in sync because you
| aren't having to remind yourself to go edit this secondary
| location when you make a quick change.
|
| I'm cautiously curious about something like this, although
| I haven't tried it personally.
| npodbielski wrote:
| Yes, seems like right now pendulum is going in other way
| and separation is no longer in fashion and now
| fashionable thing is to have everything in one place.
|
| The idea seems interesting to me just cause I do not
| really like terminals and having something more visually
| appealing and with better history and comments is an
| improvement though I am also not sure if Atuin is best
| way to achieve all of that.
| BLanen wrote:
| If the fix/solution would be easily describable and automate-
| able, it wouldn't/shouldn't be a problem anyway. I don't see
| how this solves anything.
| soupdiver wrote:
| It shouldn't but often still is... and maybe a runbook like
| this is easier to handle than a script with possibly 1000
| lines and not a single comment. Of course, in your ideal
| world maybe nothing of this applies and you never have any
| incidents ;)
| account-5 wrote:
| Ok I think I see where this is coming from. I actually think
| seeing you description that it might even be a benefit to
| none technical people with no knowledge of what's going on.
| They can follow instructions and easily execute the relevant
| code what with it all sitting together.
|
| However I don't see how it solves the obsolete or wrong
| documentation thing. You still have to make sure the runbook
| is correct, if it's not you've got the exact same problem.
|
| Having a centralised place for all your scripts is an
| advantage with inline docs. But then this is a local desktop
| version...
| jimbokun wrote:
| Seems like this is literate programming for shell scripts.
|
| Thus "Runbooks That Run."
| johnQdeveloper wrote:
| Well, what is the purpose of deployments being built in ansible
| or deployer or whatever tooling as a general rule? And then
| packaging, say, extra python scripts to perform common tasks
| then dumping it all in a git repo?
|
| Some people just like a particular workflow or tooling flow and
| build it really. Maybe it works for enough people to have a
| viable market, maybe not.
|
| I am just using a PHP deployment process for no reason other
| than feeling like it for personal projects and it handles 60%
| of the work without me needing to do anything. But any runbooks
| for it are tasks built into the tool and in the same git repo
| for the entire server deployment. I'm not gonna put it in some
| random place or a shell script that I need to remember separate
| commands for.
|
| Code, for programmers, is inherently self-documenting if you
| keep a simple functional style without any complexity with
| comments on the occasional section that isn't just "Create a
| MySQL user, roll the MySQL user's password, update the related
| services with the new password/user combination, remove the old
| user that the fired employee has credentials to on the off
| chance we failed to block them at the VPN" kind of stuff.
| elAhmo wrote:
| Looks interesting!
|
| We recently started using https://marimo.io/ as a replacement for
| Jupyter notebooks, as it has a number of great improvements, and
| this seems like a movement in a similar direction.
| floathub wrote:
| For anyone emacs-curious, you can do a similar thing with org-
| babel
|
| You can have a plaintext file which is also the program which is
| also the documentation/notebook/website/etc. It's extremely
| powerful, and is a compelling example of literate programming.
|
| A good take on it here:
| https://osem.seagl.org/conferences/seagl2019/program/proposa...
| kstrauser wrote:
| Similar with BBEdit's Shell Worksheets, which mingle prose with
| commands you can run with a keypress.
| spudlyo wrote:
| Thanks for the shout-out! I think org-babel is really well
| suited for this task, and can make some really great
| documentation. You can check out the video[0] from the talk and
| a git repo[1] with a more advanced demonstration.
|
| [0]: https://www.youtube.com/watch?v=0g9BcZvQbXU
|
| [1]: https://gitlab.com/spudlyo/orgdemo2
| floathub wrote:
| Thanks for making the presentation. I found it very useful
| when I first started messing around with babel, and I still
| come back to it from time to time.
| zelphirkalt wrote:
| Actually, in terms of capabilities, org-babel is among the most
| capable, if it is not the most capable, systems for literate
| programming. I have used it to great effect when learning from
| computer programming books. I can now go back to those literate
| programs, and understand again much faster, than originally
| when reading the books. The literate part of it answers my
| "silly" questions, that come from not remembering 100% of the
| reasoning or my own thoughts. That said, there is of course a
| learning curve, and people unwilling to learn something like
| that are better off not going that route.
| InvisGhost wrote:
| This sort of slogan says nothing about what actually makes it
| worth looking into.
| rc00 wrote:
| What more do you need than "written in Rust"?
| pm90 wrote:
| My dream tooling is for every tool to have an terminal interface
| so that I can create comprehensive megabooks to get all the
| context that lives in my head. i.e. jira, datadog, github etc,
| all in one pane.
| buremba wrote:
| You might like MCP then.
| x-complexity wrote:
| > You might like MCP then.
|
| That's entirely different to what's being desired by GP.
|
| > > My dream tooling is for every tool to have an terminal
| interface so that I can create comprehensive megabooks to get
| all the context that lives in my head. i.e. jira, datadog,
| github etc, all in one pane.
|
| My perspective on this is essentially having
| jira/datadog/github/etc be pluggable into the CLI, and where
| standard bash commands & pipes can be used without major
| restrictions. (Something akin to Yahoo Pipes)
|
| MCP is highly centered around LLMs analyzing user requests &
| creating queries to be run on MCP servers. What's being
| desired here doesn't centralize around LLMs in any sense at
| all.
| omneity wrote:
| It's actually not too far off. Yes MCP is designed for LLM
| interactions, but we observed that it's an invocation API
| that's pretty generic. So we built a package format that
| encapsulates computations and makes them accessible from
| any of MCP, REST, JSON-RPC over WS (the generic cousin of
| MCP)..
|
| We build logic once and make it automatically accessible
| from any of these consumption methods, in a standardized
| way to our clients, and I am indeed piping some of these
| directly in the CLI to jq and others for analysis.
| turtlebits wrote:
| Not sure what you mean- github and datadog already have
| official CLI tools.
| trenchgun wrote:
| Jira has official CLI as well
| https://appfire.atlassian.net/wiki/spaces/JCLI/overview
|
| and plenty of unofficial ones
| prakashn27 wrote:
| This is not an official API, this is from third party. stop
| promoting appfire products as official api.
| tecleandor wrote:
| Maybe something like wtfutil? (Although wtf development has
| been stuck for a year, but I guess that's the general idea...)
|
| https://wtfutil.com/
| rochak wrote:
| +1. Personally, I'm a fan of TUIs too that make things a bit
| more user friendly. Just imagine an internal TUI framework that
| has components for each internal service that you can lego-
| build into personalised TUI dashboard. Hmm, seems like
| something I could work on the side at work. Would be a huge
| undertaking but very interesting.
| pm90 wrote:
| Yes this is what I was thinking. This is how terraform became
| the lingua franca of iac, I was hoping that something like
| this would happen with TUIs too, so instead of all vendors
| creating their own bespoke cli, they all implemented a plugin
| of some generic TUI framework.
| npodbielski wrote:
| IMHO just an API would be enough, tool could be written on top
| of that. My ideal world would be every service, tool and
| application to have API that I can use i.e. if fridge is open
| too long (API polling or API webhook) I can send roomba to
| close it (using API of roomba). Because why not?!
|
| World of API...
| huntaub wrote:
| This is _exactly_ what I wanted for our team when I was at AWS.
| There are so many versions of operations which are just
| _slightly_ too dangerous to automate, and this provides a path to
| iteratively building that up. Congratulations!
| simsla wrote:
| When I was at Amazon (pre covid), Eider could've been used for
| that.
|
| (Hosted notebooks with IAM integration.)
| rochak wrote:
| Preface: My opinions are my own and not my employer's.
|
| Curious how long ago were you at AWS? For context, I spent the
| last few years in AWS working on an internal platform service
| whose entire purpose was to reduce operational toil by helping
| you codify your operational runbooks and execute them safely
| and automatically. Atuin Desktop is similar to that service in
| some sense but that service just offered much more features.
| perpil wrote:
| Not OP but was there 15 years (left Dec 2021) and surprised I
| wasn't aware of this. I was only aware of a few tools that
| acted like chatbots to automatically gather context or take
| action from chime/tt/alarms.
| perpil wrote:
| When I was at AWS, I built something so I could run things
| straight from the wiki. Think cloudwatch queries, aws cli
| commands etc with user inputs but without all the setup of
| securely getting the right credentials and formatting inputs.
| I've rebuilt to run stuff straight from GitHub. Here's it
| invoking a lambda function straight from a github wiki with
| user input in 4 lines of code:
| https://speedrun.nobackspacecrew.com/index.html#invoking-an-...
| scubbo wrote:
| I'm really confused by products like this and Warp Drive[0]. What
| does this add over a shell script?
|
| There is a response elsewhere in comments[1] which claims that
| this is trying to fix the problem of bad documentation, but this
| has the same fundamental problem. If you a) are responsible for
| fixing something, b) are unfamiliar with it, and c) the "fixing
| resources" - whether those are scripts, documentation, or a
| Runbook/Workflow - you were provided with by the experts are out-
| of-date; you're SOL and are going to have to get to investigating
| _anyway_. A runbook and a script are just different points along
| the spectrum of "how much of this is automated and how much do I
| have to copy-paste myself?"[2] - both are vulnerable to accuracy-
| rot.
|
| [0]: https://www.warp.dev/warp-drive
|
| [1]: https://news.ycombinator.com/item?id=43766842
|
| [2]: https://blog.danslimmon.com/2019/07/15/do-nothing-
| scripting-...
| brunoqc wrote:
| > I'm really confused by products like this and Warp Drive[0].
| What does this add over a shell script?
|
| Because everything is a start-up now.
| OhSoHumble wrote:
| This makes me think of using org mode to build runbooks.
| celera wrote:
| This looks super similar to https://runme.dev
| nathabonfim59 wrote:
| This is amazing!
|
| Exactly what I was looking for, thanks!
| sourishkrout wrote:
| Thanks for the shout-out. Co-creator of Runme here .
|
| Love runnable documentation. We don't have enough of it.
| baggiponte wrote:
| This looks so dope!
| nsonha wrote:
| Kinda related but just the other day I was thinking of the
| notebook/runbook workflow and wonder if there is a tool like this
| that also incorporates git checkpoints (either commit or stash)
| into it. Like top to bottom, associate all the blocks and
| resulting artifacts with a commit hash. Might be something to
| vibe code over the weekend.
| amirathi wrote:
| I took a stab at this ~7 years ago - https://nurtch.com/
|
| The idea has a lot of merit. We even gave a talk about it in
| JupyterCon Paris 2023 -
| https://www.youtube.com/watch?v=TUYY2kHrTzs
|
| When you have executable code in the documentation, folks want to
| follow PR-review workflow with the docs as well - which is a bit
| more team investment than editing a wiki.
|
| Good luck!
| wodenokoto wrote:
| My first thought was also "why not jupyter"? Nice to see
| someone else had the same thought!
| davidelettieri wrote:
| Our team used polyglot notebooks
| https://marketplace.visualstudio.com/items?itemName=ms-dotne...
|
| Using C# as main language this allowed us to have runbooks using
| code shared as nuget package and so being able to interact with
| our own APIs and applications as any other code that runs in
| production.
|
| Not the best experience to review but it worked for us.
| sudomateo wrote:
| Congratulations on the launch! I've been following Atuin for a
| bit and, while I'm not necessarily the intended audience for this
| runbook feature, love seeing people build fun new things.
| nikolay wrote:
| It's kind of sad the direction they took. The last thing I want
| is my runbooks being held hostage by my desktop with proprietary
| and possibly paid software.
| SomaticPirate wrote:
| What do you mean? This looks like open-source
| kunley wrote:
| Cool name, a reference to well known books
| axegon_ wrote:
| Oh, that's really neat! Thanks for sharing!
| ChrisMarshallNY wrote:
| +1 for the Sir Terry reference!
| gitroom wrote:
| i think lots of tools try to solve the same mess and i always end
| up just wanting my stuff all in one place with less hassle, so
| seeing more takes on this is cool
| exiguus wrote:
| Is it opensource?
| oscribinn wrote:
| pretty sure obsidian already has an extension for this
| sleepybrett wrote:
| This reminds me of xiki which seems to have kind of died on the
| vine.
| fitsumbelay wrote:
| This post reminded me how much I enjoy doodling with notebooks
| and UIs like observable and hacks like this --
| https://gist.github.com/kahole/651990b888c19b84d5700422daa96....
| In a really roundabout way, notebooks take me back to using
| crouton on my Chromebook and opening a terminal in a browser tab.
| Also just discovered Deno's Jupyter kernel and feel rather set
| for good times ...
___________________________________________________________________
(page generated 2025-04-23 23:01 UTC)