[HN Gopher] Show HN: Updatecli - What if Dependabot and Ansible ...
___________________________________________________________________
Show HN: Updatecli - What if Dependabot and Ansible had a child?
What if Dependabot and Ansible had a child? Well for me that could
be Updatecli. Updatecli is a project that I started to help
maintain the infrastructure of the Jenkins project. I needed
something flexible enough to update YAML with whatever information
needed. Because let's say it, everybody loves YAML. YAML is
everywhere. Run it from everywhere... Updatecli is a command-line
tool written in Golang and available for Windows, Linux, MacOSx,
amd64, arm64, thank you Goreleaser All of that to say that it runs
from wherever CI or laptop we need. As of today, Updatecli opened
over 3000 Pull requests on Github, and it evolved to update
automatically Dockerfile, Markdown, Helm Chart, and of course a lot
of YAML for tools like Puppet, Kubernetes, or Jenkins. How does it
work? Updatecli loads pipeline configurations from YAML(s) or
Golang templates then enforce the state defined by the pipeline
configuration. A pipeline run as followed: 1. Clone in a temporary
location any git repositories used by the pipeline. 2. Fetch
information for every *source* defined, and then inject them as
entry parameters into condition(s) and target(s). 3. Test that all
*conditions* defined succeed otherwise abort the pipeline. 4.
Enforce the state for every *target* defined. A state means
different things depending on the resource type, more on this
later. 5. Commit and open pull requests when needed. 6. Apply
next pipeline A Updatecli pipeline relies on resources aka
"extension" aka "plugins" to adapt pipeline behavior. By combining
them, we can easily automate scenarios for release workflow,
GitOps, dependency management, documentation update, etc. A simple
scenario could be: * Retrieve the latest Golang version * Test that
a docker image with the latest Golang version exist on Dockerhub *
If it exists, then bump the version in a YAML file and open a pull
request on GitHub with the change As of today, there are 9
extensions for "sources", 8 for "conditions", 6 for "targets", 2
for git repositories, and 1 for pull requests. A very simple
pipeline is available on ->
https://www.updatecli.io/docs/prologue/quick-start/ For more
complex pipelines, you can look for directories named
"updatecli/updatecli.d" at the root of repositories on
https://github.com/updatecli or the Jenkins infrastructure
repository such as https://github.com/jenkins-infra/kubernetes-
management/tree/... I maintain a documentation website to document
the different configuration. It's not perfect but it's available on
www.updatecli.io What's next? Well, it depends on many things.
Updatecli is since the beginning, a fun side project, I wanted to
practice Golang programming while automating tedious recurring
tasks. I built it in a way that I could reuse it across the
different projects which I maintain. It's rather simple to add new
resources so I'll keep adding them based on my needs, I welcome any
contributions that would benefit the community. More information
on https://www.updatecli.io https://github.com/updatecli/updatecli
Author : olblak
Score : 50 points
Date : 2022-02-10 13:59 UTC (9 hours ago)
| raziel2p wrote:
| this tool looks amazing. dependabot but open-source, with a
| plugin system, and you can run it locally. great idea, looking
| forward to trying it out!
| olblak wrote:
| Thank you very much. Major difference with dependabot is that
| you can use it for more use cases as you define what, how, and
| when something should be updated
| dustinchilson wrote:
| agreed, FYI dependabot is open source for non-commercial use
| and you can run it locally
|
| https://github.com/dependabot/dependabot-core/blob/main/LICE...
| pritambaral wrote:
| "Open Source, but not for X" is not open source, it's source
| available.
| g105b wrote:
| I love YAML as much as I like treading on a rake.
| [deleted]
| BiteCode_dev wrote:
| > Because let's say it, everybody loves YAML. YAML is everywhere.
|
| Nope. Nope, nope, nope.
|
| I know a lot of people, including myself, who hates YAML with a
| passion.
|
| First, they hate the format itself. It's very easy to introduce
| hard-to-find typos. Plus it has multiple versions, no way to tell
| which one you use, and depending of the parser that ends up
| eating the file, you may get weird results. It's way worse then
| just the "Norway problem". Typically, a bad monday morning could
| look like this: >>> import yaml >>>
| yaml.safe_load(''' port_mappings: 3333:33
| alternative_port_mapping: 3333:33, countries: [FR,
| US, UK, NO, IT] 1 : 1,0 2: 01
| 3: 1.0 4: 1O 5: 0b1 6: 0x1
| 7: 0i1 8: hey, 8.0: oh,
| version: [3.1, "3.1", 3.10, "3.10"] ''')
|
| Which may turn into this: {
| 'port_mappings': 200013, 'alternative_port_mapping':
| '3333:33,', 'countries': ['FR', 'US', 'UK', False,
| 'IT'], 1: '1,0', 2: 1, 3:
| 1.0, 4: '1O', 5: 1, 6: 1,
| 7: '0i1', 8: 'oh,', 'version': [3.1,
| '3.1', 3.1, '3.10'] }
|
| It's a terrible, terrible format, that has too much variability,
| and way too many foot guns. Especially since typo-writing humans
| are supposed to edit those with their little clumsy hands.
|
| But worse, YAML is mostly declarative, so no complex system can
| be described in it. Here we come to the second reason people may
| hate it: it's usually turned into an even worse DSL.
|
| A very limited, badly designed, half documented and tested DSL
| with almost no tooling to write or debug the stuff. And each
| project has a different DSL, of course.
|
| Take Ansible. If your playbook is more than 10 lines, it becomes
| a pain to write because you are basically using a programming
| language embedded into a markup language. You have no print() or
| breakpoint to dive into the mess it does. It has to reinvent ways
| to pass values around since there are no functions with
| references to parameters nor return values.
|
| Mind you, this DSL calls Python under the hood, so they could
| have just exposed a declarative, idempotent Python API, but no.
| The entire world was required to recreate syntax highlighting,
| snippets, go-to-definition and so on for it. Which will never be
| good compared to the infra for real languages.
|
| There are few reasons to decide to use YAML, apart from "this is
| the only rich format supported out of the box by this system that
| has comments". Which I totally get, but still.
|
| If one can, use JSON, TOML or XML depending of the case, if we
| need only values.
|
| And if we need more than values, let's either define a clean,
| restricted API with a full blown programming language, or, if the
| use case requires it, use CUELang (https://cuelang.org/).
|
| CUELang is cleaner than YAML on all points, even for just data.
| It can provide a schema, can validate the data with that schema.
| It can embed logic. It can import and export from YAML and JSON
| so you stay compatible with the entire devops world.
|
| And it's not Turing complete, but has useful constructs to
| repeat, branch and reuse code.
| ryan_lane wrote:
| Use a jsonschema, and validate your configs. You can have
| similar problems with most other formats, and the best way to
| make sure your configs are going to do what you expect, is to
| validate them.
| dewey wrote:
| > Nope. Nope, nope, nope.
|
| Maybe it's just me, but I read this as a joke. As much as we
| dislike YAML it's not really something we have the luxury of
| ignoring, especially in the infrastructure field.
| sockpuppet69 wrote:
| Proven wrote:
| xmorse wrote:
| Another yaml problem is that it can have both .yaml and .yml
| extensions and some tools choose to support only one
| olblak wrote:
| updatecli supports both
| [deleted]
| olblak wrote:
| Cluelang is a DSL which I already considered and that I like
| very much. It wasn't high enough in my priorities.
|
| I struggled to use the HCL.
|
| The DSL used by updatecli is not fixed in stone. It's just the
| interface exposed. The reality nowadays, is that YAML remains
| the most used DSL.
| olblak wrote:
| To add on top this. I started by supporting other DSL such as
| Json. Then I realized it was adding burden on me in terms of
| documentation since people would ask me for different
| examples.
| mdaniel wrote:
| May I suggest creating a JSON Schema for the yaml files, so
| reasonable editors can help complete the structure?
|
| I didn't study the docs to know how much this is on the "ansible"
| side of things, where conditions can be almost _anything_ , and
| thus describing possible legal values would be some monster work
| mkdirp wrote:
| I like how some are missing the point of the tool because you
| mentioned (assuming jokingly) that we all love YAML.
|
| Anyway, looks really cool. Dependabot seems to some strange
| limitations, and I'll probably look into using this.
|
| Out of curiousity, how are you linked to the Jenkins project? Are
| you just an OSS contributor or do you do this full time?
| olblak wrote:
| I worked for 5 years at CloudBees within the community team so
| maintaining the infrastructure project was one the things I
| worked on https://GitHub.com/jenkins-infra
| [deleted]
| throwaway984393 wrote:
| I like the idea of automatic proposed changes triggered by
| external events, but I'm not sure a monolithic tool with plugins
| is the best way of going about it. The Unix way to solve this
| problem would be to decompose your program into a couple of
| different programs that just do one thing, with flexible inputs
| and outputs.
|
| "Sources": what does this do? Collect information. This doesn't
| need to be a particular program, this could just be a data
| specification that any program could generate data according to
| the spec. Thus you could have a dozen different programs that
| could each generate the spec.
|
| "Conditions": This also doesn't need to be a dedicated program.
| Any program that can read the data specification can have one or
| two lines of simple logic; a couple lines of shell script would
| work fine. For complicated conditions you could use a more
| complex language, or build applications with specific features
| that are needed often.
|
| "Targets": This part seems a little complicated, could use
| deconstructing a bit more. If you want to modify a YAML file,
| that is one dedicated kind of action to take. If you want to
| commit that file to a Git repo, that's another action. Opening a
| PR is another action (PRs aren't a Git primitive). Those three
| things could be separated but run one after another, all as
| separate programs.
|
| The plugin aspect is certainly useful for being able to share
| pre-made solutions. But those "plugins" don't need to be strongly
| coupled to a monolithic application. Drone.io is a fantastic
| example of a loosely-coupled cloud-native design; every plugin is
| actually a Docker container, which is just an abstraction around
| arbitrary applications that uses environment variables and
| command-line options. (Also, the data specs absolutely needs a
| version number, and should consider backwards compatibility)
| dynamite-ready wrote:
| This does sound interesting. Regarding the idea of constantly
| updating dependencies automatically, I've heard of Dependabot,
| and haven't used it much myself, but isn't there a danger with an
| automatic update breaking things in production without
| supervision?
|
| I'd guess I'm missing something in that question?
| olblak wrote:
| You need very good tests in place to automatically change
| something or a lot of trust in your dependencies :P.
|
| It's not the case on most of my project, so I never commit
| directly to the main branch, instead I commit on a temporary
| branch then open a PR on GitHub so we just review and merge the
| changes.
| michaelmior wrote:
| Yes, an automatic update can break things. Personally, I am
| happy to have minor version updates be applied automatically if
| my test suite passes. For anything larger, I at least review
| the changelog to make sure there aren't any obvious breaking
| changes and then if the tests pass, I go ahead and deploy.
| toomuchtodo wrote:
| Depends on your risk tolerance. At a bare minimum you should
| have tests as part of your project and CI for detecting
| breaking changes from version bumps, with validating deploys to
| dev environments, and possibly human review of changes before
| deploys to prod.
| jiayo wrote:
| Another thing to consider is malicious packages being
| published. In the Node ecosystem, the last few times this has
| happened, the community reacts quickly with a speedy (<12h)
| un-publish, and the whole incident might be over before your
| development team wakes up in the morning. However, if you
| immediately and automatically build every single dependency
| upgrade on your CI systems, you are dramatically increasing
| your exposure.
|
| It's important to ensure your CI systems do not hold any
| important credentials, and are decoupled from your
| deployments. In a setup like this, at best, your source code
| may be stolen by one of these malicious packages. At worst,
| it will scrape for goodies such as `AWS_SECRET_ACCESS_KEY`.
| nmasse-itix wrote:
| I had the need for such a tool a couple of months ago and wanted
| to build a dedicated tool. And you did it ! Thanks !
| olblak wrote:
| Thanks, Feel free to open GitHub issue on
| https://github.com/updatecli/updatecli/issues was what you were
| looking for. That would be very useful
| kdeldycke wrote:
| I wrote something similar but to keep a local system up-to-date:
| https://github.com/kdeldycke/meta-package-manager#readme
|
| With Meta Package Manager you can aggregate the results of all
| package managers available on your machine: $
| mpm outdated +--------------+-------------+---------+----
| ---------------+----------------+ | Package name | ID
| | Manager | Installed version | Latest version | +-------
| -------+-------------+---------+-------------------+-------------
| ---+ | curl | curl | brew | 7.79.1
| | 7.79.1_1 | | git | git | brew
| | 2.33.0 | 2.33.0_1 | | openssl@1.1 |
| openssl@1.1 | brew | 1.1.1l | 1.1.1l_1 |
| | rake | rake | gem | 13.0.3 |
| 13.0.6 | | Telegram | 747648890 | mas |
| 8.1 | 8.1.3 | | npm |
| npm@8.0.0 | npm | 7.24.0 | 8.0.0 |
| | pip | pip | pip | 21.2.4 | 21.3
| | | regex | regex | pip | 2021.9.30
| | 2021.10.8 | +--------------+-------------+--------
| -+-------------------+----------------+ 8 packages total
| (brew: 3, pip: 2, gem: 1, mas: 1, npm: 1, apm: 0, cask: 0,
| composer: 0).
|
| Bonus point: it solves XKCD #1654 (Universal Install Script).
|
| (edit: formatting)
| 256DEV wrote:
| Impressive project, nice work! I'd use the `outdated` command
| but even just the basic listing of installed packages across
| all managers looks useful to me.
| HerrMonnezza wrote:
| Might be worth a "Show HN" of its own...
| a-dub wrote:
| interesting. so if i understand this correctly, it's like a
| bridge between all those little version numbers and hashes in
| config files for modern infrastructure and full scale automation
| (from running of scripts to provisioning a short lived test
| environment) for checking if those little numbers and hashes need
| to be bumped. cool!
___________________________________________________________________
(page generated 2022-02-10 23:02 UTC)