[HN Gopher] Streamlining CI/CD Pipelines with Code
___________________________________________________________________
Streamlining CI/CD Pipelines with Code
Author : shykes
Score : 44 points
Date : 2023-11-21 18:41 UTC (4 hours ago)
(HTM) web link (blog.matiaspan.dev)
(TXT) w3m dump (blog.matiaspan.dev)
| creesch wrote:
| I guess I don't really get dagger... It reminds me of the early
| days gulp.js sort of? The article also confused me for a hot
| minute, it enthusiastically goes into a demonstration that really
| does make it seem like things are more complex than they should
| be?
|
| Looking at the dagger docs (https://docs.dagger.io) I still don't
| really get the feeling there is a solid use case. The who is it
| for section does a poor job of winning me over.
|
| > A developer wishing your CI pipelines were code instead of
| YAML.
|
| Are there developers who wish this? Specifically developers who
| will not use it as an excuse to make an overly complex difficult
| to maintain solution instead? Not to mention that YAML pipelines
| can be maintained by a wider array of people some of who don't
| necesairly have the code skills to do the the same in code.
|
| Furthermore, the "you can't use code in pipelines" also isn't
| true. If you really can't do the things you want to do with
| standardized YAML pipeline blocks all of them support the option
| to actually run any type of code as part of the pipeline. This
| has the benefit of still offering the readability of clearly
| defined steps and stages.
|
| > Your team's "designated devops person", hoping to replace a
| pile of artisanal scripts with something more powerful.
|
| By replacing it by a pile of different artisanal scripts?
|
| > A platform engineer writing custom tooling, with the goal of
| unifying continuous delivery across organizational silos.
|
| That's something most pipeline solutions can already do if you
| want? Pipeline templates are a thing and can be enforced across
| an organization.
|
| > A cloud-native developer advocate or solutions engineer,
| looking to demonstrate a complex integration on short notice.
|
| Again, something you can already do with any modern pipeline.
|
| And maybe there are good dagger use cases, but my initial
| impressions based on this article and the documentation don't
| present them to me. Instead, I can't help but shake the feeling
| that this is one of those technologies that is looking for a
| problem to solve rather than something build to solve a problem.
| shepherdjerred wrote:
| The benefit of Dagger is that your builders are fully
| containerized, you can run CI locally, and you get to do all of
| that using your favorite language.
|
| You could achieve those goals independently using things like
| Make + Docker + keeping CI dumb, but that is a lot of plumbing
| and still has edge cases. For example, Makefiles often use
| utilities like sed, curl, etc., which behave differently on
| macOS versus Linux. It also requires a lot of discipline to
| keep CI from growing, and you need to learn the Make syntax,
| which, while useful, can be a bit of an investment, especially
| as you get to the more advanced features/syntax.
|
| This shines when you can write your pipelines in the same
| language as your app. Often, teams might have only 1-2 people
| who _really_ understand Bash/Make, but the rest of the team
| understands the language they use daily (Go, TypeScript, etc.)
|
| Dagger lets you leverage your knowledge of more common
| languages to write your CI pipeline, and additionally, these
| builds are containerized and can all be run locally.
|
| I have used Dagger a couple of times and liked it. One
| competitor in the space is Earthly [0], which has a
| Makefile/Dockerfile syntax. I've had good experiences with
| Earthly, but Dagger is a better pick for medium/large projects.
|
| [0]: https://earthly.dev/
| creesch wrote:
| Being able to run your pipelines locally is a pretty neat use
| case. That's something that often is difficult or basically
| impossible. Except for maybe GitHub actions through Act
| (https://github.com/nektos/act). I'd still lean to something
| in the yaml sphere if it eventually would be used in
| deployment pipelines and such. For example a solution
| incorporating ansible.
|
| It also seems to me that the argument you make is mostly
| focused on the building step? Earthly certainly seems focused
| on that aspect.
| SOLAR_FIELDS wrote:
| Act is also quite imperfect around its emulation and has a
| lot of edge cases. Not by any virtue of being a bad project
| or lack of effort, it's just that the way GHA works is
| slightly insane
| shepherdjerred wrote:
| Act works pretty well in my experience, but it's tied to
| GHA.
|
| Once nice thing about Earthly + Dagger is that your build +
| pipeline code are bundled together, and you can run them on
| any compute.
|
| As for YAML, it is ubiquitous and more accessible to those
| who are less comfortable writing code. One thing that I
| don't like about YAML si that for larger GHA workflows
| become hard to understand quite quickly. It's nice to work
| in "real" languages like Go/TypeScript where you can create
| and compose functions, especially if the language you're
| writing your pipeline in is the same language as your app
| -- I think that's a very powerful use case where all devs
| can feel comfortable/empowered to tweak CI/build steps.
|
| > It also seems to me that the argument you make is mostly
| focused on the building step? Earthly certainly seems
| focused on that aspect.
|
| Earthly and Dagger solve both builds and CI pipelines.
| They're essentially identical aside from the syntax that
| you use.
|
| Earthly and Dagger fit into CI in the same way, too. You'll
| have a GHA that installs Earthly/Dagger, and then calls
| your pipeline/build step. For Dagger, this is invoking your
| pipeline code. For Earthly, you'd want to define a `ci`
| target, and invoke it, e.g. `earthly +ci` (this means that
| locally you can run the entire pipeline with the same
| command). In both cases, you'd want your GHA (or whatever
| runner you're using) to be as dumb as possible, e.g. it
| should literally clone the repo, install the CLI, and run
| one shell command. Maybe it can also upload test
| reports/annotate PRs/etc, but the more you add to your
| GHA/test runner code outside of these tools, the less these
| tools can help you.
|
| Here's an example using Earthly for CI:
|
| https://github.com/shepherdjerred/glitter-
| boys/blob/main/.gi...
|
| A similar example using Dagger:
|
| https://github.com/shepherdjerred/shepherdjerred.com/blob/m
| a...
| shepherdjerred wrote:
| I'll follow up on this and say: give Earthly a try! It's
| super easy to figure out if you've ever written a
| Dockerfile/Makefile.
|
| If you like Earthly and feel limited by syntax, then try
| out Dagger.
| marcosnils wrote:
| Hi there, Dagger contributor here.
|
| One thing that generally is not immediately obvious is all
| the side benefits that using a programming language
| actually unlocks.
|
| Need type checking at build time? done.
|
| Need a package manager to version and publish your CI code?
| done.
|
| Need linting / formatting? done.
|
| Need to import functions from other places? done
|
| Need a test framework? done.
|
| Need a better collaborative IDE? done.
|
| All these things above bring massive improvements to both
| Dev and Devops engineers that need to interact with build &
| deploy pipelines daily. Generally it'll be quite difficult
| for any other configuration or custom language to provide
| these kind of tooling.
| shepherdjerred wrote:
| I completely agree, which is why Dagger is perfect for
| medium/large projects.
|
| I think Earthly is better for small projects with a
| simple build system since it's so much simpler to write
| and understand. I think Dagger is a worthwhile investment
| once your Earthfile is >200-400 lines.
| idoubtit wrote:
| I'm sorry, I can't make sense of what you wrote, those
| "side benefits that using a programming language [in CI]
| actually unlocks". Could you please give a concrete
| example?
|
| > Need type checking at build time? done.
|
| It's not unlocked by Dagger since every CI can already do
| this. And it's almost always about launching a (shell)
| command. How is that line easier in Go/any-language than
| in yaml?
|
| > Need to import functions from other places? done
|
| You mean "import functions into the CI code"? Why would I
| need that? It's usually possible through plugin systems,
| but I've never felt the need to put complex code into CI
| actions.
|
| > Need a test framework? done.
|
| Is that a test framework to test the CI code from within
| the CI? I hope I'll never want one! I guess I completely
| misunderstood...
|
| On the other side of those "benefits", I can smell
| problems. For instance, when Docker images and Compose
| systems are built from the Dagger API. If the need for
| some advanced features comes in, this extra layer could
| prove costly.
| Nezteb wrote:
| There's also Dagger's "Project Zenith" [1]. Details are
| scarce, but I asked in the Discord once and got this
| description of it: Zenith is still
| experimental and evolving, but we think that calling
| functions from the CLI may become the dominant way to run
| Dagger because of the many benefits: - reduced host env
| requirements/setup: you only need the `dagger` cli and the
| ability to run containers (no local dependencies like
| `golang`, `python`, `node`, etc - cross-language
| modules ecosystem: Any module written in any SDK is usable in
| your module written in your SDK of choice (e.g. use a Python
| mod from Go) - expressive verbs: `dagger exec`, `dagger
| save`, `daggger print`, `dagger call`, etc (names in flux a
| bit) - easier/shorter/quicker to write for most folks
|
| Apparently they're also trying to make Zenith Dagger
| consumable from non-Zenith Dagger [2] (though the terminology
| is confusing).
|
| [1] https://docs.dagger.io/zenith/
|
| [2] https://github.com/dagger/dagger/issues/5993
| baq wrote:
| YAML is the worst of both worlds. It pretends to be just simple
| configuration, but it by necessity cannot be that, because it
| gets interpreted by the CI system. The result is an unholy
| language selling itself as simple JSON with comments and less
| quote escaping. (At least nobody got the bright idea to put
| jinja in there.)
|
| The CDK people saw this and went the much saner strongly typed
| typescript JSON synthesis. It's still shit but at least you get
| your config statically analyzed before pushing it 17 times
| until it finally works (because why would you want to test
| locally, nobody does that).
| pid-1 wrote:
| > Are there developers who wish this? Specifically developers
| who will not use it as an excuse to make an overly complex
| difficult to maintain solution instead? Not to mention that
| YAML pipelines can be maintained by a wider array of people
| some of who don't necesairly have the code skills to do the the
| same in code.
|
| As someone who has been building automatic deployment systems
| for 5+ years, I do.
|
| I'd like to create libraries and workflows that expose
| infrastructure and deployment best practices to developers so
| they can spend their time solving business issues.
|
| Dockerfiles + Docker Compose + YAML template files are good
| enough for simple cases, but they will quickly break once you
| need to support different, possible configurable use cases.
|
| I've been using AWS CDK in Python to expose infrastructure
| libraries for developers and the difference is night and day vs
| using plain YAML. People just build stuff and deploy apps
| without talking to our infra team at all, object / functions
| serve as neat, statically checked, documented interfaces for
| our devs.
|
| I'm not sure Dagger is THE solution I'm looking for CI/CD, but
| having something that exposes rich declarative interfaces for
| automatic deployments is most definitely not a baseless idea.
| kjuulh wrote:
| Disclaimer: I build and maintain'ish the rust sdk of dagger.
|
| If you've ever worked with a sufficiently large pipeline, or
| set of pipelines, you face the fact that stuff breaks and you
| have to spam commits to try to get the weird yaml dsl to do the
| thing you want. Not that dagger is that much different in that
| context. But you can actually test your pipeline locally,
| shortening the feedback loop from 10 minutes to 10 seconds.
|
| We've even gone as far as integrating it into our developer
| tools, with caching we still have nice to use clis and apps. We
| just don't have to think too much about deployment as we're
| heavily invested in docker already.
|
| From a platform engineering perspective. We'd like actual
| guarantees about our pipelines when we roll them out to
| thousands of repositories. With dagger, we can do traditional
| unit tests on parts of our pipeline, and full integration test
| (or whatever you'd like to call them) on whole example
| pipelines.
|
| This is something that would suck to do, with github actions,
| bash and docker by itself. And as such isn't done often, and
| when done brittle. Been there done that.
|
| Another example is doing programmatic builds in a more
| comfortable language than Docker or bash. Lets say I'd like to
| solve rusts build/caching issues. With dagger, or other
| programmatic build tools. I can now mix docker commands, with
| computation to solve my problem. I can for example rely on
| golangs hashing libraries etc. rather than installing a bunch
| of packages, I don't want to end up in my final images anyway.
|
| Dagger shouldn't be used by developers directly. IMO the same
| as traditional pipelines, if the developers have to touch these
| files, then something is wrong. Instead it should be built and
| maintained by a platform team, with contributions by developers
| whom need specific features (which dagger allows, because you
| can just use a regular package manager).
|
| This may sound a little fanboyish, but we've just felt the need
| for less friction in traditional CI systems on actually having
| modern software engineering tools and methodologies available.
| Something which Dagger enables for us.
| eysi wrote:
| To add to what's already been said: If you think about it, CI
| pipelines are typically a complete description of how your
| system is built, tested, and deployed.
|
| Which is pretty fantastic except for how walled off they are.
| You can't really re-use these descriptions for e.g.
| development, they're not vendor agnostic, and they only way to
| run them is by pushing your code.
|
| Maybe it's a silly analogy but it's almost like being a web dev
| that doesn't have a browser and needs to send their code to a
| friend who can tell them if that font size looks good.
|
| I think we're way over due for freeing these "blueprints" of
| our system from the confines of CI and making them portable and
| flexible. And containers are the technology that's enabling
| that.
|
| Full disclaimer (as always): I work at Garden[0] where we're
| also solving that problem but taking a slightly different
| approach to Dagger (it's still a DAG). Garden config is
| declarative and the jobs (we call them actions) have a semantic
| meaning. You can e.g. have a Build action of type container or
| a Deploy action of type Helm and Garden will figure out what to
| do with it.
|
| We've also put a lot of emphasis on the inner loop development
| story with hot reloading functionality, log streaming and more.
|
| [0] https://github.com/garden-io/garden
| shepherdjerred wrote:
| Is garden suitable for smaller projects? I've stumbled upon
| it before, but it just looks so intimidating.
| ransom1538 wrote:
| Dagger looks far more confusing and weird than the docker
| compose example.
| esafak wrote:
| Would you recommend dagger for small teams or something else?
| marcosnils wrote:
| Dagger fits very nice with small and large dev and devops
| teams. Similarly to other tools, the more you grow, the better
| you'll have to organize your projects to make the best use of
| it.
| adamgordonbell wrote:
| I haven't used Dagger, but I'm familiar with the approach
| because Earthly has a similar approach.
|
| Getting your CI/CD code out of some CI-only runnable format and
| into something you can run on your local or wherever is game-
| changing.
|
| You should do it :)
| dphuang2 wrote:
| We are big fans of https://earthly.dev/! Although we haven't
| personally used Dagger, Earthly has solved our multi-service
| integration testing problem with elegance. Simple builds +
| caching baked in.
| drewcoo wrote:
| So . . . it's a complicated way to do test fakes with a
| dependency injection library? And not really anything to do with
| CD or much to do with CI except that it runs the same tests as on
| dev boxen?
|
| The article seems like it's trying to use click-baity terms and
| is frankly not the easiest to follow. LLM?
|
| https://blog.pragmatists.com/test-doubles-fakes-mocks-and-st...
| shykes wrote:
| Hi, Dagger co-founder here.
|
| Dagger is really just a streamlined implementation of two very
| common patterns:
|
| 1) Keeping your CI configuration as light and "dumb" as possible,
| by moving as much logic as possible into portable scripts. This
| minimizes "push and pray", where any pipeline change requires
| committing, pushing, and waiting for the proprietary CI black box
| to give you a green or red light. Ideally those scripts use
| containers for maximum reproduceability.
|
| 2) Replacing artisanal shell scripts, Makefiles and arcane DSLs
| with regular code, backed by a modern API. This allows using
| modern development techniques, a broader ecosystem of tools, and
| growing the pool of people on the team who can understand and
| improve the pipelines they use.
|
| To many devops teams, these patterns are just common sense, an
| everyday reality. For others, they are aspirational: maybe one
| day, if we find the time... Implementations remain very custom.
| Our goal at Dagger is to help democratize this way of creating
| pipelines, and making it a standard, so that an actual software
| ecosystem can appear, where devops engineers can actually reuse
| each other's code, the same way application developers can.
___________________________________________________________________
(page generated 2023-11-21 23:03 UTC)