[HN Gopher] Show HN: Earthly v0.6
___________________________________________________________________
Show HN: Earthly v0.6
Author : vladaionescu
Score : 95 points
Date : 2021-12-02 13:59 UTC (9 hours ago)
(HTM) web link (earthly.dev)
(TXT) w3m dump (earthly.dev)
| honkycat wrote:
| I tried using Earthly, great product.
|
| Unfortunately, we rejected it at our organization due to how
| verbose the commands are. I want my users to be able to say
| `earthly plan-stage` for a terraform repo, and for it to work.
| But you cannot.
|
| You cannot have earthly automatically use environment variables
| from your env, you have to explicitly list the commands yourself
| using flags:
|
| $ earthly \ --build-arg
| AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ --build-arg
| AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
| --build-arg AWS_DEFAULT_REGION=us-west-2 \ --build-arg
| AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \ --build-arg
| AWS_SECURITY_TOKEN=$AWS_SECURITY_TOKEN \ +plan-stage
|
| This means if I want to `terraform plan` something, I need to
| explicitly set all of my AWS environment variables.
|
| Compare this to `task plan-stage` or `make plan-stage`.
|
| It is much too verbose and will only get worse as we add features
| to our pipelines.
| adamgordonbell wrote:
| You may want to use secrets like this:
| earthly \ --secret-file config=/home/user/.aws/config
| \ +build
|
| But yes, Earthly optimizes for reproducibility, so implicitly
| pulling in ENVs is not something it does.
| honkycat wrote:
| I didn't use secrets because it is a prototype, and I need to
| explicitly give RUNs access to secret values, and I got sick
| of the boilerplate.
| AlphaSite wrote:
| It's a build tool, I'd be surprised if something like it worked
| well for non build related tasks. I suspect bazel would be
| equally verbose (if not even worse or impossible).
|
| I think you're better of using something like taskfile w/
| earthly for a really good workflow, it gives you the best of
| both worlds.
|
| (Actually I think I saw this on their slack)
| honkycat wrote:
| Yeah, that was what I was doing in the prototype stage. But
| then I thought "why do we need two task runners? " and ended
| up just going with task files.
|
| They sell it as a make replacement. Bazel constantly says
| "you don't need hazel, it is a big tool" and earthly does
| not. Which is fine.
|
| Either way, I already said earthly is a great tool. This is
| just my feedback around why it doesnt fit my use case.
| AlphaSite wrote:
| Ah sorry, that's fair. We went the other way from having
| just taskfile to having both and it seems quite nice so
| far.
| honkycat wrote:
| We may go back! Earthly is quite nice
| lessthanseventy wrote:
| could combine it with something like:
| https://github.com/bbugyi200/funky
| no_circuit wrote:
| Interesting. Dev tools post always catch my attention. For me,
| your comment regarding `build-arg` is in the same bucket as the
| comment regarding the need for a `ls` command. Ultimately I
| predict that the requested feature set will eventually be
| equivalent to what is already in Bazel -- like `.bazelrc` for
| setting up build variations and environment variables, and
| `query` for figuring out what is going on with the project. If
| Bazel has been working on these features and issues for years,
| I'd expect the same here.
|
| Earthly seems very useful to work around issues like cross-
| compiling. A simple example that I'd expect a lot of people to
| run into is running into libc/stdlibc++ differences with Alpine
| containers. That is easy to come across when doing something as
| simple as pulling in a Python pip dependency that wants to
| compile native C/C++ code. I find this extremely difficult to
| learn to do in Bazel.
|
| However if you're in the league of complex cross-compiling,
| then you may already be on CMake and/or Bazel and have the dev
| resources to define the toolchains as needed.
| wiradikusuma wrote:
| I think in your Articles / Tutorials you should show how to
| migrate(?) e.g. a "Hello World" Maven project running in GitLab
| CI to Earthly. This will help people understand it better (from a
| concept they're already familiar with).
| adamgordonbell wrote:
| Great suggestion! We have a walk-thru in the docs and some
| examples in GitHub.
|
| https://docs.earthly.dev/basics
|
| https://github.com/earthly/earthly/tree/main/examples
|
| Do those help?
| citilife wrote:
| How does this differ from Mesos?
| ghotli wrote:
| I saw that you were downvoted for asking a legitimate question.
| Can you expand on what you mean?
|
| Having done a lot of work in the past with Mesos and Docker and
| having done the deep dive on Earthly I just don't see the
| corollary between Mesos and this. Maybe there's more context
| you can provide :)
| mastax wrote:
| > The #1 reason people use Earthly today is to be able to
| reproduce CI builds locally. Earthly helps create a consistency
| layer, providing a level of guarantee that the build will run the
| same on your machine as on your colleagues machine, and as in CI.
| We call this consistency repeatability (note that this is
| distinct from reproducibility as full determinism is not
| guaranteed).
|
| > This helps avoid cases where tweaks to the CI script take long
| interaction cycles to test (git commit -m "try again..." anyone?)
|
| Now you have my attention. I haven't done much CI work and the
| long iteration times are the main reason why. I have to edit a
| file, commit, push, wait for CI to start, wait for CI to
| complete, search through 10k lines of logs to find the error,
| repeat... seriously? This is the best we can do? I assumed that I
| was missing something.
|
| I should be able to just `gh run-ci` in a repository on my
| machine. Or take one of those fancy online dev environments that
| everyone's making now and specialize it for editing CI files.
| Open an editor, let me click Run to run CI on-demand, have live
| error messages for syntax and missing variables, show a pane with
| a description of the CI jobs, show what config options are
| available for each job as it's being edited. Then once the
| changes are working I can commit and open a PR.
| adamgordonbell wrote:
| Yep. An Earthfile describes your build steps in a containerized
| format. Because of this sandboxing it can run identically in CI
| or on your local. So you just describe your build in the
| earthfile and call it from CI. This means you can iterate
| locally and it also means switching CI vendors is simple.
|
| You can see this in action with the Elixir Phoenix framework.
| They have an earthfile[1] and use github actions[2] to run it
| for CI.
|
| Even if you don't use Earthly, using a vendor neutral format to
| describe your build process can help with iteration time.
|
| [1]
| https://github.com/phoenixframework/phoenix/blob/master/Eart...
|
| [2]
| https://github.com/phoenixframework/phoenix/actions/runs/146...
| akdor1154 wrote:
| Oh geez.
|
| Dockerfiles are reason #1 for docker's popularity.
|
| Shipping 'the lot' so the artifact works everywhere is #2. This
| is incredibly inefficient for most use cases.
|
| `docker pull random-shit` is #3.
|
| Everything else it gives is largely immaterial, or maybe even a
| negative.
|
| You guys are solving #1 and #3, and providing a better
| alternative to #2. I'm very excited for you and your project!
| junon wrote:
| Nice. Been looking for something like this - a Dockerfile
| equivalent that isn't governed by the Docker folks, with whom I
| generally do not see eye to eye with.
|
| Bookmarked for later. Glad someone is thinking about this problem
| space!
| chirau wrote:
| What is your gripe with the Docker folks?
| junon wrote:
| Docker's codebase is a mess, the implementation is a mess,
| the data formats are a mess, the performance is a mess, it
| deadlocks under pressure (at least it did several years ago
| when I worked at Vercel), and I don't like its recent power
| moves to monetize people already somewhat locked into Docker.
|
| Just my opinions.
| sbysb wrote:
| But the dockerfile format has nothing to do with the docker
| codebase right? I run all my docker containers with podman
| (I recently moved over due to the monetization issues you
| mentioned), but I still use dockerfile format because they
| are (imo) good enough and also ubiquitous
| junon wrote:
| The dockerfiles are technically specified by OCI but the
| Moby project has a lot of sway in that committee. The
| Dockerfile spec has a number of deficiencies in it that
| aren't being addressed. Until recently, Docker was the
| only fully supported, ubiquitous way to build container
| images from a Dockerfile. The file even being called a
| "Dockerfile" is, in my opinion, annoying too.
|
| Podman of course allows `Containerfile` but meh. I just
| don't like the tight coupling of the Dockerfile spec and
| the Moby project, since I don't like the Moby (Docker)
| project for the reasons listed above.
| givemeethekeys wrote:
| Why is it called "earthly"?
| vladaionescu wrote:
| Hello HN. Earthly is a tool for build automation and we just had
| our biggest release yet.
|
| Earthly is a modern build solution that is similar to a sandboxed
| makefile. It works with your existing CI and the #1 reason people
| use it today is to able to reproduce a CI build locally.
|
| Today we promoted a number of important features to GA status in
| our 0.6 release including: - WITH DOCKER :
| Earthly can execute any number of containers in parallel, via an
| isolated Docker daemon that it starts up within. - User-
| Defined Commands: Extract common repetitive build steps into a
| command you can use across your projects. - Shared Cache:
| Earthly v0.6 now provides shared caching. This extends our
| existing build caching to work in ephemeral CI runners by storing
| the cache in regular container registries.
|
| We started working on this project in 2020[1] on GitHub [2] and
| while time has gone quickly the number of people and projects
| using Earthly now is truly exciting.
|
| Let me know what you think! Feature requests always welcome :)
|
| [1] https://news.ycombinator.com/item?id=22890612
|
| [2] https://github.com/earthly/earthly
| cmckn wrote:
| I love earthly! It has been a joy to use.
|
| One thing I haven't gotten right is caching. Some of my builds
| have targets they depend on that are very expensive but rarely
| change, and I haven't been able to make sense of when those
| targets are (re)computed and when they aren't. This results in
| many builds taking a long time, when the targets could be reused
| from a cache.
|
| Earthly is also missing something that I find to be a glaring
| omission from make--give me a command to simply list the targets
| in an Earthfile. 'earthly ls'? Maybe have a column or two that
| indicates whether the target saves an image or artifact. When
| building a project I didn't author, it'd be so helpful for the
| targets to be discoverable.
| vladaionescu wrote:
| The `earthly ls` idea is great. We do have shell
| autocompletions but that is not quite the same thing. I will
| add a ticket for that.
|
| Cache misses can be a bit inscrutable. It could be the buildkit
| GC is running, because disk space is getting scarce, or that
| some arg or file change caused the cache to be considered
| invalid.
|
| Caching is an area we will continue to improve. We have a
| proposal for extended cache mounts here[1].
|
| Thanks for using earthly!
|
| [1] https://github.com/earthly/earthly/issues/1399
| politician wrote:
| re ls: `https://github.com/TekWizely/run` displays targets in
| the usage command (ie `run -h`). It might be worth looking at
| their output for inspiration. (Runfiles are doing something
| similar to Earthfiles).
| jazzdan wrote:
| +1 on `earthly ls`. I've also thought it would be valuable to
| have "public" and "private" targets. When looking at a new
| codebase it can be hard to know which targets I should use when
| developing, and which are purely for internal use. Good docs
| help with this too, but they are not always there.
| vladaionescu wrote:
| Public / Private is interesting.
|
| Private would not be callable from command line and not be
| listed in `earthly ls`?
| jazzdan wrote:
| Yeah, exactly. Or even just not listed in `earthly ls`,
| maybe you'd still want to be able to call them manually for
| debugging purposes.
___________________________________________________________________
(page generated 2021-12-02 23:02 UTC)