[HN Gopher] I hate GitHub Actions with passion
___________________________________________________________________
I hate GitHub Actions with passion
Author : xlii
Score : 375 points
Date : 2026-01-14 10:53 UTC (12 hours ago)
(HTM) web link (xlii.space)
(TXT) w3m dump (xlii.space)
| spockz wrote:
| Would a tool like act help here? (https://github.com/nektos/act)
| I suppose orchestration that is hiding things from different
| processor architectures could also very well run differently
| online than offline, but still.
| c0wb0yc0d3r wrote:
| I haven't looked into act for some time but I remember it NOT
| being a direct stand in locally. Like it covered 80% of use
| cases.
|
| Maybe that has changed.
| joshstrange wrote:
| That's correct and it's linux-only (as of the last time I
| looked), you can run it on macOS but you can't run macOS
| runners (which is where I need the most help debugging
| normally, for building iOS apps).
| wannabe_loser wrote:
| it still isn't an 100% drop-in replacement
| palata wrote:
| I think really what would help is a way to SSH into the machine
| after it fails. SourceHut allows that, and I find it great.
| Mattwmaster58 wrote:
| It's generally very helpful - someone else mentioned here the
| fundamental problem is lack of a tight feedback loop. It
| doesn't perfectly replicate the GH environment, but for my use
| case that doesn't matter and it's super nice to have.
| Hamuko wrote:
| > _Now of course, in some Perfect World, GitHub could have a
| local runner with all the bells and whistles._
|
| Not by GitHub, but isn't act supposed to be that?
|
| https://github.com/nektos/act
| figmert wrote:
| Prefacing this with the fact that act is great, however, it has
| many shortcomings. Too often I've run into roadblocks, and when
| looking up the issue for it, it seems they are hard to address.
| Simpler workflows work fine with it, but more complex workflows
| will be much harder.
|
| Don't put your logic in proprietary tooling. I have started
| writing all logic into mise tasks since I already manage the
| tool dependencies with mise. I tend to write them in a way
| where it can easily take advantage of GHA features such as
| concurrency, matrixes, etc. But beyond that, it is all running
| within mise tasks.
| xlii wrote:
| act is often mentioned as a drop-in replacement but I never got
| it to replicate GitHub actions environment. I didn't try it for
| this particular case, though.
| iamcalledrob wrote:
| > For the love of all that is holy, don't let GitHub Actions
| > manage your logic. Keep your scripts under your own damn
| > control and just make the Actions call them!
|
| The pain is real. I think everyone that's ever used GitHub
| actions has come to this conclusion. An ideal action has 2 steps:
| (1) check out the code, (2) invoke a sane script that you can
| test locally.
|
| Honestly, I wonder if a better workflow definition would just
| have a single input: a single command to run. Remove the
| temptation to actually put logic in the actions workflow.
| worldsayshi wrote:
| > I think everyone that's ever used GitHub actions has come to
| this conclusion.
|
| I agree that that should be reasonable but unfortunately I can
| tell you that not all developers (including seniors) naturally
| arrive at such conclusion no.
| ImHereToVote wrote:
| I thought that's how actions are supposed to work. Python is
| king. Just use the Actions script to feed your variables.
| Conan_Kudo wrote:
| This is basically how most other CI systems work. GitLab CI,
| Jenkins, Buildbot, Cirrus CI, etc. are all other systems I've
| used and they work this way.
|
| I find GitHub Actions abhorrent in a way that I never found a
| CI/CD system before...
| embedding-shape wrote:
| > I find GitHub Actions abhorrent in a way that I never found
| a CI/CD system before...
|
| That's just the good old Microsoft effect, they have a
| reverse-midas-touch when it comes to actually delivering good
| UX experiences.
| blibble wrote:
| as usual for Microslop products: it's designed for maximum
| lock-in
|
| everything is including some crappy proprietary yaml rather
| than using standard tooling
|
| so instead of being a collection of easily composable and
| testable bits it's a mess that only works on their platform
| rerdavies wrote:
| It seems more of a cultural issue that -- I'm pretty sure
| -- predates Microsoft's acquisition of GitHub. I assume
| crappy proprietary yaml can be blamed on use of Ruby. And
| there seems to be an odd and pervasive "80% is good enough"
| feel to pretty much everything in GitHub, which is
| definitely cultural, and I'm pretty sure, also predates
| Microsoft's acquisition.
| treyd wrote:
| GHA is based on Azure Actions. This is evident in how bad
| its security stance is, since Azure Actions was designed
| to be used in a more closed/controlled environment.
| btreecat wrote:
| This is how we did things with Jenkins and gitlab runners
| before, idk why folks would do it differently for GHA.
|
| If you can't run the same scripts locally (minus external
| hosted service/API) then how do you debug them w/o running the
| whole pipeline?
| anilakar wrote:
| I assume you're using the currently recommended docker-in-
| docker method. The legacy Gitlab way is horrible and it makes
| it basically impossible to run pipelines locally.
| btreecat wrote:
| Containers all the way down
| geerlingguy wrote:
| GitHub introduced all their fancy GHA apps or callables or
| whatever they're called for specific tasks, and the community
| went wild. Then people built test and build workflows
| entirely in GHA instead of independent of it. And added tons
| of complexity to the point they have a whole build and test
| application written in GHA YAML.
| thiht wrote:
| > I think everyone that's ever used GitHub actions has come to
| this conclusion
|
| This is not even specific to GitHub Actions. The logic goes
| into the scripts, and the CI handles CI specific stuff
| (checkout, setup tooling, artifacts, cache...). No matter which
| CI you use, you're in for a bad time if you don't do this.
| frankwiles wrote:
| I actually built the last thing last weekend weirdly enough.
|
| gg watch action
|
| Finds the most recent or currently running action for the branch
| you have checked out. Among other things.
|
| https://github.com/frankwiles/gg
| drcongo wrote:
| Oh this is excellent. This is everything I wanted the `gh` cli
| to be, thanks.
|
| edit: Just a quick note, the `gg` and `gg tui` commands for me
| don't show any repos at all, the current context stuff all
| works perfectly though.
| frankwiles wrote:
| Ah sorry need to make the docs more clear. You need to run
| 'gg data refresh' to populate the local DB first.
| drcongo wrote:
| Ah, magnificent! Thanks!
| frankwiles wrote:
| Sure thing. Also just made it easier to install with
| Homebrew since people seem to be liking it.
| ZiiS wrote:
| Of all the valid complaints about Github Actions or CI in
| general, this seems to be an odd one. No details about what was
| tried or not tried, but hard to see a `-run: go install
| cuelang.org/go/cmd/cue@latest` step not working?
| g947o wrote:
| So the article is about the frustrating experience of fixing
| GitHub Actions when something goes wrong, especially when a
| workflow only fails on one platform, potentially due to how
| GitHub runner is set up (inconsistently across platforms).
|
| Took me a while to figure that out. While I appreciate occasional
| banters in blog articles, this one seems to diverge into rant a
| bit too much, and could have made its point much clearer, with,
| for example, meaningful section headers.
| arch1t3cht wrote:
| A _lot_ of the pain of GitHub Actions gets much better using
| tools like action-tmate: https://github.com/mxschmitt/action-
| tmate
|
| As soon as I need more than two tries to get some workflow
| working, I set up a tmate session and debug things using a proper
| remote shell. It doesn't solve _all_ the pain points, but it
| makes things a lot better.
| figmert wrote:
| Tmate is not available anymore, and will be fully
| decommissioned[0]. Use upterm[1] and action-upterm[2] instead.
|
| Honestly, this should be built into GitHub Actions.
|
| [0] https://github.com/tmate-io/tmate/issues/322
|
| [1] https://upterm.dev/
|
| [2] https://github.com/marketplace/actions/debug-with-ssh
| 1a527dd5 wrote:
| 1. Don't use bash, use a scripting language that is more CI
| friendly. I strongly prefer pwsh.
|
| 2. Don't have logic in your workflows. Workflows should be dumb
| and simple (KISS) and they should call your scripts.
|
| 3. Having standalone scripts will allow you to develop/modify and
| test locally without having to get caught in a loop of hell.
|
| 4. Design your entire CI pipeline for easier debugging, put that
| print state in, echo out the version of whatever. You don't need
| it _now_, but your future self will thank you when you do it need
| it.
|
| 5. Consider using third party runners that have better debugging
| capabilities
| Storment33 wrote:
| I would disagree with 1. if you need anything more than shell
| that starts to become a smell to me. The build/testing process
| etc should be simple enough to not need anything more.
| embedding-shape wrote:
| That's literally point #2, but I had the same reaction as you
| when I first read point #1 :)
| Storment33 wrote:
| I agree with #2, I meant more if you are calling out to
| something that is not a task runner(Make, Taskfile, Just
| etc) or a shell script thats a bit of a smell to me. E.g. I
| have seen people call out to Python scripts etc and it
| concerns me.
| embedding-shape wrote:
| Huh? Who cares if the script is .sh, .bash, Makefile,
| Justfile, .py, .js or even .php? If it works it works, as
| long as you can run it locally, it'll be good enough, and
| sometimes it's an even better idea to keep it in the same
| language the rest of the project is. It all depends and
| what language a script is made in shouldn't be considered
| a "smell".
| Storment33 wrote:
| > Huh? Who cares if the script is .sh, .bash, Makefile,
| Justfile, .py, .js or even .php?
|
| Me, typically I have found it to be a sign of over-
| engineering and found no benefits over just using shell
| script/task runner, as all it should be is plumbing that
| should be simple enough that a task runner can handle it.
|
| > If it works it works, as long as you can run it
| locally, it'll be good enough,
|
| Maybe when it is your own personal project "If it works
| it works" is fine. But when you come to corporate
| environment there starts to be issues of readability,
| maintainability, proprietary tooling, additional
| dependencies etc I have found when people start to over-
| engineer and use programming languages(like Python).
|
| E.g.
|
| > never_inline 30 minutes ago | parent | prev | next [-]
|
| > Build a CLI in python or whatever which does the same
| thing as CI, every CI stage should just call its
| subcommands.
|
| However,
|
| > and sometimes it's an even better idea to keep it in
| the same language the rest of the project is
|
| I'll agree. Depending on the project's language etc other
| options might make sense. But personally so far everytime
| I have come across something not using a task runner it
| has just been the wrong decision.
| embedding-shape wrote:
| > But personally so far everytime I have come across
| something not using a task runner it has just been the
| wrong decision.
|
| Yeah, tends to happen a lot when you hold strong opinions
| with strong conviction :) Not that it's wrong or
| anything, but it's highly subjective in the end.
|
| Typically I see larger issues being created from "under-
| engineering" and just rushing with the first idea people
| can think of when they implement things, rather than
| "over-engineering" causing similarly sized future issues.
| But then I also know everyone's history is vastly
| different, my views are surely shaped by the specific
| issues I've witnessed (and sometimes contributed to :| ),
| than anything else.
| Storment33 wrote:
| > Yeah, tends to happen a lot when you hold strong
| opinions with strong conviction :) Not that it's wrong or
| anything, but it's highly subjective in the end.
|
| Strong opinions, loosely held :)
|
| > Typically I see larger issues being created from
| "under-engineering" and just rushing with the first idea
| people can think of when they implement things, rather
| than "over-engineering"
|
| Funnily enough running with the first idea I think is
| creating a lot of the "over-engineering" I am seeing. Not
| stopping to consider other simpler solutions or even if
| the problem needs/is worth solving in the first place.
|
| > Yeah, tends to happen a lot when you hold strong
| opinions with strong conviction :) Not that it's wrong or
| anything, but it's highly subjective in the end.
|
| I quickly asked Claude to convert one of my open source
| repos using Make/Nix/Shell -> Python/Nix to see how it
| would look. It is actually one of the better Python as a
| task runners I have seen.
|
| * https://github.com/DeveloperC286/clean_git_history/pull
| /431
|
| While the Python version is not as bad as I have seen
| previously, I am still struggling to see why you'd want
| it over Make/Shell.
|
| It introduces more dependencies(Python which I solved via
| Nix) but others haven't solved this problem and the
| Python script has dependencies(such as Click for the
| CLI).
|
| It is less maintainable as it is more code, roughly x3
| the amount of the Makefile.
|
| To me the Python code is more verbose and not as simple
| compared to the Makefile's target so it is less readable
| as well.
| Imustaskforhelp wrote:
| > It introduces more dependencies(Python which I solved
| via Nix) but others haven't solved this problem and the
| Python script has dependencies(such as Click for the
| CLI).
|
| UV scripts are great for this type of workflow
|
| There are even scripts which will install uv in the same
| file effectively making it just equivalent to ./run-
| file.py and it would handle all the dependency management
| the python version management and everything included and
| would work everywhere
|
| https://paulw.tokyo/standalone-python-script-with-uv/
|
| Personally I end up just downloading uv and so not using
| the uv download script from this but if I am using
| something like github action which are more (ephemeral?)
| I'd just do this.
|
| Something like this can start out simple and can scale
| much more than the limitations of bash which can be
| abundant at times
|
| That being said, I still make some shell scripts because
| executing other applications is first class support in
| bash but not so much in python but after discovering this
| I might create some new scripts with python with
| automated uv because I end up installing uv on many
| devices anyway (because uv's really good for python)
|
| I am interested in bun-shell as well but that feels way
| too much bloated and even not used by many so less (AI
| assistance at times?) and I haven't understood bun shell
| at the same time too and so bash is superior to it
| usually
| quotemstr wrote:
| This thing does a global uv install when run? That's
| obnoxious! Never running stuff from whoever wrote this.
|
| Oh, and later the author suggests the script modify
| itself after running. What the fuck. Absolutely
| unacceptable way to deploy software.
| Imustaskforhelp wrote:
| Does it really matter if its a global install of uv or
| not _especially on Github Actions_
|
| Also if this still bothers you, nothing stops you from
| removing the first x lines of code and having it in
| another .py file if this feels obnoxious to you
|
| > Oh, and later the author suggests the script modify
| itself after running. What the fuck. Absolutely
| unacceptable way to deploy software.
|
| Regarding author suggest its removes itself its because
| it does still feel clutterish but there is virtually 0
| overhead in using/having it still be if you are already
| using uv or want to use uv
|
| Oh also, (I am not the Author) but I have played
| extensively with UV and I feel like the script can
| definitely be changed to install it locally rather than
| globally.
|
| They themselves mention it as #overkill on their website
| but even then it is better than whatever github action is
| quotemstr wrote:
| I'm a huge believer in the rule that everything GH
| actions does should be a script you can also run locally.
| Imustaskforhelp wrote:
| Yes I believe the same too and I think we are on the same
| goal. I think that I can probably patch this code to
| install uv, let's say locally instead of globally if
| that's a major concern. I feel like its not that hard.
| quotemstr wrote:
| It's easy enough to patch. It's the _philosophy_ that
| bugs me. We already have a huge problem with routine
| workflows pulling things from the network (often, without
| even a semblance of hash-locking) and foregoing the
| traditional separation between environment setup and
| business logic. There 's a lot of value into having
| discrete steps for downloading/installing stuff and doing
| development, because then you can pay special attention
| to the former, look for anything odd, read release notes,
| and so on. Between explicit, human-solicited upgrades,
| dev workflows should be using, ideally, vendored
| dependencies, or, if not that, then at least stuff that's
| hash-verified end-to-end.
|
| Someday, someone is going to have a _really_ big disaster
| that comes out of casual getting unauthenticated stuff
| from somebody else 's computer.
| Imustaskforhelp wrote:
| I agree with you and you raise some good points
|
| I think your reason of worrying is either that A)
| packages can update and contain malware or B) Uv's
| installation itself. might have malware if any of A) or
| B) get hacked
|
| Regarding A) I feel like uv's dependencies can be pinned
| to a certain date to make them reproducible and this can
| come of help
| (https://docs.astral.sh/uv/guides/scripts/#improving-
| reproduc...)
|
| Regarding B) I feel like they provide attestations via
| GitHub Artifact Attestations and the script could once
| again be modified to actually verify it via github
| attestations and they also provide ghcr artifacts (as
| such immutability) atleast of docker images and I looked
| further into it and it seems that you can use github
| artifacts to upload normal binary files as well so I will
| probably take a look into seeing if I can do something
| like this for uv's ghcr
|
| Effectively after A) and B) the trust just ends up being
| reliant on Github's Microsoft infrastructure (usually)
| and perhaps python infrastructure which is on fastly
|
| But I feel like this is for cases of extremely sensitive
| workflows But I feel like I might still take a look at it
| because security still feels very interesting to me and
| just because of this discussion, I can see at some
| pointers of following up on curiosity lol
|
| Anyways would love to continue our discussion and
| probably update you on trying to make a script which
| could actually be completely pinned (atleast uv binary
| instead of just running a shell script from the astral
| servers in such case)
| Storment33 wrote:
| > UV scripts are great for this type of workflow
|
| So previously when I have seen Python used as a task
| runner I think they used UV to call it. Although I don't
| think they had as a complete solution as your here auto-
| installing UV etc.
|
| Although the example you've linked is installing UV if
| missing, the version is not pinned, I also don't think it
| is handling missing Python which is not pinned even if
| installed locally. So you could get different versions on
| CI vs locally.
|
| While yes you are removing some of the dependencies
| problems created via using Python over Make/Shell I don't
| think this completely solves it.
|
| > Something like this can start out simple and can scale
| much more than the limitations of bash which can be
| abundant at times
|
| I personally haven't witnessed anytime I would consider
| the scales to have tipped in favour of Python and I would
| be concerned if they ever do, as really the task runner
| etc should be plumbing, so it should be simple.
|
| > That being said, I still make some shell scripts
| because executing other applications is first class
| support in bash but not so much in python but after
| discovering this I might create some new scripts with
| python with automated uv because I end up installing uv
| on many devices anyway (because uv's really good for
| python)
|
| Using Python/UV to do anything more complex than my
| example PR above?
| Imustaskforhelp wrote:
| I think UV scripts can/will actually install python and
| manage it itself as well and you can actually pin a
| specific version of python itself via Uv scripts
|
| I copied this from their website
| (https://docs.astral.sh/uv/guides/scripts/#declaring-
| script-d...)
|
| uv also respects Python version requirements: example.py
|
| # /// script # requires-python = ">=3.12" # dependencies
| = [] # ///
|
| # Use some syntax added in Python 3.12 type Point =
| tuple[float, float] print(Point)
|
| > Using Python/UV to do anything more complex than my
| example PR above?
|
| I can agree that this might be complex but that
| complexity has a trade off and of course nothing is shoe
| fits all but there are times when someone has to manage a
| complex CI environment and I looked at and there are some
| CI deterministic options too like invoke etc. and when
| you combine all of these, I feel like the workflow can
| definitely be interesting to say the least
|
| Once again, I don't know what really ends up in github
| actions since I have never really used it properly, I am
| basing its critiques based on what I've read and what
| solutions (python came quite frequently) and something
| recently which I discovered (which was the blog)
| pjc50 wrote:
| Using shell becomes deeply miserable as soon as you
| encounter its kryptonite, the space character. Especially
| but not limited to filenames.
| catlifeonmars wrote:
| I find that shell scripting has a sharp cliff. I agree
| with the sentiment that most things are over engineered.
| However it's really easy to go from a simple shell script
| running a few commands to something significantly more
| complex just to do something seemingly simple, like parse
| a semantic version, make an api call and check the status
| code etc, etc.
|
| The other problem with shell scripting on things like GHA
| is that it's really easy to introduce security
| vulnerabilities by e.g forgetting to quote your variables
| and letting an uncontrolled input through.
|
| There's no middle ground between bash and python and a
| lot of functionality lives in that space.
| Storment33 wrote:
| > However it's really easy to go from a simple shell
| script running a few commands to something significantly
| more complex just to do something seemingly simple, like
| parse a semantic version, make an api call and check the
| status code etc, etc.
|
| Maybe I keep making the wrong assumption that everyone is
| using the same tools the same way and thats why my
| opinions seem very strong. But I wouldn't even think of
| trying to "parse a semantic version" in shell, I am
| treating the shell scripts and task runners as plumbing,
| I would be handing that of a dedicated tool to action.
| jcon321 wrote:
| yea imagine having to maintain a python dependency (which
| undergoes security constraints) all because some junior
| cant read/write bash... and then that junior telling you
| you're the problem lmao
| moduspol wrote:
| Once you get beyond shell, make, docker (and similar),
| dependencies become relevant. At my current employer,
| we're mostly in TypeScript, which means you've got NPM
| dependencies, the NodeJS version, and operating system
| differences that you're fighting with. Now anyone running
| your build and tests (including your CI environment)
| needs to be able to set all those things up and keep them
| in working shape. For us, that includes different
| projects requiring different NodeJS versions.
|
| Meanwhile, if you can stick to the very basics, you can
| do anything more involved inside a container, where you
| can be confident that you, your CI environment, and even
| your less tech-savvy coworkers can all be using the exact
| same dependencies and execution environment. It
| eliminates entire classes of build and testing errors.
| tracker1 wrote:
| I've switched to using Deno for most of my orchestration
| scripts, especially shell scripts. It's a single
| portable, self-upgradeable executable and your shell
| scripts can directly reference the repositories/http(s)
| modules/versions it needs to run without a separate
| install step.
|
| I know I've mentioned it a few times in this thread, just
| a very happy user and have found it a really good option
| for a lot of usage. I'll mostly just use the Deno.*
| methods or jsr:std for most things at this point, but
| there's also npm:zx which can help depending on what
| you're doing.
|
| It also is a decent option for e2e testing regardless of
| the project language used.
| Storment33 wrote:
| I use to have my Makefile call out and do `docker build
| ...` and `docker run ...` etc with a volume mount of the
| source code to manage and maintain tooling versions etc.
|
| It works okay, better than a lot of other workflows I
| have seen. But it is a bit slow, a bit cumbersome(for
| langs like Go or Node.js that want to write to HOME) and
| I had some issues on my ARM Macbook about no ARM images
| etc.
|
| I would recommend taking a look at Nix, it is what I
| switched to.
|
| * It is faster. * Has access to more tools. * Works on
| ARM, X86 etc.
| pamcake wrote:
| Shell and bash are easy to write insecurely and open your
| CI runners or dev machines up for exploitation by shell
| injection. Non-enthusiasts writing complex CI pipelines
| pulling and piping remote assets in bash without
| ShellCheck is a risky business.
|
| Python is a lot easier to write safely.
| snovv_crash wrote:
| You shouldn't be pulling untrusted assets in CI
| regardless. Hacking your bash runner is the hardest
| approach anyways, just patch some subroutine in a
| dependency that you'll call during your build or tests.
| masfuerte wrote:
| My software runs on Windows, Linux and MacOS. The same
| Python testing code runs on all three platforms. I mostly
| dislike Python but I can't think of anything better for
| this use case.
| tracker1 wrote:
| You might consider Deno with Typescript... it's a single
| exe runtime, with a self-update mechanism (deno upgrade)
| and can run typescript/javascript files that directly
| reference the repository/http/modules that it needs and
| doesn't require a separate install step for dependency
| management.
|
| I've been using it for most of my local and environment
| scripting since relatively early on.
| Storment33 wrote:
| I don't touch Windows so I would not know.
|
| > The same Python testing code runs on all three
| platforms.
|
| I have no objections to Python being used for testing, I
| use it myself for the end to end tests in my projects. I
| just don't think Python as a build script/task runner is
| a good idea, see below where I got Claude to convert one
| of my open source projects for an example.
| WorldMaker wrote:
| It's interesting because #1 is still suggesting a shell
| script, it's just suggesting a better shell to script.
| Storment33 wrote:
| I had no idea 'pwsh' was PowerShell. Personally not
| interested, maybe if your a Microsoft shop or something
| then yeah.
| WorldMaker wrote:
| "pwsh" is often used as the short-hand for modern cross-
| platform PowerShell to better differentiate it from the
| old Windows-only PowerShell.
|
| I think pwsh is worth exploring. It is cross-platform. It
| is post-Python and the Python mantra that "~~code~~
| scripts are read more often than they are written". It
| provides a lot of nice tools out of the box. It's built
| in an "object-oriented" way, resembling Python and owing
| much to C#. When done well the "object-oriented" way
| provides a number of benefits over "dumb text pipes" that
| shells like bash were built on. It is easy to extend with
| C# and a few other languages, should you need to extend
| it.
|
| I would consider not dismissing it off hand without
| trying it just because Microsoft built it and/or that it
| was for a while Windows-only.
| ptx wrote:
| It's also both a larger download and slower to start than
| Java, which is not known for being light and nimble. In
| fact, PowerShell is so slow that you can both compile and
| run the equivalent C# program before PowerShell finishes
| launching. Not ideal for a shell or a scripting language.
|
| Also, the newer versions don't actually come with
| Windows, which would have been useful - instead Windows
| includes an incompatible older version that admonishes
| you to download the new version. But why would you
| download several hundred megabytes of pwsh when you can
| equally well download any other language runtime?
|
| Also, the error handling is just awful, silencing errors
| by default, requiring several different incantations to
| fix.
|
| Also, it sends "telemetry" to Microsoft by default.
|
| Also, the documentation is vague and useless. And the
| syntax is ugly.
| Rohansi wrote:
| It's actually a pretty good shell! FOSS and cross-
| platform, too.
| dijit wrote:
| I mean, at some point you are bash calling some other
| language _anyway_.
|
| I'm a huge fan of "train as you fight", whatever build tools
| you have locally should be what's used in CI.
|
| If your CI can do things that you can't do locally: _that_ is
| a problem.
| embedding-shape wrote:
| > If your CI can do things that you can't do locally: that
| is a problem.
|
| Probably most of the times when this is an actual problem,
| is building across many platforms. I'm running Linux x86_64
| locally, but some of my deliverables are for macOS and
| Windows and ARM, and while I could cross-compile for all of
| them on Linux (macOS was a bitch to get working though), it
| always felt better to compile on the hardware I'm
| targeting.
|
| Sometimes there are Windows/macOS-specific failures, and if
| I couldn't just ssh in and correct/investigate, and instead
| had to "change > commit > push" in an endless loop, it's
| possible I'd quite literally would lose my mind.
| ethin wrote:
| I literally had to do this push > commit > test loop
| yesterday because apparently building universal Python
| wheels on MacOS is a pain in the ass. And I don't have a
| mac, so if I want to somewhat reliably reproduce how the
| runner might behave, I have to either test it on GH
| actions or rent one from something like Scaleway. Mainly
| because I don't currently knwo how else to do it. It's
| so, so frustrating and if anyone has ideas on making my
| life a bit better that would be nice lol.
| Imustaskforhelp wrote:
| there is quickemu which can install mac vm on linux (or
| any other host) rather quickly, what are your thoughts on
| it (I am an absolute quickemu shill because I love that
| software)
|
| https://github.com/quickemu-project/quickemu [ Quickly
| create and run optimised Windows, macOS and Linux virtual
| machines ]
| tracker1 wrote:
| Thank you so much for this... If I could +1 a dozen times
| I would.
| Imustaskforhelp wrote:
| Thanks! Glad I could help. If I may ask, what specific
| use case are you using quickemu for? Is it also for
| running mac machines on say linux?
| tracker1 wrote:
| That's what I intend to use it for, Mac and Windows...
| I'm starting on an app that I want to work cross platform
| (tauri/rust w/ react+mui) and want to be able to do
| manual testing or troubleshooting as needed on mac and
| windows without needing a separate machine.
|
| My laptop is an M1 MacBook Air, and I do have an N100 I
| could use for Windows... I'd just assume use my fast
| desktop which even emulated is likely faster and not have
| to move seats.
| Imustaskforhelp wrote:
| yes, I think just the amount of friction it can reduce
| might be worth it in the first place.
|
| Oh btw although there are many primitives which help
| transferring files between VM's and others by having
| sshfs etc., one of the things which I enjoyed doing in
| quickemu is using the beloved piping-server
|
| https://github.com/nwtgck/piping-server Infinitely
| transfer between every device over pure HTTP with pipes
| or browsers
|
| The speeds might be slow but I was using it to build
| simple shell scripts and you can self host it or deploy
| on cf workers too most likely which is really simple but
| I haven't done it
|
| But for quick deployments/transfers of binaries/simple
| files, its great as well. Tauri is meant to be
| lightweight/produce small binaries so I suppose one can
| try it but there are other options as well
|
| Piping Serrvers + quickemu felt like a cheatcode to me
| atleast for more ephemeral vm's based workflow but of
| course YMMV
|
| Good luck with your project! I tried building a tauri app
| once for android just out of mere curiosity on linux and
| it was hell. I didn't know anything about android
| development but setting up the developer environment was
| really hard and I think I forgot everything I learnt from
| that but wish I had made notes or even video documenting
| the process
| tracker1 wrote:
| Fortunately/Unfortunately it wouldn't be a good
| experience for Phone use, maybe table as part of it will
| be displaying BBS-ANSI art and messages which lends
| itself to a larger display.
| Storment33 wrote:
| > If your CI can do things that you can't do locally: that
| is a problem.
|
| Completely agree.
|
| > I'm a huge fan of "train as you fight", whatever build
| tools you have locally should be what's used in CI.
|
| That is what I am doing, having my GitHub Actions just call
| the Make targets I am using locally.
|
| > I mean, at some point you are bash calling some other
| language anyway.
|
| Yes, shell scripts and or task runners(Make, Just, Task
| etc) are really just plumbing around calling other tools.
| Which is why it feels like a smell to me when you need
| something more.
| maccard wrote:
| > If your CI can do things that you can't do locally: that
| is a problem.
|
| IME this is where all the issues lie. Our CI pipeline can
| push to a remote container registry, but we can't do this
| locally. CI uses _wildly_ different caching strategies to
| local builds, which diverges. Breaking up builds into
| different steps means that you need to "stash" the output
| of stages somewhere. If all your CI does is `make test &&
| make deploy` then sure, but when you grow beyond that (my
| current project takes 45 minutes with a _warm_ cache) you
| need to diverge, and that's where the problems start.
| tracker1 wrote:
| Ironically, at least for a couple recent projects... just
| installing dependencies fresh is as fast on GH Actions as
| the GH caching methods, so I removed the caching and
| simplified the workflows.
| marksomnian wrote:
| In fact, uv's docs half-suggest this:
|
| > With uv, it turns out that it's often faster to omit
| pre-built wheels from the cache (and instead re-download
| them from the registry on each run). On the other hand,
| caching wheels that are built from source tends to be
| worthwhile, since the wheel building process can be
| expensive, especially for extension modules.
|
| https://docs.astral.sh/uv/concepts/cache/#caching-in-
| continu...
| embedding-shape wrote:
| Step 0. Stop using CI services that purposefully waste your
| time, and use CI services that have "Rebuild with SSH" or
| similar. From previous discussions
| (https://news.ycombinator.com/item?id=46592643), seems like
| Semaphore CI still offers that.
| ufo wrote:
| How do you handle persistent state in your actions?
|
| For my actions, the part that takes the longest to run is
| installing all the dependencies from scratch. I'd like to speed
| that up but I could never figure it out. All the options I
| could find for caching deps sounded so complicated.
| embedding-shape wrote:
| > How do you handle persistent state in your actions?
|
| You shouldn't. Besides caching that is.
|
| > All the options I could find for caching deps sounded so
| complicated.
|
| In reality, it's fairly simple, as long as you leverage
| content-hashing. First, take your lock file, compute the
| sha256sum. Then check if the cache has an artifact with that
| hash as the ID. If it's found, download and extract, those
| are your dependencies. If not, you run the installation of
| the dependencies, then archive the results, with the ID set
| to the hash.
|
| It really isn't more to it. I'm sure there are helpers/sub-
| actions/whatever Microsoft calls it, for doing all of this
| with 1-3 lines or something.
| ufo wrote:
| The tricky bit for me was figuring out which cache to use,
| and how to use and test it locally. Do you use the
| proprietary github actions stuff? If the installation
| process inside the actions runner is different from what we
| use in the developer machines, now we have two sets of
| scripts and it's harder to test and debug...
| embedding-shape wrote:
| > Do you use the proprietary github actions stuff?
|
| If I can avoid it, no. Almost everything I can control is
| outside of the Microsoft ecosystem. But as a freelancer,
| I have to deal a bunch with GitHub and Microsoft anyways,
| so in many of those cases, yes.
|
| Many times, I end up using
| https://github.com/actions/cache for the clients who
| already use Actions, and none of that runs in the local
| machines at all.
|
| Typically I use a single Makefile/Justfile, that
| sometimes have most of the logic inside of it for running
| tests and what not, sometimes shell out to "proper"
| scripts.
|
| But that's disconnected from the required "setup", so
| Make/Just doesn't actually download dependencies, that's
| outside of the responsibilities of whatever runs the
| test.
|
| And also, with a lot of languages, it doesn't matter if
| you run an extra "npm install" over already existing
| node_modules/, it'll figure out what's missing/there
| already, so you could in theory still have "make test" do
| absolute everything locally, including installing
| dependencies (if you now wish this), and still do the
| whole "hash > find cache > extract > continue" thing
| before running "make test", and it'll skip the
| dependencies part if it's there already.
| plagiarist wrote:
| If you are able to use the large runners, custom images are a
| recent addition to what Github offers.
|
| https://docs.github.com/en/actions/how-tos/manage-
| runners/la...
| 1a527dd5 wrote:
| You don't.
|
| For things like installing deps, you can use GitHub Actions
| or several third party runners have their own caching
| capabilities that are more mature than what GHA offers.
| philipp-gayret wrote:
| Depends on the build toolchain but usually you'd hash the
| dependency file and that hash is your cache key for a folder
| in which you keep your dependencies. You can also make a
| Docker image containing all your dependencies but usually
| downloading and spinning that up will take as long as
| installing the dependencies.
|
| For caching you use GitHubs own cache action.
| never_inline wrote:
| Build a CLI in python or whatever which does the same thing as
| CI, every CI stage should just call its subcommands.
| Storment33 wrote:
| Just use a task runner(Make, Just, Taskfile) this is what
| they were designed for.
| jonhohle wrote:
| I typically use make for this and feel like I'm constantly
| clawing back scripts written in workflows that are hard to
| debug if they're even runnable locally.
|
| This isn't only a problem with GitHub Actions though. I've
| run into it with every CI runner I've come across.
| never_inline wrote:
| In many enterprise environments, deployment logic would be
| quite large for bash.
| Storment33 wrote:
| Personally, I have never found the Python as a task
| runners to be less code, more readable or maintainable.
| linuxftw wrote:
| 1. Just no. Unless you are some sort of Windows shop.
| rerdavies wrote:
| If you're building for Windows, then bash is "just no", so
| it's either cmd/.bat, or pwsh/.ps. <shrugs>
| pixl97 wrote:
| I mean, if you're a Windows shop you really should be using
| powershell.
| import wrote:
| That's the only reason for sure.
| zabzonk wrote:
| I develop on Windows. And I use bash and (gnu) make -
| combination that cannot be beat, in my experience.
| c-hendricks wrote:
| All my windows work / ci runs still use bash.
| jayd16 wrote:
| Pwsh scripts are portable across mac, linux and windows with
| arguably less headache than bash. Its actually really nice.
| You should try it.
|
| If you don't like it, you can get bash to work on windows
| anyway.
| zelphirkalt wrote:
| I don't agree with (1), but agree with (2). I recommend just
| putting a Makefile in the repo and have that have CI targets,
| which you can then easily call from CI via a simple `make ci-
| test` or similar. And don't make the Makefiles overcomplicated.
|
| Of course, if you use something else as a task runner, that
| works as well.
| reactordev wrote:
| Makefile or scripts/do_thing either way this is correct. CI
| workflows should only do 1 thing each step. That one thing
| should be a command. What that command does is up to you in
| the Makefile or scripts. This keeps workflows/actions
| readable and mostly reusable.
| elSidCampeador wrote:
| makefile commands are the way
| pydry wrote:
| >I don't agree with (1)
|
| Neither do most people, probably but it's kinda neat how they
| suggested fix for github actions' ploy to maintain vendor
| lock-in is to swap it with a language invented by that very
| same vendor.
| Wilder7977 wrote:
| For certain things, makefiles are great options. For others
| though they are a nightmare. From a security perspective,
| especially if you are trying to reach SLSA level 2+, you want
| all the build execution to be isolated and executed in a
| trusted, attestable and disposable environment, following
| predefined steps. Having makefiles (or scripts) with logical
| steps within them, makes it much, much harder to have
| properly attested outputs.
|
| Using makefiles mixes execution contexts between the CI
| pipeline and the code within the repository (that ends up
| containing the logic for the build), instead of using -
| centrally stored - external workflows that contains all the
| business logic for the build steps (e.g., compiler options,
| docker build steps etc.).
|
| For example, how can you attest in the CI that your code is
| tested if the workflow only contains "make test"? You need to
| double check at runtime what the makefile did, but the
| makefile might have been modified by that time, so you need
| to build a chain of trust etc. Instead, in a standardized
| workflow, you just need to establish the ground truth (e.g.,
| tools are installed and are at this path), and the execution
| cannot be modified by in-repo resources.
| quotemstr wrote:
| That doesn't make any sense. Nothing about SLSA precludes
| using make instead of some other build tool. Either inputs
| to a process are hermetic and attested or they're not.
| Makefiles are _all about_ executing "predefined steps".
|
| It doesn't matter whether you run "make test" or "npm test
| whatever": you're trusting the code you've checked out to
| verify its own correctness. It can lie to you either way.
| You're either verifying changes or you're not.
| Wilder7977 wrote:
| You haven't engaged with what I wrote, of course it
| doesn't make sense.
|
| The easiest and most accessible way to attest what has
| been done is to have all the logic of what needs to be
| done in a single context, a single place. A reusable
| workflow that is executed by hash in a trusted
| environment and will execute exactly those steps, for
| example. In this case, step A does x, and step B attests
| that x has been done, because the logic is immutably in a
| place that cannot be tampered with by whoever invokes
| that workflow.
|
| In the case of the makefile, in most cases, the makefile
| (and therefore the steps to execute) will be in a file in
| the repository, I.e., under partial control of anybody
| who can commit and under full control of those who can
| merge. If I execute a CI and step A now says "make x",
| the semantic actually depends on what the makefile in the
| repo includes, so the contexts are mixed between the GHA
| workflow and the repository content. Any step of the
| workflow now can't attest directly that x happened,
| because the logic of x is not in its context.
|
| Of course, you can do everything in the makefile,
| including the attestation steps, bringing them again in
| the same context, but that makes it so that once again
| the security relevant steps are in a potentially
| untrusted environment. My thinking specifically hints at
| the case of an organization with hundreds of repositories
| that need to be brought under control. Even more, what I
| am saying make sense if you want to use the objectively
| convenient GH attestation service (probably one of the
| only good feature they pushed in the last 5 years).
| zelphirkalt wrote:
| Usually, the people writing the Makefile are the same
| that could also be writing this stuff out in a YAML (lol)
| file as the CI instructions, often located in the same
| repository anyway. The irony in that is striking. And
| then we have people who can change environment variables
| for the CI workflows. Usually, also developers, often the
| same people that can commit changes to the Makefile.
|
| I don't think it changes much, aside from security
| theater. If changes are not properly reviewed, then all
| fancy titles will not help. If anything, using Make will
| allow for a less flaky CI experience, that doesn't break
| the next time the git hoster changes something about
| their CI language and doesn't suffer from YAMLitis.
| quotemstr wrote:
| You're correct. It's absolutely security theater. Either
| you trust the repository contents or you don't. There's
| no, none, zilch trust improvement arising from the outer
| orchestration being done in a YAML file checked into the
| repo and executed by CI instead of a Makefile also
| executed by CI.
|
| What's the threat model Wilder is using exactly? Look,
| I'm ordinarily all for nuance and saying reasonable
| people can disagree when it comes to technical opinions,
| but here I can't see any merit whatsoever to the claim
| that orchestrating CI actions with Make is somehow a
| security risk when the implementations of these actions
| at some level live in the repo anyway.
| antihipocrat wrote:
| That's a great point. If we keep following the
| requirement for attestation to its logical conclusion we
| would end up replicating the entire server running the
| repository at the source, then the cycle repeats
| newsoftheday wrote:
| > Don't use bash
|
| What? Bash is the best scripting language available for CI
| flows.
| kstrauser wrote:
| I was once hired to manage a build farm. All of the build jobs
| were huge pipelines of Jenkins plugins that did various things
| in various orders. It was a freaking nightmare. Never again.
| Since then, every CI setup I've touched is a wrapper around
| "make build" or similar, with all the smarts living in Git next
| to the code it was building. I'll die on this hill.
| tracker1 wrote:
| Minor variance on #1, I've come to use Deno typescripts for
| anything more complex than what can be easily done in bash or
| powershell. While I recognize that pwsh can do a LOT in the
| box, I absolutely hate the ergonomics and a lot of the
| interactions are awkward for people not used to it, while IMO
| more developers will be more closely aligned to
| TypeScript/JavaScript.
|
| Not to mention, Deno can run TS directly and can reference
| repository/http modules directly without a separate install
| step, which is useful for shell scripting beyond what pwsh can
| do. ex: pulling a dbms client and interacting directly for
| testing, setup or configuration.
|
| For the above reasons, I'll also use Deno for e2e testing over
| other languages that may be used for the actual
| project/library/app.
| arwhatever wrote:
| Do you (or does anyone) see possible value in a CI tool that
| just launches your script directly?
|
| It seems like if you
|
| > 2. Don't have logic in your workflows. Workflows should be
| dumb and simple (KISS) and they should call your scripts.
|
| then you're basically working against or despite the CI tool,
| and at that point maybe someone should build a better or more
| suitable CI tool.
| zelphirkalt wrote:
| Can we have a CI tool, that simply takes a Makefile as input?
| Perhaps takes all targets, that start with "ci" or something.
| jayd16 wrote:
| #2 is not a slam dunk because the CI system loses insight into
| your build process if you just use one big script.
|
| Does anyone have a way to mark script sections as separate
| build steps with defined artifacts? Would be nice to just have
| scripts with something like. BeginStep("Step
| Name") ... EndStep("Step Name", artifacts)
|
| They could noop on local runs but be reflected in the
| github/gitlab as separate steps/stages and allow resumes and
| retries and such. As it stands there's no way to really have
| CI/CD run the exact same scripts locally and get all the
| insights and functionality.
|
| I haven't seen anything like that but it would be nice to know.
| g947o wrote:
| Is any of this unique to GitHub Actions that does not happen on
| other cloud CI platforms?
| embedding-shape wrote:
| The best CI platforms let you "Rebuild with SSH" or something
| similar, and instead of having the cycle of "change > commit >
| push > wait > see results" (when you're testing CI specific
| stuff, not iterating on Makefiles or whatever, assuming most of
| it is scripts you can run both locally and in CI), you get a
| URL to connect to while the job is running, so you can
| effectively ensure manually it works, then just copy-paste
| whatever you did to your local sources.
| cesarvarela wrote:
| There are a couple of GitHub actions that let you do this.
| alfonsodev wrote:
| do you mean https://github.com/nektos/act or there is
| something else ?
| efrecon wrote:
| No. Act is for running actions locally. What was
| mentioned is a way to insert an SSH step at some well-
| chosen point of a workflow so you can login at the runner
| and understand what is wrong and why it's not working. I
| have written one such thing, it relies on cloudflare free
| tunnels. https://github.com/efrecon/sshd-cloudflared.
| There are other solutions around to achieve more or less
| the same goal.
| palata wrote:
| I use that a lot with SourceHut: after a build fails, you
| have 10 minutes to SSH into the machine and debug from there.
| Also they have a very nice "edit manifest and run" feature
| that makes it easy to quickly test a change while debugging.
|
| Are there other platforms allowing that? Genuinely
| interested.
| embedding-shape wrote:
| > after a build fails, you have 10 minutes to SSH into the
| machine and debug from there.
|
| Ah, that's like 90% of the way there, just need to enable
| so the SSH endpoint is created at the beginning, rather
| than the end, so you could for example watch memory usage
| and stuff while the "real" job is running in the same
| instance.
|
| But great to hear they let you have access to the runner at
| all, only that fact makes it a lot better than most CI
| services out there, creds to SourceHut.
| palata wrote:
| > just need to enable so the SSH endpoint is created at
| the beginning
|
| Maybe it is, I've never tried :-). I don't see a reason
| why not, probably it is.
| gchamonlive wrote:
| I designed a fairly complex test matrix with a lot of logic
| offloading to the control mechanisms Gitlab offers. You create
| job templates or base jobs that control the overall logic and
| extend them for each particular use case. I had varying degrees
| of success, and it's not a job for a Devs side quest, that
| means I think you need someone dedicated to explore, build and
| debug these pipelines, but for a CI tool it's very good.
|
| Because you can extend and override jobs, you can create seams
| so that each piece of the pipeline is isolated and testable.
| This way there is very little that can go wrong in production
| that's the CI fault. And I think that's only possible because
| of the way that Gitlab models their jobs and stages.
| Storment33 wrote:
| > For the love of all that is holy, don't let GitHub Actions
| > manage your logic. Keep your scripts under your own damn
| > control and just make the Actions call them!
|
| I mean your problem was not `build.rs` here and Makefiles did not
| solve it, was your logic not already in `build.rs` which was
| called by Cargo via GitHub Actions?
|
| The problem was the environment setup? You couldn't get CUE on
| Linux ARM and I am assuming when you moved to Makefiles you
| removed the need for CUE or something? So really the solution was
| something like Nix or Mise to install the tooling, so you have
| the same tooling/version locally & on CI?
| turboponyy wrote:
| Exactly.
|
| "GitHub actions bad" is a valid take - you should reduce your
| use to a minimum.
|
| "My build failed because of GitHub actions couldn't install a
| dependency of my build" is a skill issue. Don't use GitHub
| actions to install a program your build depends on.
| ekjhgkejhgk wrote:
| > Even though my user base can be counted on a fingers of one-
| arm-less and second-arm-hook-equipped pirate, it's still a thing
| "One Should Do".
|
| No. It's cargo cult science.
| nish__ wrote:
| I agree. Why are you building for platforms you don't even use?
| pjc50 wrote:
| I wouldn't say that, but I would say there's no "should" here;
| it's often much more hassle than people expect and everyone has
| to decide for themselves whether the number of users is worth
| it.
| mrweasel wrote:
| The love for Github Actions dissipated fast, it wasn't that long
| ago we had to read about how amazing Github Actions where. What
| changed?
| esafak wrote:
| We used it.
| munk-a wrote:
| And we realized that the bs sales/marketing material was just
| as bs as always.
| this_user wrote:
| The typical cycle of all technologies:
|
| 1) New technology comes out, people get excited
|
| 2) People start recognising the drawbacks of the technology
|
| 3) Someone else makes an improved version that claims to fix
| all of the issues. GOTO 1
| woodruffw wrote:
| I think it's possible to both think GitHub Actions is an
| incredible piece of technology (and an incredible _de facto_
| public resource), while also thinking it has significant
| architectural and experiential flaws. The latter can be fixed;
| the former is difficult for competitors to replicate.
|
| (In general, I think a lot of criticisms of GitHub Actions
| don't consider the fully loaded cost of an alternative -- there
| are lots of great alternative CI/CD services out there, but
| very few of them will give you the OS/architecture matrix and
| resource caps that GitHub Actions gives every single OSS
| project for free.)
| palata wrote:
| I think it made CI management more accessible.
|
| Before that, most people would avoid Jenkins and probably never
| try Buildbot (because devs typically don't want to spend any
| time learning tools). Devs would require "devops" to do the CI
| stuff. Again, mostly because they couldn't be arsed to make it
| themselves, but also because it required setting up a machine
| (do you self-host, do you use a VPS?).
|
| Then came tools like Travis or CircleCI, which made it more
| accessible. "Just write some kind of script and we run it on
| our machines". Many devs started using that.
|
| And then came GitHub Actions, which were a lot better than
| Travis and CircleCI: faster, more machines, and free (for open
| source projects at least). I was happy to move everything
| there.
|
| But as soon as something becomes more accessible, you get
| people who had never done it before. They can't say "it enables
| me to do it, so it's better than me relying on a devops team
| before" or "well it's better than my experience with Travis".
| They will just complain because it's not perfect.
|
| And for the OP's defense, I do agree that not being able to SSH
| into a machine after the build fails is very frustrating.
| thewhitetulip wrote:
| Until I resd this blog I was under the impression that everyone
| wrote Python/ other files and used Github Actions to just call
| the scripts!
|
| This way we can test it on local machine before deployment.
|
| Also as other commenters have said - bash is not a good option -
| Use Python or some other language and write reusabe scripts. If
| not for this then for the off chance that it'll be migrated to
| some other cicd platform
| Nextgrid wrote:
| The way I deal with all these terrible CI platforms (there is no
| good one, merely lesser evils) is to do my entire CI process in a
| container and the CI tool just pulls and runs that. You can
| trivially run this locally when needed.
|
| Of course, the platforms would rather have you not do that since
| it nullifies their vendor lock-in.
| calgoo wrote:
| Thats what i always did for our gitlab CI pipeline, just deploy
| dedicated images for different purposes. We had general
| terraform images for terraform code, this made it easy to
| standardize versions etc. Then we made specific ones for
| projects with a lot of dependencies so we could run the
| deployment pipeline in seconds instead of minutes. But now you
| need to maintain the docker images too. All about trade-offs.
| trollbridge wrote:
| The one issue with that is there isn't a good way to
| containerise macOS builds.
| philipwhiuk wrote:
| I mean this has been an issue building for iOS forever. The
| MacOS lock-in sucks really really badly.
| pragma_x wrote:
| It's kind of their whole thing, when you think about it.
| They didn't get to where they are by playing nice with
| others. If you're supporting anything in the Apple
| ecosystem, the fix is in.
| palata wrote:
| I really like the SourceHut CI, because:
|
| 1. When the build fails, you can SSH into the machine and debug
| it from there.
|
| 2. You can super easily edit & run the manifest without having
| to push to a branch at all. That makes it super easy to even
| try a minimum reproducible example on the remote machine.
|
| Other than that, self-hosting (with Github or preferrably
| Forgejo) makes it easy to debug on the machine, but then you
| have to self-host.
| Nextgrid wrote:
| Self-hosted runners with Github is a whole world of pain
| because it literally just runs commands on the host and does
| not handle provisioning/cleanup, meaning you need to make
| sure your `docker run` commands don't leave any leftover
| state that can mess up future/concurrent builds. It doesn't
| even handle concurrency by itself, so you have to run
| multiple instances of the runner.
| palata wrote:
| That was included in my "but then you have to self-host"
| :-).
|
| As I said, I really like the SourceHut CI.
| jcgl wrote:
| Do you have any experience self-hosting SourceHut? I'd
| really like to do so, but I get weak knees every time I
| look at the docs for it.
| palata wrote:
| I don't, but I would be curious.
|
| But I'm happy to contribute (money) to SourceHut, they're
| doing a good job.
| watermelon0 wrote:
| Github supports ephemeral runners which are limited to a
| single job.
|
| You can use `workflow_job` webhook to be notified of a new
| job, after that you need to call `generate-jitconfig` API
| to get a just-in-time configuration token, and then you can
| start a Github runner in ephemeral mode with the JIT token.
|
| This allows you to orchestrate Docker containers, KVM
| instances, etc., which are used for a single time, and then
| destroyed.
|
| There are some open source projects, such as using
| ephemeral Kubernetes pods with the ephemeral runners.
| IAmBroom wrote:
| Your newsletter. I needs it.
| themafia wrote:
| Github delivers an exceptionally detailed webhook. Ironically,
| there is zero actual lock in, the CI/CD cargo cult is that
| strong they didn't even need it.
| skywhopper wrote:
| GHA's componentized architecture is appealing, but it's
| astonishing to me that there's still seemingly no way to actually
| debug workflows, run them locally, or rapidly iterate on them in
| any way. Alas.
| QuiCasseRien wrote:
| Guys,
|
| GitHub action is a totally broken piece of s !! I know about that
| broken loops cause I had to deal with it an incredible number of
| times.
|
| I very often mention OneDev in my comments, and you know what ?
| Robin solved this issue 3 years ago :
| https://docs.onedev.io/tutorials/cicd/diagnose-with-web-term...
|
| You can pause your action, connect through a web terminal, and
| debug/fix things live until it works. Then, you just patch your
| action easily.
|
| And that's just one of the many features that make OneDev
| superior to pretty much every other CI/CD product out there.
| thewillowcat wrote:
| I was so lonely in this opinion for so long, and it is great to
| see it becoming mainstream. GHA is terrible.
| weli wrote:
| Agreed
| semiquaver wrote:
| Skill issue.
| xlii wrote:
| Totally agree.
| YuukiRey wrote:
| Care to elaborate?
| palata wrote:
| To some extent I do agree that is sounds like "my build was
| failing on a platform I had never tested, and I was pissed
| because I had to debug it".
|
| But it is true that GitHub Actions don't make it easy to debug
| (e.g. you can't just SSH into the machine after the build
| fails). Not sure if it justifies hating with Passion, though.
| trollbridge wrote:
| I avoid actions for these exact reasons unless I can run the
| exact same build on another host.
|
| And that's where there's a Mac Studio that sits sadly in the
| corner, waiting for a new check in so it has something to do.
| dev_l1x_be wrote:
| Who doesn't? I use it with Mise to have a very simple locally
| tested way or running tasks.
| jerf wrote:
| I think this is a specific example of a generalized mistake, one
| that various bits of our infrastructure and architecture all but
| _beg_ us to make, over and over, and which must be resisted,
| which is: Your development feedback loop _must_ be as tight as
| possible.
|
| Granted, if you are working on "Windows 12", you won't be
| building, installing, testing, and deploying that locally. I
| understand and acknowledge that "as tight as possible" will still
| sometimes push you into remote services or heavyweight processes
| that can't be pushed towards you locally. This is an ideal to
| strive for, but not one that can always be accomplished.
|
| However, I see people surrender the ability to work locally much
| sooner than they should, and implement massively heavyweight
| processes without any thought for whether you could have gotten
| 90% of the result of that process with a bit more thought and
| kept it local and fast.
|
| And even once you pass the event horizon where the system as a
| whole can't be feasibly built/tested/whatever on anything but a
| CI system, I see them surrendering the ability to at least run
| the part of the thing you're working on locally.
|
| I know it's a bit more work, building sufficient mocks and stubs
| for expensive remote services that you can feasibly run things
| locally, but the payoff for putting a bit of work into having it
| run locally for testing and development purposes is just huge,
| really huge, the sort of huge you should not be ignoring.
|
| "Locally" here does not mean "on your local machine" per se,
| though that is a pretty good case, but more like, in an
| environment that you have sole access to, where you're not
| constantly fighting with latency, and where you have full
| control. Where if you're debugging even a complex orchestration
| between internal microservices, you have enough power to crank
| them all up to "don't ever timeout" and attach debuggers to all
| of them simultaneously, if you want to. Where you can afford to
| log every message in the system, interrupt any process, run any
| test, and change any component in the system in any manner
| necessary for debugging or development without having to
| coordinate with anyone. The more only the CI system can do by
| basically mailing it a PR, and the harder it is to convince it to
| do just the thing you need right now rather than the other 45
| minutes of testing it's going to run before running the 10 second
| test you actually need, the worse your development speed is going
| to be.
|
| Fortunately, and I don't even how exactly the ratio between
| sarcasm and seriousness here (but I'm definitely non-zero
| serious), this is probably going to fix itself in the next decade
| or so... because while paying humans to sit there and wait for CI
| and get sidetracked and distracted is just Humans Doing Work and
| after all what else are we paying them for, all of this stuff is
| going to be murder on AI-centric workflows, which need tight
| testing cycles to work at their best. Can't afford to have AI
| waiting for 30 minutes to find out that its PR is syntactically
| invalid, and can't afford for the invalid syntax to come back
| with bad error messages that leave it baffled as to what the
| actual problem is. If we won't do it for the humans, we'll do it
| for the AIs. This is definitely not something AI fixes, despite
| the fact they are way more patient than us and much less prone to
| distraction in the meantime since from their "lived experience"
| they don't experience the time taken for things to build and
| test, it is made much worse and more obvious that this is a real
| problem and not just humans being whiny and refusing to tough it
| through.
| Storment33 wrote:
| Yes exactly! I am using Nix/Make and I can prompt Claude to use
| Nix/Make, it uses these local feedback loops and corrects
| itself sometimes etc.
| jbmilgrom wrote:
| just came here to say same, they are the absolute worst
| woodruffw wrote:
| I think this post accurately isolates the single _main_ issue
| with GitHub Actions, i.e. the lack of a tight feedback loop.
| Pushing and waiting for completion on what 's often a very simple
| failure mode is frustrating.
|
| Others have pointed out that there are architectural steps you
| can take to minimize this pain, like keeping all CI operations
| isolated within scripts that can be run locally (and treating
| GitHub Actions features purely as progressive enhancements, e.g.
| only using `GITHUB_STEP_SUMMARY` if actually present).
|
| Another thing that works pretty well to address the feedback loop
| pain is `workflow_dispatch` + `gh workflow run`: you still need
| to go through a push cycle, but `gh workflow run` lets you stay
| in development flow until you actually need to go look at the
| logs.
|
| (One frustrating limitation with that is that `gh workflow run`
| doesn't actually spit out the URL of the workflow run it
| triggers. GitHub claims this is because it's an async dispatch,
| but I don't see how there can possibly be _no_ context for GitHub
| to provide here, given that they clearly obtain it later in the
| web UI.)
| echelon wrote:
| We need SSH access to the failed instances so we can poke
| around and iterate from any step in the workflow.
|
| Production runs should be immutable, but we should be able to
| get in to diagnose, edit, and retry. It'd lead to faster
| diagnosis, resolution, and fixing.
|
| The logs and everything should be there for us.
|
| And speaking of the logs situation, the GHA logs are really
| buggy sometimes. They don't load about half of the time I need
| them to.
| dweekly wrote:
| Are there solutions to this like
| https://github.com/marketplace/actions/ssh-to-github-
| action-... ?
| lawrencegripper wrote:
| I wrote something recently with webrtc to get terminal on
| failure: https://blog.gripdev.xyz/2026/01/10/actions-
| terminal-on-fail...
| maliker wrote:
| I've standardized on getting github actions to create/pull a
| docker image and run build/test inside that. So if something
| goes wrong I have a decent live debug environment that's very
| similar to what github actions is running. For what it's worth.
| Storment33 wrote:
| I was doing something similar when moving from Earthly. But I
| have since moved to Nix to manage the environment. It is a
| lot better of a developer experience and faster! I would
| checkout an environment manager like Nix/Mise etc so you can
| have the same tools etc locally and on CI.
| woodruffw wrote:
| Yeah, images seem to work very well as an abstraction layer
| for most CI/CD users. It's kind of unfortunate that they
| don't (can't) fully generalize across Windows and macOS
| runners as well, though, since in practice that's where a lot
| of people start to get snagged by needing to do things _in_
| GitHub Actions versus _using_ GitHub Actions as an execution
| layer.
| steeleduncan wrote:
| I do the same with Nix as it works for macOS builds as well
|
| It has the massive benefit of solving the lock-in problem.
| Your workflow is generally very short so it is easy to move
| to an alternative CI if (for example) Github were to jack up
| their prices for self hosted runners...
|
| That said, when using it in this way I personally love Github
| actions
| maliker wrote:
| I'm scared by all these references to nix in the replies
| here. Sounds like I'm going to have learn nix. Sounds hard.
| steeleduncan wrote:
| Gemini/ChatGPT help (a lot) when getting going. They make
| up for the poor documentation
| wredcoll wrote:
| Whata the killer benefit of nix over, like, a docker file
| or a package.lock or whatever?
| steeleduncan wrote:
| package.lock is JSON only, Nix is for the entire system,
| similar to a Dockerfile
|
| Nix specifies dependencies declaritively, and more
| precisely, than Docker (does by default), so the
| resulting environment is reproducibly the same. It caches
| really well and doubles as a package manager.
|
| Despite the initial learning curve, I now personally
| prefer Nix's declarative style to a Dockerfile
| 0x457 wrote:
| Nix is so nice that you can put almost your entire workflow
| into a check or package. Like your code-coverage report
| step(s) become a package that you build (I'm not brave
| enough to do this)
|
| I run my own jenkins for personal stuff on top of nixos,
| all jobs run inside devenv shell, devenv handles whatever
| background services required (i.e. database), /nix/store is
| shared between workers + attic cache in local network.
|
| Oh, and there is also nixosModule that is tested in the VM
| that also smoke tests the service.
|
| First build might take some time, but all future jobs run
| fast. The same can be done on GHA, but on github-hosted
| runners you can't get shared /nix/store.
| dfee wrote:
| same here. though, i think bazel is better for DAGs. i wish
| i could use it for my personal project (in conjunction
| with, and bootstrapped with nix), but that's a pretty
| serious tooling investment that I just feel is just going
| to be a rabbit hole.
| moduspol wrote:
| Me, too, though getting the trusted publisher NPM settings
| working didn't help with this. But it does help with most
| other CI issues.
| tracker1 wrote:
| Most of the npm modules I've built are fortunately pretty
| much just feature complete... I haven't had to deal with
| that in a while...
|
| I do have plans to create a couple libraries in the near
| future so will have to work through the pain(s)... also
| wanting to publish to jsr for it.
| tracker1 wrote:
| I tend to have most of my workflows setup as scripts that can
| run locally in a _scripts diorectory, I've also started to
| lean on Deno if I need anything more complex than I'm
| comfortable with in bash (even bash in windows) or
| powershell, since it executes .ts directly and can refer
| directly to modules/repos without a separate install step.
|
| This may also leverage docker (compose) to build/run
| different services depending on the stage of action.
| Sometimes also creating "builder" containers that will have a
| mount point for src and output to build and output the
| project in different OSes, etc. Docker + QEMU allows for some
| nice cross-compile options.
|
| The less I rely on Github Actions environment the happier I
| am... the main points of use are checkout, deno runtime,
| release please and uploading assets in a release.
|
| It sucks that the process is less connected and slow, but
| ensuring as much as reasonable can run locally goes a very
| long way.
| globular-toast wrote:
| So you've implemented GitLab CI in GitHub... We used to do
| this in Jenkins like 7 years ago.
| themafia wrote:
| I just use the fact that any action run can trigger a
| webhook.
|
| The action does nothing other than trigger the hook.
|
| Then my server catches the hook and can do whatever I want.
| maliker wrote:
| I wish I had the courage to run my own CI server. But yes,
| I think your approach is the best for serious teams that
| can manage more infrastructure.
| robflynn wrote:
| I am embarrassed that I didn't think to do this. Thank you
| :)
| anttiharju wrote:
| > i.e. the lack of a tight feedback loop.
|
| Lefthook helps a lot https://anttiharju.dev/a/1#pre-commit-
| hooks-are-useful
|
| Thing is that people are not willing to invest in it due to bad
| experiences with various git hooks, but there are ways to have
| it be excellent
| woodruffw wrote:
| Yeah, I'm one of those people who seems to consistently have
| middling-to-bad experiences with Git hooks (and Git hook
| managers). I think the bigger issue is that _even with_
| consistent developer tooling between both developer machines
| and CI, you still have the issue where CI needs to do a lot
| more stuff that local machines just can 't/won't do (like
| matrix builds).
|
| Those things are fundamentally remote and kind of annoying to
| debug, but GitHub could invest a lot more in reducing the
| frustration involved in getting a fast remote cycle set up.
| anttiharju wrote:
| GitHub could invest a lot more in actions for sure. Even
| just in basic stuff like actions/checkout@v6 being broken
| for self-hosted runners.
| cedws wrote:
| I've never used Nix and frankly am a sceptic, but can it solve
| this problem by efficiently caching steps?
| tagraves wrote:
| This is one of the big problems we solved with the RWX CI
| platform (RWX.com). You can use 'rwx run' and it automatically
| syncs your local changes, so no need to push -- and with our
| automated caching, steps like setting up the environment cache
| hit so you don't have to execute the same stuff over and over
| again while writing your workflow. Plus with our API or MCP
| server you can get the results directly in your terminal so no
| need to open the UI at all unless you want to do some in-depth
| spelunking.
| iguessthislldo wrote:
| I've never used gh workflow run, but I have used the GitHub API
| to run workflows and wanted to show the URL. I had to have it
| make another call to get the workflow runs and assume the last
| run is the one with the correct URL. This would obviously not
| work correctly if there were multiple run requests at the same
| time. Maybe some more checking could detect that, but it works
| for my purposes so far.
| ezekiel68 wrote:
| Does the metadata in the further call not identify the
| branch/start time/some other useful info that could help
| disambiguate this? (honest question)
| hinkley wrote:
| I've contemplated building my own CI tool (with a local runner)
| and the thing is if you assume "write a pipeline that runs
| locally but also on push", then the feature depth is mostly
| about queuing, analyzing output, and often left off, but IMO
| important, charting telemetry about the build history.
|
| Most of these are off the shelf, at least in some programming
| languages. It's the integrations and the overmanagement where a
| lot of the weight is.
| modeless wrote:
| It's insane to me that being able to run CI steps locally is
| not the first priority of every CI system. It ought to be a
| basic requirement.
| duped wrote:
| This goes against every incentive for the CI service provider
| anon7000 wrote:
| Not necessarily. For example, Buildkite lets you host your
| own runners.
| mrinterweb wrote:
| I've often thought about this. There are times I would rather
| have CI run locally, and use my PGP signature to add a git
| note to the commit. Something like:
|
| ``` echo "CI passed" | gpg2 --clearsign --output=- | git
| notes add -F- ```
|
| Then CI could check git notes and check the dev signature,
| and skip the workflow/pipeline if correctly signed. With more
| local CI, the incentive may shift to buying devs fancier
| machines instead of spending that money on cloud CI. I bet
| most devs have extra cores to spare and would not mind having
| a beefier dev machine.
| smw wrote:
| https://github.com/nektos/act
| mmcnl wrote:
| But very often the CI operations _are_ the problem. It's just
| YAML files with unlimited configuration options that have very
| limited documentation, without any type of LSP.
| smw wrote:
| https://github.com/nektos/act
|
| Lets you run your actions locally. I've had significant success
| with it for fast local feedback.
| smetj wrote:
| Its not Github Actions' fault but the horrors people create in
| it, all under the pretense that automation is simply about
| wrapping a GitHub Action around something. Learn to create a
| script in Python or similar and put all logic there so you can
| execute it locally and can port it to the next CI system when a
| new CTO arrives.
| bandrami wrote:
| This is giving "Debian systemd units call their old init.d
| scripts" energy but I kind of like it
| lionkor wrote:
| systemd units that are small, simple, and call into a single
| script are usually fantastic. There's no reason for these
| scripts to be part of another init system; but making as much
| of your code completely agnostic to the env it runs in sounds
| good regardless. I think that's the feeling you're feeling.
| palata wrote:
| I think in this case they hate the fact that they cannot easily
| SSH into the failing VM and debug from there. Like "I have to
| edit my workflow, push it, wait for it to run and fail, and
| repeat".
| rurp wrote:
| Yep, my company moved onto GH Actions a few years ago and
| this was probably the single biggest pain point. But also the
| whole system just feels awkward and annoying to work with. It
| feels like a classic Microsoft product that would never get
| traction from a standalone company but because it's part of
| Microsoft/GitHub and is "good enough" lots of people put up
| with it.
| anttiharju wrote:
| One can get the ssh access with self-hosted runners but it is
| problematic because uncovering secrets becomes trivial.
| SAI_Peregrinus wrote:
| Uncovering secrets is usually trivial. `printenv` in a
| build script does that pretty reliably.
| anttiharju wrote:
| What do you mean? Simple env prints get masked as *** in
| logs
|
| I guess one can always just echo the secret to a file and
| upload-artifact it
| tracker1 wrote:
| I had the weirdest thing happen a few days ago.. and only
| seemed to be happening in the GH runner for a PR.... somehow
| a function was duplicated in the action runner, and not in
| local or anwhere else... no idea how the corruption
| happened... it literally took me hours or pushing minor
| changes to try to correct the issue... I finally cat'd that
| file contents out and yep, the function was duplicated... no
| idea how. Had to create a new branch, copy the changes and
| then it worked.|
|
| Still no idea what happened or how to ever fix/prevent it
| again.
| vimda wrote:
| CircleCI had this exact feature
| pydry wrote:
| No, it is github's fault. They encourage the horrors because
| they lead to vendor lock in. This is the source of most of
| Microsoft's real profit margins.
|
| This is probably why they invented a whole programming language
| and then neglected to build any debugging tools for it.
| goodpoint wrote:
| It's 100% GA's fault and they do it on purpose.
| hbn wrote:
| > Learn to create a script in Python or similar and put all
| logic there so you can execute it locally and can port it to
| the next CI system when a new CTO arrives.
|
| That's a perfectly good several weeks of time un-estimateable
| busywork I could be getting paid for!
| bob1029 wrote:
| For the last decade I've been doing my CI/CD as simple .NET
| console apps that run wherever. I don't see why we switch to
| these wildly different technologies when the tools we are already
| using can do the job.
|
| Being able to run your entire "pipeline" locally with breakpoints
| is much more productive than whatever the hell goes on in GH
| Actions these days.
| juliie wrote:
| I can do that with github actions too? For tests, I can either
| run them locally (with a debugger if I want), or in github
| actions. Smaller checks go in a pre-commit config that github
| action also runs.
|
| Setting up my github actions (or gitlab) checks in a way that
| can easily run locally can be a bit of extra work, but it's not
| difficult.
| blixt wrote:
| I've gotten to a point where my workflow YAML files are mostly
| `mise` tool calls (because it handles versioning of all tooling
| and has cache support) and webhooks, and still it is a pain. Also
| their concurrency and matrix strategies are just not working
| well, and sometimes you end up having to use a REST API endpoint
| to force cancel a job because their normal cancel functionality
| simply does not take.
|
| There was a time I wanted our GH actions to be more capable, but
| now I just want them to do as little as possible. I've got a
| Cloudflare worker receiving the GitHub webhooks firehose, storing
| metadata about each push and each run so I don't have to pass
| variables between workflows (which somehow is a horrible
| experience), and any long-running task that should run in
| parallel (like evaluations) happens on a Hetzner machine instead.
|
| I'm very open to hear of nice alternatives that integrate well
| with GitHub, but are more fun to configure.
| arximboldi wrote:
| I like Github Actions and it is better than what I used before
| (Travis) and I think it solves an important problem. For OSS
| projects it's a super valuable free resource.
|
| For me what worked wonders was adopting Nix. Make sure you have a
| reproducible dev environment and wrap your commands in `nix-shell
| --run`, or even better `nix develop --command`, or even better
| your most of your CI tasks derivations that run with `nix build`
| or `nix flake check`.
|
| Not only does this make it super easy to work with Github
| Actions, also with your colleagues or other contributors.
| anttiharju wrote:
| Second the Nix approach. One can even build a github actions-
| compatible container out of a flake and have actions run in it.
| I have done so for my personal projects
| https://github.com/anttiharju/compare-changes
| dfee wrote:
| isn't this better served with something like Bazel (behind
| Nix of course?)
| anttiharju wrote:
| Could be, not familiar with Bazel. Thanks for mentioning it
| now can check it out.
| BirAdam wrote:
| In general, I've never really experienced the issues mentioned,
| but I also use Gitea with Actions rather than GitHub. I also
| avoid using any complex logic within an Action.
|
| For the script getting run, there's one other thing. I build my
| containers locally, test the scripts thoroughly, and those
| scripts and container are what are then used in the build and
| deploy via Action. As the entire environment is the same, I
| haven't encountered many issues at all.
| lionkor wrote:
| That sounds sane and also completely different from GH Actions.
| gen220 wrote:
| If you wanted a better version of GitHub Actions/CI (the
| orchestrator, the job definition interface, or the programming of
| scripts to execute inside those jobs), it would presumably need
| to be _more_ opinionated and have _more_ constraints?
|
| Who here has been thinking about this problem? Have you come up
| with any interesting ideas? What's the state of the art in this
| space?
|
| GHA was designed in ~2018. What would it look like if you
| designed it today, with all we know now?
| shykes wrote:
| I've been working for the last 5 years on an alternative called
| Dagger. Ended up building a company around it.
|
| We started from the ideal state of CI, and set out to build the
| platform that would support that.
|
| For us this ideal state of CI boils down to 4 things:
|
| - local-first (local execution should be a first-class citizen,
| with no exception)
|
| - repeatable (the same inputs should yield the same output,
| with affordances for handling inevitable side effects in an
| explicit and pragmatic way)
|
| - programmable. my workflows are software. I want all the
| convenience of a modern software development experience: types,
| IDE support, a rich system API , debugging tools, an ecosystem
| of reusable components, etc.
|
| - observable. I want all the information in one place about
| everything that happened in my workflow, with good tooling to
| get the information I need quickly, and interop with the
| existing observability ecosystem (eg. open telemetry)
|
| So Dagger is our best effort at a CI platform focused on those
| 4 things.
|
| Sorry if this comes across as a sales pitch. When you're
| building solves a problem you're obsessed with, it's hard to
| discuss the problem without also mentioning the solution that
| seems the most obvious to you :)
| jgautsch wrote:
| I've been watching Dagger with great interest, although have
| not moved production workloads to it (nor, admittedly, even
| committed an afternoon to setting up any workflows/graphs).
|
| Passive comment readers should be aware that ^shykes here
| cofounded Docker (my gratitude), so it's really worth a look.
|
| Can anyone comment on the ergonomics of Dagger after using it
| for a while?
|
| I was just looking at the docs earlier this week to consider
| a migration but got confused by the AI sections...
| shykes wrote:
| > got confused by the AI sections...
|
| You're not the only one... At some point last year, we
| discovered that CI/CD workflows and so-called "AI agent
| workflows" have a lot in common, and Dagger can in theory
| be used as an execution engine for both. We attempted to
| explain this - "great for CI/CD and for Agents!". But the
| feedback was mostly negative - it came across as confusing
| and lacking focus. So, we are rolling back this messaging
| and refocusing on CI/CD again. If you haven't checked our
| docs in the latest 12 hours, it's worth checking again:
| you'll see clear signs of this refocusing (although we are
| not done).
|
| In doubt, I recommend joining our public discord server
| (https://discord.com/invite/dagger-io) it is basically a
| support group for CI/CD nerds who believe that a better way
| is possible, and want to discuss it with like-minded
| people.
|
| Thanks for the kind words!
| Atotalnoob wrote:
| Do you have to use discord? All that information is
| locked away in a vendors system. Why not choose an open
| source chat app?
| shykes wrote:
| What alternatives would you recommend?
| dan_manges wrote:
| I've been working on this problem for the past couple of years.
| State of the art:
|
| - local CLI instead of git push to run
|
| - graph-based task definitions with automatic distributed
| execution, instead of the job/step abstraction
|
| - automatic content-based caching to skip unnecessary
| executions (happens a lot in CI pipelines)
|
| - container-based runtime (instead of proprietary base images)
| without using docker directly (too slow)
|
| There are a lot of other ways to improve the developer
| experience. Happy to chat with anybody interested, I'm
| dan@rwx.com
| wrqvrwvq wrote:
| Standard msft absurdity. 8 years later there is still no local gh
| action runner to test your script before you commit, push, and
| churn through logs, and without some 3rd party hack, no way to
| ssh in and debug. It doesn't matter how simple the build command
| you write is, because the workflow itself is totally foreign
| technology to most, and no one wants to be a gh action dev.
|
| Like most of the glaring nonsense that costs people time when
| using msft, this is financially beneficial to msft in that each
| failed run counts against paid minutes. It's a racket from
| disgusting sleaze scum who literally hold meetings dedicated to
| increasing user pain because otherwise the bottom line will slip
| fractionally and no one in redmond has a single clue how to make
| money without ripping off the userbase.
| thrown_in_dust wrote:
| Skill issue bro: https://github.com/nektos/act
| thrown_in_dust wrote:
| skill issue bro https://github.com/nektos/act
| swyx wrote:
| fyi i maintain a repo that accidentally tracks github actions
| cron reliability (https://www.swyx.io/github-scraping) - just
| runs a small script every hour.
|
| i just checked and in 2025 there was at least 2 outages a month
| every month https://x.com/swyx/status/2011463717683118449?s=20 .
| not quite 3 nines.
| dundarious wrote:
| Of course avoid Actions if you expect tasks to complete promptly
| or ever. Have we forgotten safe_sleep.sh? I don't think it was
| unique.
| grugdev42 wrote:
| I know GitHub Actions won the war, but I think Bitbucket
| Pipelines are much nicer to work with. They just seem simpler and
| less fragile.
|
| But almost every company uses GitHub, and changing to Bitbucket
| isn't usually viable.
| normie3000 wrote:
| The main downside of bitbucket pipelines is bitbucket. And the
| only significant feature I recall over GitHub Actions is that
| Pipelines support cron jobs.
| normie3000 wrote:
| Reading the rest of the thread, it looks like Actions also
| has cron jobs.
| b40d-48b2-979e wrote:
| Correct. on: schedule:
| - cron: $expression
| anttiharju wrote:
| Issue is, op is trying to use matrix strategy when with cross-
| compiling they could avoid it. I have done it for
| https://github.com/anttiharju/compare-changes (which has
| nontrivial CI pipelines but they could be a lot simpler for op's
| needs)
|
| Main issue is Rust. Writing catchy headlines about hating
| something may feel good, but a lot of people could avoid these
| pains if
|
| - zig cc gets support for new linker flag that Rust requires
| https://codeberg.org/ziglang/zig/pulls/30628 - rust-lang/libc
| gets to 1.0 which removes iconv issues for macos
| https://github.com/rust-lang/libc/issues/3248
| anttiharju wrote:
| This also helps one to avoid the expensive macos runners
| AmazingTurtle wrote:
| Ah a classic stack overflow flavored answer ala: do it
| differently so you don't encounter these issues
| anttiharju wrote:
| Does this discredit the answer?
| Imustaskforhelp wrote:
| Wrote a comment to someone here but I thought of deciding to make
| a main comment here as well
|
| Note that I don't really use github actions much but have heard
| about its architecture
|
| From my understanding, I feel like Github actions should just be
| a call to some bash or python file. Bash has its issues so I
| prefer python
|
| I recommend people to take a look at
| https://paulw.tokyo/standalone-python-script-with-uv/ and please
| tell me if something like this might be perfect for python
| scripts in github actions as this script would automatically
| install uv, get all the dependencies of python and even the
| runtime I think and then execute the python code all while being
| very managable usually and it can run locally as well
|
| The only Issue I feel like I might have with this is say why go
| something with this complex when bash exists or the performance
| concerns of installing uv but considering its github actions, I
| feel like the latter is ruled out.
|
| Bash is good as well but bash has some severe limitations. and I
| feel like Python can be good case for something like this plus
| its ecosystem is a bit mature and you could even create web
| servers or have some logs be reported to your custom server or
| automate just basically everything
|
| To me this script feels like the best of both worlds and
| something genuinely sane to build upon.
| flohofwoe wrote:
| > I feel like Github actions should just be a call to some bash
| or python file. Bash has its issues so I prefer python
|
| ...and that's how it is supposed to be used. The YAML file is
| mainly there for defining the runtime environment (e.g. the
| operating system or docker image to run in), and to describe
| the job dependency tree. The actual actions to execute should
| be delegated to scripts outside the GH YAML.
| flohofwoe wrote:
| GH Actions isn't great compared to other CI systems, but it's
| also not particularly worse until you get into the nitty gritty
| details.
|
| The most important advice is probably to put as much code as
| possible into locally runnable scripts written in a cross-
| platform scripting language (e.g. Python or Node.js) to avoid
| 'commit-push-ci-failure' roundtrips.
|
| Only use the GH Actions YAML for defining the runtime environment
| and job dependency tree.
| tracker1 wrote:
| s/Node/Deno/ - direct repo/module references, no install step
| required.
| isaacvando wrote:
| You should try RWX! You can trigger runs from the CLI for way
| faster feedback loops. The automatic caching is surprisingly good
| too. https://www.rwx.com/
| kylekthompson wrote:
| Thanks for the shoutout!
|
| If anyone has any questions or wants to give it a shot, I work
| at RWX and would be happy to chat more about it!
| jillesvangurp wrote:
| I think the root problem here is scope creep on using build
| tools, configuration management, and ci tools: don't make them do
| things they are not designed for / weak at. The issue isn't
| necessarily github ci, which is fine if you keep it simple. But
| things just getting to the stage where they are not simple
| anymore.
|
| I've seen this over and over again over the years in projects
| where things like ant, maven, gradle, puppet, ansible, etc.
| Invariably somebody tries to do something really
| complicated/clever in a convoluted way. They'll add plugins, more
| plugins, all sorts of complex configuration, etc. Your script
| complexity explodes. And then it doesn't work and you spend
| hours/days trying to make it do the right thing and fighting a
| tool that just wasn't designed to do what you are doing well. The
| problem is using the wrong tool for the job. All these tools have
| the tendency to evolve into everything tools. And they just
| aren't good at everything. Just because it has some feature
| doesn't mean it's a good idea to use it.
|
| The author actually calls this out. Just write a bash script and
| run that instead. If that gets too complicated pick something
| else more appropriate to the job. Python, whatever you like. The
| point here is to pick something that's easy for you to run, test,
| and debug locally. Obviously people have different preferences
| here. If you are shoe horning complex fork/join behavior,
| conditional logic, etc. into a Yaml / CI build, maybe simplify
| your build. Yaml just isn't suitable as a general purpose
| programming language.
|
| Externalizing the complex stuff to some script also has the
| benefit of that stuff still working if you ever decide to switch
| CI provider. Maybe you want to move to Gitlab. Or somebody
| decides Jenkins wasn't so bad after all. The same scripts will
| probably be easy to adapt to those.
|
| One of my current github actions basically starts a vm, runs a
| build.sh script, stops the vm. I don't need a custom runner for
| that. I get to pick the vm type. I can do whatever I need to in
| my build.sh. I have one project with an actual matrix build. But
| that's about the most complex thing I do with github actions. A
| lot of my build steps are just inline shell commands.
|
| And obligatory AI comment here, if you are doing all this
| manually, having scripts that run locally also means you can put
| claude code/codex/whatever to work fixing them for you. I've been
| working on some ansible scripts with codex today. Works great. It
| produces better ansible scripts than me. These tools work better
| if they can test/run what they are working on.
| JohnMakin wrote:
| I've always found things like AWS codebuild or even just a self
| hosted bare bones jenkins server far easier to work with. What is
| the advantage that github actions provide that people put up with
| it? The feedback seems almost universally negative.
| anttiharju wrote:
| > What is the advantage that github actions provide that people
| put up with it?
|
| It's the default CI system on github and you get relatively
| free compute.
| bdcravens wrote:
| Being free or practically free for many use cases. Especially
| if you consider multi-CPU builds.
| moduspol wrote:
| It is overwhelmingly less resource-intensive to maintain over
| time than any Jenkins server. And the config is in the repo
| (not the Jenkins server's config).
| mrgoldenbrown wrote:
| This is very validating as someone just trying out Actions and
| getting frustrated with the less than ideal UX.
| eYrKEC2 wrote:
| Anyone use Argo Workflows as an alternative?
| kylegalbraith wrote:
| As a founder of Depot [0], where we offer our own faster and
| cheaper GitHub Actions runners, I can assure everyone that this
| feeling is the majority and not the minority.
|
| Sounds strange to say as someone who has a product that is built
| around making GitHub Actions exponentially faster to close these
| feedback loops faster.
|
| But I can honestly say it's only really possible because the
| overall system with GitHub Actions is so poor. We discover new
| bottlenecks in the runner and control plane weekly. Things that
| you'd think would be simple are either non-existent, don't work,
| or have terrible performance.
|
| I'm convinced there are better ways of doing things, and we are
| actively building ideas in that realm. So if anybody wants to
| learn more or just have a therapy session about what could be
| better with GitHub Actions, my email is in my bio.
|
| [0] https://depot.dev/
| OptionOfT wrote:
| As much as I hate GitHub Actions, I prefer it over Jenkins and
| others, because it is right there. I don't need to go and
| maintain my own servers, or even talk to a 3rd party to host the
| services.
| tracker1 wrote:
| Like many things, the tool people reach for is the one that's
| in the box... As opposed to a trip to the hardware store,
| getting distracted for a few hours, coming back home and no
| longer in the mood to work on your project/choore.
| psyclobe wrote:
| I have spent the majority of my professional career waiting for
| ci to complete. It is hell.
| kvnnews wrote:
| This happens when there's too much logic in the build server.
| UnCommonLisp wrote:
| I take GitHub Actions over any of the other CI platforms.
| NooneAtAll3 wrote:
| This feels like a rage bait.
| akhil08agrawal wrote:
| As a PM trying to understand what was happening, it took me a
| minute. Turning this into an analogy so that it might benefit
| other non-too-techies:
|
| Original plan: You have an oven in another building that
| automatically bakes your cake. But the oven needs a mixer, and
| every time you bake, it has to wait 2-3 minutes for someone to
| bring the mixer from storage.
|
| Problem: For one specific oven (Linux ARM), the mixer never
| arrives. So your cake fails. You keep trying different ways to
| get the mixer delivered. Each attempt: 2-3 minutes wait.
|
| What you finally do: Stop waiting for the mixer to be delivered.
| Just mix the batter at home where you already have a mixer. Send
| the pre-mixed batter to the other building. Now the oven just
| bakes it - no waiting for the mixer.
|
| Translation: Stop trying to generate files in GitHub Actions
| (where it takes 2-3 minutes each time). Generate them locally on
| your computer where you already have the tools. Upload the
| finished files. GitHub Actions just uses them.
|
| Sometimes "pre-mix the batter at home" beats "wait for the mixer
| every single time."
| dwoldrich wrote:
| I am not having fun with GitHub Actions right now! Why does
| everything have to be so hard?
|
| I like being able to run self-hosted runners, that is a very cool
| part of GitHub Actions/Workflow.
|
| I appreciate all the other advice about limit my yamls to: 1)
| checkout, 2) call a script to do the entire task. I am already
| half-way there, just need to knuckle-down and do the work.
|
| I was dismayed that parallel tasks aren't really a thing in the
| yaml, I wanted to fanout a bunch of parallel tasks and I found I
| couldn't do it. Now that I'm going to consolidate my build
| process into a single script I own, I can do the fanout myself.
| Storment33 wrote:
| Jobs run in parallel, so if you used Make you could have one
| job called 'formatting' calling 'make check-formatting', one
| called linting calling ''make check-linting', one called
| 'compiling' calling 'make compile' etc.
| humanlity wrote:
| > i.e. a way that I could testbed different runs without
| polluting history of both Git and Action runs.
|
| How about writing a separate repo and testing it separately
|
| Keywords: reusable workflow/actions
| levysoft wrote:
| Interesting, but the opening line threw me off:
|
| > Sure, I still make fun of the PHP I remember from the days of
| PHP 4.1, but even then I didn't hate it. (ref. And "PHP: Training
| Wheels Without a Bike" is still in the Top 10 of my favorite
| memes.)
|
| I still use it to build my latest projects, and for me it's like
| breathing. Simple and functional. It's not perfect, but no
| language is.
| jslaby wrote:
| Yeah, no reason for PHP to catch strays here, especially if his
| knowledge is still based on 4.1. It's really a whole new
| language compared to back then.
| globular-toast wrote:
| What always surprises me in these discussions is how few people
| have used GitLab CI. What gives? It's far, far superior to GitHub
| and even existed first.
| c-hendricks wrote:
| I found gitlab ci's yaml the smallest of the 3 I've used
| (gitlab, GitHub actions, CircleCI).
|
| But does gitlab ci have anything for sharing? GitHub actions
| are built around it, and CircleCI has orbs and contexts.
|
| For example, muse's guide for gitlab involves making your own
| container and managing the cache yourself (ref:
| https://mise.jdx.dev/continuous-integration.html#gitlab-ci)
|
| GitHub actions is a couple of lines (ref:
| https://mise.jdx.dev/continuous-integration.html#github-
| acti...)
| wutwutwat wrote:
| run your ci in docker containers so github actions is just a
| orchestrator
|
| or try something like https://docs.dagger.io/use-cases#portable-
| ci
| joshka wrote:
| Slap your problem in an agentic loop, this becomes the following
| single step:
|
| 1. Here's your goal "...", fix it, jj squash and git push, run gh
| pr checks --watch --fail-fast pr-link
| olivia-banks wrote:
| At work we have a bunch of GitHub actions for integration
| testing, building models, publishing, reporting, and whatnot. It
| was horrible to maintain and look into whenever something went
| wrong, so I rewrote all the individual parts in Perl and hooked
| them together with pipes inside GHA, and it works wonders.
|
| Also, GitHub actions itself just breaks sometimes, which is super
| annoying. A couple of weeks ago, half of all the macOS runner
| images broke when using GitHub's caching system. Seems like that
| would have been caught on GitHub's side before that happened, but
| what do I know!
|
| > https://github.com/actions/runner/issues/449
|
| > https://github.com/orgs/community/discussions/180160
| cadamsdotcom wrote:
| So many engineers could put the hours spent debugging GH actions
| to use developing expertise to run their own CI. But people
| either don't believe they can, can't convince decision makers to
| let them try, or just want to fix their own problem and move on.
|
| I was convinced GH actions was best practice and it was normal to
| waste hours on try-and-pray build debugging, until one day GH
| actions went down and I ran deploys from my laptop and remembered
| how much better life can be without it..
|
| (Solo dev here - but opensource CI on an EC2 instance can be just
| as nice)
| Marsymars wrote:
| > So many engineers could put the hours spent debugging GH
| actions to use developing expertise to run their own CI.
|
| If I run my own CI, then the compliance team has to get
| involved to run various endpoint security and update management
| tools on whatever system I'm running the CI on.
| cadamsdotcom wrote:
| Then the costs of staying with GH actions need to be made
| known so they can be balanced against the cost of doing
| things differently. Of course there's a cost involved in just
| getting those numbers too.
|
| It's all trade offs.
| deepsun wrote:
| > A word of explanation. I'm building tmplr for 4 platforms:
| > Linux ARM > macOS ARM > Linux x86_64 >
| macOS x86_64
|
| Oh, here we go again. Java was invented to solve that, "Write
| once run everywhere" :] I.e. `int` means `i32` on all platforms,
| no `usize`.
| Storment33 wrote:
| Java has numerous problems, it's not as simple as use Java not
| Rust. To be honest I actively avoid any tooling written in
| Java.
| bitterbytes wrote:
| Would something like this help with the feed back loop?
|
| https://github.com/nektos/act
| wasmainiac wrote:
| Honestly part of the reason why I left my last job was because I
| had to extensively work with GitHub workflows and actions.
| Debugging was absolutely hell, especially with long running tasks
| (~1hr) that would fail with next to no debug or traceability. I
| offered many times to overhaul the system and make it easier to
| maintain, but we had no time budget. I now work for 30% less but
| am substantially happier in life without github. It sounds crazy,
| but I'm not exaggerating.
| nfw2 wrote:
| I am surprised that these sort of declarative specs are so
| popular in certain domains. Essentially you always seem to be
| putting settings into the ether and hoping they interact with
| each other in the way you expect.
|
| I prefer an api with documented contracts between its
| abstractions
| kitplummer wrote:
| I recently, for Rust targets reasons, have decided to punt GitHub
| Actions AND GitHub. Opted for Radicle. Had to figure out my own
| CI.
|
| Doc'd it here: https://revolveteam.com/blog/goa-radicle-ci/
| ted537 wrote:
| The main problem with actions is the way they advertise its usage
| "just put workflows together" is a horrible and non-debuggable
| way to do things. But even in the tech itself, caching is pretty
| stingy which can slow dev builds for fairly simple projects
| because every run will repeat some common work unless you have
| the cache perfectly configured (did u cover npm, docker, etc.
| with cache keys correctly?)
|
| Looking at these flaws, running workflows from a persistent VM of
| ur own becomes pretty tempting because you don't need to copy
| caches around and can easily SSH in.
| arialdomartini wrote:
| First time I see jj being mentioned in a post not about jj. Made
| me very happy.
| notepad0x90 wrote:
| For those who have experience, how does github actions compare to
| azure devops pipelines?
___________________________________________________________________
(page generated 2026-01-14 23:00 UTC)