[HN Gopher] Terraform module for scalable GitHub action runners ...
___________________________________________________________________
Terraform module for scalable GitHub action runners on AWS
Author : ta988
Score : 104 points
Date : 2023-12-09 05:03 UTC (17 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| leipert wrote:
| This looks cool! I will have a look how it works.
|
| It's funny because just the other day I thought about
| implementing the same for GitLab in order to learn more about
| terraform.
|
| (Note: I work for GitLab, but as a Frontend Engineer and usually
| not on CI topics)
| senzilla wrote:
| I love how, in the last few years, open source infrastructure has
| become more of a thing.
|
| We're using this Philips Labs module at $dayjob. It's a great
| piece of work!
| Grimburger wrote:
| Meta: Absolutely superb readme for an open-source project. Can
| anyone point to a better opensource readme today?
|
| That said, do many see value is in this? The current stance is
| you either self-host or accept paying github's runner rates out
| of laziness. In the end it's all very mild when it comes to extra
| costs.
|
| When things go wrong, you need people going through logs working
| out that there's nothing actually wrong with CI and in the end
| finding "oh it was a re-provisioning of compute", so...let's run
| it again and hope that goes well, actually no...let's schedule it
| for 11pm Eastern which probably isn't that optimal.
|
| I can see it working for some, but horrendous for others. Maybe
| if you have burstable CI loads it's worth doing and re-rolling on
| failure.
|
| Still, a marvellous readme. If only others in open-source would
| take note that this is how you do it.
| lukevp wrote:
| Cost isn't the only vector. Someone may want to host the
| runners within a given VPC so they have access to private
| resources / PrivateLink etc. the job API is a lot narrower of
| an attack surface area than giving your auth info to GitHub
| that has enough power to do CI/CD infra changes and such.
|
| There's also people who use GitHub Enterprise who still need to
| host a runner fleet, so they could benefit from this too.
| Grimburger wrote:
| Sure, but does my point about fallible spot not still stand?
| quickthrower2 wrote:
| I agree usually "CI is running" means "human is sorta
| blocked waiting on this"* means the compute cost to run it
| reliably is probably negligible. Whether you this on the
| Github network or your own private cloud is then an
| orthogonal concern. Spot is silly although spot with
| immediate fallback to in demand might be a neat hack.
|
| * Dependabot and AI PRs might be exceptions.
| stoobs wrote:
| This. We're using the this at my day job for the requirement
| for the runner to access private resources such as internal
| api's etc
| raffraffraff wrote:
| There's also the ability to pre-provision your runners with
| all of your base container images. Base images don't change
| that often and can be large. At an AI startup I worked at,
| self-hosting runners that used our own AMIs saved us a lot of
| time and egress costs, because so many CI steps begin by
| pulling _some_ image, whether it be the base image or one you
| built at a previous step and pushed to your container
| registry. We also ran CI steps on some projects that required
| ARM or GPU. You also have the option of keeping a runner up
| during the busiest part of the day to benefit from cached
| images. We 'd automatically rebuild the runner AMIs once a
| week and set runner lifetime to max 24 hours, ensuring
| freshness and ensuring that we didn't run into disk space
| issues.
| SOLAR_FIELDS wrote:
| We were baking AMI's like this for a little while but our
| current setup is quite even fancier - you can do everything
| in containers and then a distributed in cluster buildkit
| cache and registry be hosted and backed by S3, and not only
| do you get a remote reusable cache that syncs extremely
| fast but you no longer have to rely on external artifact
| registries being flaky, which if you pull from Dockerhub
| frequently know this is an actual significant problem.
|
| We actually looked at using this project from OP at the
| beginning of our ci journey but eventually decided on
| actions-runner-controller due to its higher level of
| robustness and configurabiltiy
| barefeg wrote:
| Could you give more details/resources on the distributed
| in cluster buildkit cache?
| baby_souffle wrote:
| I would also like to know more
| SOLAR_FIELDS wrote:
| We use Dagger's implementation. The basic approach is to
| have a buildkit engine run as a daemonset on the cluster,
| and clients specify the same docker socket that buildkit
| uses. The magic is in cache synchronization, eg only
| lazily pulling layers as the client requests them. This
| is scalable but obviously since caching is hard there are
| some complexities with efficient synchronization of cache
| layers and cache volumes. This is currently a long lived
| service that runs as a deployment alongside a bunch of
| ephemeral runners to manage the cache synchronization.
|
| There are several other different architectures that
| range from simpler to more complex. The architecture I
| recommend people start out with is a single long lived
| beefy buildkit instance that a bunch of runners share,
| since that is much much simpler to implement. It of
| course has the downside that you have to refresh/rebuild
| the cache if the instance ever goes down. For runs that
| need read/write locks on volumes (eg Gradle build cache)
| my recommendation after trial and error to rsync those to
| the runners and then rsync them back after the run
| completes so you don't have a bunch of locks fighting
| each other for the same folder.
| skwashd wrote:
| Add network egress controls and telemetry as two more
| benefits of self hosted runners. Denied network connections
| are a sign there could be a problem with a dependency.
| tlarkworthy wrote:
| Absolutely, there are lots of workloads that do not run on
| actions and require custom hardware (GPU? Big boxes) or greater
| privileges to Linux API than actions provide, or just better
| caching.
| dataking wrote:
| This.
|
| You even need to self-host if you want to test code that uses
| AVX512 outside of an emulator since the default runners do
| not support that. Same if you want to test aarch64-specific
| code paths on Linux, Windows, or macOS.
| andrus wrote:
| > That said, do many see value is in this?
|
| GitHub don't host runners for Linux x arm64, so if you need
| this, you need to self-host. You can also run custom AMIs with
| pre-installed packages, which can speed up workflows that
| depend on those packages.
|
| > When things go wrong, you need people going through logs
| working out that there's nothing actually wrong with CI...
|
| I'm on a small team who've been running the Philips Lab self-
| hosted runners for the past year. It hasn't been difficult to
| operate. Once deployed, it pretty much "just works".
|
| In my experience, the things that go wrong originate from the
| GitHub workflows themselves. We usually have to review workflow
| logs regardless of whether the workflow uses a self-hosted
| runner or not.
| Grimburger wrote:
| >GitHub don't host runners for Linux x arm64
|
| Is this not possible via QEMU like everyone has been doing
| for a very long time now?
|
| The real issue here seems to be large companies wanting their
| own syntactical turf everyone abides by that they can later
| profit from.
| Aeolun wrote:
| > Maybe if you have burstable CI loads it's worth doing and re-
| rolling on failure.
|
| We have extremely bursty CI loads. One push can kick off up to
| 60 different CI jobs. If a few people push at the same time, we
| easily have 300 jobs running in parallel. This happens the
| whole day long.
|
| It absolutely makes sense to self host this, since the cost to
| run this on Github runners would be prohibitive (Github runners
| are 8x the cost of the equivalent AWS instance I think). All
| our runners are ephemeral, so we only pay for them when they
| are actually running jobs. After all the jobs are over, the
| runners are immediately scaled to zero.
|
| I build this whole thing myself, so it's a bit sad to find
| someone had already built the whole thing before.
| candiddevmike wrote:
| ARC is great for running GitHub Actions on Kubernetes:
|
| https://github.com/actions/actions-runner-controller
|
| I think Kubernetes is a better platform than EC2 for runners,
| it's faster and more integrated with your tooling (if you're
| using Kubernetes).
| uuddlrlrbas wrote:
| Yeah we use this as well. It really is awesome and highly
| recommended and fairly straight forward to setup.
| moltar wrote:
| There are also official Heml charts available for ARC:
| https://github.com/actions/actions-runner-controller
|
| I am in the process of setting it up on a cheap Hetzner box. If
| it works, would be a great deal! You can get a 64 GB RAM box for
| 35 EUR/mo at server auctions with unlimited traffic. I don't
| mention CPU or GPU, as typically this isn't a bottleneck for my
| projects.
|
| Plus, I can configure cache sharing via host-mounted dir. E.g.
| pnpm cache can be all in one place, and be locally available to
| pods via a mounted dir. Same for the Docker image cache. This
| would speed up CI runs and also reduce network traffic by a huge
| margin.
|
| GitHub Actions effectively has no local caching. There's an
| action for caching, but it uses a blob storage for cache
| artifacts. Which then gets network fetched, gzip'ed and gunzip'ed
| each time, and from my experience this has never been a gain for
| medium to large npm projects, as they have thousands of small .js
| files in node_modules, and thus takes a long time to compress and
| decompress. I think npm edge cache servers are already so
| optimized and fast, that in my experience almost always it's
| faster to install from npm directly. I even tested this on AWS,
| where the cache was stored in S3, in the same region as CodeBuild
| (CI), and direct installs from npm were still faster by about
| 30%.
|
| So other than adding more hardware resources, local caching is
| the only way to significantly speed up GH Actions, from my
| experience, and thus you must have your runner.
| ozgune wrote:
| (Disclaimer: I'm a cofounder at Ubicloud.)
|
| Hey there, we offer Ubicloud Runners that are 10x cheaper than
| GitHub and bill them by the minute. You get a fresh VM with
| each job; and we use Hetzner as our underlying provider.
|
| If you're already setting up a Hetzner box, I'd love to get
| your input. Any thoughts or feedback for us?
|
| https://www.ubicloud.com/use-cases/github-actions
|
| https://github.com/ubicloud/ubicloud/blob/main/routes/web/we...
| (our github actions integration is also openly available)
| moltar wrote:
| Does it address my main issue of cache sharing? If cache is
| still over the network, then it's not optimal, as it's the
| bottleneck for everything. E.g. our node_modules is 3 GB, and
| once you gzip and gunzip it takes minutes. With pnpm cache
| store we can reduce this to seconds.
| xjia wrote:
| If you are interested in persistent cache, maybe take a
| look at https://dime.run
| moltar wrote:
| Pricing is better than GH but still not competitive.
|
| We use BuildJet now and it's $0.008/min for 4 vCPU and 16 GB,
| while your offer is 0.16C//min for the same (20x more
| expensive).
| ozgune wrote:
| I think you're confusing the unit in our pricing. We charge
| 0.16C//min for 4 vCPU and 16 GB, which is $0.0016/min.
| That's 5x cheaper than BuildJet's pricing.
|
| We didn't quote our prices in $ because the number of
| trailing zeros confused people. Maybe still go ahead and
| switch back to that?
| cocoflunchy wrote:
| The industry standard is $/min, so yes I think you should
| stick to it. And emphasize that it's 10x less than github
| actions pricing (most competitors are 2x cheaper instead)
| moltar wrote:
| Ah you are right. A agree with other commenters that it's
| confusing compared to industry standard.
| cocoflunchy wrote:
| Ubicloud is using c/min which makes it harder to compare...
| Buildjet is 0.8c/min and Ubicloud is 0.16c/min which is 5x
| cheaper.
| hipadev23 wrote:
| Building a business on hetzner is a hell of a bold move.
| cqqxo4zV46cp wrote:
| In the past, I used GHCR for (large) Docker layer image caching
| in CI. Now, I'm using the GHA cache. For either case, it really
| doesn't feel like I'm gaining that much, because of the time
| required to pull from the cache. Really unfortunate.
| ta988 wrote:
| In our case we are talking about 2000s of cpus with a lot of
| bursting. So buying single boxes is not really doable.
| gchamonlive wrote:
| The only frustrating part of this project is GitHub actions
| itself. It could be so much better
| (https://news.ycombinator.com/item?id=37612420). Part of the
| hassle is figuring out why on earth actions/checkout stopped
| working in your self-hosted infrastructure... Why you suddenly
| have what once were ephemeral SQL containers persisting and
| blocking specific ports in subsequent runs... And why that sweet
| python provisioning action isn't doing its job anymore...
|
| It all comes down to the instances. GitHub runners are insanely
| complex and also the best part of actions, along with community
| driven reusable workflows. They not only pack so much stuff
| without conflict, they also spin up really fast (forgot which
| virtualization infrastructure they use). So when you move to
| self-hosted you now have instances that take longer to be ready,
| increasing cold start times from pipelines, that will require
| that you install everything you need in it to run your jobs, and
| that, depending on how you set them, will require more thorough
| cleanup after each job.
|
| The scalability is really good though, and all the benefits
| already cited, like having access to private resources on your
| VPC, and delegating permissions to instance profile through IAM
| roles, make this project a godsend.
| suryao wrote:
| We maintain a little action called Action-Debugger that let's
| you ssh into a running GitHub action workflow to help debug
| pesky issues. It has a few additional features when you use it
| with our runners but works very well by itself.
|
| https://docs.warpbuild.com/tools/action-debugger
|
| https://github.com/WarpBuilds/action-debugger
| damianh wrote:
| If you are interested in having self-hosted auto scaling github
| actions runners but don't want to manage your own webhook
| handling infrastructure, https://cirun.io is worth a look.
| _joel wrote:
| I used this at the last gig. It works really well +1.
| romanhn wrote:
| I was using it for my project because the workflows required
| access to VPC resources (the database, etc). It worked... ok.
| There would be times when the runner didn't start, or parallel
| runs would not work, felt a bit random. Possibly was my config,
| there were quite a few knobs and dials.
|
| Eventually I stumbled on the idea of running the VPC-requiring
| commands from an AWS CodeBuild script, and invoking it from a
| workflow executed on a GitHub-owned runner. Works beautifully and
| I was able to remove a ton of complexity from my infra that this
| Terraform module adds.
| eez0 wrote:
| Been using this for 2+ years.
|
| My experience is that it works until it does not, and then you
| are down on a rabbit hole trying to figure out why.
|
| The only reason we keep using it is because we have other
| priorities, but once we have more spare time, this module is
| going to hell, it's not reliable at all.
| ta988 wrote:
| Interesting, the only issues we have had were when AWS didn't
| have enough capacity. And you can see that in the logs of the
| scale up lambda.
| xjia wrote:
| I had a similar experience with ARC (actions-runner-
| controller).
|
| One of the machines in the fleet failed to sync its clock via
| NTP. Once a job X got scheduled to it, the runner pod failed
| authentication due to incorrect clock time, and then the whole
| ARC system started to behave incorrectly: job X was stuck
| without runners, until another workflow job Y was created, and
| then X got run but Y became stuck. There were also other wierd
| behaviors like this so I eventually rebuilt everything based on
| VMs and stopped using ARC.
|
| Using VMs also allowed me to support the use of the official
| runner images [0], which is good for compatibility.
|
| I feel more people would benefit from managed "self-hosted"
| runners, so I started DimeRun [1] to provide cheaper GHA
| runners for people who don't have the time/willingness to
| troubleshoot low-level infra issues.
|
| [0]: https://github.com/actions/runner-images [1]:
| https://dime.run
| eez0 wrote:
| Exactly what you're are describing is what I explained to my
| colleagues as "stealing runners" :)
|
| If something fails and you don't have idle runners (hence
| wasting unnecessary resources), things start to snowball.
| neoromantique wrote:
| It's only really usable for anything that doesn't involve
| secrets, I'd be very concerned using anything third party in
| CI, let alone the runner itself. Supply chain attack senses
| tingling :).
| xjia wrote:
| Yes I totally understand the concern. We are actively
| working on SOC 2 and other compliance stuff to help with
| this. But honestly I feel the compliance requirements are
| weaker than what we actually implemented. For example
| proper secure boot and whole disk encryption (without
| sacrificing performance) are mandatory in our mindset but
| these specific things don't get reflected in compliance.
|
| Instead of being a service, I'm also open to sell the
| software+hardware solution behind it, so you can have it
| on-prem. Do you think that's something you would consider
| given the constraints on supply chain security?
| neoromantique wrote:
| We're too small for on-prem services, so not your target
| market, just shared my 2c as someone who had been burned
| by self-hosting github runners too many a time.
| aprilthird2021 wrote:
| To be fair, that is an accurate description of a lot of
| software libraries I've used at work, even some we've paid for.
| Halan wrote:
| Been using Scalesets + Karpenter and the only problem is the
| occasional GH availability taking a dip
| erulabs wrote:
| With Karpenter becoming an official EKS add on, I'm really hoping
| the new "scalesets" gets the same treatment and then we're
| somewhat close to a an out of the box ci on EKS experience.
|
| Right now I'm using Karpenter, ARC, EFS and buildkit and it's
| great, but it was also like a month of setup and is nontrivially
| complex.
| SpookyChoice wrote:
| We evaluated this but shied away from the complexity... In the
| end we build a slim version ourself containing
|
| 1. A codebuild job running a container image which starts a
| runner with custom labels
|
| 2. A lambda to receive the webhook from github and run the
| codebuild job on demand.
| Shorn wrote:
| I haven't implemented it myself - I just assumed this would be
| the way to folks would do this.
|
| What would implementing all the infra give you in terms of
| benefits over codebuild?
|
| The readme doesn't mention anything about codebuild that I can
| see.
___________________________________________________________________
(page generated 2023-12-09 23:02 UTC)