[HN Gopher] Arm64 on GitHub Actions
___________________________________________________________________
Arm64 on GitHub Actions
Author : sonichigo
Score : 221 points
Date : 2024-06-05 05:02 UTC (17 hours ago)
(HTM) web link (github.blog)
(TXT) w3m dump (github.blog)
| obviyus wrote:
| Nice! I had been using self-hosted arm runners for my projects
| lately (which GitHub makes surprisingly easy to do):
| https://docs.github.com/en/actions/hosting-your-own-runners/...
|
| Considering how compiling a couple of these projects on my tiny
| arm VPS slows down everything else, this is a welcome change!
| sofixa wrote:
| > which GitHub makes surprisingly easy to do
|
| Why surprisingly? Are there any CI/CD, let alone SaaS CI/CD
| providers, that don't make it easy?
| obviyus wrote:
| For some reason I expected a "not-first-class" experience
| while using my own hosted runner but that was not the case.
| It works and behaves exactly as their own runners do. Maybe
| s/surprisingly/extremely would be more what I meant.
|
| Just wanted to add, I loved your post on logging Nomad allocs
| using Loki. Was a huge help while I was setting up our
| clusters at $CURRENT_JOB
| sofixa wrote:
| > For some reason I expected a "not-first-class" experience
| while using my own hosted runner but that was not the case.
| It works and behaves exactly as their own runners do.
|
| Interesting, I've used GitLab extensively and hosted/self-
| hosted runners were trivial to use/run, so I would have
| expected nothing less from GitHub. They probably make a
| decent margin on the hosted runners at the cost of
| significant complexity, so if they can give that away to
| the orgs that will have their own (virtual) hardware
| anyway, with their own contracts and networking and etc.,
| why not?
|
| > Just wanted to add, I loved your post on logging Nomad
| allocs using Loki. Was a huge help while I was setting up
| our clusters at $CURRENT_JOB
|
| Glad you found it helpful! If you ever want to chat about
| it, feel free to hit me up (contacts are on my website
| available in my profile).
| saagarjha wrote:
| I mean they did before, but they were running macOS ;)
| drcongo wrote:
| Digital Ocean are going to be dead last in the transition to ARM.
| diggan wrote:
| Are GitHub and Digital Ocean competitors nowadays? Or you're
| just referring to anything that offers "run code on someone
| elses computer, somehow"?
| bdcravens wrote:
| AWS still only supports X64 on Fargate Spot instances (they do
| support ARM on the non-spot instances)
| trueismywork wrote:
| T4g is available as spot
| endgame wrote:
| GP's talking about Fargate, which is their "serverless"
| container runtime offering.
| booleanbetrayal wrote:
| We waited on this issue for 2 years before migrating off EKS
| Fargate to Managed Node Groups due to this very reason. In
| the end, it turned out to be much better of an environment
| anyway, because you didn't have to deal with Kubernetes
| oddities (like lack of Daemonsets) requiring anti-patterns
| and ultimately resulted in cheaper pricing due to better bin-
| packing. If you're doing K8s orchestration with Fargate, I
| highly recommend the switch.
| suryao wrote:
| (founder of WarpBuild - we offer hosted GHA runners) This
| is a common issue. A common pain point we see with users
| approaching some scale with self-hosting on k8s is that the
| k8s node autoscaling can become inefficient because of
| spiky loads.
|
| We have a lot of users migrating off self-hosted setups
| using `actions-runner-controller` to ours because of this.
| Essentially, not having to deal with bin-packing is more
| efficient and concurrency, uptime guarantees are nice.
| watermelon0 wrote:
| To be honest, lack of daemonsets makes sense, because you
| don't have hosts per se. Each pod is running on it's own
| Linux VM.
|
| Daemonsets are generally intended if you have multi-pod
| nodes; otherwise, you can just use sidecars.
| booleanbetrayal wrote:
| Sidecars are definitely a workaround but are hard to
| manage lifecycle for in conjunction with the primary
| container. This is now easier to do in 1.29+ with
| Sidecars officially supported via restartPolicy, but it
| was a colossal hassle prior to the advent of that. Also,
| we had noticed (maybe this has changed) that often
| logging frameworks were distro'd as DaemonSets (Helm,
| etc) and you'd have to shoe-horn the sidecar approach a
| bit.
| shpx wrote:
| It's not free.
|
| > Larger runners are only available for organizations and
| enterprises using the GitHub Team [$4 per user/month] or GitHub
| Enterprise Cloud [$21 per user/month] plans.
| sunaookami wrote:
| Seems like normal users have to wait:
|
| > We expect to begin offering Arm runners for open source
| projects by the end of the year
|
| side note: I find it mildly annoying that it's no longer "ARM"
| but "Arm"
| VWWHFSfQ wrote:
| Yeah it's confusing because the company is branded like "Arm"
| but they still call the architecture "ARM"
| joshstrange wrote:
| GitHub's runners are a joke. Switching to WarpBuild was the best
| decision I made for my CI/CD. I halved my build time and got
| cheaper minutes.
|
| GH's macOS runners particularly were complete trash.
| 999900000999 wrote:
| I understand the Mac builds, but the entire appeal of GitHub
| runners is it's literally built in. I can just add a YAML file,
| and we have CICD!
| joshstrange wrote:
| I get that, but you can run GH runners on your own hardware
| or someone else's. Switching to WarpBuild was literally as
| easy as linking my GH account and changing the running image
| from `macOS-latest` to `warp-macos-14-arm64-6x` and
| everything worked. Secrets and everything.
| palata wrote:
| > Secrets and everything.
|
| I guess you had to import the secrets, too...
| yohannparis wrote:
| Different needs, different tools! I work on a open-source
| project, and not having to pay for CI/CD with a "simple" yml
| file configuration is simpler and easy to manage.
| 01HNNWZ0MV43FF wrote:
| It's only free if your time is worthless :P every couple
| weeks I have to debug something that I can't replicate
| locally because I don't have local CI
| joshstrange wrote:
| That's the other plus with something like WarpBuild, I
| haven't used it yet but you can pause the build and SSH in:
|
| https://github.com/WarpBuilds/action-debugger
| EionRobb wrote:
| Have you tried using act for that?
| https://github.com/nektos/act Has saved me a bunch of CI
| debug pain.
| joshstrange wrote:
| If you are open source and getting free minutes then it
| really doesn't matter how long builds take.
|
| I'm still using GH Actions, just not their minutes.
| suryao wrote:
| There is a lot of inefficiency in CI today and we're out to
| solve it.
|
| Thanks for the WarpBuild love!
| candiddevmike wrote:
| Or just run your own runners on-premise. You'd be amazed at how
| fast builds can be with OTS hardware, or even something like a
| laptop or Mac mini.
| joshstrange wrote:
| I considered this for my personal business but the minutes
| are stupid cheap and to pay off a Mac mini it would take me
| many, many years to break even. For a company with
| 10's-100's-1000's+ employees writing code and needing CI/CD
| the math will be different. I came close to buying a Mac Mini
| anyways, costs be damned, to get away from GH's slow runners.
| dehrmann wrote:
| > minutes are stupid cheap
|
| At smaller scales, as long as your builds aren't stupud-
| slow, cloud providers should be cheaper just because they
| can utilize their hardware more (10x?) efficiently.
| joshstrange wrote:
| Totally agree, and I called out that my scale is small so
| cloud makes a lot more financial sense for me (especially
| as a solo founder, I don't want to manage more hardware
| than I have to already).
| aromeronavia wrote:
| Can't agree more with this, I recently migrated my company to
| use WarpBuild and our CI got 50% faster, not even kidding. We
| were using self-hosted runners which were not as efficient as
| we thought so we spawned EC2 instances and connected them to
| the github network to be runners, but we had such an amount of
| issues that we wanted to go back to non-self hosted. At the end
| a tech lead of the company suggested using Warp and we could
| not be happier with the results.
|
| Also the CEO Surya is such a great guy, we had a small
| networking issue that lasted for about two hours and they gave
| us credits for all the runners affected with that, and their
| customer support is blazingly fast.
| suryao wrote:
| Thank you so much for the kind words!
| crohr wrote:
| > We were using self-hosted runners which were not as
| efficient as we thought so we spawned EC2 instances and
| connected them to the github network to be runners, but we
| had such an amount of issues that we wanted to go back to
| non-self hosted
|
| For those hitting that same kind of issue (and you are many),
| you should check out my product https://runs-on.com. 1-click
| install, 1-click upgrades, and from 7x to 16x cheaper runners
| on AWS.
| joshstrange wrote:
| I was deploying to CloudFormation and I thought that was the
| slow part but my "build" times went from 20min to 10min. My
| iOS builds also were halved.
| jjice wrote:
| We're a small team (four engineers) and we purchased four Dell
| mini PCs to toss into our small office space. The break even
| compared to our AWS hosted runners (which were much faster and
| cheaper than the GH ones) was like four or five months, plus
| they cut the time again by over half. We have to deal with them
| if something weird happens, but it's been such a low time
| investment that it has been such a great choice.
|
| Plus, who doesn't love having some new hardware to play with?
| edweis wrote:
| Why 4 Dell Minis? You don't need a monitor for CI/CD, and why
| 4? Is it for redundancy? Performance?
|
| We are interested in doing the same setup.
| candiddevmike wrote:
| Not OP, but I went with MOREFINE AMD Ryzen mini PCs. They
| can be configured with a ton of CPU/RAM (including ECC).
| They're really compact, support 2.5 Gb ethernet, and are
| great to work with. A comparable Dell workstation with ECC
| RAM would've been significantly more expensive, and you
| have to deal with their business folks to purchase them (or
| at least I couldn't figure out how to buy them direct).
| jjice wrote:
| Wasn't my choice, but one wasn't enough so we threw a few
| more at it. So mostly for performance, but the redundancy
| is an added benefit. They were a pretty decent space,
| power/heat, performance, and noise ratio for us.
|
| Our office is a single room with six desks, so space,
| noise, and heat are a larger consideration for us. I use a
| mini PC at home for a personal server as well, but that's
| just because used enterprise equipment is often a great
| deal.
| dehrmann wrote:
| > We have to deal with them if something weird happens
|
| Check back in three years once you've had to deal with OS
| upgrades, power outages, running out of disk space, and one
| of them failing.
| abuani wrote:
| Three years sounds like a reasonably good return on
| investment. At that point the teams either big enough to
| justify paying $$ for a hosted solution, or still cheaper
| to buy dedicated hardware.
| delfinom wrote:
| You can also just rent dedicated servers from hetzner or
| ovh for far cheaper than AWS instances with more
| horsepower. More expensive than a dell mini in a closet.
| But you get datacenter Internet and power backup.
| re-thc wrote:
| > justify paying $$ for a hosted solution, or still
| cheaper to buy dedicated hardware.
|
| Always cheaper if you have at least some demand. For the
| lazy fix even just get a few mac mini and it'd work fine.
| delfinom wrote:
| It's a CI runner my dude.
|
| You just obliterate and reinstall the base system in what
| 10 minutes?
| cr125rider wrote:
| Are any of those problems challenging to deal with? Even if
| it's a few expensive man hours the ROI is still quite
| short.
| barkingcat wrote:
| if the break even was 4-5 months, it means compared to
| going to cloud vendors, in 36 months, the self hosted
| solution would have saved 7.2 times the cost!!
|
| Even if within 3 years there needs to be OS upgrades, you
| can just e-cycle the old machines and buy brand new
| minipc's (with added benefit of increased performance)
| every time you need to upgrade, and you'd still be ahead!!
| jjice wrote:
| We're about eight months in at the moment, so here are some
| current stats:
|
| - No OS upgrades (running Ubuntu 22 LTS), but regular
| update
|
| - We've had a power outage, but not during the day, so when
| they rebooted, they went right back to action. Same issue
| with a network outage in the office.
|
| - We have hit disk space issues due to not clearing all the
| data we needed to. A quick SSH and an rm or two did the
| job, plus another 30 minutes to an hour to update a cron to
| remove that portion of the cache
|
| I see what you mean for sure, and there will undoubtedly be
| more things that come up at some point, but with the
| savings, it would take quite a few engineering hours per
| month to lose the savings at this point.
|
| And I won't lie, when something goes wrong, it's a nice
| change of pace to deal with a server instead of code once
| and a while. Can't actually use that for calculations, but
| I don't hate when it happens so far.
| KRAKRISMOTT wrote:
| How do they compare to Ubicloud?
|
| https://www.ubicloud.com/
| joshstrange wrote:
| I think I skipped over evaluating ubicloud because they don't
| have macOS runners. There was another place I liked but I
| needed macOS runners they didn't have them either.
| suryao wrote:
| (WarpBuild founder here) We have had a lot of users evaluate
| multiple tools including ubicloud but chose us. Typically,
| they've mentioned our runners being faster, our caching is
| very useful, and queueing delays being lower.
|
| However, my last anecdata was from a few weeks ago. You can
| try us out and take a call for yourself.
|
| Our focus is on CI runners and efficiency and I think that
| gives us an edge on the devex. Ubicloud is definitely cheaper
| though.
| duped wrote:
| GitHub Actions is a joke. It baffles me that they've been
| breaking file permissions on upload for years and they haven't
| fixed it yet. God forbid you want to ship an executable in a
| tarball or save it for a later stage of your build.
| ugh123 wrote:
| How do you handle extremely large repos? Can these be cached on
| machines, per-customer?
| suryao wrote:
| Assuming the question was reg. WarpBuild - we provide blazing
| fast caches with unlimited sizes. These still need to be
| downloaded to machines and setup on each workflow job.
|
| We don't (yet) have support for pre-pulled repos on runners
| per customer. Are the large repo sizes despite fetching only
| the refs required for the specific context [1][2] which can
| help speed things up significantly.
|
| [1] https://www.atlassian.com/git/tutorials/big-repositories
| [2] https://stackoverflow.com/questions/48072080/how-to-
| manage-l...
| electroly wrote:
| Ditto. I came for the cheaper Apple Silicon runners (before
| GitHub Actions had them available for free) and stayed for the
| faster runs across the board. I cut a 24 minute build down to
| 10 minutes just by switching over to WarpBuild runners. Not a
| shill, just a happy customer.
| suryao wrote:
| And I a happy provider - thank you!
| deivid wrote:
| I wonder if they'll start supporting risc-v soon.
|
| I've been using github-act-runner on top of a local Nomad cluster
| to run some of my builds on risc-v, arm64 & x86-64, it's fine,
| but a bit of a hassle to set up
| compsciphd wrote:
| would only happen after azure supports risc-v VMs.
| sapiogram wrote:
| Given how long arm64 took them, my guess is "3 years after
| major cloud vendors start offering risc-v VMs".
| deivid wrote:
| I'd kinda hoped that they took so long to go arm64 because
| they generalized "1 architecture -> N architectures"
| crohr wrote:
| Much welcome addition, although not available (yet) on free
| plans. I benchmarked them and they are ~15-20% slower than AWS
| Graviton processors: https://runs-on.com/benchmarks/github-
| actions-runners/#arm64...
|
| But at 37% cheaper than x64 it's quite a better deal than what
| the outdated x64 CPUs.
| crohr wrote:
| Also a bit weird that they now seem to delegate the image
| building to third-party vendors (first GPU images, now ARM).
| Would be nice to know how they validate base images before
| distributing them to everyone.
| ozgune wrote:
| (Ozgun from Ubicloud)
|
| Looks like every GitHub Actions provider is on this thread. :)
| I also wanted to chime in with two thoughts.
|
| First, we recently wrote about how we enabled arm64 VMs at
| Ubicloud. Any feedback on the technical details is more than
| welcome: https://news.ycombinator.com/item?id=40491377
|
| Second, I feel runs-on's analysis is a bit unfair to us.
| Ubicloud's x64 / arm64 performance and queue times above are as
| good as any (we deprecated the AMD EPYC 7502 line). For
| example, RunsOn arm64 queue times are 31|42 seconds and
| Ubicloud queue times are 17|24 seconds.
|
| But the analysis says the following, "Be aware that Ubicloud
| has low CPU speeds and high queue times, which means they won't
| necessarily end up being cheaper than competitors." I fail to
| see this conclusion from the numbers above. What am I missing?
| crohr wrote:
| The queue time for x64 appears to vary quite a bit still. But
| you are correct that it is much better than what it was a few
| weeks back, when the analysis was written (it was not unusual
| to see 60s+ spikes).
|
| Also note that I specifically mentioned that "all third-
| parties are good on that front, expected [sic] Ubicloud (but
| that may change)."
|
| Also agree that now that you removed the outdated CPUs (was
| still active end of May), the analysis should mention that
| you have OK CPUs by now. I will fix it.
|
| For providers that are on this page, do not hesitate to write
| to me if you find the analysis outdated for your specific
| service.
|
| [edit] page is now up to date with new analysis
| ozgune wrote:
| Hey there, thank you for this update. It's your analysis,
| so you're free to mention Ubicloud as you'd like.
|
| Still, you updated the above statement to, "Be aware that
| Ubicloud has a slightly lower CPU speed and somewhat
| variable queue times for x64 (but improving)."
|
| Could you clarify what you meant by variable queue times?
|
| According to the benchmarks, Ubicloud queue times for x64
| were 18|44 secs at p50|p95 over the past month. I agree
| that our p95 number is higher. At the same time, RunsOn's
| AWS numbers are 31|36 secs. So I feel that driving a
| conclusion based on our p95 variance number for x64 is a
| bit unfair to Ubicloud.
|
| Thank you for compiling this benchmark btw. Now, we'll
| follow it closely.
| crohr wrote:
| RunsOn is the only on-premise solution of the bunch, so
| it is single tenant, and as such it cannot easily have
| pools of 10s of machines on standby to start within 10s.
| So on queue time I'm comparing Ubicloud against the other
| SaaS providers, and anything higher than 20s is not
| especially great in that case. Plus you have high
| variance, which I personally consider an issue.
|
| Once this improves I'll be sure to update it.
| bushbaba wrote:
| Doesn't GitHub use Azure, being owned by Microsoft. So perf
| difference could be Microsoft azure vs AWS graviton offering
| differences. Or Grubhub using an older machine type that
| provides greater capacity at lower costs.
| re-thc wrote:
| > I benchmarked them and they are ~15-20% slower than AWS
| Graviton processors
|
| As expected. Just Azure's Ampere Altra that's OLD. It's
| Graviton 2 era. We're at 4 now (sort of). Ampere One is still
| "somewhere".
| suryao wrote:
| I'm so glad that GitHub finally offers this.
|
| They're just more efficient for certain kinds of workflows - both
| cheaper and faster.
|
| If you're building for arm64 targets or android, this is a no
| brainer because it sidesteps the emulation requirements.
|
| We've been offering arm64 runners for a few months now (up to 32
| vcpu) with WarpBuild and our users love it.
|
| In general, there are lots of inefficiencies in CI starting from
| the lack of visibility, debugging ease, performance, runner
| config customization, persistence, etc. We're out to solve it and
| make the world's fastest CI cloud ecosystem on top of existing CI
| providers with WarpBuild.
| christina97 wrote:
| Does WarpBuild offer discounts for nonprofits/FOSS?
| suryao wrote:
| Not at this time. We're a small team trying to build a
| sustainably growing business with server bills to pay.
| bhouston wrote:
| Hmm... this is neat. I develop on MacBook so ARM64, and I do use
| GitHub Actions, which can not be ARM64, but unfortunately Google
| Cloud Run doesn't support ARM64 images yet:
|
| https://issuetracker.google.com/issues/303743857
| david_allison wrote:
| Excuse my ignorance, isn't `runs-on: macos-14` ARM64-based (M1)
|
| https://github.blog/changelog/2024-01-30-github-actions-maco...
| booleanbetrayal wrote:
| It's nice to know that we have an ARM64 fallback if our perfectly
| robust and cheaper self-hosting of GHA (gha-runner-scale-set*
| components) were ever to go down, I suppose, but I can't help but
| feel like I am part of a huge market segment that GitHub lost
| during the ARM64 runner absence. Seems a little too late given
| they have natively supported action-runners images in ARM64 for
| some time.
| dedene wrote:
| Can it be used to efficiently build multi-arch docker images?
| Both armv8/x64
| caleblloyd wrote:
| Maybe but I'd recommend checking out Depot.Dev for that, they
| expose Arm and x86-64 runners to buildkit/buildx. Not
| associated with them, just a happy customer.
| rahkiin wrote:
| Are there third party providers for Gitlab runners? I'd love
| windows and mac runners for my self hosted gitlab.
| frankdejonge wrote:
| One thing that consistently bugs me is how upgrading from the
| lowest type of runners to _literally_ anything else renders your
| _paid for_ included minutes useless. I do not understand who
| those wouldn 't just be multiples of the base runner, much like
| how Mac and Windows runners are. Seems like the crediting logic
| is there, but knowing software, there is probably some accidental
| complexity causing this part not to leverage it. That said, it's
| very frustrating from a paying customer perspective.
|
| The same goes for minute crediting, where splitting things up to
| run concurrently is actively discouraged because they
| individually all round up to the next minute. For example; have 3
| concurrent tasks that run 1m10s? You're billed 6 minutes. I get
| that a run would be rounded up, but come on.
| AndyKelley wrote:
| At Zig Software Foundation we rent an aarch64 Hetzner server for
| about 100 euros a month and it has 64 processors. It can handle a
| lot more CI work than the GitHub provided runners for a fraction
| of the cost.
| qwertox wrote:
| I had an issue where a Rust application, which I had compiled on
| a Radxa ROCK 5B, would nor run on a Raspberry Pi 4; I had to
| compile it again for it. Both were running Ubuntu 22.04, 64 bit.
|
| Why would that be and how would this affect these Arm64 compiles
| on GitHub?
|
| If you compile a normal application for amd64, it will work on
| any 64 bit Intel and AMD machine, unless you start using
| specialized features, which wasn't the case in the above
| mentioned case.
___________________________________________________________________
(page generated 2024-06-05 23:02 UTC)