[HN Gopher] GitHub Actions limitations and gotchas
___________________________________________________________________
GitHub Actions limitations and gotchas
Author : 147
Score : 130 points
Date : 2021-09-08 17:30 UTC (5 hours ago)
(HTM) web link (www.cbui.dev)
(TXT) w3m dump (www.cbui.dev)
| jeremyepling wrote:
| Thanks for taking the time to share your experience. I work on
| GitHub Actions and am familiar with the limitations you're
| experiencing. Below is more info on where we're at with each of
| these issues, using the numbers in your Table of Contents. Also,
| we have a lot of other new things planned for Actions as you can
| see in our roadmap.
| https://github.com/github/roadmap/projects/1?card_filter_que...
|
| 2.1 We're starting work on this in the next couple of months. We
| plan to ship it in early 2022.
|
| 2.2 We want to speed up the pace of GitHub Enterprise Server
| releases, but I don't have more info to share.
|
| 2.3 We're looking at ways to not require a GHEC account or
| "unified" license.
|
| 2.4 The limits are much higher with the GitHub hosted runners,
| but this is a current limit of self-hosted runners.
|
| 3.1 It's on our backlog. No date to share.
|
| 3.2 I haven't heard this before and thanks for sharing the
| scenario. We'll think about it.
|
| 3.3 This will ship in October.
|
| 3.4 We're doing some performance optimizations for GHES 3.4 that
| should fix this.
|
| 3.5 This shipped recently -
| https://github.blog/changelog/2021-08-25-github-actions-redu...
|
| 3.6 We have a couple API improvements coming later this year.
|
| 3.7 We're looking into this, but no dates to share.
|
| We're dedicated to making Actions a great experience. As you
| would assume, I'm very excited about the future of Actions and
| getting feedback like this helps us make it better.
| AndiSmith wrote:
| @jeremyepling I'd be interested to know if Github Actions will
| eventually replace Azure DevOps pipelines? Thanks
| jannes wrote:
| From an outside perspective it looks like neither of them is
| going away anytime soon and that they just want you to use
| the one that you feel more comfortable with.
| chatmasta wrote:
| Regarding 3.5 (actions calling other actions) - please continue
| investing in composite actions. They are a well-balanced
| abstraction for managing workflow complexity. I can see how
| they could become too powerful and encourage bad patterns (YAML
| programming), but that's a good problem to have.
| Liskni_si wrote:
| Related to 3.3 (You can't restart a single job of a workflow),
| is there any reason why the numeric id of a specific job run
| isn't available anywhere? There's GITHUB_RUN_ID, but that only
| identifies the workflow run, not the individual jobs, and this
| isn't unique across run/job restarts. Services like
| https://coveralls.io/ need an actually unique job run id, and
| they could also use the numeric id to link to the specific job
| run in a build matrix.
|
| To clarify this further, in
| https://github.com/xmonad/xmonad/actions/runs/1201348600,
| GITHUB_RUN_ID=1201348600, GITHUB_RUN_NUMBER=168, but there's no
| way to get the 3514203530 out of
| https://github.com/xmonad/xmonad/runs/3514203530 (that's one of
| the jobs of that run).
| nicois wrote:
| We have recently switched to GitHub actions and in addition to
| the above, there are two others which impact us:
|
| Sometimes the checkout action fetches the wrong commit! The SHA
| in the GitHub environment variable says one commit, but the
| actual code is different(!). Because we don't know why this
| happens we basically need to do an unshallow fetch of the whole
| repo to be sure we have what we expect.
|
| Using autoscaling self hosted runners, it is not currently
| possible to instruct the agent to finalise the current job but
| to accept none after it. This is essential to avoid broken
| workflows while scaling in. Gitlab supports this via a signal,
| but there is no equivalent.
| m_a_g wrote:
| One extremely painful "gotcha" we encountered was trying to push
| code to a protected branch from inside a workflow.
|
| With the default GITHUB_TOKEN, you can't push to protected
| branches. If you decide to use personal access tokens, you can
| push to protected branches, BUT that will trigger other
| workflows. That can cause an infinite loop of workflows.
|
| We still couldn't figure out how to push code to a protected
| branch without triggering the same/other workflows.
| chad_oliver wrote:
| > We still couldn't figure out how to push code to a protected
| branch without triggering the same/other workflows.
|
| If you include `[skip ci]` anywhere in the commit message, that
| commit won't trigger any github actions. This is a built-in
| behavior; you don't have to manually check for that string in
| any downstream actions. I spent a long time trying to work
| around the same constraint before my colleague pointed out that
| `[skip ci]` was a thing.
| simonw wrote:
| One trick that might work is to include a token such as
| 'DO_NOT_TRIGGER_ACTIONS' in your automated commit message, then
| have your other action skip any commits that include that token
| using this in the workflow: if:
| "!contains(github.event.head_commit.message,
| 'DO_NOT_TRIGGER_ACTIONS')"
| m_a_g wrote:
| good idea, thanks!
| chad_oliver wrote:
| > You can't have actions that call other actions
|
| As of August 25, you can!
| https://github.blog/changelog/2021-08-25-github-actions-redu...
| oefrha wrote:
| Also, scheduled jobs are a joke, they are routinely one to
| several hours late. I have to use the API to manually trigger
| jobs (workflow_dispatch) whenever I need scheduling that is
| remotely time sensitive.
| chrisfosterelli wrote:
| This was the first thing that came to mind for me too. I don't
| need it to be _exact_ , sure, but the scheduled jobs are more
| than an hour off more often than not. Sometimes several hours
| as you mentioned. Frequently they would also just not run at
| all!
|
| It was so bad I ended up scheduling my cron jobs to run 4 - 5
| hours early and just sit there idling until the actual
| execution time came around haha.
|
| Ultimately not running at all started to be a real pain point,
| I gave up and paid $5 for a monthly box to sit there and run
| crond to save on the headache.
| 147 wrote:
| I've heard of issues with scheduled jobs on Github.com but we
| didn't have any issues so far on GHES.
| elpakal wrote:
| Build engineer at a large Fortune 500 shop here - the largest
| impediment to us even enabling GitHub Actions is that you either
| disable them entirely for your org or allow repo admins to enable
| them _per repo_. We have several hundred repos in one org alone
| and we cannot simply enable them for everyone with admin access
| (for us if you create a repo, you get admin access).
| shadycuz wrote:
| You can run actions on your existing Jenkins infra
| https://github.com/DontShaveTheYak/jenkins-std-lib
| adwww wrote:
| > cannot simply enable them for everyone with admin access
|
| why not? is that a cost issue, perceived security,
| administration overhead?
| athorax wrote:
| Not OP, but in our case allowing essentially the entire
| company to start to use GHA would result in a huge load
| increase on our GHE deployment. Even with self-hosted runners
| per org, there is a lot of overhead that is ran on the github
| servers themselves. We are working towards rolling it out,
| but there are definitely some scalability issues.
| pojzon wrote:
| We are running dedicated agents per repo, anytime someone
| creates a repo an agent pool is created for him.
|
| Everything is running on AKS. We dont even pay for it THAT
| much, because container-agents are spinned up when needed.
| (Also few hundred repos)
| elpakal wrote:
| perceived security unfortunately. not my call
| [deleted]
| Octabrain wrote:
| In the company I am working we've decided to use Github actions
| for our CI pipelines instead of deploying any "on prem" solution.
| I've worked a lot in the past with Jenkins and Travis and I also
| played a bit at home with Github actions. Now that I am using it
| in real world scenarios I have to say that I am a bit
| disappointed. In my opinion, as soon as you try to do something a
| bit complex you end up having to implement some nasty hack. Also
| I found that the Github Actions marketplace is a bit of a sink.
| You have to spend a good amount of time browsing it for finding
| something decent that hasn't been discontinued, or is a pointless
| fork or at least is actively maintained. This happens even for
| basic functionalities.
|
| I known is a fairly recent platform but I was expecting much more
| compared to what other services offer.
| takeda wrote:
| This is why I like GitLab much more. They implemented a proper
| CI.
|
| I think what GitHub was going for is to create some kind of
| ecosystem for proprietary 3rd party applications and use that
| as a revenue. That approach only crippled the whole
| functionality.
| ak217 wrote:
| In my experience GHA far outclasses Jenkins and Travis. After
| GHA released composable actions
| (https://github.blog/changelog/2021-08-25-github-actions-
| redu...), there's not much of a comparison to be made. GitLab
| is more on par, but the docs are less accessible and they don't
| seem to have a long-term development strategy the way GHA does.
|
| As far as the limitations in the blog post, they are real, but
| most of them are limitations in features that are not available
| to begin with in Jenkins or Travis.
| cormacrelf wrote:
| Composable actions... which must be kept in an entirely
| separate repository. Yes they technically have the feature
| now, but that is a really poor showing. I won't be using it
| for anything because it takes the CI code away from where
| it's being run. It's a niche for people publishing generic
| actions eg a language specific cache, but most people are not
| in the business of producing actions for the marketplace. And
| I don't like that it's one repo per action.
| kylorhall wrote:
| I use a lot of composite actions in the same repo as well
| as composite actions across repos.
|
| Eg. `workflow1` and `workflow2` both call
| `composite/action`:
|
| .github/workflows/workflow1.yml
|
| .github/workflows/workflow2.yml
|
| .github/actions/composite/action.yml
|
| The only missing bit in that is a bit more support in a
| composite action: `if` and a few other keywords. Also, it's
| a bit annoying to access private across across repos.
| duped wrote:
| I use zero because of all the showstopping bugs and lack
| of any introspection or input validation. It saves me a
| lot of time and money.
| danielheath wrote:
| GHA definitely far outclasses Jenkins and Travis.
|
| However, having also used Buildkite, I'm not sure I could go
| back to GHA - it's slightly nicer to target, and at least as
| large a step towards reliability & predictability as Travis
| -> GHA was.
| rpdillon wrote:
| I use Buildkite all day as part of my job, and I have to
| say, it's amazing. The company is great to work with, the
| product is highly flexible for a variety of complex build
| workflows teams want to implement, and has scaled
| beautifully for 1000+ concurrent build machines split
| between on-prem and cloud. No affiliation, just a customer.
| trulyme wrote:
| > GitLab is more on par, but the docs are less accessible
| and...
|
| Interesting - I find GitLab docs easy to use while I hate GH
| ones. It takes a long time for me to find the relevant part,
| even more to determine that this is really all there is, and
| then some extra to guess which interpretation of the text
| suits the reality. It has never occured to me that someone
| might prefer GH docs to GL ones. :)
| bob1029 wrote:
| The only thing we've ever trusted GH actions with is enforcing
| check builds pass before a PR is allowed to merge.
|
| Everything else is managed via a custom tool we use for packaging
| & deploying our product.
|
| Even our simple "run this 1 build command and ensure exit code ==
| 0" action seems to have a semi-weekly issue like stuck "waiting
| for status" and other unexplained failures throughout. We don't
| want to put any more eggs into that particular basket right now.
| judge2020 wrote:
| The cache action section focuses on GHES which is a shame, since
| the GitHub.com cache action is also pretty bad.
|
| - can't manually delete cache (although they're considering
| changing that[0])
|
| - only saves cache at the end of a workflow, and only if the
| workflow succeeds. This could be solved with CircleCI's approach
| of having a save-cache step and a restore-cache step.
|
| - Cache is super slow for self-hosted runners, so it makes more
| sense to have a local cache instead of using the action
|
| - only 5gb of storage size. This was supposed to be increased via
| billable cache storage[1], but it's been on the backburner since
| July 2020
|
| - In addition to the above, you can't use a different storage
| backend on the official action (which would allow storing over
| 5GB of cache via your own storage). The best workaround is to use
| a user-provided action which utilizes the s3 api[2].
|
| 0: https://github.com/actions/cache/issues/632
|
| 1: https://github.com/github/roadmap/issues/66
|
| 2:
| https://github.com/actions/cache/issues/354#issuecomment-854...
|
| 2>
| trulyme wrote:
| - can't cache on tags [0]
|
| Caching is a joke. But then again, comparing GHA to GitLab CI
| is like comparing a pogo stick to a bicycle. Sure, both can be
| used to get there, but one ride is a bit... bumpy.
|
| (I use both and I am not affiliated to either.)
|
| [0] https://github.com/actions/cache/issues/556
| lmeyerov wrote:
| We've been loving GHA: CI/CD-as-code, pull requests capture
| deploy history (CI, stage/restage, deploy), and labeling a PR
| with 'release' is enough to generate our on-prem + multi-cloud
| artifacts.
|
| Our main gotchas are roughly:
|
| - GH-hosted runners have too little RAM/HD for big docker
| software. They push you to self-hosted runners for that, which is
| fine in theory, but GHA/Azure doesn't actually support serverless
| runners, so that falls flat in practice. We don't want to be
| turning machines on/off, that's GHA's job. We experimented with
| GHA -> Packer -> Azure for serverless, but it was slow and Packer
| frequently leaves zombie machines, so we went back to tweaking
| the low-RAM runners provided by our enterprise plan.
|
| - Security: We want contactors etc. to be able to run limited GHA
| CI jobs and use that quota, but not higher-trust GHA CD ones.
| This is tricky at a configuration level. Ex: It seems like we'd
| need to funny things like main repo for CI w/ CI secrets, and a
| separate repo for CD w/ CD secrets, and only give untrusted folks
| access to the CD-cred repo. We've thought of other possibilities
| as well, but in general, it's frightening.
|
| - Big Docker images: We do spend more time than I'd like messing
| with optimizing Docker caching as GPU containers are
| embarrassingly big (we use dockerhub vs github's due to
| sizes/pricing/etc), think both multi-stage containers + multi-
| step jobs (monorepo/microservices). I think they're in a good
| position to speed that up!
|
| I'm optimistic about these, but tricky to align with MS/GH PM
| personal team priorities :)
| yebyen wrote:
| I work on Flux, and for the reasons you mentioned we advocate
| for a strict separation between CI and CD, where CI is
| responsible for building and testing images and pushing out
| releases, and CD is responsible for production. There is no
| crossover between these roles, and this makes it very secure.
|
| You use a policy CRD called ImagePolicy to declare what
| versions should be matched and deployed automatically. You can
| have them deployed directly to production, or if it makes you
| more comfortable with your contractors, to an "auto-PR" branch
| which simply queues them up for a release engineer with the
| required juice to approve and merge the changes to prod.
|
| There is no deploy job in CI, since the deployment in prod is
| whatever happens to be on the main branch in the config repo.
| This is a protected branch which only releng can merge changes
| into, usually only when the release checks are satisfied. Flux
| (which scans any image repos) updates those manifests when
| there is a release, either directly or via pull request.
|
| This strategy I think would be able to address your Security
| concerns regarding contractors, by reducing the responsibility
| of your CI system to only CI, and not to handle CD anymore.
|
| This is the subject of my talk at KubeCon[1], although you
| might not be able to get that from the title, (I'm presenting
| this topic for a Jenkins audience, but the focus is on Flux and
| how Flux works) so I'm hoping it should be applicable to a
| broader audience, certainly inclusive of GitHub users :)
|
| [1]: https://kccncna2021.sched.com/event/lV0V/gitopsjenkins-ci-
| wi...
| orlovs wrote:
| +100000. any more or less serious case deployment must be
| decoupled from build. It brings some challenges, but deploy
| and build are two different cycles what you cant put on the
| same assembly area. It can be triggered, but it should not be
| same pipelin
| lmeyerov wrote:
| Yep -- and it goes deeper, even for CI. The CI/CD secrets
| vulnerability is just the most burning example.
|
| Ex: Network segmentation for CI. We only expect CI to
| communicate with dockerhub, conda/pypi, and the CI service
| (ex: azure). Something similar for CD. That _should_ be
| settable via GHA, but isn't. In a world of weird npm/python
| scripts and github action marketplace... scary we can't.
| Self-hosted runners can in theory do this via custom network
| policies, but it's a PITA for something ~everyone should be
| doing out-of-the-box.
|
| Edit: HN won't let me respond to the below. Imagine something
| like IP theft, wanting to mine our repo for everything
| labeled "security", dig into our version #'s for viable CVEs,
| or force-push some git commits (incl. history rewriting).
| Defense-in-depth says we shouldn't make that unnecessarily
| easy for anyone who plants a backdoor that runs during CI's
| `npm install` / `RUN xyz` phase. Network + RBAC are basically
| table stakes for almost everyone building enterprise
| software, so GHA doesn't have to reinvent the wheel here,
| just do it + make it friendly.
| yebyen wrote:
| I think those problems solve themselves when you can say
| "CI doesn't need to communicate with production at all
| anymore" - the Flux controllers live on the cluster, or on
| one cluster which is in position to communicate with others
| on the network in case of multi-cluster. They do not have
| any outward facing interfaces (except one), they live on
| the Kube API. (The Flux API is the CRD within the
| Kubernetes API.)
|
| The reason for CI to communicate with production is to
| handle deployments, right? So take away that
| responsibility, and now you can safely isolate GitHub
| Server and shunt it into a network that only has access to
| outside artifacts that are necessary, push access to an
| image registry, and that's it.
|
| Deployment status notifications can go from Flux to CI, so
| you can read about your deployments via commit status
| messages, but the communication is one-way. Changes are
| always pulled into the cluster and never pushed. You can
| set up webhooks in both directions, of course, for faster
| round trips (it makes a big difference for devs when
| iterating to not have to do a context switch and manually
| trigger a deploy, or to wait seconds not minutes and get
| immediate feedback after git push) but many will also opt
| not to do this in order to further confine and constrain to
| limit attack surfaces, it works just as well so long as
| they can live with some 5-10 minute waits for polling to do
| the job.
|
| Edit: heaven forbid we have a conversation, Hacker News
| won't allow fast replies :D thanks for giving examples
| Pathogen-David wrote:
| > but not higher-trust GHA CD ones
|
| They improved this quite a bit last June.
| https://github.blog/changelog/2021-06-22-github-actions-envi...
|
| Basically you can define deployment environments which have
| their own secrets and configure it so only authorized users can
| approve workflows which access those environments.
| lmeyerov wrote:
| Ah this is a good step. I hadn't looked since the env var
| vulns earlier in the ~year. Add in GH runner domain/vpc
| safelisting and a lot of the basics get solved!
| staticassertion wrote:
| My experience with GHA is that it can be awesome for things like
| small projects that want to enforce linters, unit tests, etc.
|
| Once you get into more complex things - like building docker
| images, storing into an artifact repository, baking amis, running
| integration or end to end tests, etc, it can be a pain.
|
| It was a great place for us to start but we've since moved to
| BuildKite.
| athorax wrote:
| Agreed. It is great for things that you want to run directly
| _against_ your code base (like linters, tests, etc.)
| chad_oliver wrote:
| The biggest limitation I'm hitting with Github Actions right now
| is that there's no real support for queueing jobs up. Github
| actions do support a 'concurrency group' primitive that will
| prevent two actions in the same concurrency group from running at
| the same time, but this only allows you to have one item
| executing and one item queued up. If you try to queue up another
| one, the first queued item will be cancelled.
|
| In our case we've got some cypress tests that we want to run on a
| specific on-premise server every time we create a pull request.
| They take about 20 minutes to run, and we're creating a lot of
| pull requests, so you have to carefully check what github actions
| are executing before you create a pull request or push new
| changes to it. I'd love support for proper queues like what
| teamcity and other CI systems have.
| jhgb wrote:
| > The biggest limitation I'm hitting with Github Actions right
| now is that there's no real support for queueing jobs up.
|
| Maybe they need to reinvent JES and JCL. :)
| aidanr wrote:
| Now that made me smile! Add in a dose of CA-7 and job's a
| goodun.
| AltF4me wrote:
| Turnstyle?
|
| We've had similar issues with Terraform deployments.
|
| https://github.com/softprops/turnstyle
| AltF4me wrote:
| Turnstyle?
|
| We've had similar issues with Terraform deployments.
|
| https://github.com/softprops/turnstyle
| throwaway20371 wrote:
| > migrate every engineering team at Venmo to GitHub Actions
|
| Oh jesus christ. I feel for you dude.
|
| We evaluated GHA, and we still are trying to use it, but there is
| a barrage of problems and limitations, including cost, lack of
| functionality, and technical issues. It's really only suitable
| (at scale) for linting, or generating Changelogs, or something
| else trivial. I use it in my OSS projects to run tests, and it's
| okay for that (though impossible to just tail a build log when
| it's large)
|
| Drone.io is still an amazingly effective system that matches GHA
| (and has _crazy_ features like _build parameters_ ) but is more
| flexible. Of course you'll have to pay for commercial licenses,
| but if it's between paying for GHA or Drone, I highly recommend
| Drone instead. Drone is stupidly easy to maintain
| (infrastructure-wise).
| shadycuz wrote:
| If you use Jenkins and want to try actions then check out
| https://github.com/DontShaveTheYak/jenkins-std-lib
|
| It let's you run actions on top of Jenkins.
| AJRF wrote:
| Has anyone got experience or resources for running a big iOS
| application on GHA? Is it possible, or is it pretty much a toy
| right now? Say you have a big CI pipeline like;
|
| * Full clean build including dependencies (support Carthage,
| Cocoapods, SPM)
|
| * Running multiple test suites that takes maybe 5+ hours for full
| suite?
|
| * Running simulators for screenshot testing
| arminiusreturns wrote:
| Does anyone know if on GHE I can bypass actions and just use the
| builtin git recieve hooks? I really enjoy playing with post-
| commit scripts.
| simonw wrote:
| "You can't have actions that call other actions" - I think it's
| possible to use the repository_dispatch trigger described at
| https://docs.github.com/en/actions/reference/events-that-tri...
| for this - you'd need a separate GitHub personal access token,
| but using that it should be possible to trigger a workflow in any
| other repository you own from an API call in another action.
| 147 wrote:
| I meant that in the context of composite actions. You can't
| package up a composite action that can call other actions, at
| least not yet.
| arccy wrote:
| Pretty new: https://github.blog/changelog/2021-08-25-github-
| actions-redu...
| jordemort wrote:
| Ooh, I didn't realize this had shipped. Off to refactor all
| my boilerplate into its own repo...
| 147 wrote:
| Thanks! This is super exciting. Unfortunately, since we use
| GHES we'll be at least 6 months away from having this
| though :(.
| akerl_ wrote:
| The issue is that once you add a PAT, you've imbued the build
| with your permissions. You can bound which permissions by call
| type (so, for example, you can make a token that can't modify
| your profile info), but you can't bound it by repository. So if
| you need the build to trigger write actions on one repo, now it
| has them for all repos.
| simonw wrote:
| Agreed, this is an infuriating limitation of PATs - I would
| desperately like to be able to create a PAT with much more
| finely grained permissions - can only trigger this single
| REST API for this specific repo for example.
|
| There's a brilliant trick for doing that described here but
| it's a lot of work to setup:
| https://blog.thea.codes/building-a-stateless-api-proxy/
|
| You can also create a brand new GitHub user account, grant it
| access to just a single repository and then create a PAT for
| that user account - annoying but it does at least let you
| scope down the permissions a bit.
| thinkafterbef wrote:
| Disclaimer - incoming self promotion.
|
| Actually, most points in the article are the basis on why we
| created BuildJet.
|
| We initially tried to solve these annoyances by creating a CI
| with speed and the YAML config as a USP. We got 4x speed and a
| much better YAML config structure, but despite these improvements
| we noticed that it people had a mental barrier to migrate to a
| new unknown CI.
|
| But like OP we always enjoyed the experience of using GitHub
| Actions, so with this in mind. We decided to build BuildJet for
| GitHub Action[1] that uses the same infrastructure but plugs
| right into Github Action as a "self-hosted" runner, which is
| automatically set up for you with OAauth. This resulted on
| average a 2x speed improvement for half the cost(due to us being
| close to the metal). Easy to install and easy to revert.
|
| [1] https://buildjet.com/for-github-actions
| epage wrote:
| I really like Github Actions but it still only feels appropriate
| for small projects. Unless I'm missing something, I didn't see a
| good mechanism for monorepos. I'm thinking in terms of there
| being shared pieces that get built/tested and then products that
| sit on top. The complication comes in test avoidance. I don't
| remember why I didn't like doing this all in one workflow with
| jobs (though it was going to require an orchestrator job setting
| variables to choose which downstream jobs to run). For chaining
| workflows/pipelines, when I looked, you could only trigger other
| workflows for master, defeating the point.
|
| Among my small, open source work, probably my biggest complaint
| is actions running in forks. Wastes a lot of resources on their
| side and limits my concurrent runners for projects in my personal
| space. For companies, depending on the setup, this would eat
| their compute minutes.
|
| Also annoying that PR actions can't post to the PR. I can
| understand there are security limitations but it makes it so a
| lot of nice features don't exist for most people.
| thedougd wrote:
| Surprised there's no mention of the inability to view logs
| generated before the page was loaded until the job completes.
| This one drives me crazy when I have long running, silent
| activity.
|
| My wishlist item would be more variants of Windows server
| versions so that we could build Windows containers for more
| versions of Windows. I realize the fault lies with Windows
| containers pinning the container base version to the host
| version, but I'm still stuck with the burden.
|
| I think GitHub Actions got the model correct, using everything as
| events to trigger any number of workflows. This is far simpler to
| maintain than a single workflow with conditionals and wait states
| that you see with other systems.
| sofixa wrote:
| Wow, reading this and all the comments makes me realise GHA is
| really far behind Gitlab CI. They've at least been playing catch-
| up since the MS acquisition, having stagnated for years before
| that.
|
| I really wonder why would anyone self-host GitHub. Gitlab has a
| much more feature rich, mature and cheap ( there's a perfectly
| usable free version) offering. Yeah, someone might prefer
| Github's UX, but is it really worth it to pay for a worse
| product?
| easton wrote:
| GitHub's UX is worth it, IMO. GitLab tries to do too many
| things and it's really hard to get my bearings (it's like Azure
| DevOps' problem times 10). And if GitHub Actions doesn't work,
| it's really easy to integrate third party CI solutions.
| sofixa wrote:
| > GitHub's UX is worth it, IMO. GitLab tries to do too many
| things and it's really hard to get my bearings (it's like
| Azure DevOps' problem times 10).
|
| That's subjective. Gitlab does many more things, but nobody
| is forcing you to use them, and unused menu buttons don't
| matter much for the main workflows ( commits, MRs, repo
| visualisation, wikis, CI/CD).
| adwww wrote:
| Yeah I liked Gitlab when we used it for everything, but once
| we started to integrate with other services (Jira etc)
| instead of the built in functionality it felt like you were
| fighting how they had designed it.
| john_cogs wrote:
| GitLab team member here.
|
| We've put a lot of effort into our integration with Jira.
| The docs [1] detail how it works and we have a blog [2]
| with some screenshots so you can see what it looks like in
| action.
|
| What kind of troubles have you encountered?
|
| 1 - https://docs.gitlab.com/ee/integration/jira/
|
| 2 - https://about.gitlab.com/blog/2021/03/25/integrating-
| gitlab-...
| roblabla wrote:
| Holy shit wat? Gitlab CI is a clusterfuck that is impossible to
| use for all but the most simple use-cases. We have a monorepo
| at work, and GitLab CI never fucking works. Here are some
| examples:
|
| - Fallback cache keys don't work (because it doesn't compose
| with cache changing cache ids). So each time someone clears the
| runner caches, the fallback cache stops working.
|
| - `if`/`rules` and `needs` don't work together, and cause the
| build to fail with a *yaml build failure*. Wat? This is even a
| documented failure mode now. This is a huge issue for
| monorepos.
|
| - gitlab-runner is full of weird behaviors, like `gitlab-runner
| unregister <runner-name>` fails if that runner was already
| deleted (through the gitlab UI), but won't remove it from the
| local config. But `gitlab-runner` has a separate command,
| gitlab-runner verify --delete, which does just that...
|
| It feels like every time I use it, I run into a bug. I've
| heavily used github actions in many projects of similar sizes,
| and have yet to come across a single bug.
| shoo wrote:
| > I really wonder why would anyone self-host GitHub. Gitlab has
| a much more feature rich, mature and cheap offering.
|
| before i understood your comment i was going to reply
| "security, data security & control" -- as a reason orgs might
| choose to self-host vs outsource hosting SaaS github.
|
| But you're probably arguing for self-hosted gitlab over self-
| hosted github.
|
| Good question! i've worked somewhere with self-hosted github,
| but they were using it prior to github actions / workflows, so
| it was a combination of self-hosted github and N different CI
| and deployment automation tools ( bamboo! GoCD! udeploy!
| jenkins! google cloud build! poor bloody operator manually
| copying deployment payload onto jump box as the bureaucracy
| won't approve automation!)
| _jal wrote:
| That's usually the wrong question, as people only make
| greenfield decisions once.
|
| If, however, you've been running GHE for years and have it
| integrated with other tooling and workflows, and your
| developers understand it, the question is,
|
| "Why would anyone rip all that out, replace it, reintegrate it
| and retrain everyone over a handful of features, most of which
| you don't care about?"
| sofixa wrote:
| > "Why would anyone rip all that out, replace it, reintegrate
| it and retrain everyone over a handful of features, most of
| which you don't care about
|
| Functioning CI/CD for free is certainly a huge feature to
| care about. For a user moving between Github and Gitlab is at
| worst a slight annoyance, the UX and DX are pretty similar;
| most third-party tooling that isn't github.com ( SaaS) only
| supports both Github and Gitlab self-hosted (i'd even wager
| there are more tools that support self-hosted Gitlab than
| Github, the first is much more popular).
| stavros wrote:
| "Because there's one feature we _really_ care about ".
|
| That's why we migrated to GitLab CI. We made some pretty
| complicated pipelines, with tens of stages, many of them
| dynamic, and it worked with minimal hassle. It was a great
| success story internally.
| boleary-gl wrote:
| GitLab team-member here. Obviously coming in with a lot of bias,
| but I wanted to address how each point relates to GitLab CI/CD's
| view of the world. I'm also thinking about writing a longer post
| with more details as I have a lot of thoughts ((tm)) about this
| topic.
|
| 2.1 Caching isn't available: GitLab has this everywhere.
|
| 2.2 GitHub Enterprise Server is behind GitHub Enterprise Cloud:
| GitLab ships the same code to GitLab.com as it does to our self-
| managed customers. This was a tough decision but has a lot of
| benefits...the central being feature parity and scalability for
| self-managed folks
|
| 2.3 Using Public GitHub.com Actions: This is a symptom more than
| the problem itself - relying on third-party plugins for build
| jobs is scary, and leads to many of the same issues we've seen in
| the Jenkins ecosystem - easy to get started, hard to maintain.
|
| 2.4 Dockerhub pull rate limiting: for self-hosted runners, you
| can use a registry mirror or Dependency Proxy to reduce your
| number of pulls from Docker Hub. The key is the entire platform
| has to be there to enable the right workflows.
|
| 3.1 No dropdowns for manually triggered jobs: GitLab also doesn't
| have drop downs, but does have the ability to pre-fill these
| values.
|
| 3.2 Self-hosted runner default labels: I think this is also more
| of a symptom than a problem. 3.3 Being able to tag and use
| runners for specific tasks is key - so I understand the
| frustration and we've spent a lot of time on this.
|
| 3.4 You can't restart a single job of a workflow: You can do this
| with GitLab.
|
| 3.5 Slow log output: I haven't seen this be a problem, and is a
| benefit of our scalability features being built into the self-
| managed code.
|
| 3.6 You can't have actions that call other actions: There are
| lots of ways to relate pipelines (parent/child, triggers. etc.)
| in GitLab.
|
| 3.7 Metrics and observability: The GitLab runner has Prometheus
| build in, and the dashboards we use to manage GitLab.com are
| partially public: https://dashboards.gitlab.com
|
| 3.8 Workflow YAML Syntax can confusing: This can be really hard
| to get right. I learned to stop worrying and love the YAML long
| ago, and I know we've got through a lot of iterations to try and
| get this right.
|
| I'd love to know where folks think I got this assessment wrong.
| And is there value in writing more about it?
|
| (edited for line spacing)
| ChrisMarshallNY wrote:
| I think GH Actions is a pretty cool idea. I don't use them,
| myself, because, every time I count myself, I keep coming up "1."
|
| When I left my last job, and started working on my own, I set up
| things like CI/D, JIRA, Jenkins, etc. These were the bread and
| butter for development in my old shop.
|
| But they are "Concrete Galoshes"[0], and work very well for
| _teams_ , as opposed to ICs. As a single developer, working
| alone, the infrastructure overhead just slowed me down, and,
| ironically, interfered with Quality.
|
| When GH Actions were first announced (I can't remember, but they
| may have been beta, then), I set up several of them, on my busier
| projects. They worked great, until I started to introduce some
| pivots, and I realized that there was actually no advantage to
| them. I ran the tests manually, anyway, and the Actions just gave
| me one more thing to tweak. It was annoying, getting the failure
| messages, when I knew damn well, the project was fine. I'd just
| forgotten to tweak the Action. I introduce frequent changes, in
| my work, and that is great.
|
| [0] https://littlegreenviper.com/miscellany/concrete-galoshes/
| jayflux wrote:
| I don't think you need to be in a team to get good mileage out
| of Actions.
|
| For example. Some open source code I need to publish to
| package-indexes or marketplaces, and will have an action to do
| that whenever I publish a release, I've found it to be very
| useful and saves a lot of time.
| ChrisMarshallNY wrote:
| I agree. It's just a difference in style. I find mine is
| quite productive, for the way that I work.
| shadycuz wrote:
| You can run GitHub actions locally but I forget the name of the
| tool.
|
| You can also run actions on top of Jenkins.
| https://github.com/DontShaveTheYak/jenkins-std-lib
| Pathogen-David wrote:
| I've never used it myself, but you're probably thinking of
| this: https://github.com/nektos/act
| cebert wrote:
| GitHub Actions are a fantastic experience for serverless
| applications. I am working on a serverless project where we use
| GitHub Actions exclusively for CI/CD as well as running automated
| tests. We rely heavily on Lambda, S3, and DynamoDB. Our client
| app is static JS files we serve over Cloud Font. GitHub Actions
| make our piplines accessible to any developer on the team. Since
| we only pay for what we use with our serverless infrastructure,
| we can even deploy each pull request to the cloud rather
| inexpensively and leverage GitHub's environments to help manage
| the cleanup for us. This allows our team members to review and
| test changes in their browser before we pull them into our
| development branch. We additionally can run Playwright E2E tests
| to verify that none of our critical user workflow scenarios have
| broken resulting from the PR changes. I love this development
| experience and would have a hard time going back to anything
| else.
| shoo wrote:
| thank you for sharing -- as i was reading through i tried to
| understand what about the workflow was specific to github
| actions vs other CI automation.
|
| > GitHub Actions make our pipelines accessible to any developer
| on the team
|
| Do you reckon this accessibility is a combination of (i)
| storing the pipeline definitions in the application's source
| repo, where application developers can find them easily, not
| hidden/scattered elsewhere in other repos or behind management
| UIs, and (ii) a relatively simple and documented pipeline
| syntax?
|
| The first example I can think of a tool that supported this
| workflow was Travis CI ~ 2011 - 2012. Appveyor offered similar
| capabilities quite early as well. Same workflow can be done
| with Gitlab, Google cloud build.
|
| > we can even deploy each pull request to the cloud rather
| inexpensively and leverage GitHub's environments to help manage
| the cleanup for us. This allows our team members to review and
| test changes in their browser before we pull them into our
| development branch
|
| Yeah, this kind of workflow is great. Another way this kind of
| workflow can be done is to create simple command line tools
| that developers can use to create and destroy temporary test
| environments running their speculative changes. In some cases,
| for rapid experimentation, it can be great to be able to spin
| up N temporary environments in parallel with different changes
| without tying it to pull requests. But I can see that tying the
| temporary environment lifecycle to the lifecycle of a PR might
| make it easier to share demos of proposed changes with
| reviewers.
|
| Out of curiosity, how reliable do you find the environment
| cleanup is? I remember building a similar create-temp-
| environment / destroy-temp-environment workflow for ephemeral
| databases running in AWS RDS driven by jenkins pipelines. It
| took a few months of tweaking to figure out how to ensure the
| RDS databases got torn down correctly and not "leaked" even if
| the jenkins master or workers failed midway through pipeline
| execution. From memory we had a bunch of exception handling in
| a jenkins groovy scripted pipeline that would run on the master
| jenkins to try to do cleanup, and even that wouldn't work all
| of the time, so we had a second cleanup job on a cron-schedule
| to detect and kill leaked resources.
| cebert wrote:
| > Do you reckon this accessibility is a combination of
|
| Yes, exactly. All of our build pipelines for a repository are
| included in the .github folder in the root of the repo. It
| makes it easier for team members to feel comfortable making
| changes and submitting a PR for them. You can setup an ACT
| container so you can test GitHub Action changes locally
| before pushing them too (see https://github.com/nektos/act )
|
| > Out of curiosity, how reliable do you find the environment
| cleanup is?
|
| So far, environment cleanup has been reliable, but I have
| noticed where it failed to cleanup some provisioned resources
| once in a blue moon. I blame this more on our code than
| GitHub Actions. I periodically review our sandbox
| environments to ensure we didn't miss deleting anything.
| BugsJustFindMe wrote:
| Is it possible yet to (without hoops) have an action workflow
| that runs on pull requests to <default_branch> without having to
| name that branch explicitly so that you can have a write-once-
| run-everywhere-forever process across repos that might change
| their default branch name? Last I saw there was a template
| parameter, but those get populated at creation not at runtime.
___________________________________________________________________
(page generated 2021-09-08 23:00 UTC)