[HN Gopher] The worst thing about Jenkins is that it works (2019)
___________________________________________________________________
The worst thing about Jenkins is that it works (2019)
Author : prabhatsharma
Score : 162 points
Date : 2023-12-03 14:40 UTC (8 hours ago)
(HTM) web link (twitchard.github.io)
(TXT) w3m dump (twitchard.github.io)
| gjvc wrote:
| Avoid Jenkins, and if you can, try
| https://www.jetbrains.com/teamcity
|
| "TeamCity Professional is free - even for commercial use - and
| has no limitations on features, number of users, or build time.
| It allows you to configure up to 100 builds and run up to 3
| builds in parallel, which is more than enough for most projects"
| xkekjrktllss wrote:
| free until it's not, of course
| aloisdg wrote:
| or enshittificated
| DarkNova6 wrote:
| Jetbrains is among the very few companies I'll trust for
| the time being
| ldjkfkdsjnv wrote:
| Jetbrains products are outstanding
| rqtwteye wrote:
| Same here. Although I am super nervous that they will
| either cash out by going public or selling to some
| investor. That will be the end of it.
| juunpp wrote:
| They aren't a silly con valley startup and they've been
| around for 20+ years.
|
| https://www.jetbrains.com/company/
| chrisandchris wrote:
| Maybe they will at one point, but JetBrains so far did
| never take VC capital (and if I remember correctly said
| "no" more than once) [1]. That doesn't tell about the
| future, but they probably had many options to cash out
| and never did.
|
| [1]
| https://www.bloomberg.com/news/articles/2020-12-18/czech-
| sta...
| devmor wrote:
| Always praying that they never will. I've been using
| their products since my first "real" developer job over
| 12 years ago and I never want to switch.
| juunpp wrote:
| "For the time being" is an understatement. The company
| has been around for 20+ years, and while I haven't used
| TeamCity, if it is anything like their other products,
| you can be certain it is built with a high level of
| attention to detail and that that won't change anytime
| soon. It is one of the few companies left I license stuff
| from.
| xkekjrktllss wrote:
| It is a capitalist enterprise, which means control by its
| very owners is contingent on making enough profit to pay
| the bills, in the very best of cases. This is not my
| opinion.
| irrational wrote:
| Yeah, but this is JetBrains. They are one of the few
| companies whose products are so much better than anyone
| else's that I'd pay for them, even as an independent
| developer. They've been around for decades and are still
| more than popular enough that making money shouldn't be
| their issue. My company of 80,000+ employees has a
| JetBrains license for every developer.
| xkekjrktllss wrote:
| ...for now.
| runiq wrote:
| > TeamCity Professional is free - even for commercial use - and
| has no limitations on features, number of users, or build time.
|
| Yeah, no. There's obviously gonna be a catch in the future.
| chrisandchris wrote:
| Maybe, yes. But it's exactly the same license / pricing since
| i first used it and that's somewhat around 5-7 years. Quite
| some time so far.
|
| I would recommend TeamCity to everyone, it's awesome.
| capableweb wrote:
| Just as a note: TeamCity originally launched around 2005
| sometime, so it's coming up to be 20 years old soon. I think
| TeamCity Professional, when self-hosting, has been free since
| always.
| AaronM wrote:
| The catch is your limited to a small number of parallel
| builds (3). Our CI pipeline would take hours to run with that
| limitation
| solardev wrote:
| To be fair, I've never seen a company turn around a licensing
| fiasco as well as Jetbrains did.
|
| Initial announcement, 2015:
| https://blog.jetbrains.com/blog/2015/09/03/introducing-
| jetbr...
|
| Community anger a couple days later:
| https://bytecrafter.blogspot.com/2015/09/how-jetbrains-
| lost-...
|
| Significant changes after hearing the feedback, a couple
| weeks later:
| https://blog.jetbrains.com/blog/2015/09/18/final-update-
| on-t...
|
| I'd trust Jetbrains -- as a company with a few decades of
| experience serving devs and only devs -- much more than your
| typical get-rich-quick cloud SaaS startup that's just looking
| for a quick exit.
|
| Yes, they need income too, but they're one of those few tech
| companies not aiming for hypergrowth, just a sustained user
| base.
|
| Even though VScode gets all the love these days, and
| Atlassian and Github and CircleCI etc. get all the enterprise
| use, they've made some really cool products of their own too
| (Space, TeamCity). And their IDEs continue to be excellent.
|
| It's not like TeamCity even has a secret catch... their cloud
| version DOES have a monthly cost already. It's only the self-
| hosted version that's free.
| gjvc wrote:
| _To be fair, I 've never seen a company turn around a
| licensing fiasco as well as Jetbrains did._
|
| Absolutely, though it was quite dramatic.
| marginalia_nu wrote:
| Yeah, I use TeamCity for both my test and production setups.
| I've only had positive experiences with it, very nice seamless
| integration with Jetbrains' IDEs as well.
| callmeal wrote:
| >configure up to 100 builds and run up to 3 builds in parallel
|
| Funnily enough, Jenkins does not have that limitation.
| gjvc wrote:
| It has plenty of others which more than make up for it
| though.
| gjvc wrote:
| The responses to this post are hilarious.
| lmm wrote:
| TeamCity is even worse than Jenkins. Jenkins randomly breaks
| and you can eventually fix it, or kludge some kind of
| workaround. TeamCity randomly breaks and you call support and
| they can't fix it.
| prabhatsharma wrote:
| Github actions works great Even AWS codebuild is good.
| sethammons wrote:
| While we are plugging ci tools, I enjoyed buildkite at my last
| gig
| dafelst wrote:
| Same, we are using it now and it is a much nicer experience
| than Jenkins
| anon23432343 wrote:
| Jenkins works...
|
| No it dosen't or maybe it does for java which I don't care about.
|
| The ui looks like from the 90s.
|
| Finding anything in bigger projects/companies is impossible... I
| have browser bookmarks to find anything.
|
| The log view is annoying at best.
|
| I have nothing good to say about jenkins, it feels like it is
| stuck in the 2000s and refuses to change.
| colordrops wrote:
| "refuses to change". Is there an active team working on it or
| is this anthropomorphism of a software project?
| xp84 wrote:
| The 90s were the last actual advance in UI design anyway.
| Design has drifted in a terrible direction, in the past decade
| or two especially. Some of this is justified (by some) by
| optimizing for mobile, but that's precisely the kind of thing I
| wouldn't want creeping into my CI tooling anyway. I don't do
| that on a phone.
| dangus wrote:
| This idea that design peaked in the 90s does a big hand wave
| over a whole bunch of horrible examples of the user interface
| from that era.
|
| Is the Space Jam website peak UI design? What about
| Microsoft's "hell of tabs" settings dialog boxes?
|
| Or the original Amazon home page:
| https://www.versionmuseum.com/images/websites/amazon-
| website...
|
| What link do I click to find my order status? Where do I go
| to search for a book by keyword? What will happen if I click
| the link for "first time customers click here?"
|
| Same deal with Yahoo!, what to the "new" and "cool" and "more
| yahoos" buttons do? http://2.bp.blogspot.com/-JAp1M-Q_0_Y/TkA
| wsyueWxI/AAAAAAAAAM...
|
| If I want to check a build status on my phone what's the
| excuse for the web page to not display nicely on that device?
| You really think there's no good reason for me to want to do
| that? Doesn't modern CSS allow you to completely customize
| the viewing experience for both platforms in the same
| codebase?
| wharvle wrote:
| Desktop program UI design mostly peaked in the 90s (search-
| to-launch becoming ubiquitous is about the only really good
| thing to happen since then). Desktop web design peaked
| somewhat later, in the two- and three-column era. '00s. It
| was nice when most sites looked and worked about the same
| way, and that way also happened to have great info density
| for a desktop screen.
|
| Mobile web design, I'm not sure it's had a peak yet. Phone
| os UI peak was iOS6.
| indymike wrote:
| Confusing web design and desktop UI design in the 90s does
| little to dispute the assertion that desktop UI design
| peaked in the 90s. In the late 90s, the browser was
| completely new - so there was a lot of skeuomorphism,
| borrowing from other media to try to make the web work
| well. Space Jam's format was familiar, though to users of
| multimedia CD-ROM and other interactive hypermedia of the
| era.
|
| The desktop application GUI - which is really what people
| are claiming hit the peak in the 90s, really did. Menus,
| windows, tabs, dialogs, scroll bars etc... all were fairly
| well settled, and users understood them. A user who knew
| Word for Windows could do pretty well using WordPerfect for
| Windows. Most day-to-day applications were pretty easy to
| figure out because discoverability was very well done, and
| wizards and how-to dialogs helped users through the rough
| bits.
|
| There was consistency between applications - save in areas
| where the OS didn't really provide GUI guidance - so
| design, CAD, and other creative apps (hi, Adobe) often had
| divergent ways of doing things and came with a steep
| learning curve. The web took off because it actually worked
| a lot like the multimedia CD-ROMs that preceded it - and
| websites were a lot easier for developers to build.
| IshKebab wrote:
| For desktop apps maybe. Not for websites though. Jenkins has
| an absolutely objectively terrible UI. Super ugly,
| inconsistent and incomprehensible.
|
| I've used Buildbot, Travis, Gitlab, GitHub and Teamcity. None
| of them are anywhere near as bad as Jenkins. Even Buildbot
| has a much much saner UI and that's a weird Python CI system
| that practically nobody uses.
|
| Jenkins is just an objectively terrible choice.
| muglug wrote:
| Needs a (2019)
|
| Richard was my manager in early 2019, and was awesome.
| bityard wrote:
| Submitter editorialized the headline as well, Actual title is
| "Life is Too Short for Jenkins"
| twitchard wrote:
| Submitter's is a way better title imo
| muglug wrote:
| You should change the title to match tbh
| kureikain wrote:
| Jenkins has improve a bot. before github action has self host
| runner, Jenkins has worker node years ago.
|
| using jenkins declarative pipeline, almost everything can explain
| in code.
|
| updating jenkins is a walk in the past.
|
| the most tedious part is dealing with no longer maitenance or
| support plugin.
|
| make sure to never expose jenkins out. use a relayer to bring
| webhook traffic in
| lacrimacida wrote:
| I had a gig (circa 2014) where Jenkins was used in atrocious ways
| where it was basically running the show. Yet there were no
| problems with Jenkins itself, it was running smoothly
| fullspectrumdev wrote:
| I've found that people will use CI tools like a hammer once
| they get them working - everything's a nail, even screws or
| bolts.
|
| You often end up with these glorious Rube Goldberg machines
| where Jenkins does everything and becomes a single point of
| failure - if the instance falls over, everything's fucked.
| FireBeyond wrote:
| Yes, one multinational company who was a customer at my last
| employer lamented their "dystopian Jenkins hellscape", "glued
| together with 300+ Python scripts".
| lmm wrote:
| I once worked somewhere that had two very different Jenkins
| instances - one that was the typical dev CI instance, and one
| that was a job scheduler for all the prod batch jobs. It
| actually worked very well.
| zdw wrote:
| All of the test runners have their issues. I've had as much
| problems with Jenkins as I've had with Github Runners and similar
| solutions.
|
| The best way out of this is to subdivide responsibilities:
|
| - Put all build/publish/test logic in Makefiles or scripts in the
| repo. This means that devs can run it locally as well. The only
| interaction between a test runner and the codebase should be
| running `make <target>`.
|
| - Put all permissions and code checkout and artifact publishing
| credentials in the test runner, but no logic. At most you would
| put processing of test output like making junit/coverage/tap more
| web-readable.
|
| That's it. Split the efforts cleanly, and things fall into place.
| Also, you can switch runners easily - less needs to be
| reimplemented in whatever runner config DSL is picked.
| nerdponx wrote:
| This is basically what we do, using TeamCity. DevOps sets up
| all the permissions and VCS integration, and helps figure out
| all the AWS IAM stuff. Then we get a nice clean little sandbox
| to run our build and deploy scripts in, which we commit to
| version control, including CloudFormation templates. It works
| really well: it's a comfortable separation of responsibilities
| that allows our team to move quickly with relatively little
| risk, and it involves almost no hands-on attention from DevOps
| once it's up and running, so they can direct their efforts
| elsewhere.
| sevagh wrote:
| >- Put all build/publish/test logic in Makefiles or scripts in
| the repo. This means that devs can run it locally as well. The
| only interaction between a test runner and the codebase should
| be running `make <target>`.
|
| This always breaks because then each platform has a specific
| way of defining env vars or secrets for those Makefiles and
| bash scripts. End result is devs still can't really run CI
| "proper" the way it's configured in a runner.
| zdw wrote:
| This is fundamentally an interface problem. As you called out
| Env vars are a pretty common and well supported interface,
| across multiple platforms.
|
| Do you have a better option to suggest?
| sevagh wrote:
| No, and I don't think any CI platform deals with it in a
| great way. I mean that once you account for the fact that
| your dev team may not have access to all of the secrets,
| the single bash script/Makefile stops being able to shield
| you from the specific CI platform, and now you have to
| start using Jenkins secret storage, or GitHub Actions
| secrets, or whatever.
| lmm wrote:
| Set up whatever maintains your secrets (vault?) so that it
| works the same way in dev, CI, and real. Have whatever
| manages your dev versions of services you depend on
| (vagrant? kubernetes-docker-whateveritisthesedays?)
| integrate with that so that you find your service endpoints
| and credentials the same way in every environment.
| maratc wrote:
| You're correct. I sometimes forward my devs the script they
| need to run locally in order to replicate what Jenkins does.
| It has the looks of make <target> THIS=THAT
| AND_THIS=THAT AND_THAT_THING=SOME_OTHER_THING ETC=ETC
|
| Sometimes it won't fit into a Slack message, so I attach it
| as a text snippet instead.
| MereInterest wrote:
| Maybe this is just me, but needing to define environment
| variables or secrets for testing seems like a massive problem
| in the test design. Does that mean the tests are dependent on
| the use of an external service, and that service can't be run
| locally?
|
| The only part of a CI pipeline I can imagine requiring
| secrets would be a release/publish step. However, those would
| only occur after the tests run successfully, so nothing up
| until that point would require secrets.
| john-tells-all wrote:
| > - Put all build/publish/test logic in Makefiles or scripts in
| the repo. This means that devs can run it locally as well.
|
| This is critical!
|
| In many teams Devs are "lazy", if any errors or weirdness
| happens in CI they just wave their hands and say "not a
| feature, not my problem". Even though they know exactly what's
| in the test code, the deploy code, and can see the issue.
|
| Letting Devs see and run the CI scripts locally gives them
| _ownership_. They don't have to fix the issues, but they have
| control and understanding which radically reduces the arguments
| and pipeline drama :)
| roland35 wrote:
| It also takes a lot less time to fix the tests locally too -
| rather than tweak it... push the change... oops it failed
| again... repeat...
| dharmab wrote:
| This still happens, but the surface area is greatly reduced
| to figuring out that the directory, variables, credentials,
| dependencies between stages, etc. are set up correctly
| Jnr wrote:
| I totally agree. Make sure developers can run the same tasks
| locally as well.
| eysi wrote:
| This is such a great point!
|
| Your pipelines should mostly be CI provider agnostic and
| runnable from anywhere, including your laptop during
| development.
|
| If you ever need to change a CI provider, you just move your
| pipelines.
|
| I'm definitely biased, I'm working at Garden[0] which allows
| you to do just that but it's mostly applicable for teams using
| Kubernetes. I paused at your comment because Garden has
| sometimes been called "a makefile for the cloud".
|
| Whatever tooling you use, having portable pipelines that you
| can iterate on and debug from your laptop is the only sane
| approach imo.
|
| [0] https://docs.garden.io/overview/use-cases#faster-simpler-
| and...
| craig wrote:
| > Warning - I'm going to rant for many, many paragraphs. My
| advice is to skim
|
| I had a good chuckle at this.
| opentokix wrote:
| Really like the one line dissmissal of gitlab ci - haha, what a
| n00b.
| ath3nd wrote:
| > "everybody should be writing and maintaining their own CI jobs"
|
| And, of course, everybody should be using whatever language and
| version they want to, and run their software on whatever cloud
| they want. The notion of this makes me super uneasy, but at the
| same time it's a good explanation why many big orgs have 100s of
| engineers when they can probably easily do with 10s of them.
| dig1 wrote:
| Jenkins is not perfect by any means, but I can say it is much
| better than existing solutions (taking pros and cons after trying
| and using many of them). Sure, it doesn't have fancy UI or flying
| buttons, nor is it rust/PWA/react/whatever-rage-is-these-days,
| but it does its job - it works, is free, and has been very well-
| maintained for ages. Download a single file and run it. Want to
| extend it? There is a plugin for everything under the sun (but be
| careful, you can easily bloat it).
|
| IMHO, I don't want bells and whistles from the CI tool. CI tools
| are dry; I see them as a car drive shaft - it must work properly,
| and I don't expect it to be pretty. If you need to showcase it,
| you are probably doing something wrong.
|
| And one more thing: I expect Jenkins to be around and free in the
| next 10 years. I doubt that will be the case for "insert-your-
| favorite-ci-from-some-company".
| 2OEH8eoCRo0 wrote:
| It's _infrastructure_. It should be boring, unchanging, and
| without surprises. It should be so reliable that we take it for
| granted.
| Nextgrid wrote:
| Agreed - I'd much rather spend more time working around
| Jenkins' peculiarities _once_ than _constantly_ deal with
| change for the sake of change because some PM wants more
| "engagement" to justify their job and promotion.
|
| Tools need to be boring and get out of my way - something
| "modern" tooling forgets since everyone's paycheck depends on
| forgetting that.
| troupo wrote:
| Jenkins BlueOcean is one of the best "fancy UIs"
| i386 wrote:
| Thanks! My team and I made that and it was a lot of fun :)
| vijaybritto wrote:
| Using that everyday all day. Thank you!!
| allannienhuis wrote:
| Thank you for building it - it's my go-to jenkins UI. Makes
| jenkins tolerable :)
| troupo wrote:
| No, thank you! It's fresh, it's infomation dense, it's
| fast, it works!
| maratc wrote:
| Thanks for that, we use it all day every day!
|
| It's too bad it's been declared "end of life", without any
| replacement in sight, and nobody these days is fixing some
| bugs that are really bothering us sometimes...
| i386 wrote:
| I really wished we finished it and made it the default
| UI. Jenkins is very extensible, so there are a LOT of
| extension points. We certainly didn't have the right
| approach to the API and devex for creating react plugin
| extension points either.
| atticora wrote:
| > I see them as a car drive shaft ...
|
| That's a good metaphor, and it's significant that there are
| large innovations happening in car drive shafts as with CI
| tools:
|
| https://electrek.co/2023/11/29/hyundai-kia-introduce-new-uni...
| nonethewiser wrote:
| Basically a dumb pipe in the "dumb pipe, smart endpoint"
| dynamic.
|
| Drive shaft is a good analogy though because its not simply
| moving data (unless we're getting really abstract).
| hospadar wrote:
| I hear you, I want my CI to be boring and just work, and using
| something old and and a little cooky is fine but...
|
| omg switching away from jenkins (in our case to gitlab CI) was
| a revalation. SO much easier to use. There were a ton of things
| we were avoiding doing in CI (or at all) that we started doing
| (easily) once we switched.
|
| > it must work properly, and I don't expect it to be pretty
|
| Too often it really _didn't_ work properly, or the gap between
| where we were and "working properly" was a mysterious foggy
| ocean with no clear path.
|
| To be fair, we drove it pretty hard, we had some jobs that ran
| thousands of tests on big clusters of nodes to validate and
| deploy huge ETL pipelines, but man it was nice to have that
| work smoothly with a nice UI that made sense with super-well-
| documented pipline commands. It did _basically_ work with
| jenkins, but the experience of troubleshooting problems and
| adding new features was a constant pain point that really
| dragged out a lot of work.
| dharmab wrote:
| The car drive shaft analogy is also apt because a drive shaft
| is a precision part that _someone_ has to think about very
| passionately, in order for others to not have to think about at
| all.
|
| ~13m30s: https://youtu.be/xyeeksFRnn4?si=924f0OZYDZk7otFL
| dangus wrote:
| I don't think the issue with Jenkins is the lack of bells and
| whistles. The UI it has is unintuitive - I always feel like I
| can't find what I'm looking for anytime I am in there.
|
| It has two different UIs (last I checked) which is kind of
| insane, blue ocean still sucked the last time I used it and the
| product can't 100% move away from the old UI (again, last I
| checked...)
|
| On top of all that it's not really fun to administer (first-
| hand experience) where almost every environment will result in
| plugin hell. I wouldn't describe the setup process "download a
| single file and run it" as I look at the documentation, either.
|
| Jenkins can't give you the "one less tool" advantage that you
| get with GitHub or Gitlab, either.
|
| Don't forget that Jenkins' competition includes free options
| with mostly high longevity.
| drewcoo wrote:
| The thing that made me want to use hosted CI was realizing how
| much time I ended up spending on Jenkins infra and debugging.
| Upgrades to master or to plugins and then testing them - bleh!
|
| GitHub's being slow? Well that's a GitHub problem. Look, it's
| been ack'd and they're working on it! Job done.
| uf00lme wrote:
| Jenkins will be around for as long as developers still code.
| I've always found that any significantly large software
| engineering office will have Jenkins running somewhere, happily
| on a local dev laptop/desktop or in a VM somewhere. I think it
| is being rarely used en-mass still which is a shame since it is
| free, open-source and easy to self-host.
|
| -
|
| I'm not sure why people are logging into Jenkins and
| complaining about the UI all the time; that just sounds like
| your pipeline needs better notifications and people are relying
| on Jenkins to look at build logs? If you want something
| prettier to look at just build out a dashboard on Grafana or
| similar, and centralise your build logs with artifacts and link
| to them with build failure notifications.
| btbuilder wrote:
| The kubernetes plug-in for Jenkins works well for us, and as the
| checkout/clone happens in the pod you don't have the workspace
| bind mount problem. We are using docker-in-docker at the moment
| but looking at buildah. We do parallel windows container builds
| as well.
|
| Jenkins is a bit like Perl IMO: there are ways to use it that
| avoid the majority of its sharp edges (eg putting logic in bash
| and just call bash, avoid groovy) if you know the perils that
| lurk.
| sigwinch28 wrote:
| > We are uding docker-in-docker at the moment
|
| You can also run a "less privileged" container with all the
| features of Docker by using rootless buildkit in Kubernetes.
| Here are some examples:
|
| https://github.com/moby/buildkit/tree/master/examples/kubern...
|
| https://github.com/moby/buildkit/blob/master/examples/kubern...
|
| It's also possible to run dedicated buildkitd workers and
| connect to them remotely.
| AaronM wrote:
| The biggest issue I have with the plugin is the container takes
| a significant amount of resources to run.
|
| I don't know how that compares to other solutions like github
| though. Being able to run a really lightweight container in
| argowf has its advantages
| pyjarrett wrote:
| I don't mind Jenkins pipeline limitations because my philosophy
| is that CI should be as simple and contain as little build logic
| as possible.
|
| Every job should be a few simple commands or a python script or
| something I can run locally. This simplifies reproducing and
| debugging issues caught by CI -- there's nothing worse than
| wasting time trying to reproduce what happened.
| demizer wrote:
| Jenkins is rock solid. Been using it for three years to deliver
| highly critical software. The development experience is the worst
| of all ci tools since no one wants to setup jenkinspipelineunit
| to test the pipelines. Dagger might be the salvation here, but
| even that tool didn't support Jenkins out of the gate. And for
| those that are reading this, the statement about pipeline groovy
| being a trap is exactly right. Avoid writing groovy at all costs.
| No one on the team wants to get good groovy and if you do, you're
| weird. The way we do it is to just use the Jenkins file to call
| the scripts (or tools) with bare minimal script code.
|
| Edit, forgot to add that I recently interviewed a bunch of
| candidates for a high paying job at a ai hardware company and
| every one of those candidates stated they worked on or near a
| project that involved moving from Jenkins to Gitlab CI.
| rightbyte wrote:
| Oh ye keep it simple. I worked as a Jenkins admin for half a
| year and one day I woke up from a nightmare and realized that
| I've made some sort of parallelization batch job processor
| communicating via outputting bat-files, in Groovy, to cut down
| build times.
| bigstrat2003 wrote:
| > And for those that are reading this, the statement about
| pipeline groovy being a trap is exactly right. Avoid writing
| groovy at all costs.
|
| I disagree with this pretty strongly. I've worked with Jenkins
| where it was all freestyle jobs and it was a nightmare to
| maintain. Pipelines written in groovy are essential to doing
| Jenkins well, imo. It's also not exactly hard to write groovy.
| It's basically Java, not some weird esoteric language nobody
| knows.
| maratc wrote:
| I'd suggest you wait until you spend a whole day on script
| approval and @NonCPS idiosyncrasies before you define Jenkins
| pipeline Groovy DSL as "basically Java". This doesn't mean
| you shouldn't be writing it, you should -- but for the
| reasons completely different from its supposed "similarity to
| Java".
| maratc wrote:
| > Avoid writing groovy at all costs. No one on the team wants
| to get good groovy and if you do, you're weird.
|
| I guess I'm weird then, having produced about 100K LoC in
| Groovy over the last couple of years.
|
| We'd move to Gitlab CI too -- if it was anywhere near
| resembling a serious tool and not a sandbox toy collection.
| loloquwowndueo wrote:
| Concourse CI is great, it felt like it would be easier to write
| Concourse resources for the custom things we needed rather than
| fighting Jenkins anymore.
| solardev wrote:
| Does Jenkins let you test workflows locally?
|
| That's the thing I hate that about CircleCI, Github Actions,
| etc... having to "commit, upload, wait" to test every minor
| change =/
| Jnr wrote:
| I put all the important steps in scripts (makefiles or as
| commands in package.json for JS projects) which can be run
| without Jenkins. Pipeline in the Jenkinsfile only runs those
| steps separately in the correct container build stage to
| collect separate step results.
|
| If done correctly, the Jenkinsfile should be easy to read by a
| person who has never written a Jenkins pipeline before.
|
| With this approach it is hard to mess things up by creating a
| monster pipeline which no one can understand, and if necessary,
| you can quickly change to different CI runner at a later point
| in time.
| happymellon wrote:
| Interesting, the side snipe about Docker
|
| > One of the mostly-false promises of Docker, as it was sold to
| me by the true believers who introduced me to it, was that, if
| you do it right, you can run the same docker image, and therefore
| have basically the same environment in production, in CI, and on
| your local development machine.
|
| I literally have never worked anywhere that _hasn 't_ used the
| same container in dev/test/prod.
|
| People who have had to deal with processes that rebuilt images
| for each environment, can you tell me what was different in each
| image?
| dijksterhuis wrote:
| > People who have had to deal with processes that rebuilt
| images for each environment, can you tell me what was different
| in each image?
|
| Tensorflow used to generate multiple images for different use
| cases: cpu only, GPU, cpu with Jupyterhub, GPU with Jupyterhub.
|
| So each release would generate minimum 4x container builds.
|
| CPU only images were useful for local testing of ml code at CLI
| (does it run) when a GPU might no be available.
|
| Jupyterhub variants for simple GUI for development work.
|
| GPU for obvious reasons (run the damn thing).
|
| I can't remember if they used to do builds for their release
| against multiple CUDA / CuDNN versions... that might have been
| PyTorch.
| AdrienPoupa wrote:
| PHP dev here, we have extensions for development that make no
| sense in production, xdebug for example. You need it for
| breakpoints and debugging in general but it should not be
| installed in production. So we extend our production image and
| install it on top of it. Similarly, we include Composer
| (package manager) in the dev image only as we only need the
| installed dependencies in production but not the package
| manager. Our dev image is a flavor of the production one,
| really.
| comprev wrote:
| Would multi-stage Docker builds not help here? Composer
| executes in one step and the result artefacts are copied into
| a "clean" PHP image without Composer installed.
| twunde wrote:
| Based on the description they are doing a multi stage
| build, but using the prod container as a base and then
| building the dev container atop that. But yes you could
| easily go the other way with dev building an artifact and
| adding it to a secure locked down container. This is less
| typical with dynamic languages that don't typically create
| a single binary, but still comes up. The downsides are that
| your prod container is now significantly different and for
| dynamic languages the fast feedback loop now has a slowish
| build step
| AdrienPoupa wrote:
| This what we are doing for the prod container that does not
| have Composer installed yes.
|
| But in development it's much easier to have it in the
| image. Additionally we do not bundle the code in the dev
| image but bind mount it in Docker Compose, which is much
| faster than rebuilding the image to test changes in
| development; PHP not being compiled allows us to do that to
| reduce the feedback loop duration.
| nullify88 wrote:
| In my experience, Xdebug absolutely made sense in production
| just not enabled by default for all requests. A lot of its
| functionality can be enabled via a cookie for a single
| session, and its made debugging production much easier as
| well as identifying bottlenecks in code or production
| infrastructure.
| AdrienPoupa wrote:
| That's certainly possible but we have other tools for that
| such as NewRelic that served us well.
| DylanSp wrote:
| At a recent job, we had slightly different containers for local
| dev; our backend containers (for a Go app) had Air [1]
| installed for live reloading, plus Delve [2] running inside the
| container for VS Code's debugger to connect to. We also had a
| frontend container for local dev, which didn't get deployed as
| a container, just as static files.
|
| [1] https://github.com/cosmtrek/air
|
| [2] https://github.com/go-delve/delve/
| twunde wrote:
| I'm general, it's an anti pattern since it's makes everything
| much more complicated. The typical reason to have it is so that
| there are dev tools in the dev container (autoformatter, linter
| etc) or support for hot reloading and then the prod container
| is locked down. The other pattern you'll sometimes see is that
| the prod container will include an agent or a certificate
| bundle, although it's more common to use sidecars for this.
|
| It becomes problematic because it then becomes easy for
| engineers to have a completely different container for dev then
| is used for prod. I recently found an issue where a dev
| container was using a completely different base and had a
| different version of node installed compared to the prod
| container
| hedora wrote:
| Multi-stage docker files mostly solve this problem:
|
| https://docs.docker.com/build/building/multi-stage/
| anschwa wrote:
| I think it's common (and important!) to build your "production"
| image once and deploy it into a dev/staging environment first.
| This means there are no re-builds between deploys to each
| environment.
|
| However, I think it's less common to run automated testing
| within the same Docker image you build and deploy to these
| environments.
|
| This is challenging because you probably don't want to include
| any build/test related packages in your production image (or
| layer) but you still want some level of confidence that your CI
| and Prod environments are the same.
|
| I have often see builds pass automated testing but fail after
| deployment because our production Dockerfile/layer was missing
| packages or shared libraries that were present in the CI
| environment.
| mkesper wrote:
| But then there have to be rebuilds, no?
| mmcnl wrote:
| Why don't you want to include build/test related packages in
| your production image? Exactly because you mention you can't
| test the Docker image without these tools in the image, you
| cannot guarantee the production image works as you would
| expect it to work. Precisely because of that I think any
| packages that are required to run the tests should be part of
| the production image.
| hardwaresofton wrote:
| Gitlab CI is still the best CI in the game IMO, but GitHub
| Actions gives it an _incredible_ run for it 's money because of
| how easy action re-use is.
|
| I meant to make a blog post about this, but here's a good a place
| as any: GitLab absolutely innovated many hard parts of CI/CD as a
| platform-native piece, but it feels like they lose slightly to
| GitHub on what GitHub does best -- social virality for
| developers. The problem with losing slightly there is that the
| advantage compounds; if developers find it easy to make and
| share, then they make and share which makes more people make and
| share and generates tons of value for a platform.
|
| It may not be safe, but being able to: - uses:
| some-developer/some-repo@some-branch
|
| Is absolutely amazing. Everything else (generally) GitLab has,
| and (likely) did first, but the social viral stuff GitHub just
| gets right in a different way than GitLab does.
|
| No one would accuse GitLab of being a "copy" in recent memory
| like they did in the bad old days, but IMHO they need to just
| copy the shit out of when GitHub gets social things (like this)
| right.
| capableweb wrote:
| > GitLab absolutely innovated many hard parts of CI/CD as a
| platform-native piece
|
| As someone who only used GitLabs CI briefly when it initially
| launched, what hard parts did they innovate on exactly?
|
| As far as I could tell, it's a run-of-the-mill CI/CD platform,
| for better or worse, but nothing I'd call "innovative".
|
| But again, maybe since the first time I tried it when it
| launched, it has changed, and I missed something really cool
| they did.
| starttoaster wrote:
| Innovative is a strong word, but they always had the best
| support for a variety of deployment archetypes, including
| baremetal, kubernetes, docker, docker+machine (autoscales VMs
| in a public cloud to run many Docker executor runners.) They
| also had JWT authentication support built into GitLab before
| GitHub even had a CI offering at all. The GitLab kubernetes
| agent makes their CI/build tool more like a CD tool for
| organizations that primarily deploy to kubernetes clusters,
| which is fairly unique for a CI tool to act as a CD tool,
| though with scripts people often just make the CI tools do
| deployments anyway.
|
| My experience is limited to GitHub Actions, GitLab CI, and
| Jenkins. I've never been in an organization that managed to
| crack the code on running Jenkins in a sane way (always a
| poorly maintained, bloated mess of plugins, unfortunately) so
| I can't say I count Jenkins as a contender in this
| conversation. One could argue it's unfair to discount Jenkins
| because of organizations that manage it poorly, but there's
| something to be said about a tool that is so easy to
| accidentally run in an unmaintainable way. I also have some
| limited experience with Drone and Circle, but not enough to
| talk about them confidently.
| dijksterhuis wrote:
| It's possible to do 'uses' on GitLab, but there's a few extras
| needed admittedly (branch name and path to file)
| include: https://gitlab.com/awesome-project/raw/main/.before-
| script-template.yml
|
| https://docs.gitlab.com/ee/ci/yaml/includes.html#include-a-s...
|
| But I agree, the social virality is what makes GitHub Actions
| what it is.
|
| I used to only visit a GL project when I needed to look through
| Nvidia cuda container builds... everything else I needed was on
| GH :shrugs:
| trilbyglens wrote:
| Currently working on a gitlab api thing for work, and I gotta
| say their API us a complete joke. GitHub runs laps around them
| in this area. Gitlab is actually worse than bitbucket, believe
| it or not.
| tomjakubowski wrote:
| Travis and Circle CI predated Gitlab, with YAML-configured
| workflows checked in to the repository and the ability to use
| custom images.
|
| GitlabCI's innovation was integration: making it a feature of
| repo hosting.
| Ayesh wrote:
| I moved from Jenkins to Travis several years ago, and
| Travis's UI and configuration was such a fresh breath. Back
| then, pretty much _every_ OS project used it because of their
| generous offerings for open source projects. I agree with
| this comment that, and from my own experience living through
| the transitions, Travis CI made a massive leap leveling up
| the modern CI we have today.
| pharmakom wrote:
| I find GitHub Actions incredibly counter-intuitive, personally.
| dzhiurgis wrote:
| It's the confusion between actions(basically third party
| plugins) and workflows(the ci that you are building for
| yourself) that got me.
| ambicapter wrote:
| Thank you for finally explaining this to me (have had to
| deal with github ci off and on for months now).
| NameError wrote:
| I think it was a mistake for Github to name the whole
| product "Actions" and then re-use the same word for a
| specific component within system. It's really natural to
| say something like "we need a push action for this repo"
| when it might be more correct to say "a push workflow"
| chologrande wrote:
| How is Gitlab CI materially different from the jenkins model?
|
| I find that the only difference is that it's YAML - so even
| harder to debug, and maintains the same model where you must re
| run an entire pipeline every commit to test functionality.
| robertlagrant wrote:
| You can run locally first to test functionality : - )
| chologrande wrote:
| oh that's nice I didn't know that! Still a stickler on yaml
| for code though
| codethief wrote:
| > Gitlab CI is still the best CI in the game IMO,
|
| Oh god, if this is the best CI in the game, I don't want to be
| part of this game anymore. I work with (as in write pipeline
| code for) Gitlab CI almost every day and it's absolutely
| horrible. Not only does YAML lack any type safety whatsoever,
| but every day I run into yet another open ticket in the Gitlab
| issue tracker because using a declarative (YAML-based) approach
| (as opposed to an imperative one) essentially means that the
| Gitlab devs need to think of every possible use case
| beforehand, so as to be able to define a declaration & write an
| implementation for it. Clearly this is impossible, so now I'm
| banging my head against a wall on the daily.
| j1elo wrote:
| I'm in the apparently small group of people who ignores 90% of
| the made & shared features, and push back against making
| internal ops to depend too much to any particular CI system.
|
| We know how to build, package, and release our software, and at
| work I firmly support that we write those operations as
| standalone scripts that in principle could even run in the
| developer's machine itself. Moving them to CI is just changing
| the machine that will run the scripts, and that's where I draw
| the line of how much coupling to accept with any given runner:
| run scripts, inject secrets, and maybe accumulate artifacts
| between jobs.
|
| There are a myriad of things done with Actions, such as
| preparing NPM, or running Docker containers. I see no point in
| them, you should be able to manually run the same commands, one
| by one, locally on a VM or empty Docker container, so at that
| point why not write them as a script. Use CI as a dumb shell
| runner. Getting to depend on all those small Actions saves 5
| minutes today, only to make migrations immensely painful
| tomorrow.
| wombatpm wrote:
| Especially if you have to run and deploy on your own servers.
| Not everything can be in the cloud.
| anymouse123456 wrote:
| Been using git hooks with in-repo Makefiles and small bash
| scripts to good effect for some years now.
|
| It's fast, simple and took a couple hours to set up. Rarely ever
| needs attention. It just works.
| ysofunny wrote:
| the worst think about Java (in general, Jenkins is Java) is that
| it solves a real problem that companies have.
|
| I put, however, that _only_ companies have the problem, not
| programmers, and not even regular (sized) engineering teams;
| except when they have a fast employee turnover speed.
|
| I do not have a very clear idea, but roughly speaking I'm saying
| that java does really well in the "too many engineers are working
| on the same code but not at the same time" problem space
| mcfedr wrote:
| Of everything I've tried, gitlab co is my favourite. It's just a
| shell script, and that makes it super simple to replicate
| yourself locally, and - I think most importantly - it's really
| obvious what it's doing. I think that's my biggest problem with
| Jenkins and GitHub and such is all the plugins that do something,
| but without digging through the source code you cannot work out
| what they really do
| mschuster91 wrote:
| I'm so much more happy with GitLab CI. The _only_ thing that is
| easy to implement in Jenkins, but an outright _horror_ in
| anything else, is the usual "helper scripts" stuff: sync the
| production database to integration (and remove PII in the
| process), run reports and data analysis... the best way I've
| figured out to implement this is to create an empty branch (git
| checkout --orphan helper_xyz) and then run pipelines against this
| branch either manually or on a scheduler, but it sucks.
|
| And this is what keeps Jenkins afloat, despite it being a
| consistent source of CVEs both in core and in its myriad of
| plugins that don't bother to follow semantic versioning so every
| upgrade is playing Russian roulette with your company.
| happymellon wrote:
| Aside from the docker snipe I've already commented on, everywhere
| that I've had to deal with Jenkins it's been an issue because
| it's always a pet.
|
| There are always issues, and like the author, it's "only" about
| 25% of the time. A quarter of all builds fail due to things
| absolutely not related to that specific build.
|
| > oh, we were resource constraint at that one second, retry and
| it'll probably work
|
| > oh, we can't replicate that error message from a tool that you
| have no visibility over, retry and it'll probably work
|
| > oh, we have no idea why something timed out, retry and it will
| probably work
|
| Now when I have a failed build I don't actually trust the build
| before I don't trust that there was an edge case in an
| integration test that failed. Because it's almost always
| something outside of the code.
|
| [Edit] downvotes but no constructive responses? The issue isn't
| necessarily Jenkins, it's the type of people who use Jenkins to
| roll their own everything.
| mqus wrote:
| > oh, we were resource constraint at that one second, retry and
| it'll probably work
|
| Had the same issues with (self-managed) gitlab CI. This is not
| jenkins-specific and usually something you can fix much easier
| in jenkins (other than just retrying)
|
| > oh, we can't replicate that error message from a tool that
| you have no visibility over, retry and it'll probably work
|
| This could maybe be better in other, more declarative CIs but
| honestly, its pretty much the same. The only thing some of them
| have is a local runner.
|
| > oh, we have no idea why something timed out, retry and it
| will probably work
|
| Jenkins has this, yes. But use another CI with docker and you
| can get the same and much worse.
|
| But in essence, yes, at our company too, jenkins is a pet. But
| this means that someone knows it from the inside and outside
| and knows which things to fix in your pipeline. Good luck with
| trying to extend gitlab CI (or others) to fix things you need
| and/or opening an issue with them.
|
| Gitlab has quite some high-profile issues sitting around for
| years with customer requests piling up. In jenkins, you'll find
| an extension that will do exactly what you want, or you could
| roll your own.
| shadycuz wrote:
| > On a previous team I had used Concourse CI to some extent, but
| I wasn't really blown away by the experience. Travis and Circle
| were mentioned. I was a fool. I should have committed to
| seriously researching some of the contenders and making a more
| informed decision, but I lacked the willpower and the
| discernment.
|
| The whole post can be summed up as he had very little CICD
| experience. Made lots of beginner mistakes, which is easy to do
| in Jenkins. Then decided to write a post where all his complaints
| about Jenkins are not only wrong but are the issues that plague
| all the other CICD tools.
|
| > So instead of writing Bash directly, you're writing Bash inside
| Groovy
|
| Why are you doing that? You have a fully featured programming
| language and you are running `sh('npm install')`. You could do
| this instead https://github.com/DontShaveTheYak/jenkins-std-
| lib/blob/mast... . How is bash inside of YAML better?
|
| > The trouble is: Groovy is a much, much worse language for
| executing commands than Bash. Bash is interpreted, has a REPL
| that is great for experimentation, does require a ton of imports,
| and has lightweight syntax. Groovy has none of these things.
|
| Groovy has a language server, linters and a vscode IDE plugins.
| They are probably not as stable or full featured as the bash
| ones, but they are available and very few take advantage of them.
| Again, how is YAML+Bash better?
|
| > The way that developers test their Groovy steps is by
| triggering a job on the remote Jenkins server to run them. The
| feedback loop is 2 orders of magnitude slower than it is for just
| executing Bash locally.
|
| This is a rookie mistake. For about 60-75% of pipelines you can
| run them locally in a docker container on your local machine. You
| can even set up hot code reload so as you change your pipeline
| the Jenkins reloads it. You can also configure the job to kick
| off a build when it reloads the code. When Jenkins is configured
| correctly it has the fastest feedback loop of any CICD tool on
| the market. GitHub actions comes in a close second since it can
| also be run locally but you cant run a "clone" of what you run in
| production, like having the same secrets, so it gets second
| place. Beside Jenkins and GitHub actions, I dont know of any
| solutions for the other tools.
|
| You can run a GitHub action on Jenkins. It's a very deep and
| complex system. It's like an iceberg and so many engineers dont
| leave the surface before deciding it sucks and one of the YAML
| CICD tools is better. Sure the YAML alternatives are EASY to get
| started with and to do basic stuff with. But they are Terrible at
| anything complex. While Jenkins is not easy to get started with,
| once mastered, you can build complex pipelines with ease.
|
| I get that I'm a Jenkins fanboy. Most of the things I mentioned
| above, I either contribute to or I'm the author of. I know
| Jenkins has issues. I know it has hurt lots of people, I read the
| complaints online. But it's still the best out there. The best
| software in the world is not written in bash or yaml and the same
| is true of the best CICD pipelines in the world. It's a shame
| very few people get to see/use those pipelines.
| twitchard wrote:
| This is "the worst part of Jenkins is that it works".
|
| You shouldn't judge a developer tool by just what is _possible_
| to do with the tool. After all, with a little turing-
| completeness it is _possible_ to do anything with anything --
| you should judge the tool by what is _easy_ to do with the
| tool. A good developer tool shouldn 't require knowledge of a
| bunch of arcana to "configure correctly". A good tool protects
| you from "rookie mistakes" and makes sane choices the intuitive
| and obvious path of least resistance. Good tools can have a
| learning curve, but they assist the learning curve by making
| their abilities easy to discover and experiment with, they
| don't require you to dig into source code or do random searches
| on github to find some random pipeline somewhere that uses the
| configuration you need, as described in the post.
| shadycuz wrote:
| I wasn't judging the tools. I use several of them and with 0
| complaints.
|
| I'm judging the article for being incorrect about it's
| specific points and for focusing on a single tool while the
| alternatives also suffer from the exact same issues.
| twitchard wrote:
| You are judging Jenkins favorably because it is _possible_
| (with a bunch of arcane knowledge) to build good CI on top
| of it.
|
| I am saying you should judge Jenkins disfavorably for being
| hard to use instead of going "skill issue" when somebody
| describes the pain points.
| dagmx wrote:
| I truly, truly dislike Jenkins but haven't found a better locally
| hosting CI system that is repo agnostic.
|
| My issues with Jenkins:
|
| 1. Groovy is a really annoying language to configure stuff in.
| I've never written it for other things but it has so many little
| idiosyncrasies that catch you out.
|
| 2. Jenkins doesn't have much in the way of resource management.
| It's easy for a job to take over the entire node and Jenkins
| doesn't have the smarts out of the box to monitor this and put
| things in a different node.
|
| 3. So many random job killing bugs that boil down to poor job
| isolation and resource tracking. Every now and then killing one
| job can take down another.
|
| 4. The UI and UX is bad. I know there are better interfaces for
| it, but they all just have such limited interface exposure of
| state beyond "running" and "failed". Estimate time is useless
| because it aggregates failed jobs.
|
| 5. Inter job dependency management and tracking is poor and
| annoying to setup. It's doable, but really should be a better
| first class citizen.
|
| In many ways, I start looking to other tools to fill my CI need
| that aren't meant for CI: Render farm managers.
|
| Film farm rendering is largely similar to CI at a structural
| level. But they have much higher throughput of individual jobs
| and have built around making the most of that. There's better
| inter job dependency tracking and resource management because of
| that, and they're built to expose more information up front to
| help non-tech savvy artists or render wranglers who need to see
| status quickly at a glance on a few hundred jobs at once.
| camgunz wrote:
| The most interesting part IMO was at the end when op was noting
| his new company doesn't have teams/engineers create and maintain
| their own jobs. I've worked lots of different places now and
| something that's always frustrated me is how reticent engineering
| orgs are to reevaluate their processes and policies. Like it
| looks like his first org could have moved on from this problem by
| adopting the second org's policy. There might be other tradeoffs,
| but IME no org even wants to hear about this stuff, and they
| definitely don't have metrics for evaluating anything such that
| they might say "huh, this isn't working."
| ebiester wrote:
| There are tradeoffs. What happens when the CI team is a
| bottleneck? What happens when you have to frontload tasks you
| might not need because you're afraid your release could be
| compromised?
|
| What happens when the CI team says no and it introduces
| complexity in your system? Or you end up doing "shadow CI" on
| your machines to avoid conflict and escalation?
|
| The complexity has to live somewhere. It is always worth
| evaluating the optimum place for your organization at any given
| stage.
| krick wrote:
| Well, to be fair, pretty much each of these points apply to all
| other CI tools in existence.
| Smaug123 wrote:
| Jenkins is, like, _uniquely_ awful though. Aside from the sheer
| staggering number of bugs in its core functionality (I
| encounter on average maybe one bug per new pipeline I write),
| it is also configured in almost the worst language I 've ever
| touched (considerably worse than Nix, for example, which is
| really saying something). "Blue Ocean is Incomplete and
| Unintuitive" is entirely correct - you have to drop back to the
| old interface for essentially everything, and also Blue Ocean
| has the absolutely unforgiveable behaviour that it responds to
| any ESC keypress, even if the OS has focus.
| theptip wrote:
| > The head of SRE championed Gitlab CI. I resisted this idea
| because I, the relatively inexperienced manager of a nascent
| team, was daunted by the prospect of trying to supplant Jenkins,
| Github, and JIRA all at once.
|
| I think this is a really big scoping mistake. There is a clean
| glide path, and I have no idea why you'd think you need to
| replace Jira.
|
| You can use Gitlab Runners in Github (https://docs.gitlab.com/ee/
| user/project/integrations/github....). Then with your mirrored
| repo you can try out the Gitlab features and realize it's better
| to use Gitlab for your repo if you're aso using it for CI. Having
| switched back and forth over the years (with tens of engineers
| using the repos), it's really not a lot of work to migrate
| between Github and Gitlab.
|
| Finally, if you like Jira, just use that? Sure there are
| advantages to having everything in one place, but if you're
| already using Jira+Github then you'll get an equivalent
| experience with Jira+Gitlab.
|
| I think this is one of those things that if you had actually
| prototyped the migration, you would find that it's much easier
| than you thought it was going to be.
| ebiester wrote:
| Gitlab is 29/month. It's hard to stomach 29 + the money for
| GitHub and Jira, depending on the features you need. (Minimum
| 11/mo)
| stickfigure wrote:
| OP's company is paying for a dedicated CI team. Unless they
| have special needs, they probably aren't concerned about the
| CI hosting costs themselves.
| fullspectrumdev wrote:
| I've worked at some places that were pretty huge, but the
| finance department absolutely fucking nickle and dime over
| _everything_.
|
| Even the smallest EC2 instances had to be accounted for
| with a business reason that they would accept, with audits
| happening every few days.
|
| Asking for a licence for something upfront was like pulling
| teeth.
| hedora wrote:
| I've encountered this too.
|
| I always wonder why the eng departments never itemize the
| time they spend dealing with finance, then bill the
| finance team for the hours.
| fullspectrumdev wrote:
| I've threatened to do this to a sales department - send
| them a bill for lost time, engineers working overtime,
| etc because they sold shit without a clue if it even was
| possible.
|
| Lead balloons have flown better than that idea did :)
| esafak wrote:
| You were in a cost center, and/or your company had no
| concept of productivity.
| fullspectrumdev wrote:
| The latter. Extremely "Penny wise, pound foolish"
| nonsense going on.
| twitchard wrote:
| Yup. I think this is the sort of perspective you get with
| experience.
| flashback2199 wrote:
| > Finally, if you like Jira, just use that? Sure there are
| advantages to having everything in one place, but if you're
| already using Jira+Github then you'll get an equivalent
| experience with Jira+Gitlab.
|
| Sometimes you don't adopt X because its likely to make people
| want to adopt Y
| enlyth wrote:
| Our company switched from GitHub / Jira to everything on GitLab
| because devops had the authority to make the switch and that's
| what they wanted.
|
| PMs now complain that the GitLab "issue board" is nowhere near
| a replacement for Jira and us devs complain that GitHub had a
| nicer UX and less stability issues.
|
| Can't make everyone happy at once I guess /shrug
| bluGill wrote:
| What management and what devs need from an issue manager is
| very different. Which is why every 5-10 years everyone wants
| to switch to a simpler tool, then slowly most of the
| complexity in the first tool gets added back as the people
| who needed that one feature complain loud enough, until the
| people who wanted simplicity become the loud voice and you
| switch to a simpler tool. It is never the tools fault above.
| (Some tools are better than others, but the real problem
| isn't the tool)
| IshKebab wrote:
| The fact that Gitlab Issues aren't a replacement for Jira is
| a feature IMO. Jira is surprisingly awful at its main job.
| For example you can't have more than 2 levels of parent/child
| tasks (compared to Phabricator which has no limit). Changing
| an issue to a task or vice versa goes via a complex batch
| update mechanism. Over-configurability means you end up with
| a gazillion different task states (Done, Resolved, Finished,
| Closed, ...). You can't do basic things like reorder the
| backlog based on priority.
|
| Perhaps most critically the interface is just insanely slow!
| It regularly causes waits in sprint meetings while we wait
| for someone to drag & drop an issue or for the page to load.
|
| The only reason it's popular is because PMs like to make
| engineers do their job for them so they can just click a
| button and get pretty graphs that they can copy & paste to
| presentations.
| postalrat wrote:
| The best part about jekins is that it works. The worst part is
| that many people dont want to work with jenkins.
| jurassic wrote:
| Having worked with a huge Jenkins deployment at a large company
| somewhat recently (>10,000 jobs), I found it worked okay enough
| that company leadership never felt it was worth the pain if
| switching. But all the friction points added up to a system that
| was rarely touched by anyone who hadn't learned where all the
| bodies were buried. Over time that meant as an engineering org we
| were underinvesting in CI; there was a lot of quality-oriented
| stuff beyond unit testing that never got implemented in CI
| because the typical dev was unaware of how to change it or
| fearful about trying to change it. The relative accessibility of
| Github Actions (and other config as code alternatives) transforms
| the average dev's relationship to CI from consumer to
| owner/developer/maintainer, and I think that is extremely
| worthwhile even if these tools bring some new problems of their
| own.
| flimsypremise wrote:
| The real failure here was the failure to recognize the sunk cost
| fallacy. My team attempted the pursue the "supported" path with
| Concourse for a recent project, and I watched them bang their
| heads for a week, realized it wasn't going well, set a meeting
| where we all discussed and decided to move to GitHub actions. Now
| everyone is happy and the project is back on track and I don't
| have a write a blog post like this one about Concourse.
| _whiteCaps_ wrote:
| I really like Concourse but unless you've got someone who's
| fully drunk the koolaid, it can be hard to a team onboard. It's
| got some very strong opinions on how builds should be
| deterministic. "Why can't I just click re-run this failed
| job!?"
|
| If you're doing something that doesn't line up with Pivotal /
| Cloud Foundry workflows, you're probably going to have a bad
| time.
| redserk wrote:
| There are a large number of quirks I've had with the Kubernetes
| runner on Jenkins causing flaky behavior, but when it works, it's
| mostly fine after figuring out Groovy. The 1 in 50 time a job
| needs to be kicked off again due to some strange error is
| annoying though.
|
| More often than not, I end up creating a sandbox folder in
| Jenkins and cloning workflows to work on them by hand without
| having 90 commits of misc errors and typos. This has helped speed
| up some iteration.
|
| Sort of off topic from the article, I think where almost every
| single CI engine falls short is the extremely limited scope they
| cater to - that every job ever made must be solved with the cargo
| cult ideology where every CI job mustn't ever have human input,
| everything must be perfectly automated CI. This is wonderful when
| it is possible to do, but sucks when it isn't feasible.
|
| Jenkins' strength is that it's actually a workflow engine
| masquerading as a CI tool, and that gives a lot of flexibility
| for a number of bespoke processes.
|
| Do you want to wait for someone to click a button to grant
| permission to run a job because the test might spin up extremely
| expensive resources and you'd really rather someone baby sit it?
| Jenkins has you covered.
|
| Or maybe it just isn't possible (or worth the time) for your team
| at a point in time to automate all input to a job, so spin up a
| Jenkins job that lets someone manually specify parameters.
|
| Maybe your flow can trigger a release of software but you want a
| human to come behind to do a manual once-over as a final check,
| then also capture who/why approved it.
|
| More CI tools need to accept the reality that not everyone can
| implement the perfect ideal of CI jobs. Some jobs will require a
| human step somewhere, be it custom input to a job or waiting for
| a response.
| Jnr wrote:
| The part about using Docker is wrong. You can use the docker
| plugin and just run any image you want, and then run the code
| inside that specified image. You can pass additional parameters
| to also run it within the context of any particular docker build
| stage.
|
| I use the same Dockerfile for production builds and for running
| all the CI related tasks, without any problems. And the same
| single pipeline is used for PR hooks and releases.
|
| Post also mentions Blue Ocean UI, but that one is deprecated and
| should not be used at all.
|
| Jenkins is far from perfect, but unless you can use Gitlab (and
| sadly you can not in every case), it is the most feature rich
| self-hosted tool out there. Just make sure you set it up with
| Jenkins Configuration as a Code plugin out of the box.
| jamesu wrote:
| I stayed away from jenkins for a long while because of its bad
| reputation, but after diving into it for a recent project I was
| surprisingly impressed how easy it was to set everything up (with
| the docker image at least), and it seemed that there was a
| solution for just about every convoluted build scenario I could
| come up with. In contrast, I've tried out other tools and
| encountered: confusing uis, too much focus on docker, broken
| interop with forges, poor documentation, too much yaml, ...
|
| I guess the flexibility with jenkins is probably both a blessing
| and a curse.
| chologrande wrote:
| The blog perfectly sums up the feelings of anyone who has worked
| deep in jenkins. It works, but goddamn it's hard not to hate.
|
| Dagger is the answer to jenkins woes. It's saved my sanity and
| made CI development tolerable again.
|
| * Real language support. No groovy, no yaml.
|
| * Debugging available in your native IDE tools.
|
| * Clear docs in any language of your choice.
|
| * Reproducible builds you can run locally. Just need docker.
|
| This all comes with the bonus that it can actually just consume
| GHA yaml with some of the tooling folks have made.
| doubloon wrote:
| at my company this is all very easy. there is no version control
| of any kind and also no automated testing. solves a lot of
| problems.
| nathants wrote:
| cloud is so good now it's hard to justify not doing something
| bespoke. ec2 spot is insanely cheaper than turnkey cicd, and
| better in almost every way.
|
| i'm delighted to pay 30% over infra cost for convenience, but not
| 500%. and it better actually be convenient, not just have a good
| landing page and sales team.
|
| this month i learned localzones have even better spot prices.
| losangeles-1 is half the spot price of us-west-2.
|
| for a runner, do something like this, but react to an http call
| instead of a s3 put[1].
|
| for a web ui do something like this[2].
|
| s3, lambda, and ec2 spot are a perfect fit for cicd and a lot
| more.
|
| 1.
| https://github.com/nathants/libaws/tree/91b1c27fc947e067ed46...
|
| 2. https://github.com/nathants/aws-
| exec/tree/e68769126b5aae0e35...
| awinter-py wrote:
| > Your editor won't syntax highlight the Bash inside Groovy. You
| can't run "shellcheck" (or any sort of Linter) on the Bash inside
| the groovy. You can't very easily execute your shell commands to
| test them.
|
| this lands, and not specifically for jenkins. CI generally is
| inscrutable and difficult to test
|
| every CI has its quirk
|
| github CI feels fundamentally worse than gitlab because job
| artifacts are native in gitlab -- this makes a million things
| easier. github's own action for attaching things to releases is
| deprecated. the upload-artifact action doesn't work in nektos/act
| (the only way to test locally)
| fireflash38 wrote:
| One of the most frustrating things about gitlab ci is testing
| rules. There's just no way to do it without sending it live,
| especially for things that might be more rare (tags, releases,
| default branches, etc). Why that isn't a priority to let people
| see what the jobs/vars of a pipeline would be without filly
| executing it is beyond me.
| awinter-py wrote:
| yeah, really need a repl for CI. have run into exactly the
| same issue, conditional cases are impossible to test ahead of
| time
| fireflash38 wrote:
| Jenkins is amazing for one thing in particular, and I've not
| really seen it replicated well elsewhere (like gitlab ci): common
| job execution. Letting people run jobs on/against shared
| equipment, rather than having people run scripts locally.
|
| I wouldn't recommend it for general CI other than that. Think of
| it like a butler, able to do small jobs, not as your complete
| home assistant.
| AceJohnny2 wrote:
| What would people recommend for a _self-hosted_ CI system in
| 2023?
|
| I'm currently leaning towards Jenkins, because neighboring teams
| are using it (with some pain due to their scale, but theirs is
| 10x-100x ours), and I have some ~10y-old experience with it.
|
| (Our builds are compiling for embedded targets, with a Makefile-
| based system)
| AceJohnny2 wrote:
| I've noticed the Avahi folks use Woodpecker CI [1], which I
| hadn't heard of before, because it supports priorities and pre-
| emption [2]
|
| [1] https://woodpecker-ci.org/
|
| [2] https://social.treehouse.systems/@marcan/111258515328251078
| kbenson wrote:
| Because not every company wants to allow their source code to
| the cloud. Basically all the reasons organization self host
| their code repositories (and a bunch do) all apply.
| danielheath wrote:
| Buildkite is mixed-hosting; source code stays on your machines,
| on-prem if you like, but build logs / history / SCM triggers
| are hosted.
|
| Unclear whether that's enough for your situation but it's by
| far my favourite because of how it lets you store the CI
| pipeline in the repository (so changes are reviewed via normal
| channels etc).
| MilStdJunkie wrote:
| You can run a local gitlab instance, so I'd assume self-hosting
| as a rider on that. Unless I've been making some _very_ bad
| mistakes.
|
| GitLab's not too bad. I had a Gradle pile I had been using but
| once I knocked out some old Java bits I wanted to sleek my
| "required tools" down a bit. I've integrated some OpenProject
| stuff into GL also. Maybe someday . . for "minimal" clients . .
| I can just bring in the GitLab and call it a day, so far as
| server stuff is concerned.
|
| That day is not today, however. Customers in my business want a
| _lot_ - like, a LOT a lot - of project toys, and homebrewing
| those in GitLab is beyond my skills.
| shitcoder wrote:
| I use drone self hosted (which is what Woodpecker CI was forked
| from in one of the other replies). It's an alright system and
| it works with my Gitea instance to build executables and docker
| images.
|
| I think though ironically my take away after setting everything
| up, was that I didn't really need to spend a lot of time to get
| it perfect, rather just getting it working was enough for me.
| thayne wrote:
| The complaint about a long feedback loop is true of every CI
| system I've used (jenkins, travis, gitlab, github actions). None
| of them really have a good way to test a pipeline locally. The
| best solution here is to put as much of the build logic as
| possible in scripts and your build system.
| dahart wrote:
| I've long had a very similar argument for why commuting in the
| Bay Area* sucks: because it's just tolerable enough that people
| keep doing it. It's not unmanageable, but it drains your life
| little by little every day.
|
| * This, of course, applies to all long commutes everywhere, like
| how it applies not only to Jenkins, but lots of other software
| too... especially Jira.
| lmm wrote:
| Jenkins is the worst CI system except for all the others.
| Everything feels like a pile of hacks upon hacks, maintaining it
| is tedious and unrewarding, new functionality is always bolted on
| and undiscoverable. But every alternative implements the first
| 20% of features that provide 80% of the functionality and then
| never gets around to implementing the other 80% (and
| interestingly every attempt to "modernize" Jenkins, such as "Blue
| Ocean" or "Declarative Pipelines" as mentioned in the article,
| suffers from exactly the same problem).
|
| I wish there was a nicer tool that worked. But there isn't.
| shitcoder wrote:
| I really resonate with this post after working with Jenkins for
| ages. In my experience I wasn't very happy with Bash in the end
| either and moved to Javascript for reliability, performance and
| tighter integration with the Cloud were deploying to. It also
| allowed us to run locally and avoid having to deploy to Jenkins
| to test.
|
| The discoverability issue is also a massive issue, declarative
| pipelines massively over promise its capability. Pipeline
| configuration was mostly still done in the UI because extensions
| had 0 documentation about how to declaratively define its
| configuration. Searching on the internet for the magic
| incantations as the author puts it is such a waste of time and I
| just compromised to get it running.
|
| I'm glad this post is out there to articulate these common
| challenges when you invest so much into Jenkins.
___________________________________________________________________
(page generated 2023-12-03 23:01 UTC)