[HN Gopher] How a Jenkins Job Broke our Jenkins UI
___________________________________________________________________
How a Jenkins Job Broke our Jenkins UI
Author : FraBle90
Score : 82 points
Date : 2021-06-03 17:36 UTC (5 hours ago)
(HTM) web link (slack.engineering)
(TXT) w3m dump (slack.engineering)
| bdefore wrote:
| Circa 2009, back when it was Hudson (I think?) I once had an idea
| to rename MyJenkinsProject to !MyJenkinsProject in order to move
| it to the top of the list alphabetically. When I hit save, the UI
| explained that this wasn't possible _and that I shouldn't be
| putting dangerous characters in my project names_. Not to be
| pushed over so easy, I tried again with a skull and crossbones
| unicode symbol () in the name. The UI immediately became
| unresponsive and wouldn't start again until the project was
| removed.
|
| edit: Interesting HN also stripped out the character:
| https://www.fontspace.com/unicode/char/2620-skull-and-crossb...
| orf wrote:
| Jenkins is endlessly flexible, but at the end of the day its
| complexity adds such an upfront cost that less flexible tools win
| outright when the metric you care about is the "ability to get
| shit done without giving a fuck about groovy sandboxing".
|
| A system like Gitlab-CI is less flexible than Jenkins but because
| it makes 99% of the use cases you have for a CI system take 99%
| less effort it wins hands down.
|
| Combine it with a Kubernetes-based executor and you have a
| scalable, isolated, reproducible and flexible CI system that
| requires basically no maintenance and most importantly is about
| as approachable and understandable as you can possibly get for a
| CI system. It's simple shell commands vs
| AbstractProxyCSPFactoryGroovyBean classes.
| bob1029 wrote:
| After a while it got easier for us to write our own code to
| automate things than to try and piece together a wordpress-like
| component system.
|
| We still have Jenkins running somewhere, but I can't remember
| the last time I needed to run a job on it.
| oblio wrote:
| The thing is, Gitlab & co. almost all mandate Docker.
|
| You don't use Docker, the list of options narrows down
| dramatically.
|
| You need to support non-Linux OSes, same thing.
|
| If your needs are limited, yeah, something other than Jenkins
| is better. If they're not... I don't think there's a better
| alternative to Jenkins.
| ArchOversight wrote:
| Gitlab CI does not mandate docker. The gitlab CI agent will
| happily execute directly on the host without starting docker
| containers.
| orf wrote:
| Gitlab CI doesn't mandate anything - you can run it on a
| "bare metal" server. However if you use a kubernetes executor
| it mandates that your builds run _inside_ a container image,
| which is fantastic for reproducibility and visibility.
| Everything is versioned and can be customised at the
| repository or branch level with very little effort.
|
| Of course this limits some more esoteric builds that might
| need dedicated hardware or for whatever reason cannot run
| inside a container, but you can mix executor types if you so
| please.
| oblio wrote:
| Do they support Windows Docker containers?
| orf wrote:
| Yes: https://lmgtfy.app/?q=gitlab+ci+windows+docker
| sytse wrote:
| Yes, GitLab CI supports Windows Docker containers, see ht
| tps://docs.gitlab.com/runner/executors/docker.html#using-
| w...
| scrozart wrote:
| Gitlab doesn't mandate anything. Also, even if your CI
| pipeline does use Docker, it doesn't matter if "you don't use
| Docker" _as a team_. Your pipeline may name an image, and
| that's it. That's way different than maintaining your own
| Dockerfiles and images.
| c7DJTLrn wrote:
| It's horrifying that Jenkins is still the industry standard. The
| whole thing is poorly documented and full of cruft and
| vulnerabilities. But there is nothing out there as flexible to my
| knowledge.
| jalk wrote:
| Its the Nagios of CI tools ;-)
| TheGuyWhoCodes wrote:
| Most of the flexibility comes from the plugins which are a
| security risk. Either they are just abandoned, have very old
| dependencies or just don't sanitize inputs.
|
| You'd think Cloudbees would take over abandoned plugins,
| integrate into the main code or just remove them from the
| repository for safety but they just let them rot.
|
| We had one of the plugins we use brake after an upgrade because
| of the dependency hell in Jenkins so we ended up contributing
| to the plugin to remove the dependency. Thankfully the
| maintainer was still around to verify our fix and update the
| plugin repository (we obviously built it locally and tested).
|
| To think that anyone could adopt an abandoned plugins (which
| could have 1M installs) and just insert some malicious code,
| with minimum or no oversight is really scary.
| Jenk wrote:
| Jenkins (when it was still just Hudson) was the forerunner for
| CI. It made a lot of mistakes but only because nothing else was
| there at the time to learn from, in my humble opinion. The only
| alternative in the early days was CruiseControl (which became
| GoCD) and of the two, Jenkins was far better and more advanced.
|
| I made extensive use of Hudson/Jenkins from 2007 to about 2015
| and it has flaws, sure, but I know it made some pretty difficult
| tasks sane for me, and was pretty straightforward building CD
| pipelines.
|
| I didn't (and still don't) like the decision to adopt groovy but
| it is better than configuring via UI and I like that it is
| imperative, at least.
|
| Some of the jobs I built using Jenkins pretty rapidly include
| full CD from svn push/git commit to production with all the
| bells, whistles, gates, and stages in-between, to managing
| failovers, and even my early foray into IAC with rudimentary
| remote exec scripts and the likes of chef.
|
| I think it became a victim of its own success. It was _flooded_
| with contributions that yanked Jenkins in all kinds of directions
| with no clear owner for direction and maintenance, which lead to
| frankly some horrific (but "working") architecture within and a
| jumbled mess of extensions vs plugins vs patches and all kinds of
| horrific UI changes, often all conflicting.
|
| I believe it was Gojko Adzic that wrote up a blog article (that I
| can't find) about 10 years ago listing some of the truly
| horrendous code in Jenkins source. Stuff like abstract classes
| typecasting themselves to derivatives to access members.
|
| Looking back Jenkins was and is clunky, messy, uncertain of its
| purpose in life. But so was just about _everything_ to do with CI
| at the time. Would I use Jenkins again in 2021 and beyond?
| Probably not but it definitely added value to the build
| technosphere.
| colek42 wrote:
| I have to estimate time and materials for a lot of DevOps
| contracts. I estimate twice the hours, or more for Jenkins CI
| work vs GitLab CI even if my engineer is an expert in groovy. The
| complexity of Jenkins adds a huge amount of risk.
| flabbergastedtw wrote:
| >Note: "hudson.slaves" is part of the interface that Jenkins
| provides; in our own code at Slack, we've fully moved over to
| "controller" and "agent" as the preferred terminology.
|
| Thankfully Slack lets us know that their teams are much more
| enlightened than the Jenkins team. This was very much holier than
| thou and unnecessary.
| geodel wrote:
| It could just be my bias against this company. But I find their
| "Engineering blogs" about fixing their poor product / process
| rather lame. I get this vibe of "How we fix our inconsistent
| metric generation problem by using AtomicInteger instead of
| Integer"
| TomBombadildoze wrote:
| Good gravy, this is a lot to unpack. It's an alarming story from
| the very beginning, and a cautionary tale of how tempting it is
| to do everything with Jenkins, even though it's an appropriate
| tool for absolutely nothing in the Year of our Lord 2021.
|
| > As part of our automation setup, we continuously run integrity
| jobs to inspect our Jenkins nodes.
|
| Why on earth would you self-host this in Jenkins? This is a
| _monitoring and alerting_ problem.
|
| > These jobs check system configurations and properties and look
| to see if any node is failing those checks.
|
| What year is it? We've solved this with immutable infrastructure
| or system integrity monitoring. Or both.
|
| > The checks automatically mark Jenkins nodes as offline when any
| of those checks fail and notifies our Mobile Build & Release team
| via a Slack message.
|
| "Mark" offline? Why not just terminate it? And why do we care if
| build nodes come and go? These should be cattle, not pets. If
| they all die at once, that's bad. If they're cycling in and out,
| that's business as usual.
|
| > When our Jenkins UI stopped working, we noticed two things:
|
| > 1. We had recently upgraded Jenkins and all its plugins to a
| newer version
|
| Did they _just now_ learn what an awful idea this is? All of this
| at once, really?
|
| This isn't so much a Jenkins problem (though let's be clear,
| Jenkins _is_ a problem) as it is a remedial engineering problem.
| The top takeaways should be "choose appropriate tools for the
| task at hand" and "don't make reckless decisions with brittle
| systems".
| jordanbeiber wrote:
| We moved to drone after many years of a love/hate, abusive
| relationship with Jenkins (junkins was the common name).
|
| Drone has this awesome feature where you can have it hook out and
| receive a pipeline on the fly. We now generate our pipelines in
| an api and this way we can write the logic in "something other
| than groovy" - typescript in our case.
|
| No pipelines required in repo and everything as code without ugly
| hacks.
|
| Never looked back even once.
| spondyl wrote:
| So just to clarify, they rolled out the latest version to
| production which broke? What's the staging environment for then?
| zmmmmm wrote:
| > It gets executed in a special Groovy sandbox to increase the
| security posture
|
| Not a jenkins user but I am really curious about what the
| perceived security issue is here? Why are there all these layers
| of protection placed on what is presumably internal
| infrastructure? I can't think of similar protections being
| applied to other CI systems where you can run arbitrary bash
| commands and containers (aka: do anything you like). It seems to
| be one of the most common pain points, but I can't quite
| understand why it's there in the first place.
| 0xbadcafebee wrote:
| I feel so bad for them. Jenkins is really a technology anti-
| pattern. I keep saying I'll do this, but I really need to write a
| series of blog posts to elucidate all the ways in which Jenkins
| is just bad for your business. If you can use _any_ alternative,
| do it, and for Bob 's sake, _pay for a solution_. Stop trying to
| cobble together some crap with shitty free tools when CI /CD is
| critical to the velocity, quality, and reliability of your
| products.
| nawgz wrote:
| Can you expand more on how it would qualify as an "anti-
| pattern"? I agree it is slow, has issues with its built-in
| coverage and capabilities, and has an oldschool UI; but it is
| at its fundamental core a pipeline runner. It is a decent
| pipeline runner even, which when it comes down to it is the
| core of each other CI product [that I've seen].
|
| So to hear it described as an "anti-pattern", when
| realistically it seems to BE the pattern - just poorly
| executed, is a bit unintuitive to me.
| sakisv wrote:
| (not the person you replied to, but allow me to give a couple
| of personal annoyances - keeping in mind that they're a
| couple of years old and things may have improved since)
|
| My main problem with Jenkins was that its architecture made
| it extremely difficult to automate its provisioning without
| having to click through the ui at all.
|
| This led to the second big problem which is that updating
| either Jenkins or one of its plug-ins when they got a new cve
| (which was every other day) was quite stressful because you
| could never be sure if something would break - especially for
| plugins that depended on other plugins (case in point this
| post)
|
| I have since moved to concourse which has a much more sane
| architecture - at least for these things.
| orlovs wrote:
| Probably not possible to describe. Jenkins is just tool, if
| used wrong it will bite to operators. Problem with jenkins is
| that it's takes time to setup workflow. You configure to run
| plugin-install-tool, JaaC, Jobdsl, Shared Libs, some
| credential store etc. But when it's up and running, oh boy,
| it's factory. I have run jenkins with 5k+ jobs. All auto gen,
| no manual interventions etc. Gitlab-CI (same as github
| actions) I like due opinionated approach. It makes things
| easier if setup is simple, but when you need exceptions or
| special cases, hack's begins.
|
| But, yes, I have seen so many badly implemented jenkinses.
| 0xbadcafebee wrote:
| 1. it isn't designed as a cloud-native configuration-as-code
| immutable service. The way it stores and loads configs, jobs,
| logs, build workspaces, etc is all 1990s tech. Every modern
| replacement does these things much better. These inherent
| design flaws set up all the later problems.
|
| 2. configuration as code is an afterthought, so it doesn't
| work very well.
|
| 3. the only way to manage Jenkins as described initially
| requires learning four different DSLs, although for
| developers to write jobs only requires learning three (JobDSL
| to load jobs from JCasC, Jenkinsfile for simple pipelines,
| Groovy for complex ones). This is ridiculous.
|
| 4. the plugins are atrocious, there's too many of them, they
| don't have good enough features much of the time, managing
| and upgrading them is always a pain.
|
| 5. CloudBees doesn't even maintain the core stuff correctly.
| The current Jenkins container comes with a new plugin manager
| which, by default, does not respect pinned plugin versions.
| That's literally the most basic thing you can do for
| operational stability. I filed a bug in January, and they
| didn't feel like fixing it, so I got them to merge a note at
| the bottom of their README instead mentioning the bug.
|
| 6. pipeline libraries are a costly maintenance and
| development pain. Having to write Groovy code just to write
| pipelines is horrible. Jenkinsfiles are, although much better
| than Groovy, still an over-complicated, unintuitive mess.
|
| 7. there's no simple way to deploy, maintain, test, and
| upgrade a Jenkins cluster. You have to maintain multiple
| clusters, increasing cost and complexity.
|
| 8. since most people don't set it up right (because it is so
| overcomplicated), the jobs, server configurations, and build
| history are not backed up, there's no version control. So
| when something goes wrong, the whole thing is hosed. Unless a
| Jenkins _expert_ took 6 weeks to set it up perfectly.
|
| 9. due to all the above problems, you end up with a million
| different Jenkinses, all in various states of insecurity,
| brokenness, and wildly different configuration, making them
| incompatible with each other. This makes for a gigantic
| maintenance cost that never ends.
|
| 10. literally all of it is completely proprietary to Jenkins.
| Unless you build it and the jobs in a very particular way
| (which makes it impractical to use) none of it can be re-used
| in a different system.
|
| That's off the top of my head. There's more reasons.
|
| The point is, organizations will invest literally thousands
| of man-hours in making Jenkins work, slowing down their
| product development, forcing everyone to use this old-ass
| over-complicated piece of junk. If they took the same amount
| of cash they could buy literally any proprietary CI/CD system
| and do everything much faster and better. But the
| organization doesn't see the hidden costs until it's too late
| and they desperately want to replace it. Jenkins is not just
| bad, it actively holds back your organization.
| nawgz wrote:
| I struggle to see how this justifies the claim that
| "Jenkins is really a technology anti-pattern".
|
| You are describing bad maintenance, bad architecture, and
| bad execution, I fully agree with this. Jenkins is clearly
| old and it has old approaches to complexity.
|
| But an "anti-pattern" implies that using it moves you
| further from your goal. When I was a noob with 2y of
| software development, primarily writing React & NodeJS
| APIs, I stood up a Jenkins VM and was able to correctly set
| up a CI/CD system building, testing, and deploying
| containerized microservice-based architectures via Docker
| and Jenkinsfile alone. I encountered extremely few issues
| with the core of Jenkins, because it is a pipeline runner
| and it has a lot of ways to run pipelines.
|
| So to me, it looks like you use the word "anti-pattern" too
| liberally, since I don't think there are any other free
| open-source pipeline runners that would meaningfully
| integrate git webhooks and clone source code easily for me.
| But perhaps you disagree because of your final opinion,
| which is that standing up production-grade CI/CD for bigger
| workload would be faster avoiding Jenkins.
|
| Still, not sure I even agree with that claim. I have seen
| it used to great success in many contexts. Does it have
| lots of problems? Yes. Anti-pattern? Tough sell.
| oblio wrote:
| > The way it stores and loads configs, jobs, logs, build
| workspaces, etc is all 1990s tech.
|
| I agree with a good chunk of what you said, but.
|
| It's files, my friend, files. Tech from the 1970's.
|
| Nothing wrong with that, inherently, and they are easy to
| inspect, repair if needed, you can use standard tools, etc.
|
| The design of the file structure is maybe the issue,
| because it makes high availability complicated, but just
| using files is not necessarily a bad idea.
| 0xbadcafebee wrote:
| Correct, in some circumstances, files are great. They
| suck for Jenkins.
|
| What are they? Lots of different things: build logs, job
| configurations, server configuration, secrets, cached
| unpacked plugins, build workspaces, etc. Some of those
| you want in S3, some you want in a database, some you
| want on fast ephemeral storage, some you want in a
| credential store. Good luck with that; only the secrets
| are doable with plugins.
|
| Where are they? Sitting on some EC2 instance's ephemeral
| or EBS storage. But you don't want them there, so now you
| have to throw a bunch of crappy wrappers in to
| occasionally move them if you want them somewhere else.
| (Even if you do JCasC/JobDSL/Jenkinsfiles for version-
| controlled configuration and secrets, you may still want
| to back up your build artifacts and logs)
|
| And them being files, it doesn't scale. Using EBS? Only
| one host can mount it (unless Nitro), so good luck
| scaling one box's workspace filesystem past one gigantic
| EBS volume, or doing master-master. And you have to clean
| up the filesystem every time a plugin or core version
| changes, or the cached version on the filesystem will
| override your container/host's newer version. Using EFS?
| Network filesystems suck (scalability + reliability +
| security + performance woes).
| aranchelk wrote:
| Modern Jenkins using JenkinsFiles and cloud provisioning can
| work pretty well.
|
| All things being equal, it's my preference to not use Jenkins
| for new projects, but anti-pattern is IMO a significant
| overstatement.
| coredog64 wrote:
| I'll note that Cloudbees offers commercial support for Jenkins.
| Having said that, I had a former employer that was a Cloudbees
| customer and the support we got was typically "Have you tried
| turning it off and back on?" That drove us back to OSS Jenkins.
| Although, AIUI, said employer has now moved to Harness.
| mdeeks wrote:
| As a counter point, we had major issues with just paying for it
| via CircleCI. Excessive downtimes, a UI so slow it was nearly
| unusable, etc. We decided at the time we couldn't bet our
| company on it and we moved to Jenkins. We have far more
| control, build time and build costs are lower, uptime is better
| (All of this at the expense of initial dev time and maintenance
| of course). Generally we are a SaaS company, but paying for a
| solution just doesn't always work when you get larger and when
| there are limited options out there.
| diroussel wrote:
| Try gitlab. You can run it yourself. Great up time.
|
| And best of all it's much easier to run and upgrade than
| Jenkins.
|
| Jenkins and it's plugins can get quite brittle and is quite
| an art to automate the installation and config of all those
| plugins.
| mdeeks wrote:
| I'm interested in trying Gitlab, but this does not look
| easy to run (from their docs): https://plantuml.gitlab-
| static.net/png/U9oDLbrlsZ0KVVUli9gNB...
| dfcowell wrote:
| Unless you're a very large organization, you're not going
| to be managing each of those components individually.
|
| The "Omnibus installation" as they call it is pretty easy
| to use. Just bring a Postgres DB and a redis instance.
| crummy wrote:
| You don't even need to BYO databases - they have an
| omnibus docker compose setup that runs everything for
| you.
|
| https://docs.gitlab.com/omnibus/docker/#install-gitlab-
| using...
| richwater wrote:
| The discussion chain is everything that's wrong with
| modern engineering. A complex topic progressively
| simplified and abstracted to just "Run this container",
| ensuring no one in your org actually understands how it
| works, how to maintain it and how to debug it.
| Aeolun wrote:
| Gitlab is a complex system. I wouldn't expect anyone to
| suddenly understand my companies' product either.
| crummy wrote:
| The GP was interested in "trying" Gitlab. If you want to
| try it out, but are wary of the time involved, isn't a
| docker-compose installation a great solution?
| zug_zug wrote:
| Since everybody seems to be hating on jenkins so much, I'll speak
| up. IMO jenkins is one of the most valuable tools at any startup
| and every good engineer uses either jenkins or something very
| similar.
|
| There's no comparison between
| managed/logged/permissioned/distributed jobs that jenkins
| provides for free vs building an overwrought service or an
| insufficient crontab. However it's a power-tool and I think a lot
| of people go in expecting something dead-simple and pretty and
| are put off by something that they need to invest in learning.
|
| Just for example, in 2 days I built an uptime checker that ran
| every 60-seconds via jenkins and triggered slack alarms/sms on
| problem, and it just worked (< 2 days total maintenance), for
| years. An equivalent service (Pingdom) quoted 10k a year.
| [deleted]
| orlovs wrote:
| Don't abuse CI/CD system as workflow automation tool. There are
| better and much easier alternatives.
| theamk wrote:
| What do you have in mind? We have a number of workflow
| automation tasks on Jenkins and while it works, it requires
| constant upkeep.
|
| An ability to share machine pool with a CI system, or having
| an integrated CI is a plus.
| dfabulich wrote:
| Such as?
| devonkim wrote:
| I think the question itself begets more questions. For
| example, I've seen Jenkins jobs replaced with an Apache
| Airflow DAG or two, but obviously that's not the right call
| for continuously deploying applications to K8S clusters,
| which is where I'd use Tekton, Harness, or Argo CD. If we
| go way back, companies used to churn out Perl and Ruby
| scripts and eventually they were replaced with anything
| from Python to Go to Rust. It always goes back to _why_
| someone finds deficiencies in what Jenkins jobs provide.
| Ask 10 people that hate Jenkins and one is likely to find
| 10 different products / solutions appropriate for them.
|
| Most folks I've seen replacing Jenkins for a job runner
| wind up using Ansible, Salt, or Rundeck if they're not in a
| giant enterprise (there's also the old HP Operations
| Orchestration system but that's buried now unfortunately).
| All of these have their own holy wars brewing and
| deficiencies, sure, but I personally prefer those warts
| over adding more maintenance issues by adding in Jenkins.
| I'd also suggest StackStorm for a more modern, async
| approach to orchestration workflows. The workflow software
| ecosystem is built around business domain specialization
| and it's kind of silly to try to go against Jenkins toe to
| toe as a company, so this is what we've got sadly. After
| all, Jetbrains makes TeamCity which is essentially a Better
| Architected than Jenkins system but it's definitely not
| very popular either.
| oblio wrote:
| I'd also like to know. Many of them are <<clunkier>> and
| have smaller communities, less active development, etc. Far
| from a clear win.
| otabdeveloper4 wrote:
| (crickets)
|
| We don't use Jenkins for CI/CD. It's probably too overkill
| for those basic tasks.
|
| But as the middle ground between crontabs and something
| like Airflow it fixed all our problems without creating new
| ones.
|
| Are there really any alternatives out there? (And no, we
| don't need Kubernetes, thanks.)
| codeduck wrote:
| I've been using jenkins for over 10 years.
|
| There is no finer system for turning quick CI hacks into
| entrenched technical debt.
|
| Jenkins gives you the freedom to do things without the burden
| of asking whether you should do those things. Jenkins permits
| you to munge a whole lot of separate domains into a single,
| creaky system.
|
| There are better tools available for most aspects of CICD and
| automated testing these days.
| dijit wrote:
| > There are better tools available for most aspects of CICD
| and automated testing these days.
|
| It's easy to say this and not cite examples.
|
| I rock perforce and windows agents, requirements of the
| environment, and we struggled very hard to find something
| that beats Jenkins.
|
| One guy even bought an Octopus Deploy license for a year (at
| the tune of a few thousand dollary-doos) and it did
| everything worse than our Jenkins setup.
|
| I actually dislike Jenkins, I'm open to suggestions here.
| devonkim wrote:
| Jenkins to me is great for prototyping certain process
| activities in a very arbitrary set of conventions, but it
| provides a lesson as to why the Java world moved on from
| tools like Apache Ant (XML-based Make) to Maven (convention
| over configuration as a tooling value). While 10 years ago
| such processes were not really standardized nowadays people
| are used to conventions like GitOps with well documented
| processes for how artifacts are built, deployed, and
| operated with container primitives after we've spent a
| _lot_ of time now trying to manage Jenkins installations
| that become incredibly brittle over time and become more of
| a liability than a force multiplier for the organization.
| So people are adopting more specialized tooling instead of
| trying to have Jenkins do everything under the sun from
| building artifacts to deploying them to deploying
| infrastructure with CD conventions.
|
| Things I've found better than Jenkins once an organization
| lets go of over-engineered processes:
|
| - TeamCity - Concourse CI - Circle CI - ArgoCD - Tekton
|
| For Windows build agents I've met plenty of people
| successful with Octopus Deploy that have hated Jenkins and
| your data point is the first I've seen that went back to
| Jenkins.
|
| Perhaps it's worth looking at Team Foundation Server /
| Azure DevOps?
| munk-a wrote:
| I think Jenkins is a very powerful tool but I would strongly
| disagree that it's quite valuable in a startup environment.
| Jenkins has struck me as a philosophical sibling as C++ - it
| offers a plethora of foot guns to the user that can also be
| used to accomplish rather good things. I would advise Jenkins
| to companies with projects that don't require agile responses
| to user requirements where the functionality desired at the end
| is very clear from the beginning - since the times I see
| Jenkins fail the most are when the configuration of jobs on it
| are being updated and reconfigured rapidly.
|
| Jenkins tends to suffer from extremely poor state management
| and experimenting with things can cause a permanent loss of
| value to companies if backups are not properly configured -
| compared with a system like ansible where recipes will grow and
| be regularly committed to a repository Jenkins doesn't have a
| nature version control system and as a result strikes me as
| extremely brittle.
|
| I almost wish the UI for jenkins simply didn't exist and it
| actually was just a whole mess of config files and shell
| scripts so that it could be locked down to a much better
| degree.
| zug_zug wrote:
| I don't care to unpack most of that. But what I will say is
| that I recommend automatically backup up jenkins nightly, I
| accomplished this simply by commiting jenkins_home to a git
| repo and pushing every night.
| oblio wrote:
| These days you can also use the Configuration as Code
| plugin and you can automate the setup, version control it,
| document it, etc: https://www.jenkins.io/projects/jcasc/
| clutchdude wrote:
| As someone who runs a large build cluster at a major corp,
| Jenkins is clunky and outdated. It's a pain to deploy and keep
| maintained. I'd rather stub my toe than figure out why some
| event isn't being properly processed in the bowels of Jenkins.
|
| However, when coupled with Kubernetes, Jenkins is more powerful
| than almost any other CI tool.
|
| You can orchestrate any iteration of build types or actions
| while easily providing the underlying resources transparently
| to your users.
|
| Notice that almost every other CI tool mentioned here is either
| 3rd party hosted or severely limited in some fashion when
| compared to Jenkins.
| Aeolun wrote:
| If you are already running in kubernetes, how is any other
| tool severely hampered if it also runs in kubernetes?
| clutchdude wrote:
| It's partially because I'm not as familiar with other tools
| but they don't expose the ability to interact with an
| entire pod within the build.
|
| Gitlab Action Services for instance is closer to Docker
| Compose.
| aranelsurion wrote:
| Seems to me people hating on Jenkins are not hating on the idea
| of a central automation service/task runner, they hate on
| Jenkins itself.
| jordanbeiber wrote:
| You could use a tool desgned for it and probably be better of,
| since upgrading jenkins and plugins, as everyone knows... is
| not always exactly pain free.
|
| Regarding your batch scheduling solution - take hashicorp nomad
| for a spin: a single go binary and your scheduled job can be
| declared in 10 lines of yaml. Won't miss a beat.
| TomBombadildoze wrote:
| This isn't a batch scheduling problem. It's a monitoring and
| alerting problem. OP needs a _monitoring and alerting
| solution_.
|
| Stop ~~~ engineering ~~~ things and choose the right tool for
| the job.
| mixologic wrote:
| "Solution" and "Job" may be overstating things here. If all
| op needs is monitoring and alerting of a couple of
| machines, then Datadog/New Relic/Ops Genie/Pager
| Duty/Server
| Density/Pingdom/AppDynamics/Loupe/Sysdig/Dynatrace (just to
| name a few in the crowded space) are all likely overkill
| and not worth the cost.
|
| A large portion of the _cost_ of many of these tools is
| spent "choosing the right tool for the job" Figuring out
| what they do, what they do well, where they overlap, where
| the company that makes the tool is headed, how hard its
| going to be to swap that tool out for a better one (or
| cheaper one) - thats a lot of expense in labor and
| training.
| jordanbeiber wrote:
| A script scheduled to test something and notify? Hardly
| "engineered".
|
| I claim using pretty much any tool other than jenkins is
| easier to maintain over time (as illustrated in this very
| article).
| Cpoll wrote:
| I use Jenkins as a glorified cron runner in some contexts. One of
| the things I hate the most is that it's difficult to define jobs
| as code (the 'Job DSL' plugin works in most cases, but if you're
| using certain plugins it's hard or impossible to configure them).
|
| What does everyone else use if self-hosting is a requirement but
| you don't have an enterprise budget?
| Aeolun wrote:
| Gitlab? Hasn't done me wrong yet, even if it feels kind of
| overkill if you need only CI/CD.
| nhoughto wrote:
| Always surprised when I see people still using jenkins, must be
| because of history? you wouldn't choose it today..?
| schoolornot wrote:
| Jenkins has always rubbed me the wrong way. The quality of
| plugins, the dated UI, Groovy. It just feels out of place and
| bloated every time I use it. Being a Java app doesn't help
| either. It reminds me of Jira where you have to hack that death
| to make it "fit in" to common workflows. I'll take any of the
| alternatives over it.
| radicaldreamer wrote:
| Despite everything you said being true, it's still one of the
| few open-source, free solutions in the mobile build space.
|
| I'm wondering if Buildkite or something newer is comparable
| today, but for a long time, Jenkins was one of the only non-
| custom ways of building an in-house iOS/Android build system.
| maccard wrote:
| I've not used buildkite (we don't use git), but we're
| moderately happy with TeamCity. It's definitely not perfect,
| but it's orders of magnitude better than Jenkins.
| MuffinFlavored wrote:
| I could be wrong but isn't Jenkins at its core just "wait for
| a job trigger (git push/commit to remote origin), then run
| some shell scripts"?
| emanuelez wrote:
| Not really. Jenkins allows a very fine control over jobs
| and steps orchestration. I realize it might appear as a
| dated glorified cron, but it's really not. There's nothing
| you cannot do with it, which is more than I can say about
| any other CI system I used.
| theptip wrote:
| Is there something about mobile builds that makes the
| standard Jenkins competitors like GitLab, Drone etc.
| unsuitable?
|
| Even hosted Gitlab (gitlab.com) lets you bring your own
| runner to use whatever arch you need to run the jobs on.
|
| I'm sure I'm missing something here though.
| handrous wrote:
| Large cross-platform native builds _generally_ , not just
| mobile, kinda suck on the usual cloud solutions. The cloud
| stuff's amazing, performs well-enough, and is easy to
| configure if your workflow is... containers, and that's
| all. It gets rough in a hurry once you're much outside
| that, and especially with large (=long build time) apps.
| Many _can_ accommodate native builds, but for anything non-
| tiny you 'll experience slow builds or go over plan limits
| (and see spending skyrocket) in a hurry. Anything that
| requires large assets at build-time, or anything else for
| which it's nice to have persistent state between builds, is
| right out. You'll end up spending a bunch of time (=money)
| re-configuring your build to fit the tools, so performance
| is merely _bad_ and not _astonishingly, uselessly bad_.
|
| You're also more likely, with native builds on cloud CI, to
| run into "yeah, we support that! (I mean... we shipped it,
| anyway, pay no attention to all the bugs and rough edges)".
| Aeolun wrote:
| The thing is, none of that is true. Even for the cloud
| hosted providers (Gitlab at least) you can install your
| own native runner and connect it to your CI system. Then
| run any weird combination of commands you want.
| js2 wrote:
| At Yahoo (Verizon Media), the CI/CD team started with
| Jenkins, then built a UI to manage all the instances, then
| wrote their own engine that replaced Jenkins entirely. It's a
| good product:
|
| http://screwdriver.cd/
| Aeolun wrote:
| The problem is that website presents me almost zero
| information about the product.
| 2OEH8eoCRo0 wrote:
| What are good alternatives?
| maccard wrote:
| We run everything through TeamCity. Not going to claim its
| the best tool in the world but it's certainly the best I've
| used, and it's fairly straightforward to work with!
| sharken wrote:
| TeamCity works well but can be a bit expensive.
|
| It has a Kotlin DSL that is quite powerful and flexible,
| but of course you need to know your way around Java to make
| the most of it.
|
| Custom plugins is also an option if you need something
| special that is not provided out of the box.
|
| Even though I have moved on to Azure DevOps Services, it's
| still good to see that JetBrains have fixed a 10 year old
| feature request.
|
| https://youtrack.jetbrains.com/issue/TW-17939
| throayobviousl wrote:
| I would rather use Jenkins for all of eternity than use
| TeamCity again. TeamCity is probably the worst CI/CD
| product out there.
| regnerba wrote:
| We use TeamCity here as well. Pretty limited in the options
| we have since we have most of our stuff in Perforce.
|
| I definitely prefer TC over Jenkins though. The Jetbrains
| support for TC is also really good and they have been
| helpful whenever we contacted them or opened an issue on
| their public tracker.
| globular-toast wrote:
| Gitlab.
| nobleach wrote:
| Both Gitlab and Bitbucket Pipelines fall under the
| "sufficient" category in my book. I've used both in large
| productions deploys and, they for the most part, get the
| job done. Gitlab has moments where it doesn't work - and
| one needs to go check the status page... and at times
| report an outage. But honestly, I still place it in higher
| regards than Jenkins.
| diroussel wrote:
| Yes gitlab so so much nicer to run on than Jenkins.
|
| Sure there is something missing for more elaborate build
| pipelines, but not enough missing to block you.
|
| The best think about gitlab vs Jenkins is that there are
| no plugins. So it's very clean what is gitlab and what is
| your config of it.
|
| Whereas Jenkins has so many plugins and config for the
| plugins that it's hard to say where Jenkins ends and your
| build pipelines begin.
| trevor-e wrote:
| Bamboo from Atlassian is really good IMO but obviously not
| free.
| orlovs wrote:
| If you are Atlassian shop, then yes. Otherwise, don't even
| bother
| eddieh wrote:
| http://buildbot.net/
| buildbot wrote:
| Oh no my username is a an actual product :/
|
| Though this looks pretty awesome!
| ArchOversight wrote:
| Buildbot as a project has been around since 2003... with
| a username like that I am surprised you hadn't heard of
| it before!
|
| Buildbot was the first CI I ever deployed at a company a
| long time ago. It came before Jenkins!
| ak217 wrote:
| GitHub Actions and GitLab.
|
| I personally found both of those to be paradigm shifts (I
| went from Hudson to Jenkins to Travis CI to GitLab to GitHub
| Actions). Travis CI introduced self-serve test infrastructure
| configuration-as-code, GitLab introduced composability and
| API integration, GHA introduced many more API features (like
| various kinds of event sources) and better composability.
| Each transition also took more hosting headaches and
| compatibility issues off my hands.
| Fordec wrote:
| +1 to this. Travis, Circle and Jenkins are entirely in the
| rear view in this area for me now. My only concern about
| Actions has become that I use it _too_ much.
| onlyfortoday2 wrote:
| why is using something too much a bad thing? LOL
| sakisv wrote:
| Concourse: https://concourse-ci.org/
|
| It takes some getting used to, but I really got to love its
| concepts for resources/inputs/outputs and how they work
| together to get you actually reproducible builds.
| jalk wrote:
| I must admit that I got put off by the lack of abstractions
| for the common use cases. The resources/input/output model
| is so primitive that it almost feels like programming a
| Turing machine.
|
| Edit: Took a look again: The situation might have gotten
| better since I last played with concourse
| theptip wrote:
| Really love the expressivity of the config syntax and the
| domain model in Concourse.
|
| Seems to be a bit harder to integrate with other systems
| (like GH check results), and alas the community is pretty
| small (which probably factors into the first point).
| sneak wrote:
| I like Drone, and run it on my own systems. It integrates
| nicely with Gitea.
| 0xbadcafebee wrote:
| Drone Enterprise is fantastic. Rough around the edges and
| the docs are really lacking, but it's just so god damn
| simple and effective.
| athenot wrote:
| +1 for drone.
|
| Note that it's intended for dockerized deploys, so it won't
| fit if you're not deploying containers.
| sneak wrote:
| That's not true. It runs whatever you want, _inside_ a
| container with your code. You can deploy from within that
| container however you like. You can run tests that just
| pass /fail, and throw away the container result. You can
| ssh or rsync things from within the container to other
| remote hosts, and throw away the CI container. It uses
| containers to ensure build consistency, but you don't
| have to use them to deploy containers just because your
| jobs run in them.
| 0xcoffee wrote:
| Links for the lazy:
|
| https://www.drone.io/
|
| https://github.com/drone/drone
| encoderer wrote:
| We (Cronitor) are seriously considering diving in here. People
| already install Cronitor to remotely monitor their background
| jobs, we have a lot of the UI and data platform challenges
| solved, and we are building a control plane to allow you to
| securely invoke your jobs remotely (via polling). I'm not
| totally convinced we should go this path vs better and deeper
| monitoring and metrics capabilities, but we're contemplating it
| because, honestly, Jenkins needs to be replaced.
|
| (Readers, if that sounds interesting, and you have some
| Kubernetes and Golang experience, we are hiring!)
| lelandbatey wrote:
| To throw my hat into the ring: I've found the best "runnable
| jobs" / "digraph of jobs" system I've ever used to be Gitlabs
| built in CI system. The UI is nice, it's got tight integration
| into our existing code workflows meaning there's a close-but-
| not-too-close association between code and jobs meaning it's
| easy to say that "the code and the jobs running that code live
| 'in the same place'" if you want that, and it still provides
| all the pipeline features you'd want if you want to use your
| CI/jobs as a cornerstone of your infra; features like the
| ability to schedule pipelines, to trigger pipelines via other
| pipelines, remote triggering of pipelines via cURL request,
| pipeline DAGs for a 'make-like' build experience, etc.
|
| I've absolutely loved using Gitlab CI for the last several
| years, and I highly recommend it.
| epage wrote:
| My experience is the opposite. Gitlab has a lot of features
| but they are all half-baked. Ever time I try out a feature, I
| feel like I get bit for doing so.
|
| A couple of cases
|
| - At first child pipelines couldn't access artifacts from the
| parent job. Now you can opt-in to it but nothing enforces
| sequencing
|
| - Some features only work with "needs" (getting parent
| pipeline artifacts) while others only work without it
| (depending on an entire stage)
|
| - When a job you "needs" doesn't exist, instead of having a
| sane behavior (e.g. skip yourself), the pipeline errors.
| Correctly propagating and maintaining all of the rules
| correctly in a moderately complex pipeline is ... difficult.
| nawgz wrote:
| > At first child pipelines couldn't access artifacts from
| the parent job
|
| Why would you use the tool like this? Make jobs / blocks
| that generate artifacts, and then chain them together into
| a workflow so there isn't hierarchical issues.
|
| Now, maybe I am working in a simplistic world, but: the
| only time I have parent / child relationships is when a
| repo or service is built and another pipeline is invoked to
| run integration tests or deployments. What situations have
| lead you to needing to pass artifacts between pipelines and
| not job stages?
|
| > When a job you "needs" doesn't exist, instead of having a
| sane behavior ... the pipeline errors
|
| You and I may work in very different worlds, but I
| personally would be quite offended if a dependency graph
| workflow executor saw a node with an explicitly specified
| dependency missing and proceeded to NOT error.
| diroussel wrote:
| I think there are missing concepts and features in
| gitlab. And the advertised features don't really match
| reality. But it's still way better than Jenkins, as you
| can work around any issues and it's clear how it works.
| It's just a yaml file.
| [deleted]
| NikolaeVarius wrote:
| I hate jenkins groovy so much
| nobleach wrote:
| Yeah, Jenkins can be really unpleasant. Groovy as a language
| isn't so bad - I just don't use it anywhere else. So... like
| a few other languages, I have to look things up every time I
| need to mess with it.
___________________________________________________________________
(page generated 2021-06-03 23:00 UTC)