[HN Gopher] The Continuous Delivery Test
___________________________________________________________________
The Continuous Delivery Test
Author : sourceless
Score : 71 points
Date : 2022-07-09 10:31 UTC (1 days ago)
(HTM) web link (sourceless.org)
(TXT) w3m dump (sourceless.org)
| alkonaut wrote:
| Extremely web centric list. Any other development (mobile,
| desktop, games, embedded, ...) will find a lot of these steps
| very weird.
| ChrisMarshallNY wrote:
| The whole idea of "Continuous Delivery" is not always one that
| meshes well with some shrink-wrap workflows.
|
| I worked for hardware manufacturers, for most of my career, and
| software was always just a "sidekick" to The Main Show. We just
| got the "Supporting Actor" nods.
|
| I'd say that 90% of HN seems to be Web/SaaS (and, these days,
| crypto), which is an excellent workflow; just not the one I do.
| Nowadays, I have some integrated stuff, but it's mostly native
| iOS/TVOS/WatchOS/MacOS deliverables.
|
| CD is nice, but I feel that CI is better, for a team. Even
| that, is overkill for the way I work. I'm spoiled, I tend to
| work alone, or very loosely-coupled. That gives me a different
| workflow from what many experience. I had to spend a couple of
| years "un-learning" a lot of stuff from my Big Team days.
|
| The way I work results in astonishingly good stuff, very
| quickly, but the scope is much narrower than what a lot of
| folks here, do.
|
| As such, I find little utility in telling others that the way
| they do things is wrong, and that they should be doing it my
| way. I do talk about how I do things, but I'm not judgmental
| about it.
|
| I do feel that software quality, in general, is fairly
| problematic, but don't claim to have the silver bullet. I know
| what works for me, and I try to stay in my lane.
| 8n4vidtmkvmk wrote:
| i work on a very large team and solo for my side hustle. love
| working solo, no red tape to fight. no waiting 24 to 48 hours
| for a teammate to review my code per iteration and up to 24
| more for it to deploy. just signed 2 contractors on for my
| solo project though and its really messing up my process. all
| the overhead is so high I'm not even sure it's worth it even
| if they were free.
| simongr3dal wrote:
| I like to imagine it as a sort of volume vs surface area
| problem.
|
| The amount of service you can deliver to costumers is your
| surface area, but the amount of work you put in only
| contributes to the volume.
|
| As the number of employees grow you will necessarily be
| less efficient, but there aren't really any other great
| ways to gain the required surface area.
| wahnfrieden wrote:
| there are perhaps interesting options with bringing in
| partner-level labor instead of buying someone's time-in-
| seat and the relative personal investment involved in
| each
|
| the big point glossed over is that wage labor, where the
| value they produce is taken from them, isn't a great
| incentive for great work. that's imo a far bigger issue
| than fundamental challenges with collaborating, and why
| so many managers need to continuously trick their
| employees into productivity
| kqr wrote:
| Really? I challenge you to find any items other than 7 and
| perhaps 8 that don't apply to the types of development you
| mention.
|
| Essentially the only thing that's different with the types of
| development you mention is that "final deployment to
| production" looks different, as it usually involves more or
| less physically transporting artifacts to your customer.
|
| But the rest is just the same. You should be doing trunk-based
| development, practise something like code review, gate on
| integration tests, feature flag, spin up virtual production
| clones, etc.
|
| In other words, the fact that getting software to your end
| users is a clumsy process does not preclude you from doing
| every other step of the process with short feedback loops.
| maccard wrote:
| Using distributed version control isn't applicable to most
| people working in games.
|
| 9) is also debatable - requiring someone to clone the entire
| application to make an infra change to a testing environment.
|
| Adding tickets to commit messages isn't necessarily a
| requirement - some work (at least in my area) is prototypey
| and maybe ill defined (the task might be to define it).
|
| Being able to deploy from your own machine is a double edged
| sword; the situation you need this is an absolute last
| resort. Enabling deployments from dev machines means
| credentials to environments, write access to infra, and
| likely skirting around normal processes.
| marcosdumay wrote:
| What?
|
| #2 and #5 exist only to support #7.
|
| #4 usefulness is completely dependent on the context.
|
| #9 is a joke, right? It's an anti-pattern for most of the web
| based SaaS applications, and poisonous to anything that isn't
| SaaS.
|
| And #12... Try that in a large regulated company and people
| may quite as well end up in jail.
| kqr wrote:
| #2 is to improve quality, shorten feedback loops, and
| simplify merges. You get those benefits whether or not
| you're able to push things to your customer at will.
|
| Both #2 and #5 really just set a quality bar high enough to
| reduce long-term maintenance costs. I don't see why you
| wouldn't want that in a non-web scenario.
|
| #12 is not talking about deploying to production, and I
| don't see why you shouldn't be able to deploy to a dev
| environment from your machine. It's nice not to have to run
| everything locally. Even for non-EU companies.
|
| So aside from #9 which is of questionable utility for any
| type of development, no, I'm not convinced you've pointed
| valid examples.
| choeger wrote:
| Pretty much everyone with more than one production environment
| will. Imagine developing Firefox. How do you do a "self-healing
| deployment" automatically whenever trunk changes?
| hoskdoug wrote:
| Indeed. Does your deploy process self-heal? Err, no, I need to
| wait two days for App Store review.
| etamponi wrote:
| 7. 7! This is by far the one that resonates with me the most. But
| for some reasons SREs think the opposite is true: the longest it
| takes to get to prod, the better. What a pain.
| bhawks wrote:
| > 11. Do you include ticket IDs in your commits or branches?
|
| This is one of the things that sound 'good' on the surface and
| are totally worthless in practice (especially when practicing
| minimal commits).
|
| In the wild this devolves into a meaningless umbrella issue
| (Deliver Feature Foo) or 1 liner issues that sometimes did not
| even match the pr contents.
|
| Just use the commit message and in pr review enforce commit
| message norms.
| whakim wrote:
| It's incredibly easy to simply add the ticket number to branch
| names (or commits if you're rebasing your branches when you
| merge). Many times the ticket isn't helpful, but "just make the
| commit/branch self-explanatory" ignores the fact that non-
| developers - product folks, designers, etc. - are far more
| likely to leave useful context in the ticket itself.
| kqr wrote:
| When there is a ticket, yes, adding the number is trivial and
| probably worth it. Where the rule breaks down, in my opinion,
| is when you have to create a ticket and replicate the commit
| message in it any time you stumble over a code
| maintainability issue and fix it in stride.
| dallbee wrote:
| This is the practice we use - link to a ticket if it
| exists.
|
| There's nothing wrong with fixing a bug you discover upon
| inspection. Forcing tickets for every change discourages
| code maintenance.
| crdrost wrote:
| This is also where automated code quality checks can
| shine. You can say something like, if it's more than 5 or
| 10 lines we expect a ticket ID, otherwise we don't.
|
| See DeGrandis's "Making Work Visible," the hazard of
| "link to a ticket if it exists" is that sometimes this
| causes us to treat tickets as an external process, but we
| absolutely want to surface tickets that track our cleanup
| of tech debt and any other procedural tasks that we are
| stuck doing.
|
| If automated tooling sounds nice but you are at the sort
| of place where "there are three priority levels: hot, Red
| Hot, and DO IT NOW," this can also be part of a pull
| request template. One extremely effective template we
| used at a previous job just had quick checklist and you'd
| check off for example "I tried this code on my dev
| cluster." Caught a lot of "whoops I am moving _too fast_
| and need to slow down a bit" issues, hah!
| lelandfe wrote:
| Yes, exactly. I treat my commit history like I'm preparing
| for a detective to investigate years down the line :)
|
| Great commit messages are just one part of the timeline.
| Ensuring there's a link between the commits/PR and the ticket
| lets the future detective get more information (frequently,
| that's "why did we do this").
| lawl wrote:
| I disagree.
|
| I have dealt with bugs that were thought to be fixed a long
| time ago, only for them to mysteriously pop up again later.
| Ticket is created and the old ticket is linked.
|
| It's really helpful to see what was done 2 years ago, including
| the attempted fix that is now still live in the code base but
| apparently doesn't work properly.
|
| That said, as always, it's not black or white. There are
| definitely cases where it doesn't make sense, but I don't think
| you should call it 'totally worthless'.
| wodenokoto wrote:
| Have one branch for every story in Jira has been incredibly
| helpful in teams I've been in.
|
| You can check in git if work has actually begun, and you can
| get from and old commit to Jira by virtue of the merge commit.
|
| I don't know how you'd do it in a non-mono repo setting
| kqr wrote:
| There's a difference between "every story has a branch" and
| "you need to create a story for every commit" which is what's
| implied by TFA.
| crdrost wrote:
| I like this, but it violates one of my norms, haha.
|
| Stories can last for a while, but long-lived feature branches
| have to be kept in sync with the main branch, otherwise they
| develop nasty merge conflicts and you are not deploying
| continuously.
|
| I think the better solution is for your versioning and your
| issue tracker to be integrated: we still expect every
| developer to merge to mainline one or twice a day[1], keeping
| merge conflicts shallow... But they should mention what issue
| they are working on in the commit, and the integration dumps
| that information into the tracker automatically.
|
| [1] This is not hard to make safe, though it makes a lot of
| devs skittish at first. See Three-Flow for a non-CD version,
| https://www.nomachetejuggling.com/2017/04/09/a-different-
| bra... as a stepping stone. One key thing is that code review
| needs to become much lighter than it is at many
| organizations, "I don't care how you did it as long as it can
| be turned off with a feature toggle and does not break prod
| or introduce security vulnerabilities when I merge it: those
| key things are what I am looking for."
| kqr wrote:
| > I don't care how you did it as long as it can be turned
| off with a feature toggle and does not break prod or
| introduce security vulnerabilities when I merge it
|
| Does this not lead to poorly-written features that drag
| down future development yet cannot be turned off because
| they have grown important for the business?
|
| At least that's my experience with not regulating at least
| a basic level of code quality in the review. What are you
| doing differently to prevent that?
| theptip wrote:
| I think it can be useful to encourage (not require) this
| practice since auto-linking between Jira and GitHub/Gitlab is a
| nice feature. Jira can now show you what releases a given
| feature is deployed in, etc.
|
| If you are in a regulated environment or selling to large
| enterprises you may find that having a SDLC policy of "no
| changes without ticket to track" is a more easily defensible
| control against unauthorized code changes (you'll get asked
| about this in SOC2 and security questionnaires). The policy
| "anyone can make a change if they find a reviewer to approve"
| might not float.
|
| Mandating ticket IDs can be really annoying though, for example
| if I want to make a few-line no-op refactor to clean up some
| code, I don't want to have to create a ticket. So it's
| definitely a tradeoff.
| kqr wrote:
| My previous team used to have ticket IDs on every commit but
| just decided to drop that requirement for commits that are
| truly self-contained and documented properly in the commit
| message.
|
| That approach makes a lot of sense to me. Link the commit to
| the ticket when it enhances future reading, but not out of
| blind application of principle.
| Merad wrote:
| I've yet to work at a company that has managed to figure out a
| strategy to get the majority of devs to write clear and
| descriptive commit messages the majority of the time. At least
| if they include the ticket number I can look at the ticket to
| understand what they intended to do. Of course whether they
| actually did it or not is another matter.
| winter_squirrel wrote:
| KronisLV wrote:
| > This is one of the things that sound 'good' on the surface
| and are totally worthless in practice (especially when
| practicing minimal commits).
|
| I feel like I disagree here.
|
| Turning on Git Annotations in any JetBrains IDE and seeing who
| made changes, when and which Jira issue necessitated these
| changes right there is pretty useful. All of the sudden, you
| can easily say: "This method had been changed in 5 different
| commits in the last 3 years, what I probably need to read
| before making my own changes are issues ABC-123, ABC-236,
| ABC-455, ABC-855 and ABC-1233, to understand the business
| context and historical limitations behind all of this."
|
| > Just use the commit message and in pr review enforce commit
| message norms.
|
| I fully agree with this, but unless you squash your commits or
| rebase, finding out which MR/PR a particular set of changes
| belongs to isn't always immediately doable, especially when
| looking at things locally instead of the GitLab/GitHub UI.
|
| Furthermore, the barrier of entry is higher: I always describe
| my changes and add a bit of context in the merge request/pull
| request description, sometimes even with GIFs or MP4 video
| demonstrations of what happens as a consequence of them. And
| yet, the majority of other developers have no desire to do that
| - I've regularly seen people leave those empty, and even commit
| messages are sometimes like: "try fix" instead of "make Ansible
| ensure that file permissions are set to have the newly created
| directory be readable by the group that will run the app".
|
| In contrast, everyone is capable of adding a simple identifier
| to the issue management system and there are very few arguments
| that they can make against adding a few characters to the
| beginning, as opposed to: "But adding descriptions would take a
| lot of time and slow down development, I don't think we really
| need those because you should be able to read the code and
| understand everything."
|
| Sure, you can enforce it in a top down manner, but it can be
| like pulling teeth at times, so you might as well ensure that
| _at least_ the lowest common denominator is in place, before
| trying to find better approaches.
|
| Exceptions to this might be small changes that don't really
| correspond to an issue, then you have two choices:
| - create an issue yourself so the work doesn't become
| untraceable in the issue management/billing system you use
| - just leave a short text description without issue identifier
| xorcist wrote:
| > ABC-855 and ABC-1233, to understand the business context
| and historical limitations
|
| This should be what a commit message is. Commit messages
| should describe the rationale behind the change.
|
| Ticket numbers are great for linking commits together. But
| they should not have to be relied on to find out why a change
| was made. Ticketing systems may come and go but commit
| messages persist.
| KronisLV wrote:
| > Ticket numbers are great for linking commits together.
| But they should not have to be relied on to find out why a
| change was made.
|
| In theory I agree. In practice, I've yet to see it work out
| that way - there are always discussions with clients/other
| departments in the ticketing system which will never appear
| in the commit messages. References to logs,
| screenshots/animations of the issue and other things that
| give the _full_ context to why that change matters.
|
| That's also why I believe that code alone is not enough -
| you need code comments to explain not what the code does,
| but why it does it that way etc., it's just that ticketing
| systems can provide even more of the surrounding
| information which also doesn't fit either in commit
| messages, or in code comments.
|
| > Ticketing systems may come and go but commit messages
| persist.
|
| I'd say that everything comes and goes, but in most cases
| you won't have to worry too much, since most information
| will be kept around in some capacity, like migrating SVN
| revision messages to Git commit messages (though even that
| wasn't always quite possible, in the case of non-standard
| repository layouts, since most migration scripts broke)
| etc.
|
| Though with how deeply integrated some companies out there
| are with solutions like Jira, I doubt they'll ever rid
| themselves of it.
| satyrnein wrote:
| I'd like to see a workflow that meets all the criteria. I was
| surprised to see the item about deploying directly from your own
| machine; that seems to contradict the other goals which point to
| automated pipelines that deploy on merge.
| TheP1000 wrote:
| This is a very common pattern especially with the cdk. Having a
| locally deployed stack makes development in the cloud so much
| easier.
| kqr wrote:
| I feel like a lot of people misread that point. It's not about
| deploying to production (that sounds nuts!) It's about spinning
| up and deploying to a full dev environment. This is in part to
| not have to run everything locally, but mainly to exercise
| deploy automation regularly.
| dijit wrote:
| This feels like yet more cargo-culting.
|
| Even the first item on this list is hotly contested, with
| Facebook/Microsoft/Google and Co. using centralised mono-repos,
| google itself using a server model similar to Perforce called
| Piper.
|
| To be clear here, I'm not saying they're right, but with it feels
| un-scientific to make a blanket statement that they're wrong.
| theoldlove wrote:
| You can easily have a monorepo with a distributed control
| system.
| dijit wrote:
| Yes. But I'm referring to the article:
|
| > 1. Do you use a distributed _version control_ system?
|
| Edit: the parent originally said "distributed build system"
| Morgawr wrote:
| Just because it's a monorepo it doesn't mean it's
| centralized?
| michaelt wrote:
| Yes. It's the accompanying suggestion of using github or
| gitlab that means it's centralised.
| gregmac wrote:
| I'm not sure what you're getting at here. Mono-repo is
| entirely a different concept then centralized (eg: SVN) vs
| distributed (eg: git).
|
| I'd also seriously question an org not using git,
| specifically, despite claiming "cargo cult". Literally
| every developer knows it, it's got a huge choice of
| tooling, and every service/tool (CI/CD, issue tracker, etc)
| has an integration. IMHO, you better have a damn good
| reason to be on something else to justify the headaches
| involved of being _not_ git.
| fmajid wrote:
| Even Microsoft manages Windows using Git today
| maccard wrote:
| > I'd also seriously question an org not using git
|
| That's a very damning statement. Many places use
| perforce, for example.
|
| Git doesn't handle binaries and frankly git struggles to
| scale. Try checking out a _large_ git repo and running
| some common operations on it.
| chrisoverzero wrote:
| Read this as "Do you use git?" This is opposed to svn, cvs,
| something home-grown, or passing thumb drives back and
| forth among developers.
| maccard wrote:
| Hg, svn, perforce, plastic are all feasible alternatives
| to git.
| UweSchmidt wrote:
| > 9. Does your Infrastructure as Code live alongside the service
| it hosts?
|
| That means minor changes to some test infrastructure have to go
| through the strict review process, because the gitlab-ci.yaml is
| in the part of the main code. Last time this happened to me I
| found it annoying, and I don't think the code quality guardians
| care about some CI config anyway.
| xorcist wrote:
| I will be first in line to suggest to _not_ split repos unless
| required. It is my experience that deployment, admin and node
| provisioning (and everything that is generally put under the
| not always useful IaaC umbrella term) is one of those
| exceptions. Often documentation is, too.
|
| It all depends on what the release flow looks like. A useful
| rule of thumb when components can share a repository is when
| the branch model is identical. If that's the case for codified
| release and provisioning processes and the code that is
| shipped, then it is likely that you are not taking full
| advantage of what that deployment and provisioning code can do.
| If not anything else, then at least test versions must deploy
| production code, and test code deploy production versions.
|
| Again, in my experience across multiple organizations, it is a
| good idea to keep that code both forwards and backwards
| compatible. It should be natural that is sometimes says things
| like "if build number greater than x then set parameter y".
| While it may feel a bit dirty for some, that type of logic is
| what deployment code _does_ , and it is much more maintainable
| than keeping branches around for the same result.
|
| If I get to say one thing to new devops/release engineer type
| people, that's probably it. That, and the value of clear and
| concise commit messages. Which goes double for this type of
| code.
| davidmurdoch wrote:
| Changes to CI config for release automation are probably vetted
| and manually tested more than anything else in the projects I
| work on.
___________________________________________________________________
(page generated 2022-07-10 23:01 UTC)