[HN Gopher] DevOps is broken
___________________________________________________________________
DevOps is broken
Author : davydog187
Score : 372 points
Date : 2022-10-20 14:16 UTC (8 hours ago)
(HTM) web link (blog.massdriver.cloud)
(TXT) w3m dump (blog.massdriver.cloud)
| CraigJPerry wrote:
| Hard disagree but Mandy Rice-Davies applies.
|
| Some interpretation of DevOps may be BS, e.g.
|
| >> Need a database? File a ticket with DevOps.
|
| That does seem like a bit short of the mark. E.g. that should be
| automated provisioning in 2018, never mind in 2022.
|
| Counter points as to why DevOps is not BS:
|
| Today there's just an acceptance that you version your code in
| VCS, it wasn't always that way. "Hey, this doesn't look right, i
| know you said you based your change on gui-app.latest-final2.zip
| but was that Steve or Laura's version of latest-final2?". If you
| didn't work through this period you'll struggle to believe how
| common it was for shipping products not to be fully up to date in
| VCS or not to have VCS at all. DevOps changed this.
|
| Continuous integration? No there were people hired as "merge
| masters" or "build managers", i promise i am not making this up.
| DevOps changed this, the idea that you wouldn't do at least CI or
| perhaps CD is unexpected today.
|
| Deployment automation? Sure, you email it to the ops team, send a
| few more emails with attachments late on Friday with ammendments
| and hope that they deploy the right one. Automated deployment as
| far as the developer was concerned. The ops person on the other
| end? Sure they had a batman's belt full of hand crafted tools and
| scripts but it was definitely pets not the cattle DevOps has made
| us strive for.
|
| Testing automation? The testers sit on level 3 not next to dev on
| level 4, there's about 4 banks of desks over by the cupboards,
| that's all the testers. They have lotus notes databases with
| checkboxes to confirm when they test something. If they find an
| issue a regression report will arrive with the dev team in under
| a week.
|
| I could go on an on. Platform teams are great when used
| correctly. You can say something similar about DevOps.
| mikkergp wrote:
| >> Need a database? File a ticket with DevOps.
|
| > That does seem like a bit short of the mark. E.g. that should
| be automated provisioning in 2018, never mind in 2022.
|
| Serious question, as I think this has been part of my thought
| process in the challenges of platform engineering: what does it
| mean to automatically provision a database?
|
| I can think of lots of different examples that are insufficient
| in one way or another (I think I'm mostly talking UX here, and
| how many questions the user has to answer in one way or another
| / infrastructure as code, not should the user have to apt-get
| install postgres, which should I think rather obviously be
| automated.) But if infrastructure as code is defined as
| automation, this can conflict with the developers who don't
| want to learn terraform and thus still leads to "file a ticket
| with devops"
| CraigJPerry wrote:
| If we're talking in the context of an organisation which has
| or is considering having a devops team and / or a platform
| team, like in the article then yeah i agree you're not going
| to be apt installing a db.
|
| There will be some mandated platform choices so that the org
| has a fighting chance of managing complexity and
| knowledge/skills within the db platform team.
|
| I'd expect you'll have to specify in a web form or maybe an
| api call that you need a document db / oltp / olap and you
| want it in region A and of size medium and you want an
| indefinite expiry date on this lease. I wouldn't expect to
| have much more freedom than that, e.g. backups, point in time
| restore options etc will all be standardised. I'd expect to
| be immediately given an appropriate pre-built instance that
| had been provisioned and kept warm waiting for the next
| request of a db of these specs.
|
| That allows the developer minimal friction but also allows
| the db platform team to say we provision a postgres for
| document db use cases not a mongodb because it simplifies
| platform ownership (and we just mail a link to some docs
| explaining how to use a postgres as a document db to the user
| account that requested the instance).
| [deleted]
| l0b0 wrote:
| Just some anecdotes around VCS use...
|
| At uni back in ~2000 was the first time I ever heard of version
| control. It was a bit of a curiosity, nobody seemed to be using
| it.
|
| In 2004, engineering at the place I worked (you've heard of it)
| were developing PL/SQL _in production_ with no VCS.
|
| In 2007, I worked in IT at the same place. They were developing
| in SharePoint without VCS. When I asked if I could use VCS, I
| was told to use a 1996 or so Visual SourceSafe which was more
| temperamental than Subversion.
|
| The _first_ place I worked that used VCS was in 2008, and only
| because the devs had started using it _against_ the wishes of
| the principal developer.
|
| The world moves fast and slow at the same time.
| projektfu wrote:
| Funny, I thought DevOps was introduced as a way of having the
| specialists be part of the same team, so that the dev side would
| keep ops in mind and the ops side would be getting dev help in
| automating. Creating a separate DevOps team seems like a manager
| reading an article and implementing without understanding.
|
| The problem in a lot of orgs is having various priesthoods that
| have their own goals that aren't aligned with other teams/the
| business. For example:
|
| - hardware purchase
|
| - software purchase
|
| - DBA
|
| - ops/infra
|
| - networking (firewalls esp)
|
| - security
|
| - HR/recruiting
|
| - business analysis
|
| - front end/design/ux/dev
|
| - back end dev
|
| - technical documentation
|
| All of these are roles and they can have specialists, but you
| probably want them aligned and rewarded for keeping the business
| running, not serving specific measurement goals like uptime, to
| the detriment of selling product. These priesthoods often have
| their own religious virtues that they espouse, like 3rd normal
| form or low TCO, that they pursue in absence of directions and
| understanding of their role in the business.
|
| It's easy to see the problem but wicked hard to prevent it or fix
| it. I have a small organization and it's difficult to get people
| really on board with a vision.
| nikolay wrote:
| Very shortsighted article ignoring a lot of responsibilities
| developers don't want to assume and want somebody else to be on
| the hook for costs, secruity, breaches, on-call rotations, SOC-2
| compliance - you name it. With freedom comes responsibility!
| AzzieElbab wrote:
| DevOps is like everything else. It is what you make out of it
| tootie wrote:
| A lot of this rings very true but I think it's missing the actual
| nobel intent of DevOps that is still true even if it's been
| hopelessly corrupted. He touches on a lot of the counter examples
| that lead to the rise of DevOps. Things like ops/admin teams
| making themselves a gatekeeper or bottleneck to delivery. That's
| a classic problem in traditional IT departments. Dev is
| incentived to ship features, Ops are incentived to avoid
| downtime. The two goals conflict. DevOps says we all work
| together to ship features that don't cause downtime. That's a
| philosophical aim and not a new discipline. The maturity and
| sophistication of the tools and practitioners is orthogonal. If
| you hire a team and set them apart from Dev and give them an
| incentive to avoid downtime and call them DevOps then you've
| accomplished nothing. If you hire SREs or platform engineers or
| whatever else you call them and align their incentives to dev and
| dev to them then congratulations.
| imaurer wrote:
| I feel like I need some definitions to understand this article
| and discussion. What is dev? What is ops? What is devops? What is
| system administration? What is sysadmin?
|
| I'm 47 and what is described seems like the old problems that I
| thought devops solved.
| travisgriggs wrote:
| DevOps is a way of creating a specialization label so that
| people can differentiate each other and create more roles to
| fill at a staffing level.
|
| Since you're 47 (I'm 52), it's the equivalent of hiring an
| "AutoConf engineer" in the late 90s.
| thealienthing wrote:
| I'm also confused by this article. Perhaps this is more
| applicable to companies that develop software that is very
| tightly coupled with the web and cloud services. I work in the
| embedded industry and from my understanding (as naive as it may
| be), DevOps has simply meant the automation of building,
| testing and releasing our products to a customer accessible
| endpoint and is something that is life changing. It massively
| cuts back on manual human interaction getting our software to
| our customers. Perhaps since our product is not nearly as
| monolithic as most products that really hook into DevOps
| processes that can match the actual product in terms of its
| scale and complexity, DevOps is something wonderful that I
| would never define as 'bullshit'.
| AlgorithmicTime wrote:
| mherdeg wrote:
| How many companies are there where a Dev team writes code and
| "throws it over the wall" for Ops to deploy and maintain? I hear
| people talking about this as an anti-pattern all the time, but I
| haven't been in enough workplaces to see what this looks like
| when it happens recently.
| lovehashbrowns wrote:
| We had this pattern in a previous role. Essentially, there was
| a repo with yaml config files in it. A dev team would write
| their code they wanted to deploy, configure the yaml files, and
| infrastructure would be deployed in AWS. That infrastructure
| was owned by the operations team.
|
| This caused as many disasters as you'd expect. For example,
| when something breaks (like ECS, or a bad AMI is deployed), the
| dev team is stuck waiting on the operations team to fix it
| because the infrastructure is owned by the ops team.
|
| I was part of transitioning to a different model where the same
| general concept applies; a bunch of yaml leads to
| infrastructure being deployed. But that infrastructure was
| fully owned by the dev team that deployed the code.
|
| It only made things generally easier, because then if the
| deployment code broke (i.e. the code responsible for converting
| the yaml files to infrastructure, like terraform modules), we'd
| have to fix it.
|
| Ultimately it was just a fancier and more guided version of
| Kubernetes.
| [deleted]
| cies wrote:
| DevOps to me is:
|
| * Infra as code
|
| * Infra code sits in a repo
|
| * Infra code is managed by the (senior) devs
|
| * No separate ops team, or ops role, is needed: it's a
| responsibility of the devs now.
|
| * Monitoring production: also by the devs -- if an issue arises
| "we" take turns in solving them as devs.
|
| DevOps is no longer managing your own hardware, but using
| hardware-behind-an-API in the cloud.
| Lutger wrote:
| DevOps is bullshit in the same way that Agile is bullshit. So,
| it's actually not at all, but just has become so fashionable that
| the term is sufficiently abused to become almost useless.
| g051051 wrote:
| 100% agreement. Agile and devops have done more to wreck
| software development than anything I've seen in more than 30
| years in the industry.
| mikebenfield wrote:
| I think you misunderstood your parent post's point.
| g051051 wrote:
| Rereading it a few times, I think you're right.
| jbverschoor wrote:
| Bigdata?
| drewcoo wrote:
| Agree. Both devops and agile were about empowering developers,
| the ones who actually make the products people use. And then
| managers and management consultants got hold of the terms and
| mutated the ideas and they ossified into "best practices" that
| get in everyone's way.
| qzx_pierri wrote:
| OP that newsletter button on my mobile browser is covering up
| words in the article. What the fuck.
| davydog187 wrote:
| Thanks for the heads up! Its gonna get turned off in a moment
| 0xbadcafebee wrote:
| DevOps is still a great idea, just like Lean and Agile are a
| great idea. But a great idea isn't enough. A million people have
| "great ideas" for businesses all the time. How many of those
| succeed? You can't just have a great idea, you have to execute
| really well on the great idea.
|
| Most people don't know what DevOps is. Of the very very few that
| do know what it is, they are powerless to get other people
| onboard with the idea, because people are lazy and don't want to
| learn things or change how they work. Even if DevOps is great, if
| the executives in your company don't _force it down everyone 's
| throat_ with business policies, training, etc, nobody will ever
| actually _do_ it, because nobody really _wants_ to.
|
| Platform engineering is not gonna solve what DevOps is failing to
| solve. It's just another silo. "Let's empower developers" sounds
| great. But developers still lack most of the knowledge to
| maintain a complex system. Give them a million super-powered
| tools and they will still screw things up. Most developers I meet
| today don't even understand the concept of DNS. That's not
| exaggeration - they literally don't understand how hostnames
| work, record types, zone delegation, authoritative records, ttls,
| much less propagation or transfers. And you want to, what, give
| them a fancy tool to change the system that they don't
| understand? You still need operations, in any business, not just
| tech. Somebody has to be paid to care about the boring shit that
| keeps the business working. There is no way to automate away that
| responsibility, in any complex system in the world. A platform
| eng team is just adding another team on top of the Ops team you
| will always need.
|
| What would _actually_ solve a lot of this - and nobody is going
| to like this - is boring-ass business management best practice
| from the 50s. W.E. Deming. Lean. Six Sigma. The stupid shit that
| MBAs nerd out over? That stuff works. High-performing businesses
| that don 't just pay it lip service, but _actually_ do PDSA,
| _actually_ train their workforce, _actually_ continuously improve
| their process, and make better business outcomes. But who among
| the tech nerds wants to listen to that? They just want to play
| with their toys and have no responsibility. "Build me a platform
| so I don't have to use my brain."
|
| People have been studying businesses for the better part of a
| century. There is no easy way out of the morass. No single team
| or tool or paradigm will make things better. Until you consider
| _everything_ , holistically, and put into place a barrage of
| different solutions, and actually teach people to do their jobs
| better, the actual outcomes of the work won't improve.
| zoomzoom wrote:
| Couldn't agree more top-level.
|
| The other thing that this misses is that the deployments to the
| cloud are only half the battle. CI/CD pipelines, Dev
| environments, and other SDLC phases like planning and testing are
| just as important as terraform. Making the infra as code better
| and more reusable is a great step in the right direction but only
| part of the puzzle to making software development workflows
| better in the big picture.
| mschuster91 wrote:
| One thing I miss:
|
| _Most developers have no fucking clue how Linux, networking,
| storage or whatever works under the hood_. They know how to
| develop whatever stack you 're at, but stuff like latency, packet
| loss, redundancy factors, backup policies, monitoring or other
| classic ops topics are completely beyond the comprehension of 99%
| of developers.
|
| "DevOps" usually means some C-level execs say "fire the expensive
| neckbeards that have the time to properly understand a system"
| followed by them saying one of
|
| - "oh fuck, someone managed to compromise a service and because
| no one knows what the fuck firewalls are / Kubernetes doesn't
| come with ones OOTB the hacker got complete control of
| everything"
|
| - "oh fuck, production is down because someone fat-fingered in
| Elastic Beanstalk which recreated the environment, dropped the
| RDS database and there were no backups" (I've been personally
| bitten in the arse by their definition of "recreate" - all I
| wanted it to do was to replace the damn EC2 instance)
|
| - "oh fuck, we're seeing insane AWS bills because someone DDoS'd
| us and no one created a cost limit or a sensible limit for
| autoscale or a simple dumb monitor that alerts someone"
|
| - "oh fuck, we're seeing an insane AWS bill because someone got
| his AWS access credentials stolen and some shithead spun up a ton
| of g3.xlarge instances to mine shitcoins"
|
| Other C-level execs see constant issues between "ops and dev
| teams" because their team leads are friends of the silo model and
| decide that instead of getting rid of the dumbass managers
| they're getting rid of the ops team because "cloud", with the
| exact same result.
| travisgriggs wrote:
| I have bewilderingly tried to discern why software development
| continues to grow more and more complex.
|
| It wasn't always like this. There was a time when we talked about
| languages and OSes and libraries as if they made a difference on
| how much you could get done with as little people and cognitive
| load as possible (the claims were very much overrated, but the
| point was we acted like it mattered).
|
| And then it started ballooning. It seems to me that much of that
| coincides with a lot of new money being dumped into the economy,
| and software moving past the necessary-evil-so-how-do-I-drive-
| down-my-costs and on to the gold rush of you must have a web app
| for your service that rivals a video game in complexity and
| visual finesse. It seems to me that as long as their is so much
| free money sloshing around venture capital funding, that it was
| inevitable that the process of making software would complexify
| to soak up the extra cash. After all, you can only add so many
| levels and varieties of managers to add value. After that comes
| the variegation in roles of software development.
|
| That's my working theory anyway.
| yamtaddle wrote:
| I sometimes marvel at how few people worked on various pieces
| of commercial software in the 90s. You'd have 4x the headcount
| and still take twice as long, today. And it'd probably be
| webshit instead of native because "productivity matters more
| than performance".
| woeirua wrote:
| There is this moronic obsession with "scaling" in this
| industry, and it just blows my mind. You can have a wildly
| successful and profitable company while serving just a few
| hundred customers that requires nothing more than a basic LAMP
| stack.
|
| Meanwhile people are now spending millions of dollars and years
| of person-hours building MVPs in the cloud that won't ever go
| anywhere because the business model sucks.
|
| Focus on delivering value to your customer first. How you get
| there is quite literally irrelevant to your customer.
| bsenftner wrote:
| My theory is we need to add "software history" to computer
| science education programs. How many developers whose careers
| began after 2000 have ever hand written or know the power of a
| Makefile? I've worked at research labs and major animation
| studios that back in the 90's the entire infrastructure was
| fully automated thru Makefiles, and it purred like a well fed
| cat. There are hundreds of thousands of forgotten perfectly
| fine software tools, ignored simply because our industry has
| absolutely no respect for it's own history or lessons painfully
| learned beyond pulling up disparaging stories for new marketing
| angles.
| ogogmad wrote:
| Very interesting comment. What older software should devs
| know about?
|
| My problem with Make and similar tools is I don't want to
| learn more ad-hoc syntax to accomplish something nearly
| trivial. But I don't know if there's any alternative.
| bsenftner wrote:
| Original Make was butt simple: any lines that begin without
| tabs are a filename followed by list of other file
| dependencies, files whose modified date must be older than
| the filename at the start of the list. Than any lines of
| text immediately beneath that first line starting with a
| tab character are the shell commands to be executed in
| order that brings the first file up to date against it's
| dependencies. That was it, that is the entire original
| Makefile syntax. Then software vendors started adding
| extensions...
| bsenftner wrote:
| I guess as far as other, older software: I'd need to know
| your interests or field. How I write software has
| completely changed at least 5 times, and now I'm deep
| diving on my 6th (docker/k8s) to deliver the same things I
| was delivering back in the 80's with software development
| process one. Actually, add 3 more complete re-wiring of how
| I write code - I forgot working on game consoles: every new
| console is a complete start from zero how to write, debug,
| package and ship software, software that is not that much
| different than was delivered with the last platform and dev
| process.
| acedTrex wrote:
| While make is very powerful it's certainly not a tool that is
| nice to use. Theres a reason people built up layers and
| layers of complexity on top of it.
| HenriTEL wrote:
| As projects grow the Makefile becomes pretty much a big bash
| script, but with a specific syntax that I always struggle
| with.
| confidantlake wrote:
| It has blown me away too. There was a time where it seemed like
| the default "devops" strategy for startups was to use Heroku.
| Developers could spend all their time working on code. Worked
| at a startup a few years ago. We had 2 devs, and one devops guy
| building out Kubernetes, Docker, running on Aws. Had to
| constantly spend me time with him troubleshooting why this or
| that thing wasn't working rather than writing application code.
| We had like a 100 concurrent users.
|
| Now I am working at a large org for an internal application.
| There is an ops team I push code to github and Jenkins runs the
| CI and deploys it to a dev environment. I push a button in
| Jenkins to deploy to production. In two years there has been
| one ops related issue where I had to bump up the memory limit
| from 2GB to 4GB. There is a single server and a database. Setup
| works fine for a few thousand concurrent users. My skills are
| in development and understanding business reqs, not mucking
| about in config files. There are people that are good at that
| and enjoy doing that, let them handle it.
| jeremyjh wrote:
| > We had 2 devs, and one devops guy building out Kubernetes,
| Docker, running on Aws. Had to constantly spend me time with
| him troubleshooting why this or that thing wasn't working
| rather than writing application code. We had like a 100
| concurrent users.
|
| I'm glad you said this. 2-dev shop here and we've used Heroku
| the last five years. We spend all our time developing and
| supporting our application and almost never worry about
| infrastructure. I keep getting tempted to move to K8s; Heroku
| is expensive for what we get and seems to have suffered some
| serious brain-drain. I've dabbled with K8s in side-projects
| and I can geek out on all the terraform and yaml stuff for
| days. But its probably not a great idea to inflict it on a
| 2-dev shop.
| nathanaldensr wrote:
| I completely agree that 20+ years of cheap money have
| absolutely caused much of the complexity we see today. That's
| because people aren't working on problems that matter; instead,
| they're just trying to make VCs rich--effectively passing the
| buck on to the greater fool. I think this era is coming to an
| end, and we should see a drastic reduction in software
| development costs with the end of cheap debt.
| bsenftner wrote:
| We may see a strange hang over from this period: over the
| last 20 years we've created tech billionaires and tech
| trillionaire corporations - both entities that are not used
| to being told "no" and will spend their own money to extend
| this party that is their reality. Case in point: FB/Meta and
| their doomed Metaverse.
| ogogmad wrote:
| I hope I'm not being pedantic here, so please forgive me.
| Didn't interest rates go to zero after 2008? So that's 14
| years. I guess 8 years before that there was also Dot Com.
|
| Are you talking about interest rates? Thanks.
|
| Even if you're wrong, it's an interesting theory.
| hotpotamus wrote:
| https://www.readmargins.com/p/zirp-explains-the-world
|
| Is probably one of the better explainers.
| Jorchime wrote:
| I think it is because of what is described through Conway's
| law: https://en.wikipedia.org/wiki/Conway%27s_law
| Any organization that designs a system (defined broadly) will
| produce a design whose structure is a copy of the
| organization's communication structure.[2][3] -- Melvin
| E. Conway
|
| The problem isn't necessarily software itself, but how we
| organize people (more than 2 or 3), how we communicate, how we
| mirror operations, expectations, etc. in software.
|
| Scaling sustainable software feels like an "unsolved" Problem,
| because the society hasn't figured out how to organize better.
| ArjenM wrote:
| Small teams have honestly always been successful to the flaw
| that they outgrow themselves constantly in my experiences.
| boppo1 wrote:
| > money being dumped into the economy
|
| This is my theory too, and as someone learning to program
| (hopefully as a career), I'm very worried what a new paradigm
| of tightening will do to the field. I feel like _LOTS_ of
| people who think their stack is secure are going to get dropped
| because at the end of the day, lots of the B2B SaaS doesn 't
| actually deliver anything to the world of material needs.
| user3939382 wrote:
| My theory is a variation on that, which is that the money has
| placed so many more people in the space you inevitably end up
| with more people working in different directions, the JS
| ecosystem especially comes to mind here.
| debacle wrote:
| Because you have, in 90 (99?) percent of instances a smart
| person (probably IQ 120+) doing a stupid menial job that
| requires a level of competence that only someone smart can
| provide.
|
| So to prevent themselves from ending it all, they invent things
| to do to keep themselves sane while they churn out CRUD apps
| all day long for decades. Sometimes you get truly amazing
| software, but most of the times it's just reinventing the
| wheel, _but worse_.
|
| Even worse, some of them become infatuated with doing things
| "Like Google," and errant CTOs enable them, leading to a 6
| person team supporting a piece of software that could be
| replaced with WordPress, to the betterment of the primary
| stakeholder.
|
| I've been consulting for over a decade now. The number of times
| a potential client has told me they've spent into the six
| figures (or eight) for something that free software does out of
| the box but better is depressing. The worst part? I usually
| can't convince that person they've been absolutely swindled,
| and so I have to let them keep on their merry way.
| hotpotamus wrote:
| > So to prevent themselves from ending it all, they invent
| things to do to keep themselves sane while they churn out
| CRUD apps all day long for decades
|
| Or they spend the spare time bullshitting on web forums.
| ldjkfkdsjnv wrote:
| The complexity of what we are building keeps expanding. Theres
| no large conspiracy. We have a lot of tooling to build things
| that were near impossible to build in the past.
| kuramitropolis wrote:
| >Theres no large conspiracy
|
| Yes, there is a network of small to medium sized ones.
|
| > things that were near impossible to build in the past.
|
| Were they necessary in the past?
| ldjkfkdsjnv wrote:
| This is like asking whether innovation is necessary.
| Capitalism pushes us towards it. New forms of efficiency
| and value are unlocked by moving further out on the
| software complexity curve. Not everything thats complex
| works well, but some of it does solve the problem
| kuramitropolis wrote:
| What's "capitalism" in this context, and what's its
| actual connection to software development practices? What
| are "forms of efficiency" and "forms of value", and how
| does complexity "unlock" them?
| acedTrex wrote:
| Capitalism is a company that is attempting to solve X
| problem or do Y thing etc. Forms of efficiency are "how
| fast can they solve that problem". And complexity doesn't
| unlock anything. It's just inherent in the problem space
| a company takes on.
| sensanaty wrote:
| This just reads like a "Damn kids, get off my lawn!" type
| comment to my admittedly young mind. Same with a lot of similar
| comments I see floating around HN, especially ones that have
| anything to do with web development.
| imetatroll wrote:
| I tried out cloudflare for the first time yesterday using a
| personal website. Previously I was just using a domain
| registrar and setting DNS to digital ocean. cloudflare is a
| very different/complex beast, but I can imagine that the pretty
| interface alone brings in customers.
|
| I don't like this hyper-expansion either, but when you have an
| army of monkeys on typewriters, well, a lot gets "produced".
| j-krieger wrote:
| To be fair, cloudflare is among the services I will defend
| forever. Their ddos protection is by far the best on the
| market.
| _fat_santa wrote:
| > I have bewilderingly tried to discern why software
| development continues to grow more and more complex.
|
| It's because of growth, software development today is both
| simpler and way more complex than before and that's entirely
| due to growth in the sector. In many ways it's simpler than it
| ever was, I'm writing an API using Lambda/API Gateway in AWS
| and it blows me away how quickly I am able to get services
| stood up and configure my API. But in another way it's so much
| more complicated, for this same API I spent 2-3 weeks
| experimenting and researching IAM roles and how all my AWS
| resources would interact with one another.
|
| I would say the floor of software development has become way
| simpler, deploying a site to Netlify is way easier than dealing
| with webservers of the past. But while the floor is lower than
| ever before, the ceiling is in the stratosphere, with extremely
| complex systems that you can string together in the "Public
| Cloud".
| insane_dreamer wrote:
| > writing an API using Lambda/API Gateway in AWS and it blows
| me away how quickly I am able to get services stood up and
| configure my API
|
| > for this same API I spent 2-3 weeks experimenting and
| researching IAM roles and how all my AWS resources would
| interact with one another.
|
| same experience here, and trying to understand/test un/poorly
| documented AWS behavior with public/private APIs and EC2
| resources has been a huge time sink :(
| dilyevsky wrote:
| SaaS model requires software to be always online (strict sla)
| and you need to be able to update it without downtime or very
| little downtime. That's the reason. For example, telecom
| software that always had these requirements was always complex
| rglover wrote:
| I like the Uncle Bob explanation: the number of developers has
| a doubling rate of ~5 years due to the constant entry of new
| developers [1]. Over time, the number of inexperienced
| developers far outweighs the experienced ones. Inexperienced
| developers naturally gravitate toward complexity because they
| don't know any better.
|
| Couple that with a social drift toward hyper-specialization.
| That sort of hierarchy naturally creates minds that don't think
| at the _systems_ level. This is the general theme of
| Buckminster Fuller 's Operating Manual For Spaceship Earth: the
| shift toward hyper-specialization (or narrow focus) has long-
| term disastrous consequences.
|
| When you couple inexperience with narrow focus, you get messes.
|
| Part of this is to blame on the industry-standard thinking that
| because someone works for Company X, they're a competent,
| logical engineer (and should be granted
| authority/responsibility over essential products/projects).
| This is the downfall of "code tests" and "whiteboard coding."
| They don't evaluate for systems level thinking and so a
| developer with poor creativity and logic skills slips through
| the cracks because they're great at eeking performance out of a
| function which impresses the lollipop guild.
|
| [1] https://blog.cleancoder.com/uncle-
| bob/2014/06/20/MyLawn.html
| agumonkey wrote:
| society also changed, tech is trendy, ecobronerds have the
| wheeldrive
| jesuscript wrote:
| Your guess is as good as anyones at this point. There's just
| many over corrections that resemble strong levels of
| grandstanding. React is complex? Okay, let's ditch JavaScript
| and just do basic server rendered pages.
|
| It's a laughable over reaction (pun unavoidable).
|
| These are manic states the dev community enters, and all
| pragmatism is lost in our discourse on the solution.
|
| Cooler heads are not prevailing. There is certainly a solution
| in the middle.
|
| _Devops is bullshit_ is also an over correction. It's worth
| saying that making infra more accessible is a democratization
| of that entire sub-field. How do we preserve the good part of
| that?
| jahewson wrote:
| I semi-agree but my take is that money can buy you people, it
| can buy you lines of code, more moving parts, more stuff. But
| it can't buy elegance, efficiency and good design - those have
| to be nurtured. Most engineers nowadays haven't ever seen a
| codebase that isn't a hot mess (no offence folks). The demand
| for talented and experienced engineers eclipses the available
| supply. The industry is growing too fast for that. The
| engineers that really are good are needed in leadership, or
| become founders.
| boppo1 wrote:
| Any open source codebases that meet your approval?
| nimbius wrote:
| >platform engineering and enabling developer self-service.
|
| in any sufficiently large company platforms arent engineered,
| they are acquired or purchased/licensed based on their viability
| as an "enterprise grade" asset that drives business success and
| reduces cost through identifiable if not meaningless KPI and
| record keeping.
|
| In any sufficiently large company self-service is supplanted by
| rigid controls, authorizations, approvals, and annual reviews.
| this is done in the service of jira and the need to make-pretend
| work by an ever growing cavalcade of pseudoworkers who recognize
| self-service as the killing stroke of their career.
|
| it can then be said, grudgingly and with scorn, that devops seems
| designed by its very definition to operate as an antipattern to
| some of the worlds largest, most successful corporations.
| dsr_ wrote:
| Over the last ten years, the market has tried to kill off the
| hardware, systems, network and security people, and mostly
| succeeded.
|
| As a result, it's relatively easy to find someone who advertises
| as "full stack devops" who has never actually operated any
| infrastructure more complex than a LAMP webserver cluster. And
| it's hard to find a senior sysadmin who has enough years of
| experience to understand and troubleshoot all of your
| infrastructure from layer 0 up. People with that experience have
| moved into management or consulting or retirement, and there are
| no jobs for new folks.
| vigeek wrote:
| Based on my own experiences, one of the most disappointing
| elements of devops is their lack of understanding how things
| work. They run to GitHub and pull other peoples recipes to
| build out certain environments. Even when they build automation
| from scratch, they miss tons of core OS level tunables and
| rather than adjusting those, they'll just add more servers to
| the mix.
|
| I worked at a place that had an elastic search cluster with 30
| nodes, because they kept hitting open file limits, when 8
| servers could easily handle the traffic with basic tuning.
| Melatonic wrote:
| It cracks me up everytime I talk to some very competent and
| senior devs who have zero knowledge of hardware - cloud has
| spoiled us!
|
| Also the amount of devs who have had to work in shops with
| really crappy on prem infrastructure setups is also amazing - I
| guess I have just been lucky to work at places with solid infra
| breakalot wrote:
| The problem is that devs simply can't accept that ops exist.
|
| We spoiled devs way too much.
|
| Software are still being thrown over the wall and ops takes all
| the blame.
|
| The problem is that many devs can get away with spawning servers
| and doing the easy parts of ops, when it requires rules and
| discipline, then we all know what happens, over engineering and
| security holes.
| [deleted]
| smcleod wrote:
| Co-opted, branded and misinterpreted by the enterprise -
| "#DevOps" is not DevOps as it was intended. Pretty spot on.
| outworlder wrote:
| > You've got a DevOps team.
|
| Yeap. Largest red flag there is.
| arpa wrote:
| Well, to be fair, most of the engineering is bullshit.
| drewcoo wrote:
| Software. Not engineering. Software.
|
| "The best minds of my generation are thinking about how to make
| people click ads."
|
| https://quoteinvestigator.com/2017/06/12/click/
| gtirloni wrote:
| TL;DR; s/DevOps/Platform/
| GiorgioG wrote:
| Managing complexity should be job 1 for technical leadership.
| We've failed utterly when it comes to DevOps. The model should be
| Heroku-like deploys for 99% of apps/companies. Instead we have an
| army of DevOps experts for k8s, helm, terraform, etc. Changing an
| environment variable now requires a DevOps person and/or a PR in
| your GitOps setup. The feedback loop for troubleshooting issues
| in these environments is too long.
|
| This is me right now: https://i.kym-
| cdn.com/entries/icons/original/000/019/304/old...
| jamisteven wrote:
| This is one of those things that just drives people out of tech
| and leaves the ones still in it super frustrated. So many
| buzzwords and PR non-sense from non-tech mgmt.
| dathinab wrote:
| PaaS is grate, sure it doesn't fully eliminated oprations but it
| makes it much easier for you to do what the article described.
|
| Like most things it comes at a cost (for a good PaaS mainly the
| running cost).
|
| But I increasingly come to believe that for small startups which
| fundamentally don't have the resources to do what the article
| describes or anything close to it using PaaS and keeping
| operational complexity as low as possible is the way to go.
| omginternets wrote:
| bradhe wrote:
| I have a hard time trusting someone who calls something
| "production-grade."
| davydog187 wrote:
| I too have a hard time trusting people who use words
| hnthrowaway0315 wrote:
| I think DevOps team maintains infra and monitor ops so that other
| teams can "plug-in" their applications. For example DevOps team
| sets up Terraform but dev teams plug-in their own repos and
| configurations to use that private enterprise Terraform
| "cluster". Another example: DevOps team sets up Airflow clusters
| but Data engineer teams use them.
|
| At least this is how the role works in my company.
| at_a_remove wrote:
| At this one place of employment, I had been struggling to divest
| myself of system administration duties for some time. I can _do_
| them, but I don 't particularly _like_ them. I have a few things
| I 'm good at in that area (troubleshooting, being paranoid), but
| I really despise middle of the night calls, early morning
| patching, and so on. I had finally gotten to a good place.
|
| And then someone newly promoted decided to rewrite everyone's job
| titles and I was suddenly DevOps, sucked back in.
| manv1 wrote:
| I've worked with a "devops" team that spend hundreds of thousands
| of dollars on puppet consultants and never got it working.
| Another team spent it's time badmouthing various technology
| choices, but then didn't realize that our customer's onsite
| deployment dictated the technology choices and chose the wrong
| tech.
|
| The real reason devops exists (what used to be called Application
| Engineering) is because development is pretty clueless about the
| customer's runtime environment. They don't really understand the
| tuning necessary to create a production system.
|
| Example: very few developers understand how big your DB
| connection pool should be, because they tend to only test one
| connection scenarios. That's assuming they're actually know
| enough to use a connection pool. And they almost never handle
| failover scenarios.
|
| DevOps should be the purview of the senior engineers. If you're
| building without an understanding of your deployment environment
| then you're screwed. And more importantly, you're not taking
| advantage of platform features.
|
| As an example, deploying to SSDs (which everyone should be doing)
| means your database performance has just gotten an order of
| magnitude better for free. You need to retest and get rid of a
| lot of those performance-related changes.
|
| Let's put it this way: one monolithic Java/spring application I
| worked on was basically a SOAP server with a UI that took up gigs
| of RAM. It was that big because the scaffolding required to
| handle all that was, well, huge. But really, it was essentially a
| web server that served pages to connected clients. All that other
| shit was overhead...so on AWS it got transformed into a few
| lambda functions and REST apis. Without an understanding of the
| deployment environment (and the possibilities associated with
| that) it would never have happened.
|
| TL;DR: senior engineers should be the devops people, because how
| and where you deploy software should determine how you should
| make software.
| cosmiccatnap wrote:
| "DevOps is broken"
|
| Said the 4th article on the subject this month.
| notabee wrote:
| We're building up to the next buzzword. Give it a year or two.
| bob1029 wrote:
| I don't understand why we can't take more ownership of our work.
| Even if you are not immediately and directly compensated for
| every inch that you go above and beyond, you are still in a
| position to make yourself irreplaceable in terms the business
| cannot ignore.
|
| Think about the long play. You join a startup with a broken, hot-
| potato-style "devops" process. Instead of saying "not my problem"
| all day, you can take some ownership of the items slightly
| outside your space and try to arrive at a better solution. If you
| do this often enough and with enough persistence, the end
| customer will eventually see benefit. At some point, management
| will likely notice the correlation as well. Even if they don't,
| you have gained far more experience than you would have otherwise
| and can maybe go start your own damn company, realizing up to
| 100% of the value of your labor.
|
| This is why I try hard, even if someone doesn't make me.
| dinosaurdynasty wrote:
| We can't take more ownership of our work because we do not in
| fact own it, and the business will never let you actually own
| it.
|
| All I've gotten from trying hard is burnout and sickness. At
| this point I can say the only reason I work for others is
| because I need money to live.
| bob1029 wrote:
| > and the business will never let you actually own it.
|
| This is not true in many startup environments. You'd have a
| hell of a time getting me to work on a new project without
| some sort of equity arrangement.
| dinosaurdynasty wrote:
| Does the equity come with voting rights? Can you fire the
| C-suite? Can you legally leave and fork the project?
|
| Without stuff like this it's hard not to see equity as
| simply a different way of getting paid.
| bob1029 wrote:
| > Does the equity come with voting rights?
|
| It does in my case.
|
| I started out as a junior developer. I am now _in_ the
| C-suite.
|
| Equity is also not a replacement for salary, which we
| clearly just take for granted these days.
|
| Taking an adversarial stance in business is the fastest
| way to wind up nowhere. Being able to "fire the c-suite"
| is not a good thing. If you have to do that, you likely
| don't have a viable business in the first place.
| javier_e06 wrote:
| The name meant something a while ago: Something along the lines
| of:
|
| "Turning the carnival bumping cars into a ferry's wheel or maybe
| the teacups"
|
| Today's Dev-Ops is a Teacher's Lounge Room with a big suggestion
| box by the door.
|
| One more shortcoming in the long list of modern needs.
| bvrmn wrote:
| > The problem is most engineers don't want to do operations work.
|
| My experience tell that most engineers don't want to do any work
| and it's ok.
| insane_dreamer wrote:
| I think of DevOps as "deploying and maintaining infrastructure on
| which applications run", vs SW Engs as "building applications".
| Not sure if that's the "accepted" definition or not, but works
| for us. YMMV.
| pkrumins wrote:
| I identify as a sysadmin.
| giantg2 wrote:
| "DevOps Is Bullshit"
|
| Wait until you meet DevSecOps...
| arvindamirtaa wrote:
| I LOLd for this one.
| charles_f wrote:
| Oh, yes, the big shift of "everything's the responsibility of
| devs". Wait till we also need to do product analysis and we
| finally get to it.
|
| ProAnalDevSecOps
| PubliusMI wrote:
| Or Dev ML Ops...
| vasco wrote:
| There must be some a rite of passage or initiation ritual that I
| haven't been good enough for yet where you need to write an
| opinion piece about the term DevOps, because otherwise I'm not
| sure why in 2022 we're still having to read these articles.
| Whoever wrote the first one has trolled an entire generation of
| engineers.
| dsr_ wrote:
| Because, like Agile, it was seized on as a label for "whatever
| reorg I want to introduce in my three year term at this company
| before I leave for the next one".
| errantmind wrote:
| Why was the title changed from the original?
| [deleted]
| [deleted]
| ndneighbor wrote:
| I had a very weird sensation while reading this where I was
| shaking my head disapprovingly at this article. Specifically when
| they were talking about the commoditization of DevOps.
|
| Speaking personally, I used to work on a Platform Engineering
| team for a major multinational that rhymes with ay-do-bay. And
| their requirements for the workloads that we needed to be built
| and hosted where very unique because you have teams running
| completely different languages and toolchains, esp. from
| acquisitions. I can argue there that the template K8s setups
| wouldn't work.
|
| In the author's case, the only reason why they feel the way they
| do is because we finally have some sense of standardization on
| what Infra looks like for most companies. It's no longer a
| question for most folks if they should adopt Docker, we have
| accepted images into our lives. Same for K8s (after a certain
| point of scale). So uh... yea. Catchy blog title to sell some
| thin layer on top of K8s in which it doesn't solve the root issue
| that they are talking about.
| thundergolfer wrote:
| It seems like "rhymes with ba-do-bay" is meant to be obvious
| but I have no idea what company that is
| ndneighbor wrote:
| adobe, but I live my life in perpetual fear of lawyers. Fun
| times back then.
| zwilliamson wrote:
| It takes really good engineering leadership to build a platform
| team. With the wherewithal to follow through on migrations and
| marketing of the platform. Also, sourcing the right people
| internally and elevating them. These engineers become true full
| stack (the whole OSI model) engineers and their work most
| definitely shouldn't be thankless or taken for granted by product
| leaders. I've seen efforts like this fall apart in very ugly ways
| due to a lack of leadership.
| lucidguppy wrote:
| Certain articles need a list of "required reading" before reading
| the article.
|
| You should at least read the "dev ops handbook" before reading
| this article.
|
| The article should then talk through its points using the book as
| a reference.
| gitpusher wrote:
| I'm not sure I agree with author. Sounds like they have worked at
| shitty places that are using the word "DevOps" but really they
| are doing things the old way.
|
| My company practices "DevOps" and it feels great:
| - Infra team build self-service tools (build, deploy, scale,
| observe) - Infra team write good documentation for these
| tools - Dev use tools to build, deploy, and monitor their
| apps - Dev not use use words like: AWS, k8, Envoy. These
| are abstracted by tools. - if problem with app (very
| common) Dev fix it using the tools - if problem with tool
| (very rare) Infra fix it
|
| We have no build engineers, release engineers, etc. However we do
| have a rotation (similar to on-call) whereby Dev is responsible
| for releasing code that week.
|
| Sure there are sometimes problems, frustrations, etc. No system
| is perfect. But you are getting paid lots of $$$ so shush with
| your whining & instead help improve the system.
|
| For context my company is mid-size ~150 engineers
| shadowgovt wrote:
| This echoes the philosophy of Google Site Reliability
| Engineering, which (this is key) is an engineering discipline.
|
| The job of DevOps is not to close tickets. That'd be like driving
| a car by shouting directions at someone lying on the floorboards
| holding a wrench to the steering pinion.
|
| The job of DevOps is to build a steering wheel (and ideally,
| teach SWEs how to drive... at least enough that they understand
| what a "road" is and why it's a pleasant experience for everyone
| if you stay on it. If the road doesn't go where they need to be,
| _then_ it 's time to file a ticket, but that ticket had better be
| "Build a new road," not "Offroad this one car to the cabin in the
| woods and call it a job well done").
|
| The raw hardware of an enterprise deployment is so flexible it
| solves _nobody 's_ problem. DevOps is in the business of writing
| the operating system for a mega-computer physically represented
| by hundreds to possibly millions of heterogeneous computers. It's
| a process of continuous growth to make that work.
| tootie wrote:
| Well they explicitly use the term SRE and not DevOps and I
| think that's very intentional and related the gist of the
| article. DevOps was never meant to be a role played by a person
| or a team. It's meant to reflect aligned incentives of dev and
| ops. Whether you have an SRE team or a platform team or a bunch
| of kitchen sink teams.
| hintymad wrote:
| > Congrats, that's not DevOps. I'd wager most of what they are
| doing is using Terraform and YAML to do menial tasks for the
| engineering team. > Need a database? File a ticket with DevOps.
|
| Shouldn't the "DevOps" team build APIs and consoles so eng teams
| can provision their resources without knowing TFE/YAML from a
| mile away? Really, this is year 2202. Please, learn from AWS and
| Netflix. Do software development. Get rid of the god damn
| ticketing process. Friends don't let friends touch TFE/YAML or
| whatever configuration files.
| dijit wrote:
| I've long held this opinion but I consistently get drowned out.
|
| DevOps has different meaning depending on who you're talking to,
| even some definitions that _appear similar_ are different in
| nuanced but important ways.
|
| All "devops" as a job title has done has muddy responsibilities
| and given many folks the wrong impression of what an operations
| discipline should be.
|
| There is also a lot of rewriting of history that gets thrown in,
| similar to how when people talk about cloud then the _only_
| alternative is to start making CPUs by hand and begin building
| your own nuclear reactors. It's the _idea_ of what came before,
| not the reality, that people seem to be defensive of.
|
| It's honestly exhausting to discuss.
|
| So instead I became CTO so I can solve this mess properly, I
| don't hire devops, I hire infra engineers, build engineers,
| release engineers and: backend engineers.
|
| Roles so simple that you already have a clue what they do, which
| is sort of the point of job titles.
| NikolaNovak wrote:
| >>the only alternative is to start making CPUs by hand
|
| Agreed. For some applications the cloud difference is
| significant; for many (most?) others though, "Cloud" is just
| rebrand of "Hosted". And even for more cloudy offerings, while
| I'm in a very specific and different part of IBM, some of the
| old timers/architects/powers-that-be keep trying to explain "We
| had that in _1969_!!! " :-D
|
| Agreed also at rewriting of history when it comes to
| development/support/operations models. My dad has been IT
| director and he chuckles when I talk to him about "new and
| exciting paradigms" which he of course sees as turning a circle
| to what they had in 70's and 80's :)
| softfalcon wrote:
| The two of you have validated my very existence as a coder. I
| keep jokingly telling everyone that we're often going around
| in circles. No one ever believes me (lol!)
| kennend3 wrote:
| > My dad has been IT director and he chuckles when I talk to
| him about "new and exciting paradigms" which he of course
| sees as turning a circle to what they had in 70's and 80's :)
|
| As someone with 20+ years in IT, I agree - a lot of these
| "new and exciting paradigms" are not new at all.
|
| My personal favourite is how many large multi-nationals are
| now building in-house clouds?
|
| WTF is the difference between an "in-house cloud" and a
| shared-use datacenter from the 1990's?
| chevdev wrote:
| An in-house cloud will just be a bunch of commodity servers
| running a hypervisor that gives you an API that allows you
| to automate the provisioning of infrastructure.
|
| I am guessing that in the 80s you weren't writing
| Infrastructure as Code to define exactly what resources you
| needed for your software, having it all set up
| automatically, and so on.
| pfarrell wrote:
| "There aren't any new problems. Just new engineers" -
| A sig I read a long time ago
| willcipriano wrote:
| > WTF is the difference between an "in-house cloud" and a
| shared-use datacenter from the 1990's?
|
| A couple million in salaries and bonuses.
| hinkley wrote:
| The only possible value over the last iteration is if they
| cleaned house and the new team manages to be more
| permissive than the old one. I'd much rather have on-prem,
| but the sad fact is that for cloud I just need a signed
| check. For on-prem I also need buy-in from other divisions
| before I can even start experimenting with a new service.
|
| I still have the exact same problem with respect to never
| having exactly the ratio of CPU to memory that would make
| my app happy.
| adamsb6 wrote:
| I think the difference is the interface.
|
| The interface to the shared-use datacenter, if you're
| lucky, is a spreadsheet that declares the static resources
| you own and a remote hands guy that can tackle things
| beyond the capabilities of your remote KVM. If you need
| more capacity you need to work with the datacenter folks to
| order physical machines that might show up in a few months.
|
| The interface to the in-house cloud is an API. In most
| instances, developers are completely abstracted away from
| the physical infrastructure and don't need to take a lock
| on some human in the datacenter to get their work done.
| toast0 wrote:
| > WTF is the difference between an "in-house cloud" and a
| shared-use datacenter from the 1990's?
|
| An in-house cloud sounds like the mainframe installed in
| the raised floor computer room at the school district
| office I worked at in the late 90s; of course, the 12 foot
| long Unisys mainframe was replaced with a Unisys 4U pentium
| pro box pretending to be a mainframe, and then there was a
| lot of extra floor space.
|
| If you're running your own 'cloud' in a (shared) colo, I
| dunno that that's really in-house. I guess it's still
| 'private cloud' though.
| ElectricalUnion wrote:
| > WTF is the difference between an "in-house cloud" and a
| shared-use datacenter from the 1990's?
|
| Can't call a mainframe a bunch of buzzwords like "Hyper-
| converged, high availability, on-premise software as a
| service cloud platform".
| hinkley wrote:
| Turn-key cloud technology.
| NikolaNovak wrote:
| I'm pretty sure mainframe guys are calling it exactly
| that:
|
| https://www.ibm.com/ca-en/products/z16
|
| warning: buzz word alert!
| doctor_eval wrote:
| I had a couple of telco customers that ran their own
| internal cloud, and I'd say the main difference is that
| it's way more expensive than the public cloud, heaps less
| flexible, and when you need compute it has to go through an
| approval process.
|
| I mean ... sometimes I shake my head in wonder, and other
| times I just shake my head.
| mannykannot wrote:
| Agreed; I have always thought of it as a rationalization of bad
| practices.
| ftlio wrote:
| > I hire infra engineers, build engineers, release engineers
| and: backend engineers.
|
| I always thought DevOps the "function" just meant this, and
| being a DevOps engineer at a small company meant you did these
| with decreasing emphasis, where by the time you're in the
| backend it's just helping enforce logging, tracing, other
| observable components.
|
| Has worked for me in hiring and being hired and almost everyone
| I know understands this.
| kodah wrote:
| I get called an "Infrastructure Engineer" sometimes, I've also
| had the title of SRE and SWE. I don't really feel like any of
| these actually fit me.
|
| I do work on Cloud Infrastructure at times, but honestly it's
| the smallest part of what I do. It's usually addressed in the
| architectural designs and I have to touch it incrementally.
| What I do much more often is writing tools, daemons, and
| services for distributed systems. I end up calling myself a
| Distributed Systems Software Engineer to reflect the idea that
| what I work on is systems and software, and most of them are
| non-monoliths (from a systems perspective).
|
| Do you have any thoughts on Systems Engineers or the title that
| I prefer to call myself?
| dijit wrote:
| In the games industry we have a role called "tools
| programmer" which is a person who works on tooling to
| accelerate development, sometimes these can be extensions to
| the editor for a specialisation for a game (say, a tyre
| builder for a racing game) and sometimes they can be more
| broad (a distribution tool using libtorrent so that a new
| build of the game can be shipped worldwide very quickly).
|
| There are people who might refer to you as a backend
| programmer (if the focus is services and daemons), or
| platform engineer (if the focus is developer velocity).
|
| Largely it depends what your primary focus is.
| edmcnulty101 wrote:
| I suspect most places just use the term DevOps to try to get
| two jobs out of one person.
|
| Then they also tend to have to do their own security and spec
| out their own tickets and then do Project Management.
|
| Most tech people should be called
| DevSecOpsBizAnalystProjectManager.
| lumost wrote:
| I don't know, every time I see a title split people start
| saying that's "X's job", you can't do that because it's "y's
| job", or I don't want to be involved in "Z".
|
| Fewer titles can avoid these discussions.
| dijit wrote:
| Sounds like a bad culture.
|
| That won't be fixed with a new job title.
| rovr138 wrote:
| Looks like you already identified the people that should be
| in those roles.
|
| If they're not doing it or was overlooked for some reason,
| then that's an issue but not really an argument to not do it.
| lumost wrote:
| Since when are managers incentivized to put people where
| their interests align? particularly when there is a
| perceived role transition.
| bgro wrote:
| I don't understand how seemingly 90% of developers, can't do
| anything outside of their narrow scope of experience. Writing a
| shell script? Throw a temper tantrum, that's devops job! Have
| to work with legacy code? I can't believe this!
|
| Especially seniors and beyond. They force leetcode interviews
| they somehow pass or are grandfathered through and gatekeep
| "trash devs" by slamming gotchas about how the whiteboarded
| code discussion doesn't technically compile and how
| "returnAverage()" isn't a method ("did you know you have to
| write methods before they work? What does returnAverage()
| supposedly even do? Return a random character in the
| alphabet?")
|
| I get excited about working in different tech areas. I'm
| exceptional at fixing other's bugs and maintaining code.
| Absolutely nobody seems to be hiring for this though. It's all
| about college exam trivia and leetcode.
|
| When I do get into these companies, I have to work with people
| throwing around casual racism with HR joining in and f-slurs
| like it's corporate 4chan. I ask if anyone can help look at a
| critical bug I discovered, and nobody speaks up. Even when it
| turns out it was their last code change that caused it and they
| are the sole master expert in this area, and they were just
| working on organizing their desktop instead.
|
| "I can try to look at the SQL problem, since nobody else spoke
| up." Then I'm explaining the basics of SQL to some guy sitting
| around blank faced and it turns out he was hired because of his
| 10+ years of expertise in SQL. His whole job is to tackle
| problems like that.
|
| Meanwhile I'm out here looking for jobs when panic cuts happen
| and it takes forever because I'm drained from all the gotchas
| and gatekeeping in these interviews.
|
| There's no way I could live with myself if I put this
| experience onto others. I lose sleep when I fail to call out
| someone talking over a quiet person on my team. I've never seen
| anybody else stand up for anybody, however.
| apalumbi wrote:
| I like to build up my teams with people that act and think
| like you. Sometimes I need a sniper and when I do I'll get
| one. But my teams need to be excited about building and
| running the systems that we make.
| int0x2e wrote:
| You hit the nail on the head - at many places, the hiring
| process seems to be great at blocking people who are great at
| solving real world problems that aren't as good at whiteboard
| leet-coding.
|
| If you truly enjoy this part of the job, you can go for a
| freelance career - build a network and reputation as the
| person you call to solve the hard stuff, and do mostly that,
| no leetcode interviews required...
| hinkley wrote:
| I learned SQL because my coworker was stuck on a bug she
| couldn't figure out and 'knew' I'd be able to solve her
| problem.
|
| I just kept asking questions until I found the one she
| missed.
|
| I also ended up becoming a bespoke VxWorks admin because the
| guy who volunteered was never available, and someone asserted
| my code wasn't working well on VxWorks, so of course I had to
| know enough to do benchmarks. I fixed a few problems but the
| real issue was the hard drive wasn't doing DMA due to the
| kernel not recognizing the processor revision number.
|
| Somewhere between those two events I realized that if my part
| of the project is great but the whole project is on fire,
| nobody cares about my stuff. I don't get points for being
| right and the team being wrong. I mean, I do for some people,
| but I still feel bad at the end of the project, and those are
| the 'points' I have to live with the most.
|
| The way that played on on the latter project is that once we
| got our shit together, we started work stealing from our peer
| orgs. Volunteering to carve off little pieces of interface
| between us and 'take care' of this bit of data handling here
| and that one over there. I came to realize that the Org
| subconsciously knew this and any past success they had was
| due to self-organization and collective work-stealing.
|
| Throw people at it, turn a blind eye to precise mission
| statements, and hope for the best.
| llanowarelves wrote:
| Sounds like medium or bigger sized company office politics.
|
| At smaller companies the title tends to be only a title and
| many people are basically full-stack and know a bit of
| everything. And you're trading hats to get jobs done.
| doctor_eval wrote:
| I agree with this but I'd add that it's also important that the
| roles aren't overly siloed. A front end engineer, for example,
| should be able to and even encouraged to look at backend code,
| and even submit PRs for it.
|
| And more importantly, the backend engineers should be
| encouraged to _be grateful_ for the PRs (even if it's not
| acceptable for some reason). The more eyes on the code, the
| better.
|
| I think developers naturally tend to specialise (at least for a
| time) but ultimately we all need to understand and contribute
| to code regardless if it's front end, backend, build, test, ...
| duxup wrote:
| Agreed, anytime I talk with someone about DevOps ... we end up
| having to hash out the entire process to really know what
| either of us are actually talking about. Otherwise you have
| these situations
|
| "Yea the DevOps guy messed up the widget and nobody notic---"
|
| "Wait, what is the DevOps guy doing even touching that
| widget.... what is even DevOps to you?"
|
| "Bro that widget IS DevOps."
|
| -silence-
|
| Same applies to the topic of "micro-services".
| soperj wrote:
| SOA is just Spaghetti Architecture. I haven't seen an
| implementation that convinces me otherwise.
| rileyphone wrote:
| Microservices should be resource-oriented, not service-
| oriented. NetKernel takes it to the extreme, it's a shame
| it hasn't seen broader adoption.
|
| http://resources.1060research.com/docs/IntroductionToResour
| c...
| duxup wrote:
| I feel like it has its place in large organization. But
| being a large organization means the odds of spaghetti are
| approaching 100% no matter what you do.
| Kinrany wrote:
| Any general purpose architecture allows spaghetti, there's
| no way around it.
| usrusr wrote:
| Has there been an architecture movement that formally
| embraces the spaghetti, aiming for peaceful coexistence?
| Detractors will certainly point at SOA and shout "that
| one!", but I mean one that openly admits..
| jedberg wrote:
| What is your definition of Spaghetti Architecture? Netflix
| had a good SOA that enabled rapid development and had
| strong cut lines between services, with no way to access
| the data of a service without going through the service's
| API.
|
| I think that's where most people go wrong. They put a bunch
| of services in front of a shared database, which means that
| they don't have to go through a service's API to get to
| it's data, and that's what breaks everything.
| eropple wrote:
| This is really important, and I've gotten a lot of
| quizzical looks when making this assertion over the
| years: data is owned by one and only one service. If two
| pieces of code assert ownership of it by mutating that
| data or looking past the public encapsulation of that
| data, then that code is the same service.
|
| If you see a queue between two services, that is usually
| an indication that some ownership is being transferred
| (even briefly), and that is a critical point where you
| need introspection and observation.
| maxrev17 wrote:
| This is it exactly!
| wikibob wrote:
| > _Data sovereignty per microservice_
|
| > An important rule for microservices architecture is
| that _each microservice must own its domain data and
| logic_ . Just as a full application owns its logic and
| data, so must each microservice own its logic and data
| under an autonomous lifecycle, with independent
| deployment per microservice.
|
| https://learn.microsoft.com/en-
| us/dotnet/architecture/micros...
| JamesBarney wrote:
| The big issue I've seen is that takes a lot of work, so
| people cut corners. The problem is of course when you cut
| corners with microservices and rely on a shared database
| for instance, suddenly you're dealing with 40% of the
| costs of a microservices and 0% of the benefits.
| GauntletWizard wrote:
| Shared database can be a reasonable microservice
| boundary, especially when using database-as-queue or
| database-as-mucroservice. In the former, services a, b
| and c can insert and query but only D can update, i.e.
| any service can creat a work-order but only D can mark it
| completed. I. The latter, nobody can read or write and
| all access to tables is done through stored procedures.
|
| I don't recommend either, and it's still a code smell,
| but with clear definitions they can work.
| JamesBarney wrote:
| The benefits of the microservice pattern are you can
| build separate teams responsible for different business
| logic, and they can have their own deployment schedule.
| You don't get either of those benefits when a b and c
| have to coordinate on their work-order creation business
| logic, and a, b, c and d all need to be deployed at the
| same time anytime the schema changes.
|
| Btw some of the confusion might be what I mean by shared
| database. I didn't mean two services sharing a limited
| set of tables, using the database as a rabbitmq
| replacement. I meant sharing the backing database of the
| microservices. It sounds like we probably agree I just
| wasn't very clear about what I meant by sharing a
| database.
|
| (Ironically I am literally using a database as a queue to
| share data between two services we are running in prod. I
| don't think of it as a microservice because it's not
| separate teams, it just because our monolith is hosted on
| a platform that doesn't support certain libraries so we
| had to role those libraries onto a separate platform that
| does support them.)
| GauntletWizard wrote:
| I think we're in violent agreement - A database _can_ be
| a queue and that doesn 't link things as the same, but
| it's easy to break those promises and you need to be
| clear in that promise to begin with. If two
| "microservices" have to move in strict lockstep, they're
| not microservices, they're components of a larger
| service.
| lamontcg wrote:
| Yeah, services should be formed around the needs of data,
| and you shouldn't run multiple services in the same
| image/container just to split the code up. It shouldn't
| ever be about the code.
|
| Amazon's SOA architecture at least all started out about
| pulling bits of data spread over hundreds of servers into
| services that could take advantage of caching.
| throwdbaaway wrote:
| I have seen the "data service" design pattern that tries
| to work around this, by having a CRUD microservice to
| front a database, providing hundreds of APIs to
| read/write the data for 50 other microservices. If one of
| these CRUD microservices goes down, everything breaks.
| dustymcp wrote:
| this devops is also a dumping ground for anything else not
| happening on a developers computer which they should control
| but wont because "That is devops job" mostly this happens
| alot with node projects for some reason..
| chrisweekly wrote:
| "anything else not happening on a developers computer which
| they should control but wont"
|
| Ok, I've seen this, but IME (24y in industry, the last 6 as
| a consultant) in the vast majority of cases, it's more like
| "things devs should control but CAN'T [bc CICD etc are
| silo'd and owned jealously by an overburdened ops team
| unable or unwilling to facilitate self-service]".
| outworlder wrote:
| Some of the 'jealousy' may also come from bad
| experiences.
|
| It only takes a few instances of people royally messing
| stuff up in production for their deployment rights to be
| stripped away - and other groups catch the fallout too,
| as new "procedures" get implemented.
| dijit wrote:
| "Safety rules are written in blood" applies to parts of
| the organisation too.
|
| Sure, there are some overbearing procedure monkeys who
| really want a process on everything, but a lot of
| "protection" rules are there because something really bad
| (and expensive, financially or reputationally) has
| happened before.
| duxup wrote:
| Humans are kinda terrible when they can just "not my
| problem" things.
| sbf501 wrote:
| "Hey, did the front-end new-hire upgrade the Linux distro
| yet?"
| duxup wrote:
| Side rant:
|
| There was an angry Ask HN post a while back about how
| terrible the "new guys" are that are out there.
|
| It then complained that the new hire wrote a horrible
| authentication service.
|
| I thought it was an intentionally absurd post about the
| expectations put on new some rando new guy to write
| something important / they shouldn't be working alone on
| ... but they were serious.
| boppo1 wrote:
| >microservices
|
| I never miss an opportunity to share my favorite piece of
| comedy this decade:
|
| https://youtu.be/y8OnoxKotPQ
| dijit wrote:
| it's too close to reality.
|
| There is a good talk that is also comedic as it comes from
| the perspective of someone who wants to _fail_ at doing
| microservices.
|
| https://www.youtube.com/watch?v=GWgRw5jiYy0
| smcleod wrote:
| Thank was pretty funny, thanks.
| brightball wrote:
| The extent of devops for your developers should be:
|
| 1. Push code to an automated pipeline
|
| 2. I understand that the automated pipeline may need input from
| me in order to run successfully.
|
| Beyond that, you need other people building the "platform" that
| they're deploying to (for the most part). Probably the ideal
| example here is Heroku.
|
| Your devs shouldn't be required to do all of the actual ops
| stuff. They need access to an ops platform.
| tetha wrote:
| > Your devs shouldn't be required to do all of the actual ops
| stuff. They need access to an ops platform.
|
| This becomes a lot more apparent once you're dealing with B2B
| customers and their nonfunctional requirements.
|
| We've had situations in-house when devs built some simple
| microservice to handle some connection to a customers BI/DWH
| system - it's just 2 days of spring boot chugging to wrangle
| APIs around without state, nothing bad. But then, the
| customer started blasting that team with SLAs, backup
| questions like RTO, RPO, retention, regulation adherence of
| retention, the whole stack of IT security down to the
| physical access control.. That poor PO was caught just like a
| deer in the headlights.
|
| This is some operational responsibility we're taking over for
| our dev-teams. We're providing the persistence, and this
| includes a defined backup and recovery strategy, a security
| strategy and such. And this also includes experience in
| dealing with these insane questionaires, and correctly
| pricing absurd requirements for custom backup solutions. And
| in fact, "we will have to schedule a discussion with ops
| about this" has ended quite a few of these requirements with
| "oh.. it's not that important". Intimidation with long job
| titles and appeals to external authority do work.
| ghaff wrote:
| To grossly oversimplify, there are essentially two views of
| DevOps.
|
| 1.) What's probably the traditional view which is breaking
| down the walls between devs and ops, developers carrying
| pagers, etc. I.e. at least in an idealized world, there are
| no devs and ops--only DevOps.
|
| 2.) As you suggest (and which probably more closely matches
| how "DevOps" works especially in larger organizations), an
| internal (or external) operations team provides a platform
| that developers can use. Developers are still going to be
| exposed to _some_ operational details, but a lot of them are
| abstracted away.
| dilyevsky wrote:
| Highly dependent on the product. Anything infrastructure
| related you'll have a really bad time with that model
| wschoenberger wrote:
| The article has a solid solution ;)
| tremon wrote:
| I expect my developers to troubleshoot a failing pipeline
| too. The pipeline is part of the "ops platform" they're
| expected to work with.
| Melatonic wrote:
| I agree - I think it it is much better to use titles like these
| and with even a short JD describing what technology is used
| (many of which could be "devops" tools for infra engineers)
| everything becomes much more clear.
| indymike wrote:
| > I don't hire devops, I hire infra engineers, build engineers,
| release engineers and: backend engineers.
|
| This is a great way to do it. There seems to be a correlation
| between unnecessary product complexity and unnecessary
| corporate complexity. Being direct about roles goes a long way
| towards simplifying corporate complexity.
| LudwigNagasena wrote:
| That depends on the complexity of the development lifecycle,
| doesn't it? If a single person can maintain it, having three
| different roles would mean unnecessary corporate complexity.
| tremon wrote:
| Not sure what you mean with "development lifecycle"
| exactly. Aren't maintenance and development both part of
| the product lifecycle?
| dimitrios1 wrote:
| Roles are muddy because life is muddy...and because people
| can do more than one thing at a time. Also because save for a
| few roles, businesses rarely need people 100% allocated to
| one role or task. Responsibilities are nice, and job titles
| mean people _mostly_ do x or y.
|
| My favorite example is that of a restaurant. You hire
| waitors, diswashers, bartenders, line cooks, prep cooks,
| hosts and hostesses, and managers. And your waiters might be
| 100% focused on taking orders during prime dining hours, but
| at the end of the night, they help clean up assuming other
| roles. Your bartender might be busy making drinks, but they
| take a table or two if they can. Your prep cook might finish
| the prep early, and come on the line to help out. Your
| manager focuses on expediting everything and keeping the ship
| running smooth, but also can fill holes at time.
| willcipriano wrote:
| > There seems to be a correlation between unnecessary product
| complexity and unnecessary corporate complexity.
|
| That's Conway's law.
|
| "Any organization that designs a system (defined broadly)
| will produce a design whose structure is a copy of the
| organization's communication structure."
|
| https://en.m.wikipedia.org/wiki/Conway%27s_law
| zeruch wrote:
| Conway's Law is sorely overlooked. It really does have
| broad applicability in the industry, and sometimes you can
| use it to evaluate vendors better (read: this product is a
| bit haphazard...the vendor probably is also. Warning.)
| notabee wrote:
| Most of the useful insights in the Devops movement can be
| derived from knowing Conway's law and thinking about
| organizational patterns holistically. As usual the success
| stories get cargo culted by others without understanding
| the principles and thought processes that led there, and
| usually fail because every organization is different and
| has different needs. Change is hard. This will happen with
| every shiny new movement like Devops or Agile.
|
| I'm sure we're due for a new one soon which will follow the
| same path, since people seem to be admitting that Devops
| has problems on here more and more. Will that cause deeper
| reflection? Probably not for many orgs, because soon some
| consultants will dream up a new brand of silver bullet to
| slay the immortal monster of organizational dysfunction and
| that's much more exciting.
| pfarrell wrote:
| Totally agree. The best succinct summary of Conway's Law
| I've heard is "you ship your org chart."
| hinkley wrote:
| Which is a good way to word it because if your org chart
| is screwy enough you won't ship at all.
| writeinpencil wrote:
| I like "the org chart is the asymptote," i.e. it's the
| best you could possibly do, and your reality will
| actually be worse. (see Casey Muratori)
| jmull wrote:
| I'm convinced the use of the term "DevOps" continues only
| because of its incredible polymorphic vagueness...
|
| Two intelligent, sincere, experienced tech people can discuss
| devops, in detail, for a good length of time, and still be
| talking right past each other without really noticing.
| xen2xen1 wrote:
| It exists so people who dole out money can make one person do
| two jobs.
| llbeansandrice wrote:
| Are you hiring? I was hired as an infrastructure engineer and
| I'm currently writing typescript for the frontend of an
| internal product.
| dijit wrote:
| Yes, I am, the project is mostly in stealth mode but please
| feel free to drop me a CV:
|
| jan [at] competition [dot] company
|
| If you want more information on the project we're working on
| the site is https://rennsport.gg
|
| We're building a hard-core racing simulator game with a
| backend which can persist car ownership in a way that feels
| authentic. (IE; not just tied to _a_ game).
| vlunkr wrote:
| DevOps is part of my job description. I was fuzzy on what it
| meant before, after reading this article, I have no idea what
| it means. As a general term, it's far too vague, when in
| practice it seems to just mean that you'll be splitting your
| time between writing regular application code and writing
| infrastructure as code. Oh and you get to be on-call.
| [deleted]
| chasd00 wrote:
| i'm working on an RFP response that has the word "DevSecOps"
| sprinkled around here and there. That's an even more
| subjective and ambiguous term than devops :/
| guhidalg wrote:
| It means you're gonna be doing dev, ops, security,
| compliance, UX, UI, cost optimization, build, test,
| release, and on and on... good luck!
| [deleted]
| usrusr wrote:
| Do those different breeds of engineers work in one team or in
| separate organizational branches?
|
| The problem the term 'devops' is addressing, the way I
| understand it, isn't that of insufficient individual jack-of-
| tradesness, but that of too much separation between (internal)
| organisations. It should be perfectly possible to build a
| "devops" team from deeply specialized experts and turning it
| into a job description seems quite a stretch to me.
|
| But you might actually want some jack-of-trades types
| nonetheless, because those isolated organizations weren't
| completely without merit: they are good at solving that problem
| affectionately called the bus factor. A single ops guy in a
| devops team is a single point of failure and the mitigation
| preparation of keeping some of the not-so-ops peers
| sufficiently in the loop will be much more dependent on organic
| motivation than the counterpart in a specialist company branch.
| There, substitutability would be much easier to ensure with
| formal process.
| calvinmorrison wrote:
| DevOps to me is all about bridging what used to be a huge gap
| between operations and development. Developers need to know top
| 10 OWASP. One cannot simply say "it works locally". Ops people
| need to understand how your tools and programs work at a high
| level at minimum.
| Izmaki wrote:
| These two talks will change your view on DevOps forever:
|
| - https://www.youtube.com/watch?v=zwSNjVTF168
|
| - https://www.youtube.com/watch?v=MnyvgFDh-kw
| calvinmorrison wrote:
| not if I don't watch it.
| dijit wrote:
| Developers and Operations have been working together for a
| long time.
|
| Some teams historically worked fine together, others didn't.
|
| I think the tools we have these days are much better (for
| ops) to ship what works on developers machines.
|
| But if your organisation had a culture of not working
| together then devops didn't really do much.
|
| and anyway, you've leaned into my point about it meaning
| different things to different people.
| la64710 wrote:
| I think everyone can agree that for most complicated things
| there are many sides to a story that makes up the present state
| of the system. I think taking a very opinionated stand in any
| complicated subject by itself negates the possibility of a
| mature and complete analysis of any subject. So the title
| itself of this article is unfortunate.
| kuramitropolis wrote:
| Shouldn't a mature and complete analysis result in a
| conclusion, i.e. an opinionated stand backed up by correct
| reasoning?
| otabdeveloper4 wrote:
| "Sysadmin" got rebranded as "DevOps", because "techsupport" got
| rebranded as "sysadmin".
| Karellen wrote:
| Yeah, I've often seen "DevOps" described as "System
| Administration, but automating everything you can with
| scripting or whatever", to which my first thought was "WTF do
| you think Sysadmins _do_ , if not exactly that?!?"
|
| It's like they'd never heard a BOFH tell someone, "Go away
| now, or I will replace you with a small and uncomplicated
| shell script."
| notakio wrote:
| Yerp. As a former *nix admin, around the time that "DevOps"
| became a term, I read it as "system administrator, who also
| now has to fix developer code." Which, honestly, was already
| part of the job at a lot of places I've worked, owed to the
| age old problem of people who "test" something on their weird
| personal desktop environment, then hand it off and shrug off
| any questions with "well, it works on my desktop". That said,
| the part that I found offensive was that I was doing 2 jobs,
| but only getting paid for one.
|
| Then came DevSecOps. Around that time, I switched to just
| Security, and while I miss the ability to make and push
| changes to hundreds of thousands of machines to get things
| done, I don't miss any of the pressure or blame that
| automatically got lumped onto the sysadmin shoulders every
| time anything went wrong, and the complete lack of
| appreciation of all the times nothing went wrong, that were
| entirely the result of a tireless, efficient systems
| administration team.
| deepstack wrote:
| glad someone said it. I mean all the CI/CD pipeline are just
| in the old day *nix/bsd days bash/perl/awk/sed/python/ruby
| (fill in the blank) scripts. One problem when IT/Software
| development become more main stream is that everything got
| another layer of obfuscation especially in corporate cultures
| and amongst sales and recruiters and managements. Development
| Operation or pipeline sounds sooo much better bash scripts.
| ardfard wrote:
| Just bash script isn't enough if there's no automated
| process that run it every time developers make changes to
| your main/release repository trunk. If you still run it
| manually to verify your build, then it's not a CI. If
| you're still doing manual release deployment, then it's not
| a CD.
| cr4nberry wrote:
| > DevOps has different meaning depending on who you're talking
| to, even some definitions that appear similar are different in
| nuanced but important ways.
|
| This is always a dead giveaway that something is a buzzword
|
| Same for rest. Sometimes when people use it, it just means JSON
| + http requests. Other times it's supposed to be some kind of
| architectural style
| guhidalg wrote:
| You can always point people to the original paper that
| defined REST: https://www.ics.uci.edu/~fielding/pubs/disserta
| tion/rest_arc...
| 331c8c71 wrote:
| Should one also point people to the original vision of OOP
| (by Alan Kay) when they mention the term?
| guhidalg wrote:
| Yes, at least when there's disagreement about what OOP
| means. You and I can disagree about it, but we can more
| easily answer "Does it match what Alan Kay wrote about?"
| with a yes or no.
| [deleted]
| tetha wrote:
| Yep, this has been the hardest fight in our "culture shift
| towards devops" - getting the idea and wording right.
|
| After a lot of discussion, we mostly realized that we get the
| most value if we define more specific operational roles. We now
| have the idea of infra-ops, and product-ops. infra-ops is
| providing a deployment platform - a container runtime and
| persistences. product-ops on the other hand is responsible for
| deploying and running the different products from development
| on this deployment platform.
|
| And this is giving people good ideas. Some products are very
| simple without harsh requirements. In these cases, one of the
| backend devs just takes over the product-ops role by setting up
| a deployment pipeline, a job and some migration handling. Other
| products are bigger - for example we are providing some of our
| systems as essentially managed systems to invididual tenants
| and customers. In such a case, parts of the product operations
| is with the product team - such as writing jobs, releasing
| containers and artifacts, and other parts of the product-ops
| role takes place in the managed services consultancy. These
| products are currently looking at wordings to split up the
| product-ops role according to their needs in their context. And
| that's totally fine - the infra-ops role is also divided
| depending on the system the engineer is working on. A postgres
| admin is an infra-operator, for example.
|
| And all of this results in a devops oriented culture of
| cooperating across team boundaries through automated processes.
| biggu wrote:
| You hire build engineers and release engineers seperately? Does
| your product demand that level of specialization for these 2
| functions? Just curious
| dijit wrote:
| I work on AAA video games, so unfortunately yes.
|
| It's ok to have build-engineer doing release engineering if
| it's simple enough, but doing build engineering properly
| (Source control, artifact control, shared caches, dependency
| management of the compiler outside of the standard toolchain)
| is quite a large job.
|
| Release engineering itself gets complicated when you take
| into consideration the different target platforms, most games
| get published on Playstation, Xbox and one of a handful of PC
| platforms (notably steam), plus the online systems and of
| course the development environments and internal release
| systems which are ubiquitous.
|
| Build Engineering usually is the step between developers and
| QA, and release engineering is the bit after QA.
|
| Both interact a lot with QA so could be rolled into the same
| role.
| chrismarlow9 wrote:
| This is the answer.
|
| I am interviewing right now and my previous experience includes
| devops roles and SRE roles, so I get contacted for both by
| recruiters. After hearing about the responsibilities and
| examples for these roles I can only come to the conclusion that
| titles are a waste of time. Even the "level" of a title of
| Staff, Principal, Lead, etc are a waste of time. Just call the
| job what it is and if you can't decide on what it is at least
| to an 80% level, maybe you're asking too much for that job?
| dogleash wrote:
| >DevOps has different meaning depending on who you're talking
| to
|
| 1000% this.
|
| I think it's an instance of how the conversation in software
| always talks about solutions as if they're end-all-be-all
| answers, rather than explain a problem in specifics and why
| that made the solution the right answer at the time.
|
| Then that "solution" becomes a buzzword version of itself, and
| popular buzzwords are tools leaders use to overcome
| institutional inertia. Which is a good thing, to get over that
| inertia. But then what comes misses the insight and
| understanding that eventually turned into a buzzword.
| Izmaki wrote:
| You hire those engineers and then you have them implement and
| support Continuous Delivery for your business, yes? DevOps is a
| silly term, but one that companies seem to use for
| "infrastructure engineers who help us continuously deliver
| value to our customers, efficiently and pain-free". Give your
| people whatever title you feel comfortable with - but
| essentially they are "doing the DevOps". :P
| Izmaki wrote:
| Did I get downvoted because I'm right and the truth hurts or
| is there a disagreement somebody chose to not share? Hmm...
| burnished wrote:
| You basically assert that the other person is categorically
| wrong without argument but by rereredefining words. It
| doesn't read as productive to me.
| xtracto wrote:
| Some time ago, i was talking to an MBA people /project manager
| guy was ho had a completely different definition of what
| "devops" meant. His meaning was some kind of project management
| approach.
|
| We have a way to prostitute words in this field... agile, QA,
| devops.
| zerkten wrote:
| >> I became CTO so I can solve this mess properly, I don't hire
| devops, I hire infra engineers, build engineers, release
| engineers and: backend engineers.
|
| This makes a lot of sense, but you need to realize that this is
| only a tiny part of a successful organization. There are many
| setups like that which you describe across many industries that
| have failed reasons beyond the role definitions. It's not
| enough to establish the organization. You have to keep the
| behaviors in check over time when you hire a build engineer
| with different aspirations.
| rubyist5eva wrote:
| "devops" is just when developers do operations, and most
| developers generally don't want to because they are more
| interesting in writing software. If your job is just to manage
| infrastructure and you use terraform, you're still just a
| sysadmin. It's a "boring" job but someone has to do it.
| fleddr wrote:
| "From full stack engineering to DevOps practitioner, our industry
| loves to pretend everyone can do everything."
|
| To me, this gets to the heart of the matter. As example, I
| present the required skillset for a front-end engineer:
|
| https://frontendmasters.com/guides/front-end-handbook/2018/
|
| See the index on the left. Do you need to know everything about
| all of these things for every single project? No. But as you grow
| into a senior, you'll be touching almost everything in that list.
|
| It's an absolute explosion in complexity. Web development once
| was barely considered engineering, now it's one of the most
| complicated roles in the industry. Consider also that almost
| everything on that list is constantly evolving, this list being 4
| years old.
|
| Has all this added cognitive load and complexity resulted in
| massive productivity wins and dramatically better outcomes (UX,
| quality)? I'd say no, or at least it's questionable.
|
| My point being is that this is already too much. I work in teams
| with a distribution like this: senior (20%), medior (50%), junior
| (30%). So the vast majority of them are median. And the median
| programmer is severely lacking against our ever growing demands.
| It's crude, but the typical programmer really sucks at
| programming.
|
| So if next you're going to add even more to this pile with all
| sorts of devops and funky cloud tooling, the issue becomes clear:
| we're over-asking.
|
| We over-value flexibility and scaling but ignore its dramatic
| costs.
|
| "Devops" is just a made up word.
| g051051 wrote:
| > The problem is most engineers don't want to do operations work.
|
| Glad to finally see someone else saying this.
| taylodl wrote:
| I disagree. DevOps was intended to break the organizational
| mindset of having this group of people here doing this set of
| activities and this other group of people over there doing that
| set of activities when in reality both groups of people are
| needed to work together and deploy software. It was about
| breaking down those organizational silos. Anybody creating a
| dedicated "DevOps" team was way off the mark, unless that team
| was a coaching team whose job it was to help other teams become
| self-sufficient. Unfortunately the fact that an engineering team
| had responsibility for their software from implementation, to
| deployment, to operations was a news flash to large organizations
| having traditional IT staff. The DevOps mindset has led to much
| better operational excellence.
|
| The DevOps mindset has also led to the Internal Developer
| Platform this article discusses. Honestly, I don't see how
| traditional IT organizations are going to easily arrive at such a
| platform without having adopted the DevOps mindset first.
|
| So DevOps isn't bullshit, but it's not the end goal either. It's
| a necessary step needed on your journey for getting somewhere
| better.
| ozim wrote:
| I think that was not mentality. Maybe after years of working
| that way it turned out into mentality.
|
| That was employee utilization approach where you hire 1 DBA and
| he runs all DB stuff because hiring DBA for each team does not
| make financially sense as there is not enough day to day work
| for DBA specialist on a project/product.
|
| Other stuff is that DBA/SysAdmins have to have access to
| customer - company data so you still need separation of duties
| (no devs access on prod systems) and it is easier to make guy
| part of "OpsTeam" and give him access across all systems than
| get "Ops" person configured per project\product.
|
| So why "DevOps" if you get operational overhead and you cannot
| "utilize employee 100%"? Because in most companies delivering
| new features is more important than making Joe Dbaer closing
| tickets like in factory because we learned it actually is not
| efficient when "important feature X" is delayed because Joe was
| doing his job fine but feature X was in queue.
| [deleted]
| WolfOliver wrote:
| see: "The Big DevOps Misunderstanding"
|
| [01] https://linkedrecords.com/the-big-devops-
| misunderstanding-84...
| irrational wrote:
| I hate devops. We used to have a dedicated systems team. Now
| programmers are expected to both write code and manage their own
| cloud infrastructure. These are two entirely different skill
| sets.
| chasd00 wrote:
| "The growing zeitgeist is that "platform engineering is the
| future."[1][2] And given that I co-founded a product in the
| space, I sure hope so! "
|
| oh, now i understand this blog post..
| counttheforks wrote:
| > The problem is most engineers don't want to do operations work.
|
| There's your problem. You have people who build stuff without
| caring where and how it runs. Recipe for disaster.
| cr4nberry wrote:
| Where and how it runs can get incredibly complicated. It isn't
| a matter of not caring about where it runs, but having a finite
| amount of time
| hden wrote:
| Sounds like a cultural problem and/or hiring problem.
| bluedino wrote:
| > You have people who build stuff without caring where and how
| it runs.
|
| _In the old days_ , developers would also configure the design
| of the system it ran on.
|
| "Alright, we're going to use some fast CPU front end nodes with
| smaller, faster disks, and then have a load balancer, our
| database server is going to have the most RAM, now let's set
| these resources up on a CDN to serve faster, this memacache
| server is really going to lighten the load..."
|
| I think the idea was to use tools that automatically did all
| this work for you, but it just ended up creating another tier
| of people that configure this side of it.
| mandelbrotwurst wrote:
| Equally common I think are engineers who care where and how
| things run, but are either disincentivized from or even
| disallowed from working on those aspects of systems.
| icedchai wrote:
| It's always been like this. I'm a backend engineer that also
| does cloud operations / systems administration / network
| engineering / "devops" work. The job titles have changed, but
| the truth remains: (most) developers want to develop, not deal
| with infrastructure.
| quintu5 wrote:
| Agreed. I get a rush from designing a system from the ground
| up and bringing it to life with all the infra, but only when
| that's necessary. If it's just a basic CRUD app without any
| major performance or resiliency requirements, I'm more than
| happy to do as little of that infra work as is necessary to
| get it into production and move on with my life.
| dunno7456 wrote:
| > There's your problem. You have people who build stuff without
| caring where and how it runs. Recipe for disaster.
|
| Spot on, but not only that, also decisions made based on number
| of hands in a room.
| Merad wrote:
| I absolutely care where and how my code runs. And yet I don't
| give a damn about writing terraform, managing IAM, VPCs,
| keeping up with backups, upgrades, etc. I want specialists who
| know what they're doing to deal with that stuff.
|
| I'm also not convinced that the average junior or mid level
| application dev needs to know where or how their stuff beyond
| some high level concepts (i.e. our app uses auto scaling).
| lucasyvas wrote:
| It is more nuanced. I care, but my skillset demands I spend
| more time on the Development side, so Operations _must_ take
| the backseat in my mind because of this incentive structure.
| Asking one person to do two (actually three) job functions is a
| scam.
|
| Most developers have to do Frontend, Backend, and Ops. These
| have wildly different mindsets and feedback loops and not
| enough time exists. Don't hate the player, hate the game. The
| orgs are fucked, not the workers.
| jayd16 wrote:
| I think the argument is that calling ops different is like
| saying error handling is another department. To be a complete
| dev you must understand the code in situ regardless of your
| specialty.
| makestuff wrote:
| It is like your doctor being the anesthesiologist, the
| recovery room nurse, and the surgeon all at once.
| dilyevsky wrote:
| Race car driver vs bus driver. Race car driver knows how
| their car functions and works with engineers/mechanics on
| improvements. Bus driver don't give a shit (no offense to
| bus driver). Decide which company you are - a race team or
| a bus line
| Multicomp wrote:
| I will extend this analogy to say I am a race car
| mechanic but I'm having to pick up how the driver drives
| the car in order to make the car perform better, with the
| eventual goal of being able to drive the car myself (ops
| -> devops -> dev) so I can do any of the above roles.
|
| Software devs who close their mind to the lower parts of
| the stack give me the opportunity to learn and do their
| jobs as well, becoming a more valuable employee as a
| result.
| dilyevsky wrote:
| > becoming a more valuable employee as a result.
|
| That's iyho. The management of a bus line company doesn't
| see it that way ;)
| danielvaughn wrote:
| Thank you, yes this is the perfect analogy. I think every
| developer should have some kind of baseline understanding
| of devops stuff. But it's a whole field in and of itself,
| and is also innovating fairly rapidly.
|
| There are only so many hours in the day, it's hard enough
| to stay on top of my core skill set.
| hbarka wrote:
| Bad analogy. How about your mom asking you to help wash the
| dishes and take out the garbage?
| Smaug123 wrote:
| That is an _extremely_ bad analogy. The purpose of chores
| is to spread the load onto an otherwise unproductive
| member of the household. (And to teach said household
| member how to do them.)
|
| Unless you're arguing that engineers are unproductive
| members of a company, I suppose.
| makestuff wrote:
| Well I didn't have to get an education to do that and if
| I mess up washing the dishes I didn't cause a death in
| the case of a doctor or a multi million dollar outage in
| tech.
| wruza wrote:
| She doesn't impose multiple business-critical schedules
| on that, so you can just wash them, pick your nose for a
| while, take out the garbage and go play games. There is
| nothing wrong with doing related jobs solo when situation
| allows, but this example is also far from learning and
| using "modern" full-stack of stacks or doing surgeries.
| lolinder wrote:
| > Asking one person to do two (actually three) job functions
| is a scam.
|
| The word "scam" is overused. It's a scam if it was a bait-
| and-switch. It's not a scam if it was advertised this way up
| front. In the latter case it's just a job whose description
| you don't like.
| lucasyvas wrote:
| It is a scam. It should absolutely be assumed that you
| _could_ perform these job functions - it makes you a way
| better engineer! But being _expected_ to is a scam because
| the skill ceiling on each is so amazingly high that only a
| scam artist would expect someone to be able to perform all
| three to a specialist level. The same issue happened when
| QA departments got the boot. I deeply understand testing
| strategies (both manual and automated) but this is an
| important enough job that it deserves a dedicated person.
| Not having this person is exclusively because the business
| is trying to save money and make the worker easily
| replaceable.
|
| It is not better for the department members or the software
| itself. It is only good for the bottom line. Yes, if you
| know it all you'll be valued higher - but the company is
| still reaping the majority of that benefit from you and not
| rewarding you with 3 salaries. You should not perform all
| the roles because it's economically idiotic to do so. You
| could contract out all 3 services and make a shit ton more
| money than you would performing them in a salaried
| position.
|
| That is a scam.
| lolinder wrote:
| I perform all three roles for one company. I agreed to it
| upfront. I do it in less than 40 hours a week at a very
| respectable salary. I get maybe 1 after hours call a year
| for an Ops task. My boss knows that I won't do QA or Ops
| nearly as well as I do Dev, and he knows that QA and Ops
| time come out of Dev time (not personal time) and he's
| okay with that because that's the stage we're in.
|
| If you think _that_ is a scam, I don 't know what to tell
| you.
| briandon wrote:
| But isn't this the norm in many other fields? The engineers who
| design automobiles, aircraft, kitchen blenders, etc. aren't
| expected to maintain and repair the cars, planes, blenders.
| etc. after they're put into use. They go on to design other
| products. Architects and construction/civil-engineering firms
| design and build large structures and then hand them off to
| others to maintain. Authors generally write a book and then go
| on to write the next book rather than dedicating themselves to
| carefully curating a list of errata for a single book. There
| are academic authors who do play a part in churning out new,
| slightly modified editions of major textbooks every few years
| but they're not on-call and being woken up in the middle of the
| night to fix a punctuation error on page 982 of the physics
| text they co-authored.
| brundolf wrote:
| It's a context-switch. It's a huge, unavoidable context-switch
| squarely in the middle of your workflow. And _those_ are a
| recipe for disaster:
| https://thinkingthrough.substack.com/p/context-switching-cos...
| erokar wrote:
| You'll spread yourself too thin as a developer if you need to
| do a lot of ops too. Specialisation is a good thing.
|
| You don't want a doctor who brags that she can do heart
| surgery, perform talk therapy and treat your skin rash. You
| want someone how knows a lot about your specific problem
| because she's prioritised it over other things.
| rr808 wrote:
| You mean people relaxing WFH in a unbelievably great job market
| dont like being on call 24/7 and supporting random stuff they
| didn't build. Sounds like we need a recession.
| Test0129 wrote:
| I don't understand. I want to develop code. I don't want to
| become an AWS/S3/Github/Jenkins/Action/terraform/etc expert. I
| know enough of this to be dangerous but not at a level that
| passes as professional. Yet I am regularly tasked with
| maintaining the full deployment of code.
|
| There's a reason to have a team of people doing this "DevOps"
| work. Just like we have a team of people who do SRE. It creates
| a standard and a single point which all work flows through.
| Then you don't wander onto a new project only to realize they
| use $BESPOKE_DEPLOYMENT_METHOD because "it's what we used 6
| months ago". Or worse, you don't have a developer playing with
| a massive, nuclear powered, foot gun like Terraform and
| accidentally destroying infrastructure.
|
| Making DevOps/DevSecOps/$BUZZWORD the responsibility of
| developers is a cost-cutting measure not a responsibility
| measure.
| c0mptonFP wrote:
| A developer who doesn't have a holistic view on their systems
| is a liability.
| [deleted]
| 0xbadcafebee wrote:
| DevOps came about because of developers who "just wanted to
| write code". They would write something, then throw the dead
| cat over the wall and say "you figure out how to run it".
| That... doesn't really work. Somebody needs to explain to the
| Ops people how to run the code. Hence: DevOps... a way to get
| Dev and Ops to avoid throwing dead cats over walls.
|
| If you don't want to think about AWS S3, GitHub
| Actions/Jenkins, Terraform, etc, .... then we need to work
| together. All those tools and services exist because all the
| software developers are sitting in their sandbox, and don't
| want to come out and play. The systems and tools that we run
| your code with... suck. A lot. We need programmers to make
| the systems better. We (in Ops) are a little busy with trying
| to just figure out how to run your apps without them falling
| down. We don't have a lot of time to reinvent the state of
| the art of computer systems.
|
| For example, we need a distributed operating system. Not some
| fucked-up kludge of a monolith of microservices overseen by a
| company that has more engineers than brains... but an honest
| to god, stable-ABI, simple, composeable, stable, general
| operating system. We need Linux to come out of the box, ready
| to run distributed applications, in a way that doesn't
| require a PhD. Once we have that, then you - yes, you, the
| developer! - will be able to make applications that
| automatically scale so easily that we will never need to
| utter the phrase "container" ever again. You will rarely ever
| need us again, because the system will just be so simple, so
| general, that anybody who can use the terminal can build and
| deploy applications without ever learning anything outside of
| your programming framework.
|
| But we need _you_ to make that distributed operating system.
| Until you do, we will just have more stupid kludges, more
| bizarre unnecessary complexity, in the futile attempt to
| constrain all the crazy shit we want to do with technology,
| while trying to run your apps _for you_. Please, I 'm begging
| you - put me out of a job.
| dr-detroit wrote:
| mikkergp wrote:
| It just doesn't work for the most part. Maybe as an Ops
| person, I just want to do Ops, I don't want to have to
| understand your code, but Lambda has specific limitations on
| how long code can run for. I can't allocate CPU and Memory
| resources in Kubernetes without a deep understanding of the
| application. S3 has limitations on how files can be
| distributed and accessed. Integrating with CI gets
| complicated quickly and requires understanding the code being
| integrated. A lot of the time building things in terraform I
| spend three times as much time getting the information out of
| a developer as they would doing it themselves. I mean yes,
| there's 60% or so of my job that involves working on lower
| level infrastructure that doesn't touch a dev, and we do need
| ops engineers for that, but the other 40%? It works both
| ways, a good "devops engineer" needs to understand code, but
| if we don't have a shared language we both just end up
| banging our heads against the wall.
|
| If you're building a crud-app in a common framework with low
| volume, sure you can toss that over the wall.
| mattgreenrocks wrote:
| This makes me wonder if ops allocating compute resources is
| really a good use of time if you're needing precise details
| of an app (which can and do evolve). This isn't a slam
| against ops, either, it's a knock against the tech itself
| that it forces all this incidental complexity on you.
| mikkergp wrote:
| Yeah, I mean, fundamentally it's so complex because you
| have to make tradeoffs and people hate tradeoffs
|
| "I don't want to have to worry about what machine my app
| runs on" vs "kubernetes is to complex"
|
| "Dependencies change to often" vs "I don't have time to
| maintain this thing I wrote myself"
|
| "I just want the infrastructure to figure out what I
| need" vs "I want to be able to build whatever I want with
| a bespoke language/framework/database/architecture"
|
| > it's a knock against the tech itself that it forces all
| this incidental complexity on you.
|
| If I were to say that Kubernetes is the magic secret
| sauce that fixed all the incidental complexity I would
| get laughed out of the room. There is no magic secret
| sauce to the incidental complexity, the more we try to
| fix it the more we create (or the cheap-fast-good
| problem. This is probably easy if there are no cost
| limitations)
| Sakos wrote:
| Our devops runs the infrastructure, but details like "what
| resources do we allocate where" are done primarily by the
| software devs. I don't really see the conflict here. I
| don't know and I don't care about how to put together the
| infrastructure required so I can change the CPU allocation
| on a Kubernetes pod, but I also don't expect devops to know
| jackshit about our code.
| Spivak wrote:
| I think you make a strong case for ops who can dev, but a
| fairly weak one for devs who can ops so I think you and the
| parent are actually agreeing. And this mirrors my
| experience pretty well, I need to know the code to be able
| to ops effectively but it's much rarer that devs need to
| know how to ops to dev effectively.
|
| And in some ways this is by design, I want to have some
| distance between dev and ops because it gives me the
| freedom to rearrange infrastructure transparently. I can
| move workloads between Lambda, ECS, and EC2 based on the
| observed performance characteristics without anyone being
| the wiser.
| mikkergp wrote:
| This is going to depend on a lot of things. Size of
| company, cloud native or not, org structure. I mean
| everyone would love to live in the google world where a
| team of SRE's run everything. But even in the world where
| a Devops engineer is embedded on a team there's bus
| factor to consider.
|
| I think most modern AWS services are more the equivalent
| of an API or a microservice than they are a server, and
| you need to understand the limitations of the services
| you integrate with. If you're a cloud native company and
| don't have a mature platform engineering team, devs are
| going to have to know alot about AWS.
|
| If the developer has all the information necessary to
| create an S3 bucket, lambda function, kinesis stream, etc
| does it make sense for them to offload 10 pieces of
| information to me, or for them to learn HCL and interact
| with it themselves, especially if it is something they do
| often. Especially if there's a central dev/ops team and
| they're the limiting factor. Devops taking over every
| infrastructure change for a broad team of devs is
| inefficient and expensive. It's also probably frustrating
| for devs that are aware of the above factors. Lots of
| devs I know would prefer to do it themselves.
|
| And all of this is to mention that infrastructure and
| integrations with infrastructure are not static. Should I
| be reviewing all PR's to a system that touch the client
| code for an AWS service, because the dev team doesn't
| want to learn how that service works? Maybe, but in the
| end I don't know if this makes anyone happy.
|
| Platform engineering is certainly the goal. Where ops
| creates a platform and dev just consumes that platform.
| But I don't know if it is realistic. Every system on
| rails is great until you try to take the rails off, and
| most devs I know hate rails :-) Fundamentally "devops" is
| all meant to solve human problems, not tech problems so
| it will have to be dynamic.
| Test0129 wrote:
| At companies I've worked for where they actually do this
| the DevOps people are generally assigned to a team. So you
| might have 10 developers on a project, and one guy managing
| _just_ the devops exactly to solve this problem. In smaller
| companies /etc I can see where this is a problem. I agree,
| and there's nothing wrong with knowledge sharing. This
| could even be as simple as having PR descriptions including
| benchmarks/code/etc and making sure the devops people make
| their points clear during design/planning.
|
| In theory, devops shouldn't be responsible for maintaining
| the _performance_ of code. The specification should say
| what it should run on, the devops guys set up a pipeline
| and manage that thing, and the developers are the ones
| taking heat for not hitting that goal. If devops guys are
| taking the heat for that it sounds more like cost-cutting
| measures flowing the other direction.
| mikkergp wrote:
| > DevOps people are generally assigned to a team
|
| Yeah and this is a great solution (aside from the bus
| factor), maybe the person I responded to was really
| concerned less about knowledge and more about expertise
| (the word they did use in their was expert) At a company
| I worked with we called this T-shaped engineers. Deep in
| one thing, but broadly knowledgeable. Devs have to have
| knowledge of ops, but not "expertise" that is ultimately
| what ops is for, we may just ultimately be fighting over
| where the knowledge line is sufficient and what
| constitutes "expertise" :-). I for instance think
| terraform is not that much of a footgun and provides good
| rails to be used by developers.
| kennend3 wrote:
| > Making DevOps/DevSecOps/$BUZZWORD the responsibility of
| developers is a cost-cutting measure not a responsibility
| measure.
|
| My background is large multinationals so my view here is a
| bit bias but i don't think cost cutting is the driver.
|
| Large orgs get large change management processes and
| procedures. Over time, these change management teams become
| overwhelming behemoths with minds of their own.
|
| I think "DevOps" was designed as a way to "bypass" the
| bureaucracy?
|
| "We just use this CI/CD pipeline and no need to sit on a 3
| hour change management review call..."
| jayofdoom wrote:
| Almost everywhere I've worked that I helped run software in
| production for would have a step in CD which filed an
| automatically-approvable change req with automation; just
| like the automated deploys.
|
| It becomes just robots pushing around paper for compliance.
| mritun wrote:
| > I don't understand. I want to develop code. I don't want to
| become an AWS/S3/Github/Jenkins/Action/terraform/etc expert.
|
| They're tools for the job, like your compiler and the
| language you use to program.
|
| That's like saying "I like to use Python and couldn't care
| less for Java". It's fine to disagree with the team's choice
| of the tools, but one needs to eventually commit to the
| choice, even it's not your preferred one!
|
| There's an old adage that "if you're writing clever code then
| you may not be clever enough to debug it". It's true!
| Operations requires deep understanding of the code running in
| production, the business rules and the customer. The person
| who will operate your code will eventually be smart enough to
| develop it entirely too, eventually cutting out the "dev".
| I've personally seen this happen over and over.
| scarface74 wrote:
| Alan Kay said that "Everyone who is serious about software
| should make their own hardware." How can you be a good
| developer if you don't understand the architectural
| limitations and choices?
|
| When I design a backend system, I need to think about how the
| front end developers are going to interact with it. My data
| storage characteristics and scaling. I need to know am I
| designing anything that's hard to deploy. How will logging
| work and be aggregated. I have to be able to think about the
| entire system.
|
| It's not just "cost cutting" at a certain point in your
| career you are expected to know more than just "how to code".
| I'm not saying learn AWS. But I would expect any senior
| developer to know about what their code runs on top of .
| Test0129 wrote:
| I don't think you're disagreeing with me. Developers should
| know what their code runs on. They shouldn't have to add
| managing that to an already full schedule of work. That's
| the difference.
| scarface74 wrote:
| Back when I was in the real world [1] working for a
| startup, I would do your typical serviceless solution
| with Lambdas, S3,SQS, etc. I couldn't just use ClickOps
| and create everything on the console and expect someone
| else to recreate everything with IAC. I had to know how
| to do it.
|
| I think to push back is rightfully coming from the "ops"
| part. I consider "creating the
| CloudFormation/CDK/Terraform" code as part of
| "development" as part of coding.
|
| If you use Docker, wouldn't you consider creating the
| Dockerfile as part of development?
|
| Yes I knew AWS pretty well by the time I left and I
| needed to know it to be a good developer in that context
| and designed most of the processes around it. But I
| refused to do "operations" - ie "infrastructure
| babysitting"
|
| There is a huge distinction between "I don't think I
| should have to know how everything works" and "don't call
| me in the middle of the night when something goes down ".
|
| [1] I'm the first to admit that I left the "real world"
| once I started working in the cloud consulting department
| at $BigTech
| Aperocky wrote:
| > Yet I am regularly tasked with maintaining the full
| deployment of code.
|
| Same shoes, but I have different perspectives. I can figure
| out where it's not working and if it's my/our team's area of
| responsibility then we go fix it.
|
| Since we handle infrastructure (as code) and deployments in
| team, along with all the development, most of the problems
| are handled by us. Unless it's clear that it isn't, e.g. some
| API that we consume that keeps throwing 500, can't fix that.
|
| Our operations is helped by 100s of automated tests and 1000s
| of metrics. I always thought that this is DevOps, but it
| sounds different from what most of the people here are
| alluding to.
| williamcotton wrote:
| Well what about Google Sheets? Lots of people are able to build
| things in Google Sheets without knowing where and how it runs.
|
| Perhaps most of our organizations are using the wrong tools and
| that the way we divide responsibilities is the problem.
|
| Imagine if:
|
| 1.) instead of general purpose languages being used by product
| developers we had domain-specific languages and tooling that
| were agnostic to things like deployment, memory-management,
| data storage
|
| 2.) computer engineers built the domain-specific languages and
| tooling and focused on things like deployment, memory-
| management and data storage.
|
| DevOps is generally solving for general purpose development
| while developers are generally solving domain-specific problems
| with general purpose tools. This seems to be a large part of
| the disconnect!
| sicp-enjoyer wrote:
| That's an interesting idea. I wonder what constraints led us
| to the current division of labor, and which ones are
| artificial.
| williamcotton wrote:
| I have no idea!
|
| I'm currently working on a small team of computer engineers
| who support a custom Turing-complete DSL for an
| "operations" team that both programs and directly
| interfaces with clients.
|
| It doesn't seem that different from what Apple or Microsoft
| do, which is write custom languages and tools for
| developing on their platforms, albeit specifically for
| general purpose programming.
| wry_discontent wrote:
| I totally don't want to do operations work.
|
| I've done some, and it feels like reading an arcane text in
| another language. I've seen the thing I want before, so I know
| there exists a magical YAML incantation that brings it about
| but I have no idea how to even start looking for it. It
| devolves to an extremely slow feedback loop of "did that work?
| No."
|
| I've never found anything that explained how I'm supposed to
| create a systematic approach to creating anything I need.
| sgarland wrote:
| Can't the exact same argument be made for code, though?
|
| "That didn't work, and the traceback (if I read it) isn't
| very helpful. Guess I'll try something else I found on SO."
|
| You learn ops by doing it, preferably in an environment where
| when you inevitably do it wrong, it doesn't break prod.
| kevin_thibedeau wrote:
| That used to be a separate job called sys admin.
| francisofascii wrote:
| Previously we had better division of labor IMHO. 1) UX people
| (Photoshop, CSS) who made the front-end look nice, 2)
| Developers (JavaScript, C#, Ruby, Java) who wrote application
| code and business logic, 3) DBAs who controlled the database,
| and 4) Sys Admins (Unix, Windows Server) who deployed the built
| code and ensured it ran correctly and securely in production.
| Now some of us seem to think this should all be the same
| discipline, and if you don't care to, you are a bad engineer.
| alexjplant wrote:
| Sometimes the engineers _do_ care about the deployment and
| operations of their apps and management doesn't let them
| administer them in production under the guise of "separation of
| responsibilities" or "regulatory compliance". It makes the ops
| team unhappy that they're on the hook for stuff they didn't
| build and the dev team unhappy because they're steering their
| metaphorical ship by proxy.
|
| I'm not sure why this happens but it does and I don't like it
| (as somebody who's very M-shaped).
| kbenson wrote:
| It's because often your software engineers know software
| development and just enough administration and operations to
| be dangerous, and your administrators know operations and
| just enough software development to be dangerous.
|
| Letting them wade into others responsibilities often
| introduces a lot of that danger.
|
| System administration is an entire discipline with a history
| and way of thinking and approaching problems at a systems
| level. Software engineering is the same, but with an approach
| that often emphasizes handling problems in the application.
|
| Letting solutions be put into applications that are best
| handled at the system level (where system isn't just an OS,
| but might be a complex network of them), or put into the
| system when it could easily be handled by the application,
| caused inefficiency and problems.
|
| The solution is to either hire only very accomplished people
| that can do both and make the right decision of where to put
| solutions, or hire people that can do one or the other and
| put them together with a few (maybe one) person that has a
| good grasp of both and can make executive decisions when the
| right time and place is for specific solutions (an
| architect).
|
| The latter is easier to hire for.
| g051051 wrote:
| > You have people who build stuff without caring where and how
| it runs.
|
| Why would someone who wants to build stuff not care about where
| and how it runs? Operational requirements are the same as any
| other requirements.
| LaGrange wrote:
| Yup. I basically came here to quote the same line answering
| with "And I don't want to work with them."
|
| Though mind you, the article touches - but doesn't go into -
| the real reason things go shit. It's not ops, it's not devops,
| because your knowledge silo can too, and in my experience
| (that's longer than the writers, lol) _will be_ pressured.
|
| If the estimates are consistently too low (because honest ones
| would render many - maybe most - projects non-viable), the
| requirements are often bullshit (because otherwise the lies in
| the estimate would be too obvious, and everyone needs to agree
| on the lie) and and rewards are not the ones the jobs claims to
| be (in every workplace, your job is to improve your resume.
| That's all. That's the only thing that, materially, matters for
| your career. I can care about software I write, but if anything
| I end up punished for it) - then no amount of renaming things
| and shifting org charts around or motivational reading for
| managers is going to fix that. Nor will a Cloud or latest
| orchestration software.
| BiteCode_dev wrote:
| The architecte and the carpenter can be the same person on a
| small house, but I don't think it should on a big office
| building.
| AcerbicZero wrote:
| I was semi salty reading this....but they are spot on in a lot of
| ways. There are times where my code looks like a bad 4 year old
| wrote it, but 10 minutes later I'm in a conversation where I have
| to explain basic security concepts, like not leaving everything
| wide open to the internet, to some random Sr Software Dev.
| For every operations person without software development skills,
| there are FORTY engineers without cloud operations skills. If you
| are going to build an internal platform, you'll need experts with
| overlapping experience in both fields working together.
|
| I guess I just need to find a role with more inter-team
| collaboration; Being able to mostly self teach is great, until
| you have no one to learn from anymore.
| make_me_rich wrote:
| How fucking annoying that stupid Newsletter Button is on mobile!
| davydog187 wrote:
| Agreed! Its gonna get turned off in a moment
| jacquesc wrote:
| Agreed, it covers up the article text. Complete BS
|
| Reader mode on Safari is a savior in these cases.
| papito wrote:
| Part of why I am fed up with this industry is that I no longer
| know what my role is. It seems that I have to:
|
| * Develop new features
|
| * Fix bugs
|
| * Sit in standup, "alignment" meetings, and any meeting created
| by a Zoom-promiscuous engineer (let's hop on a call!)
|
| * Reply to dozens of Slack messages per day
|
| * Write documentation
|
| * Develop and maintain the CI/CD pipeline
|
| * Be an expert on Observability (aka Debugging in Production)
|
| * Have Galaxy Brain level knowledge on the entire cloud setup in
| all environments
|
| And now I am thinking, if being an engineer involves all that,
| why not just find a gig with pure DevOps and not worry about
| product development?
| justin_oaks wrote:
| I'm not sure if having two teams is always going to be better
| than having one DevOps team, but my experience in having two
| teams is that it's rare to have the incentives aligned. The
| author of the post pointed out that dev teams will cut corners
| and throw broken applications over the wall to ops to deal with.
|
| When ops gets woken up at 2am because someone in dev cut corners,
| what happens? Does dev feel the pain? Almost never.
|
| The same thing happens when outside contractors develop code.
| They often provide a buggy and undocumented mess, and then no
| longer work on the project. They never feel the pain, so they're
| not incentivized to provide good code.
|
| Until we find ways to align incentives, we're going to keep
| getting crap whenever more than one team is involved.
| jollyllama wrote:
| So a symbol i.e. "DevOps" gets corrupted. What do you do? Make a
| new symbol, like the author is suggesting? Or try to reclaim the
| term? This happens time and again and I find the question
| fascinating.
|
| FWIW, "Platform Engineering" used to be something different than
| what the author is suggesting.
| ChrisMarshallNY wrote:
| When I ran a fairly small team of engineers, I created what I
| called an "Infrastructure Engineer." I staffed it with a fairly
| junior, but still brilliant, engineer.
|
| He rapidly became the most popular member of my team.
|
| His job was to commoditize configuration management, and strip
| away as much of the overhead from the coders, as possible. He
| didn't do release management, and we didn't really work automatic
| testing into our release workflow. This was because Japan did not
| trust auto-testing, so each engineer did their own unit and
| harness testing. Japan also wanted each engineer to make their
| own "official" release, as opposed to having a CD system spit it
| out.
|
| There were reasons. I didn't necessarily find them that
| compelling, but they were the boss, so I gave them what they
| wanted.
|
| Japan liked him, as it gave them one single person to talk to,
| and he also helped them to streamline their own infrastructure.
| In fact, he is the only employee that I ever had (including
| myself), that traveled to Japan before being there a year.
|
| For myself, I find using things like Fastlane, JIRA, and Jenkins,
| aren't actually helpful, for a one-man shop. I tend to do a lot
| of stuff by hand.
| henning wrote:
| > In production, we are running in containers, but developing in
| the container is too slow, so the team leans towards asdf and a
| README full of stuff to copy, paste, and pray. During a sprint,
| an engineer adds convert (ImageMagick) to the mix to support
| manipulating images and forgets to update the Dockerfile, and
| then production goes down.
|
| Wow, it's like you can prove anything with a contrived example!
| turtlebits wrote:
| I like doing "devops"/operations work. Getting a service out the
| door with well thought out infrastructure, plus
| observability/alerting/runbooks to be handed off to an Ops/SOC
| team is a great feeling. Then you work on the next service.
|
| If you don't have some sort of Devops team (yes, the term is too
| broad) to push back on infrastructure complexity and ensure that
| logging/metrics/alerting/documentation is done, developers just
| won't do it.
| kris-nova wrote:
| Platform Engineering is the future.
| Jtsummers wrote:
| DevOps has suffered the same fate as Agile. In particular there
| is one thing that both had in common that got lost somewhere in
| most implementations: cross-functional teams [0].
|
| Agile teams were supposed to be composed of not just devs doing
| everything (quickly, because agile is about raw speed, right?)
| but people competent in the various aspects of the system
| development, potentially deployment, customer needs, etc. working
| together as multidisciplinary teams to meet a particular
| objective. The purpose of this was to break the silo that is
| common because it feels natural to many managers (the same sort,
| I presume, who don't like when their mashed potatoes touch their
| fried chicken on the plate).
|
| Silos impede communication and promote the "throw it over the
| wall" approach to product/system development. A system engineer
| (or team of) made the design after sales (and possibly only
| sales) talked to the users. Throw that design over the wall and
| let the devs build it. Devs throw it over the wall to test, maybe
| there's a volley. Eventually it's tossed to ops. A goat is
| sacrificed and maybe it works.
|
| Multidisciplinary teams are able to communicate across those
| boundaries because instead of the role-silo the roles are all in
| the same team, working (more clearly) towards one common
| objective. But then businesses managed to fuck it up. They got
| rid of test, devs do _all_ testing now. Devs do all the database
| logic. There are no UI /UX experts anymore, it's all full-stack,
| and on and on.
|
| DevOps was supposed to be the same. It was supposed to take that
| Agile cross-functional team and add in the sysadmin/operators
| (among other things). The critical problem being solved was the
| two (really more, but at the limit) silos of dev and ops failing
| to communicate. A Friday release with Ops spending the weekend
| rolling things back because Dev wasn't there and didn't even know
| the system wouldn't work as intended. Maybe their test
| environment was too different from the operational environment,
| the reasons matter a bit but there are too many to enumerate.
|
| Instead, businesses did what they do best, they fucked it up.
| Again. They said, "Take the sysadmins and teach them to code. If
| they can't, dump them in the woods. The devs will takeover the
| world." Then they started making "devops" job listings and full-
| stack grew to encompass a new set of skills.
|
| [0] https://en.wikipedia.org/wiki/Cross-functional_team
| znpy wrote:
| > The problem is most engineers don't want to do operations work.
|
| Nah, the problem is that most (software) engineers don't have the
| skills to do operations work.
|
| Let's not fool ourselves: software development is 95% development
| (writing code/docs/test etc) and 5% system administration
| (getting your local mysql or whatever up and running etc) whereas
| operations is usually 95% screaming at the machines (various
| linux tasks, writing glue scripts, infrastructure, networking
| etc) and 5% development (the aforementioned glue scripts, writing
| internal docs).
|
| The fields do overlap, but very little.
|
| They require two different skill sets.
|
| They require two different mindsets, too: a software engineer is
| usually optimistic (works on my machine, will work in prod too)
| whereas a sysadmin/operations person is usually pessimistic
| (what's our disaster recovery strategy?).
| f1shy wrote:
| > The problem is most engineers don't want to do operations work.
|
| I was convinced this was exactly what devops wanted to address
| and solve.
| fHr wrote:
| Well sure I would like to do DevOps as well as a software
| engineer but I'm getting drowned in agile ceremonies already,
| meanwhile the devops guys has like 10 hours a week more time as
| he doesn't have all the agile
| re-/pre-/postrefinement/retro/intro/outro/daily/weekly etc. They
| do like bidaily devops meet or 1on1 calls and care for their
| infra.
| mjr00 wrote:
| > If the "DevOps" team ships a Postgres RDS instance it will run
| fine forever, that is until an application starts using it. All
| of a sudden a cascade of N+1s hit, the CPU spikes, and queries
| grind to a halt. Who is woken up? And why does this always happen
| at 2 AM? In this scenario, there is nothing for operations
| personnel to do, yet here they are.
|
| This is definitely a symptom of a broken model and not what I
| would call devops. IMO the most important tenant of devops is "if
| you build it, you run it," meaning the appdev team that decided
| to use Postgres RDS is the one getting woken up at 2am.
|
| It's also, in my experience, one of the best ways to reduce
| masturbatory engineering decisions and get people to focus on
| picking boring technology that works. Coding up a serverless
| application in Rust that's using a CockroachDB backend at a
| Python/MySQL shop would get a lot of engineers excited, but those
| people would be less excited knowing they're going to be the ones
| paged at 2am when this new and exciting architecture falls over
| in an unfamiliar way (as opposed to Python/MySQL, where a wealth
| of operational knowledge at the org has already been built up).
|
| Similarly, it naturally reduces architecturally complexity.
| Younger senior engineers love drawing boxes of queues, multiple
| microservices, event buses, etc to show off their skill in
| creating the ultimate engineering fantasy, but once you throw
| enough late night operational incidents at a senior engineer,
| suddenly the preferred architecture becomes "an executable
| running on a box that I can SSH into when things go wrong."
| Spivak wrote:
| > This is definitely a symptom of a broken model and not what I
| would call devops.
|
| I'm I the crazy one here? This kind of work is my bread and
| butter as a "devops" person. PagerDuty fires, I find the bad
| query, match it up with the most recent PRs to find where the
| n+1 got introduced and either patch it right there at 2AM with
| the on-call manager's approval or roll it back. Then we have a
| postmorterm in the morning with the team.
|
| I'm the person positioned the best to do this work because I'm
| god in my little ops domain, have the most visibility and the
| biggest toolbox of potential fixes.
| mjr00 wrote:
| This sounds like what I mean with building it and running it
| being devops, yeah -- the fact that you have access to make
| pull requests or code patches, or even know where the code
| repository exists in the first place, shows that you're at
| least somewhat involved in building the application. Contrast
| this to what was traditionally ops and is now "SRE"; in those
| roles, applications are usually black boxes where an ops
| person doesn't know/care how they're developed, because
| they're responsible for the overall health of the system,
| which could be managing 100 applications made by 50
| development teams.
| roboben wrote:
| > When was the last time you saw a product manager high-five the
| ops person and say, 'fast fucking autoscaler!'?
|
| That's me. I did that.
| Coryodaniel wrote:
| You're my kinda PM.
| Spivak wrote:
| And I know it's somewhat exaggerated but if the product people
| don't have visibility into your wins as an ops person then you
| need to make a more conscious effort to talk about them.
|
| Once you peel away the jargon, and tbf doing this is a skill
| that requires practice, non-techies do think ops work is cool.
| roboben wrote:
| Disclaimer: I am an Ops engineer turned product manager.
|
| But I don't like this rift. There are many ops people having
| a good understanding of product and vice versa. It's just the
| stereotype which gets reproduced all the time, including in
| this blog post.
|
| Sure, if management doesn't get that there is value (and
| cost!) in proper ops work and having product management
| attached to it, then all is turned to "DevOps" anyways
| jimcavel888 wrote:
| pronik wrote:
| DevOps is an education and communication position. You won't get
| the devs to do ops, since they are good at development and don't
| know much about ops and the same applies to ops in reverse. You
| need people (ideally, on each product team) who understand enough
| of both sides to efficiently communicate problems and facilitate
| solutions. Before you make your devs write provisioning scripts
| for databases (which they can do, but probably won't get the
| trade-offs of different parameters), you might want to explain
| the difference between running database migration scripts on
| application server startup on a single-instance dev laptop and on
| a distributed master-master replicated database with multiple
| application server instances. You increase awareness, you provide
| platforms, you communicate. That's what DevOps and especially
| DevOps teams do. They are glue between people of different
| competence areas. The alternative is Peter's principle applied to
| both sides.
| ny711 wrote:
| DevOps is outdated; in fact, most stuff nowadays in engineering
| is starting to get outdated due to how fast technology is
| evolving
| agtorre wrote:
| It also seems like we cycle through ideas, so what was outdated
| a few years ago becomes interesting again due to a new
| perspective on modern technology as well as new developers
| entering unfamiliar spaces.
| shinzui wrote:
| DevOps (the term), like Agile, made sense when it was introduced
| but lost its meaning as the industry evolved.
| tomrod wrote:
| The article was interesting. It seems like it misses that the
| complaints of why DevOps is dead is a culture problem, which I've
| found are hard to solve by throwing new tech at the problem.
| erulabs wrote:
| Call the person or the team or the practice whatever you'd like:
| DevOps is the merger of development and _business operations_. It
| is not developer + sysadmin. Companies with no "devops"
| (people/team/practice) build big sticky balls of mud. If they're
| lucky, they hit product-market-fit before the mud dries - if not,
| well, this is why most software companies die and why most large
| organizations cannot make progress. DevOps simply plots the
| trajectory of our drying ball of mud, and attempts to keep it
| moist for as long as possible.
|
| I don't care if I'm by myself, if I have a team, if I have a
| devops title - all I want is to prevent the existential death-by-
| garbage-fire that eventually consumes all technology. If you're a
| developer and you're _not aware_ that you 're marching slowly
| towards a complexity-cliff, that's fine - we just have different
| jobs. If you're a developer and you're aware commits aren't by
| default equivalent to progress, congrats, you're an
| SRE/DevOps/Senior Engineer/Platform/whatever.
___________________________________________________________________
(page generated 2022-10-20 23:01 UTC)