[HN Gopher] Shift Left Is the Tip of the Iceberg
___________________________________________________________________
Shift Left Is the Tip of the Iceberg
Author : yatrios
Score : 122 points
Date : 2024-11-19 14:06 UTC (8 hours ago)
(HTM) web link (semiengineering.com)
(TXT) w3m dump (semiengineering.com)
| 0xbadcafebee wrote:
| For those not aware, Shift Left[1] is (at this point) an old term
| that was coined for a specific use case, but now refers to a
| general concept. The concept is that, if you do needed things
| earlier in a product cycle, it will end up reducing your expense
| and time in the long run, even if it seems like it's taking
| longer for you to "get somewhere" earlier on. I think this[2]
| article is a good no-nonsense explainer for _" Why Shift Left?"_.
|
| [1] https://en.wikipedia.org/wiki/Shift-left_testing [2]
| https://www.dynatrace.com/news/blog/what-is-shift-left-and-w...
| leoc wrote:
| So "shift left" is roughly equivalent to "tests first" or
| "TDD"?
| marcellus23 wrote:
| I would say tests first/TDD is a form of shifting left, but
| it can encompass more than that.
| 01HNNWZ0MV43FF wrote:
| It can also mean manual testing earlier. Valve credits
| playtesting early and playtesting often for their success
| with Half-Life, Half-Life 2, and other games.
| Fuzzwah wrote:
| Not just when they play tested but how. Silently watching
| someone play, only taking notes. No debrief.
|
| Source: the week I spent in the Valve offices in 2005,
| having Valve staff playing our in development HL2 mod.
| CodeAndCuffs wrote:
| Was it Dystopia, by chance?
| mehagar wrote:
| More broadly it includes moving activities that are normally
| performed at a later stage so that they are earlier in the
| process. The idea is that defects found later in the process
| are more costly.
| hitchstory wrote:
| That's just one example of it.
|
| Other examples:
|
| * Replacing automated tests with (quicker) type checking and
| running it on a git commit hook instead of CI.
|
| * Replacing slower tests with faster tests.
|
| * Running tests before merging a PR instead of after.
|
| * Replacing a suite of manual tests with automation tests.
|
| etc.
| 0xbadcafebee wrote:
| Or, again more generally:
|
| - implementing security features earlier (DevSecOps)
|
| - implement tracing and metrics/analysis tools earlier, use
| them to test and debug apps earlier (as opposed to laptop-
| based solutions)
|
| - building the reliable production model earlier (don't
| start with a toy model on your laptop if you're gonna end
| up with an RDS instance in AWS; build the big production
| thing first, and use it early on)
|
| - add synthetic end-to-end tests early on
|
| The linked article is talking about Shift Left in the
| context of developing semiconductors, so you can see how it
| can be applied to anything. Just do the needed thing
| earlier, in order to iterate faster, improve quality,
| reduce cost, ship faster.
| js8 wrote:
| So aside from automation (which also has tradeoffs), when
| you shift left, what gets shifted right?
| saynay wrote:
| The knowledge needed to do the 'shift-left' tasks. The
| downside of making your developers take on more tasks
| that used to be handled by someone further down the chain
| is they need to know how to do those tasks.
| hammock wrote:
| Why can't you shift the "someone further down the line"
| left along with the task?
| ElevenLathe wrote:
| This is interesting. Seems like it would mean something
| like having QA or infosec pair program with app devs. I'm
| sure somebody does this and am curious how it works out
| in practice.
| crabbone wrote:
| No, not really. It's a concept from the bygone era, before
| X-as-a-service took over a lot of software categories. It was
| intended to minimize friction between the QA and the
| development teams at the time the product was handed to QA.
| If the product had multiple defects, incrementally
| discovered, it would have to travel back and forth between
| the QA and the developers, stalling the development of the
| next version and generally creating a lot of noise and
| inconvenience due to the surrounding bureaucracy.
|
| There's no need to write tests upfront for you to shift left.
| All shift left means is that testing happens during
| development. Whether you start by writing tests and then
| write the actual program or the other way around -- doesn't
| matter.
| coryrc wrote:
| No evidence most of the activities actually save money with
| modern ways of delivering software (or even ancient ways of
| delivering software; I looked back and the IBM study showing
| increasing costs for finding bugs later in the pipeline was
| actually made up data!)
| coryrc wrote:
| To be more specific, let's say I can write an e2e test on an
| actual pre-prod environment, or I can invest much development
| and ongoing maintenance to develop stub responses so that the
| test can run before submit in a partial system. How much is
| "shifting left" worth versus investing in speeding up the
| deployment pipeline and fast flag rollout and monitoring?
|
| Nobody I've worked with can ever quantify the ROI for
| elaborate take test environments, but somebody made an okr so
| there you go. Far be it we follow actual research done on
| modern software... http://dora.dev
| cogman10 wrote:
| In fact, in my experience, these elaborate test
| environments and procedures cripple products.
|
| I'm firmly of the opinion that if a test can't be run
| completely locally then it shouldn't be run. These test
| environments can be super fragile. They often rely on a
| symphony of teams ensuring everything is in a good state
| all the time. But, what happens more often than not, is one
| team somewhere deploys a broken version of their software
| to the test environment (because, of course they do) in
| order to run their fleet of e2e tests. That invariably ends
| up blowing up the rest of the org depending on that broken
| software and heaven help you if the person that deployed
| did it at 5pm and is gone on vacation.
|
| This rippling failure mode happens because it's easier to
| write e2e tests which depend on a functional environment
| than it is to write and maintain mock services and mock
| data. Yet the mock services and data are precisely what you
| need to ensure someone doesn't screw up the test
| environment in the first place.
| coryrc wrote:
| There are many reasons you want to be able to turn up
| your whole stack quickly; disaster recovery is just one
| of them. And if you can turn up your environment quickly
| then why not have multiple staging environments? You
| start with the most recent of yours and everyone else's
| prod version, then carrots combinations from there
|
| Obviously this is for large-scale systems and not small
| teams.
| jeltz wrote:
| You are not wrong but I have had many experiences where
| mock services resulted in totally broken systems since
| they were incorrectly mocked. In complex systems it is
| very hard to accurately mock interactions.
|
| Personally I think the real issue is not the testing
| strategy but the system itself. Many organizations make
| systems overly complex. A well structured monolith with a
| few supporting services is usually easy to test while
| micro service/SOA hell is not.
| wrs wrote:
| No argument, but there can be limitations on how much you
| can speed up the deployment pipeline. In particular, the
| article is about integrated circuit development (actually
| about systems made of many ICs), where a "production
| deployment" takes months and many, many millions of
| dollars, and there's not much you can do about it.
|
| I heard a story decades ago about a software team that got
| a new member transferred in from the IC design department.
| The new engineer checked in essentially zero bugs. The
| manager asked what the secret was, and the new engineer
| said "wait, we're allowed to have bugs?"
| crabbone wrote:
| Shift-left comes from the world before everyone was selling
| services. In that world shipping service packs was the way
| to fix problems discovered after a release, and releases
| were years apart.
|
| From a QA perspective, I greatly regret that the world of
| infrequent releases is mostly gone. There are few kinds of
| products that still hold onto the old strategy, but this is
| a dying art.
|
| I see the world of services with DevOps, push on green etc.
| strategies as a kind of fast-food of software development.
| A kind of way of doing things that allows one to borrow
| from the future self by promising to improve the quality
| eventually, but charging for that future improved quality
| today.
|
| There are products where speeding the rollout is a bad
| idea. Anything that requires high reliability is in that
| category. And the reason is that highly reliable systems
| need to collect mileage before being released. For example,
| in storage products, it's typical to have systems run for
| few months before they are "cleared for release". Of
| course, it's possible to continue development during this
| time, but it's a dangerous time for development because at
| any moment the system can be sent back to developers, and
| they would have to incorporate their more recent changes
| into the patched system when they restart the development
| process. I.e. a lot of development effort can be
| potentially wasted before the system is sent out to QA and
| the actual release. And, to amortize this waste, it's
| better to release less frequently. It's also better to
| approach the time when the system is handed to QA with a
| system already well-tested, as this will minimize the back-
| and-forth between the QA and the development -- and that's
| the problem shift-left was intended to solve.
|
| NB. Here's another, perhaps novel thought for the "push on
| green" people. Once it was considered a bad idea for the QA
| to be aware of implementation details. Testing was seen as
| an experiment, where the QA were the test subjects. This
| also meant that exposing the QA to the internal details of
| the systems or the rationale that went into building it
| would "spoil" the results. In such a world, allowing the QA
| to see a half-baked system would be equivalent to exposing
| them to the details of the system's implementation, thus
| undermining their testing effort. The QA were supposed to
| receive the technical documentation for the system and work
| from it, trying to use the system as documented.
| coryrc wrote:
| Last rant: everybody is testing in production, but only some
| people are looking at the results. If you aren't then there's
| better ROI to be found than "shifting left" your e2e tests.
| 1propionyl wrote:
| Are you referring to the IBM Systems Science claims (likely
| apocryphal) in the Pressman paper, or Barry Boehm's figure in
| "Software Engineering" 1976 paper which did include some IBM
| sourcing (literally drawn on) but was primarily based on
| survey data from within TRW?
|
| It baffles me that anyone would continue to promulgate the
| Pressman numbers (which claim ~exponential growth in cost)
| based on... it's not entirely clear what data, as opposed to
| Boehm's paper which only claims a linear relative cost
| increase, but is far more credible.
| 98codes wrote:
| What helped me finally remember "left of what, exactly" is to
| imagine a timeline, left to right. Shifting left is moving
| things earlier in that timeline.
| DeathArrow wrote:
| Do the hard things first?
| Buttons840 wrote:
| Do the things that are hard _and_ important first.
| hammock wrote:
| Sounds like the bottleneck concept from Goldratt's The Goal
| infogulch wrote:
| I'm familiar with both concepts and they don't seem related
| imo. Maybe if you applied bottleneck theory to process design
| in the abstract? Sorry I can't evaluate this comparison
| further, I must have run out of my quota for Abstract
| Reasoning Tokens today.
| hammock wrote:
| "Shifting left" is like moving the fat kid on the hike to
| the front of the line
| rusk wrote:
| _"a stitch in time saves nine"_
|
| _"prevention is better than cure"_
|
| [ante-] _"closing the stable door after the horse has bolted"_
| bigs wrote:
| Sounds like a similar/parallel thought to the project
| management waterfall paradigm whereby the earlier you get
| things correct (left) the less costly it is in the long run or
| conversely if you have to go and re-do things later on (right)
| you're in for a shock (either cost, time or quality).
| tromp wrote:
| Funny how correct is not associated with right. The normal
| association is reflected in he Haskell Either datatype for
| describing computations that either run into some error (Left
| error) or run successfully producing a value (Right value).
| polishdude20 wrote:
| Sounds like the exact thing we never did at my previous
| employer. Just do the bare minimum to get it out the door and
| then fix it later.
| infotecht wrote:
| Shift Left is about doing more work earlier in a process for
| efficiency -- it sounds like you at your previous employer
| were shifting right instead?
| bloodyplonker22 wrote:
| I had management who were so enthused about "shift left" that
| we shifted far left and broke the shifter, or perhaps, mis-
| shifted. Now we spend too much time developing test plan and
| testing and arguing about PRD that we actually deliver more
| slowly than competitor by a lot.
| hinkley wrote:
| Another flavor of this I've encountered is people in the
| management chain who can only think about faults per time
| interval and not about faults per interaction.
|
| So you make a tool that prevents errors and speeds up a
| process, now people use it four times as much, and now they
| wonder why they're only seeing half as many faults instead of
| an order of magnitude less.
|
| We are humans. We cannot eliminate errors, we can only
| replace them with a smaller number of different errors. Or as
| in your case, a larger number of them.
| hinkley wrote:
| > Optimization strategies have shifted from simple power,
| performance, and area (PPA) metrics to system-level metrics, such
| as performance per watt. "If you go back into the 1990s, 2000s,
| the road map was very clear,"
|
| Tell me you work for Intel without telling me you work for Intel.
|
| > says Chris Auth, director of advanced technology programs at
| Intel Foundry.
|
| Yeah that's what I thought. The breathlessness of Intel figuring
| out things that everyone else figured out twenty years ago
| doesn't bode well for their future recovery. They will continue
| to be the laughing stock of the industry if they can't find more
| self reflection than this.
|
| Whether this is their public facing or internal philosophy hardly
| matters. Over this sort of time frame most companies come to
| believe their own PR.
| talldayo wrote:
| Intel has had a few bad years, but frankly I feel like they
| could fall a lot lower. They aren't as down bad as AMD was
| during the Bulldozer years, or Apple during the PowerPC years,
| or even Samsung's early Exynos chipsets. The absolute worst
| thing they've done in the past 5 years was fab on TSMC silicon,
| which half the industry is guilty of at this point.
|
| You can absolutely shoot your feet off trying to modernize too
| quickly. Intel will be the laughingstock if 18A never makes it
| to market and their CPU designs start losing in earnest to
| their competitors. But right now, in a relative sense, Intel
| isn't even down for the count.
| buildbot wrote:
| Intel has failed pretty badly IMO. Fabbing at TSMC might
| actually have been a good idea, except that every other
| component of arrow like is problematic. Huge tile to tile
| latencies, special chiplets that are not reusable in any
| other design, removal of hyperthreading, etc etc. Intel's
| last gen CPU is in general faster than the new gen due to all
| the various issues.
|
| And that's just the current product! The last two gens are
| unreliable, quickly killing themselves with too high voltage
| and causing endless BSODs.
|
| The culture and methods of ex-Intel people at the management
| level is telling as well, from my experiences at my last job
| at least.
|
| (My opinions are my own, not my current employers & a lot of
| ex-Intel people are awesome!)
| talldayo wrote:
| We'll see, I mostly object to the "vultures circling"
| narrative that HN seems to be attached to. Intel's current
| position is not unprecedented, and people have been
| speculating Intel would have a rough catchup since the
| "14nm+++" memes were vogue. But they still have their fabs
| (and wisely spun it out to it's own business) and their
| chip designs, while pretty faulty, successfully brought x86
| to the big.LITTLE core arrangement. They've beat AMD to the
| punch on a number of key technologies, and while I still
| think AMD has the better mobile hardware it still feels
| like the desktop stuff is a toss-up. Server stuff...
| _glances at Gaudi and Xeon, then at Nvidia_ ...let 's not
| talk about server stuff.
|
| A lot of hopes and promises are riding on 18A being the
| savior for both Intel Foundry Services and the Intel chips
| wholesale. If we get official confirmation that it's been
| cancelled so Intel can focus on something else then it will
| signal the end of Intel as we know it.
| Tostino wrote:
| I mean they were on 14 NM until just about 2022, those
| memes didn't come from nowhere. And it's not even that
| long ago.
| phkahler wrote:
| >> successfully brought x86 to the big.LITTLE core
| arrangement.
|
| Really? I thought they said using e-cores would be better
| than hyper threading. AMD has doubled down on hyper
| threading - putting a second decoder in each core that
| doesn't directly benefit single thread perf. So Intels 24
| cores are now competitive with (actually losing to) 16
| zen 5 cores. And that's without using AVX512 which Arrow
| Lake doesn't even support.
|
| I was never a fan of big.little for desktop or even
| laptops.
| astrange wrote:
| It's working well for Mac laptops, although I'd rather
| people call it "asymmetric multiprocessing" than
| "big.LITTLE". Why is it written like that anyway?
|
| (Wikipedia seems to want me to call it "heterogeneous
| computing", but that doesn't make sense - surely that
| term should mean running on CPU+GPU at the same time, or
| multiple different ISAs.)
|
| Of course, it might've worked fine if they used symmetric
| CPU cores as well. Hard to tell.
| hinkley wrote:
| I thought big.little was an Arm thing. The little
| Rockwell chips I have running armbian have them.
| hinkley wrote:
| In nearly every generation of Intel chip where I needed
| to care about whether hyperthreading was a net positive,
| it was either proven to be a net reduction in throughput,
| or a single digit improvement but greatly increased
| jitter. Even if you manage to get more instructions per
| cycle with it on, the variability causes grief for
| systems you have or want telemetry on. I kind of wonder
| why they keep trying.
|
| I don't know AMD well enough to say whether it works
| better for them.
| selimthegrim wrote:
| What else would they focus on?
| hangonhn wrote:
| Fabbing at TSMC is an embarrassing but great decision. The
| design side of Intel is the revenue/profit generating side.
| They can't let the failures of their foundries hold back
| their design side and leave them hopelessly behind AMD and
| ARM. Once they've regained their shares of the server
| market or at least stabilized it, they can start shifting
| some of their fabbing to Intel Foundry Services, who are
| going to really suck at the beginning. But no one else is
| going to take that chance on those foundries if not Intel's
| design side. The foundry side will need that stream of
| business while they work out their processes.
| zusammen wrote:
| As soon as I saw "shift left" I knew I wanted to double down.
| vardump wrote:
| This kind of joke might multiply...
| zusammen wrote:
| Bit hack humor: sometimes it pops and counts--but sometimes
| it's rot-ten.
|
| That might have been a bit mask-off. I'm sorry.
| jounker wrote:
| There's a rot in here somewhere
| munchler wrote:
| Way to manage up.
| game_the0ry wrote:
| In my org [1], "shift left" means developers do more work sooner.
|
| So before we clearly clarify product requirements, we start the
| build early with assumptions that can change. Sometimes it works,
| sometimes it does not, which just means we end up net neutral.
|
| But an executive somewhere up the management chain can claim more
| productivity. Lame.
|
| [1] I work at a bank.
| Spivak wrote:
| I think that's a myopic view. Getting something anything in the
| hands of your potential users that's even vaguely in the ball-
| park of a solution shaped thing gives you extremely valuable
| information both on what is actually needed and, to me more
| importantly, what you don't have to build at all.
|
| I consider it a success when an entire line of work is scrapped
| because after initial testing the users say they wouldn't use
| it. Depending on the project scope that could be 6-7 figures of
| dev time not wasted right there.
| game_the0ry wrote:
| Much of what you are saying does not apply to consumer
| banking.
|
| "Build fast and break things" works in big tech, but its a
| serious risk in financial services.
|
| That being said, we are being forced to "build faster and try
| not break things."
| massung wrote:
| I've never been in banking, but I've been in games (AAA)
| and biotech most of my career.
|
| I agree with the previous commenter. It isn't about
| breaking things out even moving fast (in my experience),
| but about getting "the juices flowing".
|
| I've found that when there's a request for some new tool or
| thing, people have a very difficult time defining [all] the
| real world needs. They know they need X, but don't remember
| - or even realize - they also need Y, Z, etc. Or worse,
| many people have been conditioned through corporate culture
| that they can't even ask for things that would make their
| lives easier.
|
| Getting something basic in someone's hands quickly with the
| desire of creating a loop of feedback, feature, feedback,
| ... gets the end user more focused and engaged with the
| final product, which gives the engineers more focused
| requirements. I've found it also gives the engineers the
| opportunity to make suggestions and get more invested.
| mrkeen wrote:
| I'm in fintech right now, experiencing that pain. We do a
| sort of "reverse waterfall".
|
| We respond to spikes in increased exceptions, then read
| logs to try to figure out why something "went wrong". If we
| can change some code to fix it, we do so. But since we
| don't have confidence in our fixes, we have to move further
| back to the design and guess how the system is supposed to
| work. I'm hoping next month we start a dialogue with with
| upstream and ask how we're supposed to use their API.
| from-nibly wrote:
| In software shifting left often also means putting more tasks
| into the hands of the person with the most context, aka, moving
| deployment from ops to developers.
|
| The main benefits you get from this is reduced context switching,
| increased information retention, increased ownership.
|
| But it has to go with tooling, and process that enables that. If
| you have a really long deployment process where devs can get
| distracted then they will lose context switching between tasks.
| If you make every Dev a one man army that has to do everything on
| their own you won't be able to ship anything and your infra will
| be a disjointed mess.
|
| They key thing is reducing toil, and increasing decision making
| power within a single team/person.
|
| From the sound of the article they might be just squishing two
| teams together? What's the advancement that made the two steps be
| able to happen at the same time?
| yoelhacks wrote:
| The shift right parts of the article are more interesting...
| lizknope wrote:
| I'm in chip design and our VP's and C level execs have been
| saying "shift left" for the last year. Every time someone in the
| Q&A asks "What does shift left mean?" No one talks this way
| except for executives. We just joke "The left shift key isn't
| working on my keyboard" or some other nonsense.
| peanut-walrus wrote:
| Can we please stop naming things after directions? I don't want
| to shift left/right/up/down and my data center has no
| north/south/east/west. Just say what you actually want to say
| without obfuscating.
| sophacles wrote:
| I guarantee your data center has a north, south, east _and_
| west. I 'm willing to bet that the north, south, east and west
| of your data center are the same as the north, south, east and
| west of the office I'm sitting in!
| bsder wrote:
| Imagine the world of software if your editor, compiler, virtual
| machine, etc. each cost a million dollars a year per programmer.
|
| This is _reality_ in VLSI CAD.
|
| Now you understand why everything in hardware engineering is
| stupidly dysfunctional.
|
| We don't need "shift left". We need tools that don't cost a
| megabuck.
___________________________________________________________________
(page generated 2024-11-19 23:00 UTC)