[HN Gopher] Department of Defense: Software Is Never Done (2019)...
___________________________________________________________________
Department of Defense: Software Is Never Done (2019) [pdf]
Author : jasonhansel
Score : 95 points
Date : 2023-01-28 16:07 UTC (6 hours ago)
(HTM) web link (media.defense.gov)
(TXT) w3m dump (media.defense.gov)
| lumb63 wrote:
| I want to read more of this later. After 4 years in DoD
| contracting, at a quick glance, some of this hits the nail right
| on the head:
|
| >Software is made by people and for people, so digital talent
| matters. The company I was at did not acknowledge this. To them,
| engineering hours were fungible. Two weeks from an engineer who
| is doing the bare minimum and coasting to retirement (in DoD
| contracting, there are lots of these type) is the same as a mid-
| level engineer who is up-to-date on the latest technology and
| working their hardest to push the project forward. The latter is
| usually paid less than half the former in the industry.
|
| >Software is different than hardware (and not all software is the
| same). What works in other industries doesn't always work in
| defense. It's a unique space. CI? Good luck running automated
| "intercept a hypersonic missile" tests. This doesn't get enough
| attention. However, there is a lot that can be done in pure
| software testing, even for DoD applications. They view it too
| much like hardware development and that hamstrings their software
| efforts.
|
| The bigger issue that is industry-wide that I noticed in my time
| there, is that the incentive system in the industry is broken.
| Contractors take an educated guess up front on what it will cost
| them to build a system for the government, and the government
| chooses from those bids, weighing capabilities, cost schedule,
| etc. The contractor who wins, then tracks their progress to
| completion by how many hours they have worked on the contract,
| relative to their total. This is how they get paid. For instance,
| if it is predicted to take 100 hours to deliver a system, and
| that system is $1 million, each hour would earn the contractor
| something like $10,000 (obviously there are other non-labor costs
| and this is a hand waving simplification, but to an extent is
| true). So, there is limited (bonuses that are tiny relative to
| the value of the contract) incentive to come in on time. There is
| a great incentive to over-hire, pay staff to sit around doing
| nothing, and collect money for it. This disincentivizes
| innovation, because you don't want to have a predicted cost that
| is astonishingly low for your next contract; you want it to be
| just low enough to undercut your next competitor. That's how you
| maximize profits, and us taxpayers get a terrible deal.
| giaour wrote:
| There are multiple contract types, and you're correct that the
| "cost-plus" paradigm (where the contract covers the
| contractor's costs plus a profit margin) has some perverse
| incentives. I saw a number of smaller efforts at Medicare use
| the "firm fixed price" paradigm (where the total cost of the
| contract is agreed to up front), which allows the contractor to
| keep whatever they save by delivering early or using a smaller
| than predicted staff but makes the contractor liable for any
| cost or staffing overages. That system seemed to produce better
| aligned incentives, but it's more difficult to apply to larger
| projects.
| gilbetron wrote:
| I was a DoD contractor (the coding type, not the shooting type)
| for almost 15 years, and this is a really interesting read. I've
| heard some software devs state that "the waterfall method was
| never really used". Those devs were not DoD software contractors
| for almost 15 years. DoD was and is all about the waterfall
| model. I have spreadsheet PTSD from seeing so many waterfall
| diagrams and project manager's Gantt charts. I was working on a
| peanut-size project for the Navy (around $5-10million over 5
| years) when Agile began picking up steam (mid 00s). I introduced
| the team to it and we began doing Agile to decent effect (at
| least in comparison to waterfall), and introduced it to our Navy
| sponsors. I remember some higher-ranking individual ask, "wait,
| so this method can actually deliver software?" Or something like
| that. Of course, Agile is no panacea, but the iterative nature
| was waaaay better than what the DoD was doing.
|
| I really like how they talk about "individuals of talent" in the
| document. Much of the DoD focuses around process, to great
| effect. The problem is that you can't take a process and put
| software drones on it and deliver software of any complexity.
| That goes against much (but not all!) of the DoD paradigm, it is
| interesting to see they might be changing.
|
| Of course, they will meet great resistance, after all, the first
| job of a successful contractor is to extract as much money as
| possible from the government with as little work as possible.
| "Never finish a contract" was the mantra we had learned from the
| giant contractors.
|
| Glad to be out of there, but it was a really interesting
| experience!
| xxEightyxx wrote:
| I'm a current DoD contractor (the coding type) and the
| agile/waterfall mentality still holds up. Different departments
| and different projects within each department will have
| different methodologies for planning and delivering products.
|
| However, where I am, there is a huge emphasis on "getting shit
| done" well ahead of schedule. I'm lucky to work with smart and
| motivated folks who together, work well and we are wildly
| successful.
|
| My biggest gripe, shared among many of my coworkers, is the
| level of micromanagement. Multiple business analysts, product
| owners, stakeholders, SCRUM masters, managers, etc. who all
| share essentially the same job - delegating and scheduling
| work. It makes every project 10x more difficult and complex
| than it otherwise should be.
|
| An example just from this past week: Project A: we are
| delivering a single product. Three managers, two SCRUM masters,
| multiple business analysts. We engineers have four different
| JIRA/task-boards to keep up with. All of them overlap, three of
| them are essentially the same board in different spaces. Each
| day we have 2-4 hours worth of meetings spread across
| stakeholders amongst those three boards where we regurgitate
| the exact same information.
|
| It's repetitive, backward, and a time-waste. Some engineers are
| currently "revolting" by saying what I'm saying now - in the
| meetings themselves. We've had multiple blowups with
| management.
|
| Having meetings and discussions spread so wide leads to a lot
| of confusion because us engineers appear to be the only conduit
| with which the analysts and managers are communicating. Often
| times, the different stakeholders will ask for the same
| features in different meetings, other times they ask for
| different or conflicting features which leads to confusion and
| the need to schedule more meetings to clarify as they are not
| communicating amongst each other.
| kqr wrote:
| > I've heard some software devs state that "the waterfall
| method was never really used".
|
| What they mean by that is that the person who invented the
| phrase "waterfall method" did it specifically to tell the world
| about something that did _not_ work. In fact, even going back
| to the NATO software engineering conferences of the 1960s, they
| were discussing how development needs to be agile.
|
| But sure enough, few big organisations got that memo.
| Jtsummers wrote:
| DOD codified Royce's waterfall (which he, himself, did not
| name in that paper) in a set of documents on officially
| endorsed software acquisitions processes. There were others,
| but Waterfall was one of them. Its critical failing is that
| it's an idealized process that cannot withstand uncertainty
| or change, but it looks really pretty in charts so people
| selected it without serious thought put into it.
| johndhi wrote:
| I haven't read the doc, but I worked on lawsuits (I'm a lawyer)
| that at their core were asserting that agile software
| development isn't as safe or bug proof as waterfall. Basically
| in the medical device (or adjacent) space the idea asserted was
| that 'fail fast' was illegal and didn't have enough safety
| checks.
|
| Agile software development is a tough pill to swallow for non-
| engineers. The lawyers and business people can't wrap their
| head around the idea that building iteratively could actually
| reduce bugs and improve quality - because the way they work is
| via process planning docs and checklists.
| Jtsummers wrote:
| What a bizarre claim. Agile doesn't mean "cowboy coding"
| (I've met as many cowboy coders in Waterfall, may it die a
| fiery death, as Agile projects).
|
| Nothing about Agile says, "Don't ever test and validate this
| system before deploying." Anyone who believes that is,
| frankly, a moron.
| johndhi wrote:
| In my experience it was less a lack of intelligence and
| more results-oriented thinking (they thought we were wrong
| so they found a way to argue it).
|
| If you have twenty versions of a thing and multiple
| deployments to fix and fine tune it, that means to this
| person there are 20 plus examples of "mistakes" you made in
| building it. They were looking for evidence that the
| software was built with "poor quality controls" and
| ironically interpreted iteration as evidence of poor build
| quality.
| arcbyte wrote:
| The ONLY way to get quality is through quantity. That's the
| key insight.
| hamandcheese wrote:
| Quantity of... dollars?
| xboxnolifes wrote:
| Iterations.
| no_wizard wrote:
| I'm betting releases and iteration cycles they're tied
| to. The most agile place I ever worked shipped code most
| days of the week in some fashion. For control reasons we
| released Tuesday-Thursday not Mondays or Fridays. So was
| nearly every day something went out.
|
| I'm including bug fixes here but nothing got stalled. We
| had a working iteration loop of Test -> Develop -> Test
| -> Review -> QA -> Release
|
| Lots of automated checks, investment in infrastructure
| and good architecture made this possible. Ironically one
| of the smallest companies I ever worked for
| __MatrixMan__ wrote:
| "Breaks when you try to use it" is a quality that I can
| usually achieve on the first try.
|
| Or did you mean some quality in particular?
| he0001 wrote:
| This is so true. They don't teach agile in management classes
| and 'planning' generally means years, not t-shirt sizes. I
| believe that also many economic theories and practices don't
| regard agile practices. I've always felt that the business
| and their planning are at odds with the agile ways.
| jasmer wrote:
| Such a great point.
|
| I think one way to go about it is to just put the Agile bits
| inside a Waterfall.
|
| Which is frankly how it works anyhow.
|
| You can characterize early sprints at 'research'.
|
| Later the sprints are just 'sub projects'.
|
| I think people could be convinced that 'feature refinement'
| is a real thing. Put it out in the field, doesn't work the
| way we want it, and so it needs to be refined.
|
| I know that I would hide 'Agile' to anyone outside the team
| and just talk in simple, easier to understand phases.
|
| It's also possible to put checklists in those phases, and if
| you say someting like '75% features' at such and such phase
| and assume '25% of that 75% will be reworked' it's not
| irrational either.
|
| But you are right, culture matters.
| no_wizard wrote:
| I think this is true even in "civilian" industries. Can say
| doing this for over a decade now that I have worked at
| exactly one place that followed anything close to what was
| laid out in the Agile manifesto.
|
| Every other place always sings the praises of agile and/or
| scrum but it's always just waterfall with ceremonies.
|
| I feel like the collective of how software is developed
| evolved little from circa 2006
|
| The notion that gets talked about at Netflix is "context not
| control" and I think it would go along way if everyone looked
| at software development management this way. Too many people
| try to control software development lifecycles instead of
| just providing engineers enough context to get their work
| done and trusting that reasonable people will make reasonable
| decisions
| beckingz wrote:
| Large organizations value predictability.
| no_wizard wrote:
| IMO it's not this simple. Culture has more to do with it
| than anything. See my note about Netflix having this
| mantra of "context not control". Managers are less
| inclined to be domineering in that setting and allow
| people to do the work in a reasonable way because they
| are given the context of why their work is being done in
| the first place.
|
| Not perfect but a more than above average way to approach
| the problem yet this kind of thinking or its permutations
| have not caught on at all. It's still top down process
| oriented models
| kbenson wrote:
| > The problem is that you can't take a process and put software
| drones on it and deliver software of any complexity. That goes
| against much (but not all!) of the DoD paradigm, it is
| interesting to see they might be changing.
|
| My (outside) impression has changed the more I've heard over
| the years, to the point that I assume the DoD and military are
| acutely aware that the drones aren't really interchangeable,
| and that at every level they cultivate people under them they
| know "get shit done" and hand the important projects to them.
| Those people in turn have those under them that "get shit done"
| they hand the important things to, and so on, all the way down
| to privates in the military. The important thing being that
| those people that get shit done are given a lot more leeway in
| how they accomplish things (which might later become doctrine
| of successful), and this is specifically how those orgs evolve
| over time.
|
| Again, that's my outside perspective from someone that's
| interested whenever I hear it and reads/listens to history
| sources (podcasts,etc).
| Jtsummers wrote:
| > to the point that I assume the DoD and military are acutely
| aware that the drones aren't really interchangeable, and that
| at every level they cultivate people under them they know
| "get shit done" and hand the important projects to them.
|
| Within military troops, yes. That's an active goal,
| especially on the officer side where the whole career path is
| pointed towards increasing levels of responsibility and
| leadership. The problem is not in the troops, but in the
| program offices.
|
| I was on a spectacularly failed Waterfall (big-W, they were
| explicit about what they were doing) where they handed the
| task of managing a large, multi-billion dollar software
| system (from the DOD side) to a 1st Lt history major. She had
| no preparation for the role. Software acquisition in DOD is,
| mostly, a clusterfuck. I felt bad for her because she
| legitimately did not know anything more than the basic legal
| and financial side of things, she had no technical
| comprehension and had to trust people (contractors) who were
| lying about the system status and quality (what was
| delivered, after she had moved on, was years late and had a
| fraction of the target capabilities).
|
| Program offices are not generally composed of people who have
| real-world experience with software development. Remarkably
| they have great engineers, good finance people, but the
| software person (often singular, even in programs that are
| almost exclusively composed of software systems) is usually
| not from a software background or a straight out of college
| EE major (more often than not, occasionally a CS major).
| kbenson wrote:
| > usually not from a software background or a straight out
| of college EE major (more often than not, occasionally a CS
| major).
|
| Ugh, I can see that being a problem. I remember my
| practical experience in projects after graduating with a CS
| degree, and it was close to non-existent (a class in
| development methodologies without experiencing it yourself
| is just a survey of all the things you can do badly if you
| were to be in charge).
| jdeaton wrote:
| Just out of curiosity, how does someone with in-demand, highly-
| transferable skills like software development find themselves
| working in a place they generally dislike for 15 years without
| leaving sooner?
| scythmic_waves wrote:
| The 10 recommendations found on the "cheat sheet" (page xv):
|
| Line of Effort A (Congress and OSD):
|
| Refactor statutes, regulations, and processes for software
| * [A1] Establish one or more new acquisition pathways for
| software that prioritize continuous integration and delivery of
| working software in a secure manner, with continuous oversight
| from automated analytics * [A2] Create a new
| appropriation category for software capability delivery that
| allows (relevant types of) software to be funded as a single
| budget item, with no separation between RDT&E, production, and
| sustainment
|
| Line of Effort B (OSD and Services):
|
| Create and maintain cross-program/cross-Service digital
| infrastructure * [B1] Establish and maintain
| digital infrastructure within each Service or Agency that enables
| rapid deployment of secure software to the field, and incentivize
| its use by contractors * [B2] Create, implement,
| support, and use fully automatable approaches to testing and
| evaluation (T&E), including security, that allow high-confidence
| distribution of software to the field on an iterative basis
| * [B3] Create a mechanism for Authorization to Operate (ATO)
| reciprocity within and between programs, Services, and other DoD
| agencies to enable sharing of software platforms, components, and
| infrastructure and rapid integration of capabilities across
| (hardware) platforms, (weapon) systems, and Services
|
| Line of Effort C (Services and OSD):
|
| Create new paths for digital talent (especially internal talent)
| * [C1] Create software development units in each Service
| consisting of military and civilian personnel who develop and
| deploy software to the field using DevSecOps practices
| * [C2] Expand the use of (specialized) training programs for
| CIOs, SAEs, PEOs, and PMs that provide (hands-on) insight into
| modern software development (e.g., Agile, DevOps, DevSecOps) and
| the authorities available to enable rapid acquisition of software
|
| Line of Effort D (DoD and industry):
|
| Change the practice of how software is procured and developed
| * [D1] Require access to source code, software frameworks, and
| development toolchains --- with appropriate IP rights --- for
| DoD-specific code, enabling full security testing and rebuilding
| of binaries from source * [D2] Make security a first-
| order consideration for all software-intensive systems,
| recognizing that security-at-the-perimeter is not enough
| * [D3] Shift from the use of rigid lists of requirements for
| software programs to a list of desired features and required
| interfaces/characteristics to avoid requirements creep, overly
| ambitious requirements, and program delays
| [deleted]
| jasonhansel wrote:
| A question I have: do people working on DoD software ever get to
| use it themselves? I assume many of them don't (if they're
| working on, say, a weapons system).
|
| This situation often (IMHO) makes it much harder to deliver high-
| quality software. No amount of surface-level UI testing replaces
| the knowledge you get from actually using a system yourself.
| mLuby wrote:
| > In a year replete with reports on artificial intelligence,
| quantum computing, and blockchain, may seem mundane to write
| about software, but software is the foundation of all things
| digital, and _the Department does it exceedingly poorly._
|
| I take issue with the title. Software is constantly "done" and
| then new requirements arrive, causing a new version of the
| completed software to be built. A suspension bridge is considered
| "done" even if it's getting continuously repainted.
| gilbetron wrote:
| You are thinking in terms of software in the non-government
| world. Contractors try really hard never to finish software so
| they can keep extending their contracts. This document is about
| the military world, and if you don't have experience in it, it
| can be hard to interpret!
| sylware wrote:
| military world only?
|
| Look at how the TV series industry is hard trying with
| cliffhangers or story telling arcs without a closure...
|
| This issue runs much deeper:
|
| It is how the economy operates: you need a roof to protect
| you, every day, you need to eat, every day, until you die in
| decades. Many economic activities, if done honestly and
| fairly, cannot justify a permanent income.
| jongjong wrote:
| As a software developer with 15 years of experience who produces
| completed software products, I'm wary of this mindset because
| it's often used as an excuse to tolerate or reward incompetence.
|
| Software can be considered done with respect to any given set of
| requirements. Now if the requirements change, then of course, the
| software may need to be updated - but even then, it should be
| designed in such a way that the impact of requirement changes are
| minimal. Software should, to some extent, anticipate requirement
| changes. Of course this is not easy, but to claim that it's
| impossible is to admit defeat. You need highly skilled people
| making those architectural decisions.
|
| Some software is subjected to constantly changing requirements
| and these can be said to never be done but that's far from being
| the case for all software. Libraries and modules are software too
| and these should be generic enough to handle requirement changes
| without code changes since they should not concern themselves
| with the business domain.
|
| I have completed open source projects which did not need to be
| updated in years and worked just as well. I've worked on other
| projects which had to be updated because of vulnerabilities found
| in dependencies but these did not require any code changes on my
| end.
|
| I consider it a huge red flag if some library was working fine
| for 5 years and then people decided to change its interface in a
| non-trivial way.
|
| I was tolerant when async/await was rolled out to major
| programming languages; this affected and continues to affect
| dependencies but it's a meaningful conceptual improvement and
| helps with future maintainability. I hope that programming
| standards organizations will slow down after that in terms of
| changes which affect function interfaces because these decisions
| have huge flow-on impacts.
|
| It's totally possible to produce completed software (especially
| libraries and frameworks) but the industry has to aim for it and
| expect it.
|
| The current goal of industry seems to be job creation, not the
| creation of robust working products. Accepting that software can
| never be done right from the beginning is going to be a disaster.
| It should only be tolerated in the context of requirement
| changes.
| jasonhansel wrote:
| As a fellow software developer, I respectfully disagree.
|
| While a piece of software may _satisfy_ its requirements, it
| will never satisfy them perfectly or optimally. All software
| contains bugs, unexpected failure modes, and undiscovered
| security vulnerabilities. I am convinced that there are
| essentially zero exceptions to these claims.
|
| Furthermore, the environment in which software runs must change
| over time. This causes "bit rot" as (say) dependencies stop
| receiving security updates or platforms become unsupported.
|
| Finally, there's the human factor. When software runs on old
| technologies, it will become progressively harder for new
| developers to understand. There will consequently often be a
| steady decline in available talent.
|
| For these reasons, even if the requirements of a piece of
| software _never_ change, it will _always_ require ongoing
| maintenance to keep it running, much like a piece of physical
| infrastructure.
|
| (edit: typo)
| jongjong wrote:
| If performance requirements change, then that would count as
| a change of requirements in my books. It should be possible
| to decide ahead what an acceptable target is. Also, it's not
| going to affect every part of the software.
|
| About changes in tech and environments. It's possible to
| build portable software. Whether or not software needs to be
| portable should be treated as a requirement. I disagree with
| any argument which suggests that it's not possible to foresee
| this.
|
| It's not always the case that old technologies are harder to
| understand. This is mostly because software is still a
| relatively new area in the grand scheme of things. There are
| some old projects which I find easier to understand.
| Especially those built with languages which stood the test of
| time like C/C++, Java, Python, JavaScript...
| closeparen wrote:
| Requirements aren't from God. They're just a tool for
| facilitating transactions with contractors. Whatever the
| software does or is used for, can almost always be
| optimized further. When you have a team of FTE SWEs owning
| a problem, that's their charter. Some amount of upfront
| documentation can help align engineers and stakeholders on
| each of the steps taken towards their broader goals, but
| that's in instrumental tactic, not the goal itself.
| Projects are judged on their impact, not their adherence to
| original plans.
|
| It's "done" when no one can think of any ROI-positive
| iterations... until something changes.
| cushychicken wrote:
| I didn't read through the whole thing, but I did skip to the
| section of the ten recommendations and each one makes a whole
| heap of sense to me.
|
| Anyone know if Congress has done any of these things in the last
| three years, since this report has come out?
| dathinab wrote:
| I know it's not what they mean and I agree with the general idea.
|
| But for very small very well contained libraries I would argue
| that they sometime _are_ done (for many years, not forever; in a
| sense that any changes you can come up with add no relevant value
| at all, including no relevant perf. improvements or better
| maintainability).
|
| At least if written in a language with nice forward and backward
| compatibility, cross platform support and a stable "base"
| ecosystem.
|
| (So not C,C++ due to not default package manager/build system and
| too much divergence between platforms (e.g. unsigned long), not
| JS because their core build tooling changes to much (through it's
| getting better!), etc.)
| legrande wrote:
| > But for very small very well contained libraries I would
| argue that they sometime _are_ done
|
| https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_a...
|
| > As stated by McIlroy, and generally accepted throughout the
| Unix community, Unix programs have always been expected to
| follow the concept of DOTADIW, or "Do One Thing And Do It
| Well." There are limited sources for the acronym DOTADIW on the
| Internet, but it is discussed at length during the development
| and packaging of new operating systems, especially in the Linux
| community.
___________________________________________________________________
(page generated 2023-01-28 23:00 UTC)