[HN Gopher] Verified Spec-Driven Development (VSDD)
___________________________________________________________________
Verified Spec-Driven Development (VSDD)
Author : todsacerdoti
Score : 127 points
Date : 2026-02-28 16:58 UTC (6 hours ago)
(HTM) web link (gist.github.com)
(TXT) w3m dump (gist.github.com)
| politician wrote:
| This is a decent approach. My concern with TDD is that writing
| tests necessarily implies designing an API upon which those tests
| operate. Here, the agent is instructed to "not write code, write
| tests", and yet, in doing so it defines an API. This will cause
| the AI to hallucinate the API. Layering in yet more tests on top
| of this will cause that API to deform in strange ways that pass
| tests but that the adversary will not be able to cope with
| because it runs too late in the VSDD process.
|
| I've seen this exact process play out in my own work. The AI
| generates code and tests that pass with high code coverage and
| honors invariants set by spec. I look at the code and find a rats
| nest / ball of mud that will cost 10x more tokens to enhance
| should I ever need to add a feature.
|
| So, I think you're on to something, but I think the process might
| be discounting extensibility and resilience under change.
| NeutralForest wrote:
| > My concern with TDD is that writing tests necessarily implies
| designing an API upon which those tests operate
|
| It really forces you to do outside-in testing; I usually
| describe the kind of API I want when chatting with the agents.
| For example, the CLI options, the routes that might be useful
| for an API, etc.
|
| > I look at the code and find a rats nest / ball of mud that
| will cost 10x more tokens to enhance should I ever need to add
| a feature.
|
| Agreed, I don't know if there are good forcing functions to
| avoid complexity. The providers have a huge incentive to have
| you waste your tokens (for example when it re-outputs the
| complete file when you ask for a tiny change).
| bonesss wrote:
| "Test Driven Design" is another way to frame it.
| 0x457 wrote:
| In "proper" TDD you you supposed go:
|
| - write a test for method that does not exist, it just calls
| the method and nothing else
|
| - write method that does nothing
|
| - add/extend test that uses that method <-- this very loop
| starts
|
| - modify method until tests passes
|
| - go back to loop start until you're done
|
| I always hated it. When I work with LLM i first massage
| interface that tests, then tests, then implementation until
| all these tests pass.
|
| > for example when it re-outputs the complete file when you
| ask for a tiny change).
|
| well with sonnet 3.5 and 4.5 (can't say about 4.6) it often
| will get stuck in a loop trying to update just the required
| parts and iether waste tons of tokens doing these updates or
| waste tons of tokens to a point where restring file from git
| is required. Tokens get wasted regardless.
| skydhash wrote:
| I like tests, but I don't bother with TDD because it's so
| ceremonial. I design the API, or at least sketch it out
| (using a whiteboard or drafting some notes, and doing
| research). Then I iterate and refine. I only bother with
| tests once I can commit or when it's no longer viable to
| tests manually (edit-compile-run cycle). And a lot of time
| I follow the table pattern.
|
| https://www.oreilly.com/library/view/simplicity/97988886517
| 0...
| DaylitMagic wrote:
| I see what you're saying. Modularity and interfaces are really
| important between the different aspects of what's being
| developed. And it is worth putting time into the question "if
| another will use this, what would they potentially use it for,
| and why?". It doesn't mean that it needs to be built now - but
| considering that and ensuring that the planned code executes
| against that is a good strategy.
| sjbr wrote:
| Nice. It can work with something like
| https://github.com/github/spec-kit ?
| beders wrote:
| My employer is trying to convince us to embrace spec-kit. But
| we are a Clojure shop: we iterate fast and produce results. We
| don't sit around and write specs and then hope working code
| plops out.
| adastra22 wrote:
| With AI tools, spec-driven development is the lowest latency
| option.
| _pdp_ wrote:
| Everything in this post stems from the assumption that you
| already know what you're doing, which is probably true for things
| you've built before. But I hope we can agree that you can't spec
| out something you have no clue how to build, let alone write the
| tests before you've even explored the boundaries of the problem
| space. That's completely unreasonable.
|
| My second point is that this approach is fundamentally wrong for
| AI-first development. If the cost of writing code is approaching
| zero, there's no point investing resources to perfect a system in
| one shot. What matters more is how fast you can explore the
| edges. You can now spin up five agents to implement five
| different versions of the thing you're building and simply pick
| the best one.
|
| In our shop, we have hundreds of agents working on various
| problems at any given time. Most of the code gets discarded. What
| we accept to merge are the good parts.
| politician wrote:
| "Most of the code gets discarded." If you don't mind sharing,
| what's your signal-to-token ratio?
| kvdveer wrote:
| How do you propose we measure signal? Lines of code is
| renowned for being a very bad measure of anything, and I
| really can't come up with anything better.
| politician wrote:
| The OP said that they kept what they liked and discarded
| the rest. I think that's a reasonable definition for
| signal; so, the signal-to-token ratio would be a simple
| ratio of (tokens committed)/(tokens purchased). You could
| argue that any tokens spent exploring options or refining
| things could be signal and I would agree, but that's harder
| to measure after the fact. We could give them a flat 10x
| multiplier to capture this part if you want.
| mirekrusin wrote:
| I'm going to call it out as bullshit, you can't dig out
| "what you like" from "hundreds agents running all the
| time".
| _pdp_ wrote:
| One of our projects has 1.2K open pull requests.
|
| https://i.postimg.cc/Jnfk9b8g/Xnapper-2026-02-28-22-25-42
| .pn...
|
| We probably accept 1-2 per day.
|
| I personally discard code for the tiniest of reasons. If
| something feels off moments after I open the PR, it gets
| deleted. The reason we still have 1.2K open PRs is
| because we can't review all of them in time.
|
| The most likely solution is to delete all of them after a
| month or two. By that time the open PRs on this project
| alone will be at least 10-20 more.
| DaylitMagic wrote:
| If you don't mind the question with regard to your second
| point, couldn't what you've done in your shop be also used
| here? There's no reason why 'try to develop it five different
| ways and pick the best parts out of each' is incompatible with
| the 'VSDD' concept; seems like it could be included?
| tikhonj wrote:
| > _you can 't spec out something you have no clue how to build_
|
| Ideally--and at least _somewhat_ in practice--a specification
| language is as much a tool for design as it is for correctness.
| Writing the specification lets you explore the design space of
| your problem quickly with feedback from the specification
| language itself, even before you get to implementing anything.
| A high-level spec lets you pin down which properties of the
| system actually matter, automatically finds an inconsistencies
| and forces you to resolve them explicitly. (This is especially
| important for using AI because an AI model will silently
| resolve inconsistencies in ways that don 't always make sense
| but are also easy to miss!)
|
| Then, when you do start implementing the system and inevitably
| find issues you missed, the specification language gives you a
| clear place to update your design to match your understanding.
| You get a concrete artifact that captures your understanding of
| the problem _and_ the solution, and you can use that to keep
| the overall complexity of the system from getting beyond
| practical human comprehension.
|
| A key insight is that formal specification absolutely does not
| have to be a totally up-front tool. If anything, it's a tool
| that makes iterating _on the design of the system_ easier.
|
| Traditionally, formal specification have been hard to use as
| design tools partly because of incidental complexity in the
| spec systems themselves, but mostly because of the overhead
| needed to not only implement the spec but also maintain a
| connection between the spec and the implementation. The tools
| that have been practical outside of specific niches are the
| ones that solve this connection problem. Type systems are a
| lightweight sort of formal verification, and the reason they
| took off more than other approaches is that typechecking
| automatically maintains the connection between the types and
| the rest of the code.
|
| LLMs help smooth out the learning curve for using specification
| languages, and make it much easier to generate and check that
| implementations match the spec. There are still a lot of rough
| edges to work out but, to me, this absolutely seems to be the
| most promising direction for AI-supported system design and
| development in the future.
| virgilp wrote:
| Nothing of what you write here matches my experience with AI.
|
| Specification is worth writing (and spending a lot more time on
| than implementation) because it's the part that you can still
| control, fully read, understand etc. Once it gets into the
| code, reviewing it will be a lot harder, and if you insist on
| reviewing everything it'll slow things down to your speed.
|
| > If the cost of writing code is approaching zero, there's no
| point investing resources to perfect a system in one shot.
|
| THe AI won't get the perfect system in one shot, far from it!
| And especially not from sloppy initial requirements that leave
| a lot of edge (or not-so-edge) cases unadressed. But if you
| have a good requirement to start with, you have a chance to
| correct the AI, keep it on track; you have something to go back
| to and ask other AI, "is this implementation conforming to the
| spec or did it miss things?"
|
| > five different versions of the thing you're building and
| simply pick the best one.
|
| Problem is, what if the best one is still not good enough? Then
| what? You do 50? They might all be bad. You need a way to
| iterate to convergence
| manmal wrote:
| This. Waterfall never worked for a reason. Humans and agents
| both need to develop a first draft, then re-evaluate with the
| lessons learned and the structure that has evolved. It's very
| very time consuming to plan a complex, working system up
| front. NASA has done it, for the moon landing. But we don't
| have those resources, so we plan, build, evaluate, and
| repeat.
| zozbot234 wrote:
| That "first draft" still has to start with a spec. Your
| only real choice is whether the spec is an actual part of
| project documentation with a human in the loop, or it's
| improvised on the spot within the AI's hidden thinking
| tokens. One of these choices is preferable to the other.
| giancarlostoro wrote:
| Thats why I have AI do a write up about the system I want to
| build, I then review it all. If it looks good I use it as my
| prompt.
| zppln wrote:
| > But I hope we can agree that you can't spec out something you
| have no clue how to build
|
| Eh, of course you can. You can specify anything as long as you
| know _what_ you want it to do. This is like systems engineering
| 101 and people do it successfully all the time.
| theptip wrote:
| There's a real tension here.
|
| If you are vibe-coding, this approach is definitely going to
| kill you buzz and lose all the rapid iteration benefits.
|
| But if you are working in an existing large system, vibe coding
| is hard to bring into the core. So I think something more
| formal like OP is needed to reap major benefits from AI.
| zozbot234 wrote:
| This is just AI-written slop, but _even_ if you 're vibe
| coding and want to go for rapid iteration, you _still_
| benefit by having the AI write out a broad plan of what it 's
| going to do and looking it over before telling it to
| implement it. One-shot vibe coding is totally worthless, but
| the more you're aware of what the AI is thinking about and
| ready to revise its plans, the better it can potentially do.
| noosphr wrote:
| If the price of code is zero then changing the spec also costs
| zero in terms of code and. This is what always was the problem
| with specs before. You'd write one, run it through the prover,
| write the code, then have to throw out the whole thing because
| there was a business case you didn't account for.
|
| Now the bottom 98% can be given to a robot with a clear success
| signal other than 'it looks about right'.
| baq wrote:
| code is orthogonal to spec. you can iterate on the code and
| iterate on the spec. the spec is not meant to be constant, it's
| a form of ECC for the artifacts of the coding pipeline.
| _pdp_ wrote:
| A lot of interesting replies below this comment that I won't be
| able to respond to individually.
|
| I'll just leave this here:
|
| https://en.wikipedia.org/wiki/P_versus_NP_problem
| robot-wrangler wrote:
| That seems barely related and settles nothing? Bottom line is
| simple, saying "you can't spec out something you have no clue
| how to build" is saying you cannot desire coldness unless you
| understand how to build a refrigerator. It's just the
| difference between _what_ and _how_. If you don 't know the
| difference between implementation and specifications, just
| try a whole day of answering "what" and "why" questions with
| "how" answers and see how it goes.
| _pdp_ wrote:
| Writing tests for a known solution (verification) is
| straightforward. But speccing out and testing something you
| haven't even figured out how to build yet (discovery) is a
| fundamentally harder problem.
|
| Try speccing out a flux capacitor. I'll wait.
|
| https://chatbotkit.com/reflections/verification-is-easier-
| th...
| DaylitMagic wrote:
| Some random (hopefully additive and helpful) thoughts:
|
| Many companies have older code bases / databases that can be
| somewhat well defined (and somewhat not). If things have been
| slowly iterating over 35 years, there's a lot of undocumented
| edge behavior that may occur; it may be beneficial to have a step
| before Edge Case Catalog where there's some kind of prompting to
| catalogue how the inputs and outputs work, and then find the
| different inputs and outputs - and then confirm that with Input A
| and Output A that it works as expected. (Legacy systems often
| have weird orchestration that nobody remembers.)
|
| (Sub-note: This is somewhat part of the provable properties
| catalog; while this step could be placed there, it would require
| a re-run of edge case catalog build potentially, which isn't a
| bad thing.)
|
| A small note that I personally think is a good idea is better
| code commenting than has been outlined here - the spec itself
| should be woven into the code with potentially slightly over-
| commenting for each aspect, code spec gets lost. The code itself
| should serve as context, especially in the TDD stage.
|
| I think it's implicit but may be worth overtly stating that for
| the Code Quality check in Phase 3 that it also checks on a zero-
| trust basis, and doesn't include things like hardcoded keys.
|
| I'm not sure what Chainlink is (sorry!) but I like the ideas
| outlined around the decomposition - but it misses stringing
| everything together end-to-end in the way outlined here (it asks
| to create each part, but never actually weaves the whole
| together).
|
| Something not covered - is sequencing work and decomposition of
| work. A spec can create multiple dependencies within itself,
| requiring things to be worked on in a specific order.
| galoisscobi wrote:
| I think this word salad doesn't have enough buzzwords. Throw in a
| few more acronyms too.
| desireco42 wrote:
| Claude or something different... there is life beyond Claude I
| assure you and it is quite good and colourful.
| mitchbob wrote:
| Upvoted for the Sarcasmotron.
| esafak wrote:
| This is AI slop not worth my time. What would be interesting is
| if the author shared her practical experience in implementing it.
| Let's see some of those specs. What tricky bugs did it catch? The
| author's latest repo hasn't even been passing CI, so what does
| that say? https://github.com/dollspace-gay/Tesseract-
| Vault/commits/mai...
| relativeadv wrote:
| yes, all of the typical signs and symptoms appear to be there.
| Lists upon sublists of verbose overengineered plausibly
| thoughtful writing.
|
| If you can't be bothered to write it, why should i be bothered
| to read it?
| dollspace wrote:
| The runners are failing because it moved them from github
| hosted to self hosted and its requiring fixes, but you would
| know that if you actually paid attention to the commits and
| werent just looking for cheap dunks. Have a good one.
| esafak wrote:
| I looked at the past few pages of your repo's history and
| half the time it is broken. Is this continued failure despite
| using a verified spec, or did you not use one? If not which
| repo should we look at instead? I am sorry but I see nothing
| from you to engage here.
| vielite1310 wrote:
| I would like to be enlightened myself if RPI,BMAD or any spec-
| driven approaches actually worked for any mid/large scale
| projects, without wasting millions of tokens of course :)
| SirensOfTitan wrote:
| LLM-assisted development feels a lot like trend-driven
| development. When dealing with technique and heterogenous prompts
| and goals, it's easy to gain somewhat of a gambler's fallacy with
| respect to a particular technique.
|
| Spec-driven development feels pretty questionable to me. I'm sure
| it works fine for feature work that is predictable or has been
| done before, but then I wonder why you'd waste your time with it.
|
| Prior to LLMs, the whole vibe was to iterate rapidly toward a
| working thing so you can see what works and what doesn't. Why
| would we abandon that strategy as an industry when the cost of
| writing code is ostensibly getting cheaper?
|
| If I'm using LLMs at all, I'm using them to do a breadth search
| of prior art or ideas, then I'm doing what I might call a
| prototype onion: successive clean room attempts at a novel
| problem, accumulating what I learn at each attempt in each
| successive prompt. I usually then take the prototype and write
| the final version myself so I'm properly internalizing the idea.
|
| Ultimately a lot of this prompt work feels like procrastination.
| It is not about understanding where these tools is useful and
| where they are not but trying to have them consume every aspect
| of the work.
| getnormality wrote:
| Or maybe people who like talking much more than they like code
| are now very excited about the possibility that talking has
| eaten software development.
|
| This is exactly backwards. For many tasks, formal languages are
| better, more real, more beautiful than English. No matter how
| many millions of tokens you have, you will never talk the
| formulas of Fermat, Euler, and Gauss into irrelevance. And the
| same is true of good code.
|
| Of course, a lot of code is ugly and utilitarian too, and maybe
| talking will make it less painful to write that stuff.
| skydhash wrote:
| > a lot of code is ugly and utilitarian too,
|
| And as everyone who can abstract well knows: Ugly code that
| have staying power have a reason to be ugly. And the best
| will be annotated with HACK and NOTE comments. Anything else
| can be refactored and abstracted to a much better form. But
| that requires a sense of craftsmanship (to know better) and a
| time allowance for such activity.
| pron wrote:
| If you come up with a strategy that seems to "solve programming",
| then you know _for certain_ there must be a flaw in it, and you
| need to identify where it is that corners must be cut and how.
|
| Computer science is an introspective discipline because it
| studies the essential difficulty of problems regardless of the
| process taken to solve them, and programming itself (i.e. the
| problem of producing a correct, or correct-enough program) is
| such a problem that can be, and has been studied. The question of
| learning whether a program X satisfies some correctness property
| P is known as the model-checking problem, and we know that
| answering it with _certainty_ is intractable. For example, some
| properties that are true for some program would take no less than
| 10 minutes to verify (regardless of how that verification is
| done), others will take no less than 10 hours, others no less
| than 10 months, others no less than 10 years and so on, and we
| don't know ahead of time whether the proprty is true, and if it
| is, where on this spectrum it falls.
|
| So suppose you decide some property _must_ be proven with full
| certainty, the question becomes, how long do you wait before
| giving up waiting for the validation and what do you do when you
| give up? If you then decide that you're okay with less than 100%
| confidence, what approach do you take and how much confidence do
| you actually have? The problem with that is that the answer to
| that question often requires a deep understanding of the
| implementation. I.e. if you have two programs, X and Y, that
| compute the same function, one less-than-perfect approach would
| give you 99% confidence with one of them, but only 10% confidence
| with another.
| twoodfin wrote:
| More, "If your LLM comes up with a strategy to 'solve
| programming' ..."
| Robdel12 wrote:
| I've gotten the absolute best results from LLMs just acting like
| the software engineer I've aspired to be the past 15 years.
|
| Normal dev things. Scope the ticket properly, break it down. Test
| well. Write the correct docs.
|
| LLM specific things are going to be gone next week
| twodave wrote:
| I feel this very recently. I just pretend the agent is a junior
| dev and tell them to do things I don't have time for. Reviewing
| the changes at my convenience is a lot like checking in on a
| junior dev, too. On the other hand I do feel like I get better
| results with the same teeing up that a junior dev requires, so
| I try to remove as many unknowns/dependencies as possible (or
| else explicitly tell it to leave some things as stubs) before
| sending him off to do something
| beders wrote:
| > Define the contract before writing a single line of
| implementation. Specs are the source of truth.
|
| There is only one source of truth and that is the source code. To
| define and change contracts written in an ambiguous language and
| then hope the right code will magically appear, is completely
| delusional.
|
| Iteration is the only game in town that is fast and produces
| results.
| choeger wrote:
| If I am not mistaken, the verification is problematic here. It's
| run too late.
|
| A piece of code that satisfies a single test will most likely
| _not_ be probable to adhere to the spec.
|
| Worse, the whole spec can only be correctly implemented in total.
| You cannot work iteratively by satisfying one constraint after
| the other. The same holds for the test cases. That means that
| satisfying the last test or fulfilling the last constraint will
| take _much_ more work than the first. The number of tests passed
| is not a good metric for completion of the implementation.
| melvinroest wrote:
| I've been doing something less formal. I stumbled upon Riaan
| Zoetmulder's free course on deep learning and medical image
| analysis [1] and found his article on spec-driven development
| [2]. He adapts the V-Model by specifying three things upfront:
| requirements, system design and architecture. The rest gets
| generated. He mentioned a study where they show that LLM
| assistance slowed down experienced open source devs on large
| codebases. The model doesn't know the implicit context. And to me
| that's the thing! An LLM should have an index of some sort.
|
| So I vibe coded my own static analysis program where I just track
| my own function calls. It outputs a call graph of all my self-
| defined functions and shows the name (and Python type hints) of
| what it is calling (excluding standard library function, also
| only self-defined stuff). Running that program and sending the
| diff from time to time seems to have helped a lot already.
|
| [1] https://www.riaanzoetmulder.com/courses/deep-learning-
| medica...
|
| [2] https://www.riaanzoetmulder.com/articles/ai-assisted-
| program...
| xcubic wrote:
| I am not following, can you give a concrete example of your
| workflow?
| alpaylan wrote:
| You cannot escape from the human verifying the properties you
| want verified mechanically. This only gives you leverage in
| specific scenarios where specification is much simpler than the
| implementation.
| WestN wrote:
| Short take: replace TDD with BDD, and might add DDD as a spice.
| Otherwise this is a fairly good article.
|
| Why not TDD? Since a lot of developers use LLMs to create tests
| today, plus a lot of the training data contains information on
| how to do this. Making it something that it either can figure out
| to do by itself or that it will cheat. Both equally bad.
|
| A somewhat controversial take is that you should simply avoid
| writing tests which the LLM can produce by itself, similar to how
| we in the last week removed the agents.md file.
| chrisweekly wrote:
| could you say more about removing agents.md?
| supermdguy wrote:
| Probably referencing this:
| https://news.ycombinator.com/item?id=47034087
| jatins wrote:
| The gist is 100% AI written
| https://www.pangram.com/history/9d89ebba-cdba-40e1-b569-9ae1...
| daveac wrote:
| In a perfect world I can see this happening. But with AI
| increasing or output I think the real bottleneck is work
| sponsors, business logic and requirements
| discovery/translating/sign off.
|
| I am seeing more teams and features being rolled faster than
| before but then discovering that the sponsors (those requesting
| features and change) either don't invest the time up front or
| with timely feedback loops and work stalls or has to be redone as
| business does not see the results until it's either live or about
| to go live.
|
| This has always been the case but I think AI tooling has moved
| the bottleneck
| rsrsrs86 wrote:
| That's how I do it, minus the marketing.
| teiferer wrote:
| I expected formal verification to be part of this. That could not
| be fooled and is rock-solid, unless you cheat in your
| specification. Swap your AI "verifier" out for that and I'm on
| board.
| mpalmer wrote:
| Looks moderately interesting, but I refuse to upvote vibe-written
| submissions. Of course documents like this have their uses, but:
|
| - They cannot easily be attributed to a human author, and
| therefore debates and discussions on the substance of the ideas
| tend not to get too far
|
| - They (tend to) take well-established concepts, glue them
| together, and describe the result in reverential tones,
| regardless of the relative triviality of the solution. Hard to
| rule that out here tbh.
|
| I am not saying any of that is what's happening here. What I am
| saying is that I'm not going to waste my time reading something I
| can't easily vet for quality.
|
| Author (not OP) has written plenty of its* own words on Bluesky
| over the last few weeks. If it's written anything longhand about
| this stuff I'd be interested to read it. But for now "anti-slop
| bias" designed into the system has not reached the prose.
|
| *respecting pronouns
| FrankRay78 wrote:
| No much different from what I did manually when employer
| outsourced development to India.
| Animats wrote:
| Is the posting a description of a real system, or just
| imagination? Is there a link to something that makes this real?
| zozbot234 wrote:
| Imagination? More like hallucination - the AI-generated kind.
| jFriedensreich wrote:
| Its an interesting direction if you see it under the umbrella of
| diminishing costs: You build a product once with vibe coding and
| a design/ product hat. Once you know what works you rebuild it
| 100% in a framework like this. You do this every time from
| scratch when the tech debt or the mismatch between architecture
| and needs are too big.
| johnnyAghands wrote:
| Does anyone know what Chainlink is?
___________________________________________________________________
(page generated 2026-02-28 23:00 UTC)