[HN Gopher] Context engineering
___________________________________________________________________
Context engineering
Author : chrisloy
Score : 76 points
Date : 2025-11-02 08:52 UTC (14 hours ago)
(HTM) web link (chrisloy.dev)
(TXT) w3m dump (chrisloy.dev)
| elteto wrote:
| Are there any open source examples of good context engineering or
| agent systems?
| calebkaiser wrote:
| Any of the "design patterns" listed in the article will have a
| ton of popular open source implementations. For structured
| generation, I think outlines is a particularly cool library,
| especially if you want to poke around at how constrained
| decoding works under the hood: https://github.com/dottxt-
| ai/outlines
| voidhorse wrote:
| There is nothing precise about crafting prompts and context--it's
| just that, a craft. Even if you do the right thing and check some
| fuzzy boundary conditions using autoscorers, the model can still
| change out from beneath you at any point and totally alter the
| behavior of your system. There is no formal language here. After
| all, mathematics _exists_ because natural language is notoriously
| imprecise.
|
| The article has some good practical tips and it's not on the
| author but man I really wish we'd stop abusing the term
| "engineering" in a desperate attempt to stroke our own egos and
| or convince people to give us money. It's pathetic. Coming up
| with good inputs to LLMs is more art than science and it's a
| _craft_. Call a spade a spade.
| qrios wrote:
| I agree with you one hundred percent.
|
| But: Interestingly, the behavior of LLMs in different contexts
| is also the subject of scientific research.
| satisfice wrote:
| My thoughts exactly. The author is saying we should think
| strategically about the use of context. Sure. Yes. But for that
| to qualify as engineering we need solid theory about how
| context works.
|
| We don't have that, yet. For instance experiments show that not
| all parts of the context window are equally well attended.
| Imagine trying to engineer a bridge when no one really knows
| how strong steel is.
| skeeter2020 wrote:
| or how wide the river is year round
| chrisweekly wrote:
| "Context crafting", ok, sure. I think a lot of expert
| researchers (like simonw) would agree.
| calebkaiser wrote:
| I think it's fair to question the use of the term "engineering"
| throughout a lot of the software industry. But to be fair to
| the author, his focus in the piece is on design patterns that
| require what we'd commonly call software engineering to
| implement.
|
| For example, his first listed design pattern is RAG. To
| implement such a system from scratch, you'd need to construct a
| data layer (commonly a vector database), retrieval logic, etc.
|
| In fact I think the author largely agrees with you re: crafting
| prompts. He has a whole section admonishing "prompt
| engineering" as magical incantations, which he differentiates
| from his focus here (software which needs to be built around an
| LLM).
|
| I understand the general uneasiness around using "engineering"
| when discussing a stochastic model, but I think it's worth
| pointing out that there is a lot of engineering work required
| to build the software systems around these models. Writing
| software to parse context-free grammars into masks to be
| applied at inference, for example, is as much "engineering" as
| any other common software engineering project.
| amonks wrote:
| long shot, apropos of nothing, just recognized your name:
|
| If you are the cincinnatian poet Caleb Kaiser, we went to
| college together and I'd love to catch up. Email in profile.
|
| If you aren't, disregard this. Sorry to derail the thread.
| calebkaiser wrote:
| Hello friend!
| grigio wrote:
| I'd like a RSS feed of this blog..
| vladsanchez wrote:
| It's available, https://buttondown.com/chrisloy/rss but it's
| not in sync with the blog, just a single 2024 entry found.
| :shrug:
| aeve890 wrote:
| Are we still calling this things engineering?
| skeeter2020 wrote:
| "professionally trained & legally responsible for the results"
| is definitely not the same thing as what we used to just call
| "good at googling".
| aeve890 wrote:
| I'd say this shit is even worse that "good at googling".
| Literal incantation for stochastic machines is like just two
| notches above checking the horoscope.
| calebkaiser wrote:
| Based on the comments, I expected this to be slop listing a
| bunch of random prompt snippets from the author's personal
| collection.
|
| I'm honestly a bit confused at the negativity here. The
| article is incredibly benign and reasonable. Maybe a bit
| surface level and not incredibly in depth, but at a glance,
| it gives fair and generally accurate summaries of the
| actual mechanisms behind inference. The examples it gives
| for "context engineering patterns" are actual systems that
| you'd need to implement (RAG, structured output, tool
| calling, etc.), not just a random prompt, and they're all
| subject to pretty thorough investigation from the research
| community.
|
| The article even echoes your sentiments about "prompt
| engineering," down to the use of the word "incantation".
| From the piece:
|
| > This was the birth of so-called "prompt engineering",
| though in practice there was often far less "engineering"
| than trial-and-error guesswork. This could often feel
| closer to uttering mystical incantations and hoping for
| magic to happen, rather than the deliberate construction
| and rigorous application of systems thinking that
| epitomises true engineering.
| timr wrote:
| There's nothing particularly wrong with the article -
| it's a superficial summary of stuff that has historically
| happened in the world of LLM context windows.
|
| The problem is - and it's a problem common to AI right
| now - you can't generalize anything from it. The _next
| thing_ that drives LLMs forward could be an extension of
| what you read about here, or it could be a totally random
| other thing. There are a million monkeys tapping on
| keyboards, and the hope is that someone taps out
| Shakespeare's brain.
| calebkaiser wrote:
| I don't really understand this line of criticism, in this
| context.
|
| What would "generalizing" the information in this article
| mean? I think the author does a good job of
| contextualizing most of the techniques under the general
| umbrella of in-context learning. What would it mean to
| generalize further beyond that?
| simonw wrote:
| Yes, and we've also decided that they deserve the title
| "engineering" _more_ than software engineering does.
|
| Most engineering disciplines have to deal with tolerances and
| uncertainty - the real world is non-deterministic.
|
| Software engineering is easy in comparison because computers
| always do exactly what you tell them to do.
|
| The ways LLMs fail (and the techniques you have to use to
| account for that) have more in common than physical engineering
| disciplines than software engineering does!
| timr wrote:
| lol. who is "we"? I honestly can't tell if you're being
| serious.
|
| I'm going to start a second career in lottery "engineering",
| since that's a stochastic process too.
| simonw wrote:
| The "we" was a tongue-in-cheek reference to the "we" in the
| original question:
|
| > Are we still calling this things engineering?
| timr wrote:
| Yeah, I understand the symmetry, but...it begs the
| question.
| scuff3d wrote:
| Lol. This has to be a troll. No way someone seriously wrote
| this and meant it.
| simonw wrote:
| Little bit of both.
| voakbasda wrote:
| In the absence of a clear indicator, either interpretation
| could be possible:
|
| https://en.wikipedia.org/wiki/Poe's_law
| cadamsdotcom wrote:
| Yep. Consider woodworking - the wood you use might warp over
| time, or maybe part of it ends up in the sun or the thing
| you'll make gets partly exposed to water.
|
| Can you make a thing that'll serve its purpose and look good
| for years under those constraints? A professional carpenter
| can.
|
| We have it easy in software.
| dingnuts wrote:
| Woodworking is to civil engineering as being an IT help
| desk rep is to being a software engineer. Woodworking isn't
| engineering either. If you build a system with aspects you
| can measure and predictably tune, you're engineering. If
| you're making skilled alterations to an existing structure
| or system without applied math or science, you're partaking
| in a craft.
|
| Software engineering blurs the lines, sure, but woodworking
| isn't engineering ever.
| mpalmer wrote:
| Physical engineers might scoff good-naturedly at an attempt
| by project managers to refer to work scheduling as "logistics
| engineering".
|
| But they really shouldn't because obviously scheduling and
| logistics is difficult, involving a lot of uncertainty and
| tolerances.
| timr wrote:
| Uncertainty and tolerance implies that you have a
| predictable distribution in the first place.
|
| Engineers are not just dealing with a world of total chaos,
| observing the output of the chaos, and cargo culting
| incantations that seem to work for right now [1]...oh wait
| nevermind we're doing a different thing today! Have you
| tried paying for a different tool, because all of the
| _real_ engineers are using Qwghlm v5 Dystopic now?
|
| There's actually real engineering going on in the training
| and refining of these models, but I personally wouldn't
| include the prompting fad of the week to fall under that
| umbrella.
|
| [1] I hesitate to write that sentence because there _was_ a
| period where, say, bridges and buildings were constructed
| in this manner. They fell down a lot, and eventually we
| made predictable, consistent theoretical models that guide
| _actual engineering_ , as it is practiced today. Will LLM
| stuff eventually get there? Maybe! But right now we're
| still plainly in the phase of trying random shit and seeing
| what falls down.
| bdangubic wrote:
| exactly why calling this engineering is downright
| _criminal_
| voidhorse wrote:
| I completely agree that much of software engineering is not
| engineering, and building systems around LLMs is no better in
| this sense.
|
| When the central component of your system is a black box that
| you cannot reason about, have no theory around, and have
| essentially no control over (a model update can completely
| change your system behavior) engineering is basically
| impossible from the start.
|
| Practices like using autoscorers to try and constrain
| behaviors helps, but this doesn't make the enterprise any
| more engineering because of the black box problem.
| Traditional engineering disciplines are able to call
| themselves engineering only because they are built on
| sophisticated physical theories that give them a precise
| understanding of the behaviors of materials under specified
| conditions. No such precision is possible with LLMs, as far
| as I have seen.
|
| The determinism of traditional computing isn't really
| relevant here and targets the wrong logical level. We
| engineer systems, not programs.
| empath75 wrote:
| This is completely backwards. Engineers built steam engines
| first through trial and error and then eventually the laws
| of thermodynamics were invented to explain how steam
| engines work.
|
| Trial and error and fumbling around and creating rules of
| thumbs for systems you don't entirely understand is the
| purest form of engineering.
| aeve890 wrote:
| >The ways LLMs fail (and the techniques you have to use to
| account for that) have more in common than physical
| engineering disciplines than software engineering does!
|
| Ah yes, the God given free parameters in the Standard Model,
| including obviously the random seed of a transformer. What if
| just put 0 in the inference temperature? The randomness in
| llms is a technical choice to generate variations in the
| selection of the next token. Physical engineering? Come on.
| andai wrote:
| >just set temp to 0 to make LLMs deterministic
|
| Does that really work? And is it affected by the almost
| continuous silent model updates? And gpt-5 has a "hidden"
| system prompt, even thru the API, which seemed to undergo
| several changes since launch...
| simonw wrote:
| It famously does not:
| https://thinkingmachines.ai/blog/defeating-
| nondeterminism-in...
| Scipio_Afri wrote:
| Hey Simon, do you have any posts diving into how one
| might be able to deal with evaluating LLMs or Machine
| Learning models in general when reproducibility is so
| difficult given non-determinism? Pytorch has an article
| on it https://docs.pytorch.org/docs/stable/notes/randomne
| ss.html but then doesn't really go into how one would
| then take this deterministic result, and evaluate a model
| that is in production (which would very likely need for
| performance reasons the non-determinism features
| enabled).
|
| While this affects all models it seems, I think the case
| gets worse for in particular LLMs because I would imagine
| all backends, including proprietary ones, are batching
| users prompts. Other concurrent requests seem to change
| the output of your request, and then if there is even a
| one token change to the input or output token, especially
| on large inputs or outputs, the divergence can compound.
| Also vLLM's documentation mentions this:
| https://docs.vllm.ai/en/latest/usage/faq.html
|
| So how does one do benchmarking of AI/ML models and LLMs
| reliably (lets ignore arguing over the flaws of the
| metrics themselves, and just the fact that the output for
| any particular input can diverge given the above). You'd
| also want to redo evals as soon as any hardware or
| software stack changes are made to the production
| environment.
|
| Seems like one needs to setup a highly deterministic
| backend, by forcing non-deterministic behavior in pytorch
| and using a backend which doesn't do batching for an
| initial eval that would allow for troubleshooting and
| non-variation in output to get a better sense of how
| consistent the model without the noise of batching and
| non-deterministic GPU calculations/kernels etc.
|
| However then, for production, when determinism isn't
| guaranteed because you'd need batching and non-determism
| for performance, I would think that one would want to do
| multiple runs in various real-world situations (such as
| multiple users doing all sorts of different queries at
| the same time) and do some sort of averaging of the
| results. But I'm not entirely sure, because I would
| imagine the types of queries other users are making would
| then change the results fairly significantly. I'm not
| sure how much the batching that vLLM does would change
| the results of the output; but vLLM does say that
| batching does influence changes in the outputs.
| simonw wrote:
| This is _so hard_! I don 't yet have a great solution for
| this myself, but I've been collecting notes about this on
| my "evals" tag for a while:
| https://simonwillison.net/tags/evals/
|
| The best writing I've seen about this is from Hamel
| Husain - https://hamel.dev/blog/posts/llm-judge/ and
| https://hamel.dev/blog/posts/evals-faq/ are both
| excellent.
| andai wrote:
| >But why aren't LLM inference engines deterministic? One
| common hypothesis is that some combination of floating-
| point non-associativity and concurrent execution leads to
| nondeterminism based on which concurrent core finishes
| first. We will call this the "concurrency + floating
| point" hypothesis for LLM inference nondeterminism.
|
| Dang, so we don't even know why it's not deterministic,
| or how to make it so? That's quite surprising! So if I'm
| reading this right, it doesn't just have to do with LLM
| providers cutting costs or making changes or whatever.
| You can't even get determinism locally. That's wild.
|
| But I did read something just the other day about LLMs
| being invertible. It goes over my head but it sounds like
| they got a pretty reliable mapping from inputs to
| outputs, at least?
|
| https://news.ycombinator.com/item?id=45758093
|
| > Transformer components such as non-linear activations
| and normalization are inherently non-injective,
| suggesting that different inputs could map to the same
| output and prevent exact recovery of the input from a
| model's representations. In this paper, we challenge this
| view. First, we prove mathematically that transformer
| language models mapping discrete input sequences to their
| corresponding sequence of continuous representations are
| injective and therefore lossless, a property established
| at initialization and preserved during training. Second,
| we confirm this result empirically through billions of
| collision tests on six state-of-the-art language models,
| and observe no collisions.
|
| The distinction here appears to be between the output
| tokens versus some sort of internal state?
| aeve890 wrote:
| Strictly speaking, it should work. We don't have a _real_
| RNG yet and with the same seed any random function
| becomes deterministic. But behind the blackbox of LLM
| providers who know what's tunned processing your request.
|
| But my point stands. The non-deterministic nature of LLMs
| are implementation details, not even close to physical
| constraints as the parent comment suggest.
| Scipio_Afri wrote:
| There is inherent non-determinism in all machine learning
| models unless you explicitly configure pytorch or other
| frameworks to do determinism (https://docs.pytorch.org/do
| cs/stable/notes/randomness.html). However, this is very
| unlikely to be done in models that are being run in
| production due to performance and other issues.
| dingnuts wrote:
| The tools mechanical and civil engineers use are predictable.
| You're confusing the things these engineers design, which
| have tolerances and things like that, with the tools
| themselves.
|
| If an engineer built an internal combustion engine that
| misfired 60% of the time, it simply wouldn't work.
|
| If an engineer measured things with a ruler that only
| measured correctly 40% of the time, that would be the apt
| analogy.
|
| The tool isn't what makes engineering a practice, it's the
| rigor and the ability to measure and then use the
| measurements to predict outcomes to make things useful.
|
| Can you predict the outcome from an LLM with an "engineered"
| prompt?
|
| No, and you aren't qualified to even comment on it since your
| only claim to fame is a fucking web app
| skylurk wrote:
| Civil engineers deal with contractors who would misfire
| 100% of the time if they could get away with it.
| simonw wrote:
| > No, and you aren't qualified to even comment on it since
| your only claim to fame is a fucking web app
|
| Whoa, where did _that_ come from?
| skylurk wrote:
| I know, right? I did not predict that output either.
| j45 wrote:
| Engineering how to engineer things might be engineering in some
| ways.
| sgt101 wrote:
| Why would I believe that any of this works? This is just some
| blokes idea of what people should do.
|
| There is no evidence offered. No attempt to measure the benefits.
| calebkaiser wrote:
| Most of the inference techniques (what the author calls context
| engineering design patterns) listed here originally came from
| the research community, and there are tons of benchmarks
| measuring their effectiveness, as well as a great deal of
| research behind what is happening mechanistically with each.
|
| As the author points out, many of the patterns are
| fundamentally about in-context learning, and this in particular
| has been subject to a ton of research from the mechanistic
| interpretability crew. If you're curious, I think this line of
| research is fascinating: https://transformer-
| circuits.pub/2022/in-context-learning-an...
| dwaltrip wrote:
| Imagine the gall of someone who just goes on the internet and
| writes something.
| alecco wrote:
| This looks AI generated slop.
___________________________________________________________________
(page generated 2025-11-02 23:02 UTC)