[HN Gopher] Prompting LLMs is not engineering
___________________________________________________________________
Prompting LLMs is not engineering
Author : Bluestein
Score : 64 points
Date : 2025-07-04 21:38 UTC (1 hours ago)
(HTM) web link (dmitriid.com)
(TXT) w3m dump (dmitriid.com)
| VirgilShelton wrote:
| It's just a buzz word but sadly it will (And already has)
| effected real engineers :(
| LtWorf wrote:
| I don't think they were good engineers to begin with anyway.
| dlevine wrote:
| My theory is that, when done properly, it's much closer to
| science than engineering.
|
| And by "done properly," i mean done in a regimented way with
| evals to verify that a wide range of inputs produce the desired
| outputs.
|
| Prompting is much closer to discovering the properties of an
| already existing system than building something using engineering
| methods.
| ankit219 wrote:
| "Done properly" anything can be science/engineering. Just make
| a process, give it a KPI on how well it's performing, and you
| have an engineering. Movie making done properly is where
| audience is engaged with every act, you test screen it, see
| audience reaction, tweak, have a specific set of evals, and
| then you have something scoring high on the metrics. Part of
| that happens today, but you would not call that science. It's
| not what people mean when they call something as science or
| engineering. Your's is too broad of a definition to mean
| anything specific.
| NotAnOtter wrote:
| My definition of science is something like "a methodical
| approach to reach some truth" whereas engineering is something
| like "a methodical approach to reach some functionality". And I
| think some version of that is pretty universally accepted.
|
| So by that definition, prompt engineering is much closer to
| engineering than science. That said, I would consider it closer
| to product development than either of the above two; I don't
| count 'tell an llm you'll torture it until your website is
| hopefully less buggy' a methodical approach.
| crowcroft wrote:
| This sounds like the same kind of issue 'real' engineers have
| with software 'engineering'.
| mystraline wrote:
| Doesn't matter that much.
|
| Management _thinks_ they can do our jobs by replacing us with a
| shell scrip... LLM prompt.
|
| We're always moving up and down the tech ladder, solving problems
| we thought were solved, and, we fix them. But some prompt is
| supposedly going to replace understanding of your infra?! Hardly.
|
| But you'll save money for this quarter. Past that is definitely
| up for discussion.... With your shareholders.
| labrador wrote:
| Prompting is more art than engineering. But understanding the
| basics of LLMs, which are engineered systems, helps you get
| better results. I think of it like being a good interviewer:
| you're not controlling the conversation, but you know how to
| guide it. A good prompter, like a good interviewer, knows how to
| ask the kind of question that gets a meaningful answer. I think
| of LLMs like a giant library and my task is to steer it to the
| right sections for a good answer.
| andy99 wrote:
| No,the prompt is a hyperparameter. If you're not treating it as
| such and systematically experimenting, it's just superstition.
| labrador wrote:
| You're view of it is correct for agents but for my purposes,
| which are mostly brainstorming, researching and asa thinking
| tool, I have GPT 4o memory systems turned on. This allows it
| to be more personable and allows crafting of answers more
| attuned to my meaning. System and chat history memory
| obviously introduce global state which is not exactly a
| tunable hyperparameter for more consistent results. You
| should probably turn it off for agents so your view is
| accurate.
| Bluestein wrote:
| > hyperparameter
|
| That's an interesting way to put it. indeed.-
| sincerely wrote:
| I think as people get increasingly aware of the nebulous
| consistency of hosted LLMs, local models will become more
| valuable
| ThrowOregonAway wrote:
| Technically you're not an engineer unless you're licensed and
| bonded and belong to a professional engineering society, or you
| drive a train, or have invented a unique engine. Period, end of
| discussion.
|
| I know, I know. California changed their law so coders could puff
| themselves up and put on airs and make their resumes look more
| impressive. But this is a sad anomaly.
| crustycoder wrote:
| I'm licensed by The Engineering Council in the UK for doing
| computery stuff, and by the relevant European body. I've been
| writing software for a living for most of my 40+ year career,
| and still do. So Chartered Software Engineers do exist.
|
| But perhaps not in CA.
| ThrowOregonAway wrote:
| We have no such thing in the US. There is no license for
| software "engineering" here, it's a made up term.
| gnabgib wrote:
| There is in Canada, licensed and bonded.
| rapatel0 wrote:
| As someone with a PhD in electrical engineering, I care exactly
| zero percent about a paid for stamp from an org calling an me
| engineer or not.
| ThrowOregonAway wrote:
| This is why the planes are falling out of the sky now, this
| attitude of yours. As an eminent master and founder of the
| entire field of electrical engineering, I am much more
| qualified than you to have an opinion on this matter as well.
| LtWorf wrote:
| Well my country didn't change the law, so I still think it's
| very weird how every american is an engineer.
| gundmc wrote:
| I have no problem taking issue with using the term "engineering"
| for this, but this reads like OP does not believe that you can
| get better results based on the way you structure your prompt and
| what information you include. That is wild to me and should be
| obvious to anyone who spends more than a few hours using LLMs.
|
| For what it's worth, I think about the "engineering" in
| "Prompt/Context Engineering" almost more akin to how it's used in
| "Social Engineering". You are influencing the model to produce a
| desirable result.
| bn-l wrote:
| You won't get far _without_ engineering if all you can do is
| prompt.
| floppyd wrote:
| This article is just gatekeeping the word "engineer" for no
| reason. When I was developing an agent recently I picked a model
| (thus fixing most of the "uncertainties" mentioned) and started
| to construct the system prompt, from simpler to more complex,
| sometimes back to simpler, discovering that to this particular
| model the name choice means a lot, observing how often I was or
| wasn't getting desired results, etc etc. I think it would be
| challenging to find a definition of the word "engineer" that
| wouldn't include this process, even if it, obviously, wasn't as
| complex as some other engineering projects.
| exe34 wrote:
| Honestly if we call software "engineering" in the first place,
| without any kind of certification/liability, I don't see the
| problem with applying it to everything else.
| umanwizard wrote:
| Indeed. Engineer is a silly word to gatekeep. Its original
| literal meaning is just someone who builds engines (in the
| original, broader sense of "engines" which basically means
| "machines"). The appropriation of the term to mean formalized
| categories with certifications etc. came hundreds of years
| later.
| LtWorf wrote:
| I doubt that's the origin, since "ingegno" means cleverness
| in italian...
| cwillu wrote:
| A word being fuzzy does not mean it has no meaning and that
| it can therefore be used for anything. A word being used in
| a particular (but fuzzy) way hundreds of years ago does not
| mean that it can't have a different (and fuzzy) meaning
| today. Yes, words drift, but the vague fuzzy notion that
| engineering meant 20 years ago still has value, and if the
| word has lost that meaning with no replacement, that's a
| problem.
| voidhorse wrote:
| All sorts of human activities involve processes like this. Art,
| for instance.
|
| Engineering is about _precision_ it 's not about this fuzzy
| sort of "iterate til it works" approach. Sure, iterations is
| involved, but knowing the precise conditions and bounds under
| which a system functions in a specific way is what engineering
| is actually about. I would not want a person building a bridge
| to use your process of iteration and guess work.
|
| Yes, under this idea a lot of software engineering isn't actual
| engineering.
| TJSomething wrote:
| Under this idea, most engineering isn't engineering. Some of
| the oldest engineering is ad hoc battlefield engineering to
| construct siege engines and building tunnels to deliver
| explosives. That frequently only needed to work one time.
|
| With bridges, you only need that high confidence because
| there are high costs and risks. Also, the stakeholders are
| usually governments, who require very predictable results.
| All that effort is worth it because the artifact will be
| useful for a long time for a lot of people.
|
| It might be okay if some widget only lasts for 6 months. So,
| you empirically shave off material until it's as cheap as
| possible while failing at an acceptable rate.
|
| The cost of shipping is low for software, so the risk profile
| is even more different. This can be shifted for high stakes
| software and I think there are some issues there, but many
| things are shaped more like Facebook than aircraft control
| systems.
|
| I think the core of engineering is in evaluating these
| tradeoffs and figuring out where you can expend effort most
| efficiently.
| StrLght wrote:
| You've described what sounds a lot like experimentation:
| observing how a black box behaves on different inputs.
| Conducting experiments is closely related to engineering -- I
| don't think anyone would claim different. But engineering _is
| not limited_ to experimentation alone.
| handfuloflight wrote:
| What level of threshold of illustration do you need to reach
| for it to be properly said that you are drawing?
|
| What's the analogous threshold to say you're engineering?
| StrLght wrote:
| Art is subjective. It isn't about checking boxes and
| counting how many formal criteria it meets.
|
| Engineering is broad, it uses many different methods and
| techniques. All I am saying is that I wouldn't call someone
| a mathematician if they can only find the result of 2+2.
| tiahura wrote:
| _gatekeeping the word "engineer" for no reason_
|
| There was value in the word Engineer connoting professionalism
| and accountability. Checkers and draftsmen were checkers and
| draftsmen, not engineers.
|
| Then, anyone who knew the difference between a patch cable and
| a crossover cable became a network engineer.
| NotAnOtter wrote:
| Words have meaning. I'm not a prescriptivist but when you have
| an army of formerly PM's, Designers, etc all at once exclaiming
| "I'm an engineer too!" that should signal that maybe the thing
| they're doing isn't really engineering.
|
| Do you count making a square space splash page 'engineering'?
| Tools improving to the point that the barrier of entry plummets
| is great. That doesn't mean you're now engaging in the same
| fundamental task that happened before things got easier/
| echohack5 wrote:
| If the thing standing in your way to engineering a great
| product was syntax, and a bunch of gate-kept and inconsistent
| linux cli tooling, then its fine to call it engineering.
|
| I dunno, do you read bytecode?
| GeneralMayhem wrote:
| I don't think that's the point. If non-technical people are
| able to make a product happen by asking a machine to do it
| for them, that's fine. But they're not engineering. It
| simply means that engineering is no longer required to make
| such a product. Engineering is the act of solving problems.
| If there are no problems to solve, then maybe you've
| brought about the product, but you haven't "engineered" it.
|
| I don't think that memorizing arcane Linux CLI invocations
| is "engineering" either, to be clear.
| NotAnOtter wrote:
| If the thing gating your from creating a great product was
| the engineering, then I would call you a great product
| designer and not an engineer. If you suddenly can create
| products with a new tool - you didn't change, the tools
| did.
|
| You're still a great product designer and not an engineer.
| baal80spam wrote:
| In my org they told us clearly: either we will learn prompt
| engineering, or they will replace us with people who already know
| it.
| jonplackett wrote:
| I don't mind people criticising AI.
|
| It's bad in loads of ways.
|
| But it also has some good bits.
|
| To call it all snake oil is just as much bullshit as to claim it
| will solve all humanity's problems.
| NotAnOtter wrote:
| To claim AI is useless is ignorant. To claim AI is as good as
| 99% of people claim it to be, is also ignorant.
|
| A lot of people have drank the 'exponential growth' kool aid
| but don't have enough understanding of the underlying tech to
| realize (1) there could be some fundamental limits to how good
| LLM based AI can be. Or AI in general for that matter. (2) a
| big part of why it got so good so fast is because we started
| pumping trillions of dollars of microchips & electricity into
| it. The hardware and energy consumption cannot continue
| exponentially.
| lloydjones wrote:
| This article seems like LinkedIn style controversy-bait, or is
| written from a place of extreme pedantry.
|
| Evals can come pretty close to getting "deterministic" output
| from LLMs, and I'd argue that this is reasonably considered
| engineering.
| nikanj wrote:
| And taking photographs is not painting, but it doesn't mean it's
| not a form of visual art that requires skill to do well
| rapatel0 wrote:
| There are plenty of engineering disciplines that need to have
| statistical approaches to solving problems. Prompt engineering,
| should be approached by creating benchmark datasets and defining
| measures of efficacy and reliability.
|
| More generally, IMHO engineering is use of physical and
| informational tools to build a solution. Tools may have
| unpredictability and reliability concerns. Its the job of an
| engineer to utilize the power of these tools while overcoming the
| reliability issues that might be present.
|
| Example: Semiconductor manufacturing involves shooting gases at
| piece of silicon. There are all sorts of random scattering,
| distributional anomalies, and patterning problems that arise
| largely at random. I think you would still call it engineering.
| simonw wrote:
| > _" Prompt engineers" will tell you that some specific ways of
| prompting some specific models will result in a "better
| result"... without any criteria for what a "better result" might
| signify._
|
| That's what evals are for. The best developers working on LLM
| applications are the ones who are addressing the problem
| described in this quote. Here's a recent thread about that:
| https://news.ycombinator.com/item?id=44430117
| socalgal2 wrote:
| I just tried Gemini Canvas which I've seen lot of cool results
| but I failed and I found it infuriating, similar to the worst
| version of trying to navigate a phone maze to talk to a real
| person.
|
| First, an example I saw online
|
| https://www.reddit.com/r/WebVR/comments/1lh9d3h/webvr_game_c...
|
| I've seen others. They seemed impressive.
|
| In particular, I thought it might be fun to ask Gemini to make a
| fireworks display for the 4th of July. What I imagined was
| Fantavision
|
| https://www.youtube.com/watch?v=pD2AnDhyynM
|
| I didn't expect it to get there but I did hope one a scale of 1
| to 10 (10 = Fantavision) it would get to 6 ish. I'd say it got to
| 3 of 10.
|
| What I got was some cubes with yellow planes on the side for
| buildings. This means things are massively inefficient since
| instead of a simple textured cube for each building, each
| building is 1000s of polygons, 2 for each window. Lots of Z
| fighting. I asked for a river with reflections. The reflections
| were practically invisible. I think it was generating reflection
| fireworks under the water plane, not actually "reflecting" the
| ones above. Because they were under the plane they were faded out
| because of blending which is not how reflections of fireworks
| work with water. Then, the fireworks themsevles were horrible,
| made with 1 pixel particles, no glow, no trails, no substance.
| They were further very un-random in their velocities and looked
| nothing like real fireworks.
|
| I tried for 90 minutes to get it to make something better but
| mostly failed. I did manage to get it to stop using pixels for
| particles and use textured quads.
|
| It's first version started with "click to launch fireworks". That
| didn't work at all, blank screen. I told it I didn't want to
| launch firework, I just wanted them to auto-launch. It made a new
| one, this time it had a caption "with auto launching fireworks"
| (and still blank). I told it there was no point in such a caption
| as it would be obvious from seeing the fireworks that they were
| auto-launching. It got rid of that caption but still a blank
| screen. I started over, this time adding "auto-launching, auto
| camera" type elements to my intial prompt.
|
| I tried several times to get it to make an interesting city
| around both sides of a river (because this was supposed to be an
| NYC fireworks display). It failed in all kinds of ways. I'd asked
| it to fly the camera around the fireworks and never have it go
| behind any buildings. I put the camera in the buildings so you
| couldn't see any fireworks. I told it "not behind the builings".
| It moved the camera further way, twice. I finally got it to put
| the camera only above the river between the two shorelines (if
| you can call a plane with some cubes on both sides a river with
| shorelines). The other way it failed was it just got too slow.
| The way it was inefficiently making buildings mean it started
| running at 3fps. I started over.
|
| At least twice it started with code that had an error. I'd click
| "Fix Error" and it would fail to fix it. I'd click "Fix Error",
| repeat 5-6 times. Eventually give up and start over with a
| slightly different prompt.
|
| I tried asking Gemini to write the prompt as someone suggestion.
| First use gemini to write the prompt. Then paste that into to
| Gemini Canvas mode. It didn't help nor generate anything
| noticable different than my hand written prompt.
|
| The issue I spent the most time on, it would "launch a firework"
| (which it referred to as a rocket). The "rocket" (a particle? a
| sphere? no idea) would go straight up verically (as in y +=
| velocity * deltaTime). As it did it would leave a trail of static
| exhaust particles just hanging in the air motionless. The
| firework would then explode and all of the static hanging in the
| air exhaust particles from the rocket trail would unfreeze fall a
| little and fade out.
|
| My goal was to get those particles to fall and fade as they were
| emitted, not wait for the rocket to hit it's peak. I tried 15 or
| so prompts but it utterly failed to understand the problem. It
| kept tweaking the velocity of the rocket trail particles after
| the firework explodes. It never got that them hanging static in
| the air was the issue no matter how many times or ways I
| explained the issue. Ever time it would just adjust the velocity
| of the trail particles, after the firefox exploded and make up
| nonsense about how it "solved the issue now".
|
| If I'd gotten that working I'd have asked it to stop launching
| the rockets straight vertically but give them a parabolic simple
| physics based arc.
|
| I could have done the entire thing myself in 15-30 minutes and
| got exactly what I wanted. Instead I faught with non-
| understanding LLM right out of the a Sci-Fi comedy movie.
|
| I know this will work some day and I've seen examples of it
| working. Still ... ugh for now.
___________________________________________________________________
(page generated 2025-07-04 23:00 UTC)