[HN Gopher] A prompt pattern catalog to enhance prompt engineeri...
___________________________________________________________________
A prompt pattern catalog to enhance prompt engineering with ChatGPT
Author : aliparlakci
Score : 114 points
Date : 2023-06-05 13:45 UTC (9 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| avereveard wrote:
| Couple more that I found useful:
|
| >List all the entities in the text that are ambiguous. Entities
| are ambiguous if you don't understand them, if they have multiple
| meanings and you can't decide which one from the context, or are
| phraseal terms whose meaning can be different in different
| cultures.
|
| This allows the LLM to identify parts of previous prompt that the
| AI did not fully understand.
|
| >List all the entities and relationships to other entities. Be
| precise, avoid duplicates, list each relationship only once.
|
| This allows to compress the past context for longer tasks.
|
| >Make a list of Google searches needed to build a factual answer,
| one sentence per search.
|
| You get the list then
|
| >For each sentence extract the main entity, decide whether to
| find more information about it on Google, wikipedia, ..., answer
| in json
|
| This is my "ghetto" agent, it's not iterative, but it's stable
| enough to handle the unexpected
|
| > It's the context enough to answer factually the user question?
| Answerer yes or no. Only answer yes if you are sure you can
| answer
|
| To make the agent able to loop, or ask user for more context.
| aktuel wrote:
| [flagged]
| [deleted]
| trollied wrote:
| Is it just me that thinks calling it "Engineering" is absolutely
| ridiculous?
| sebzim4500 wrote:
| Given the term 'software engineer' is now almost universally
| accepted (outside of legal jurisdictions where engineering =
| liability) I don't see why prompt engineering is a bad term.
| tobr wrote:
| Matches well with the definition of the engineering method
| provided by Bill Hammack, "the engineer guy" [1]:
|
| > Solving problems using rules of thumb that cause the best
| change in a poorly understood situation using available
| resources.
|
| 1: https://youtu.be/_ivqWN4L3zU
| simonw wrote:
| Lots of other people think that, but I disagree.
|
| Have you tried getting great, repeatable results out of an LLM?
| It requires great depth of knowledge - about both how LLMs work
| and the specific topic you are trying to build against - plus a
| methodical process in figuring out what works and what doesn't.
|
| I see no reason not to label that "engineering".
|
| I also think it's important to distinguish between prompting
| and prompt engineering. Prompting is when people type prompts
| in a box. Prompt engineering, by my own definition, is when
| developers build further software on top of LLMs.
| 2muchcoffeeman wrote:
| So is medicine "human engineering"?
|
| Then again, I'm a software engineer that doesn't consider my
| job engineering and I hate the title.
| sandkoan wrote:
| Isn't it? Wrangling a poorly understood system with partial
| knowledge and experimental skill seems like engineering to
| me.
| ttul wrote:
| Engineering is the application of science; as the science here
| is weak at best, calling it engineering seems a stretch.
| Perhaps "prompt author" would be more appropriate.
| itsuka wrote:
| How about prompt construction? I first heard about it from
| the maintainer of LangChain, and I also came across it in
| Azure docs: https://learn.microsoft.com/en-
| us/azure/cognitive-services/o...
| madelyn-goodman wrote:
| I think that there is a science here that people are working
| to discover. Prompt engineering is slowly turning into its
| own language and the more it's studied like this I think the
| more validity there will be to calling it engineering.
| browningstreet wrote:
| 1. It's already taken hold.
|
| 2. It looks great on LinkedIn.
| arroz wrote:
| It's not ridiculous
|
| it's just like the search tool engineers that would engineer
| the string to put in your google search
|
| /s
| valgaze wrote:
| I recognize there's plenty of catnip here when it comes to
| calling this "engineering" or not, however, whatever you want to
| call it (prompt fiddling?), the techniques are crucial if you
| want to achieve reasonably consistent output from current-state
| LLMs. As models improve concerns about context window limitations
| will be reduced and it will be easier to discern user intent.
|
| These are good straight-to-the-point guides:
|
| - Prompt Engineering by BrexHQ: https://github.com/brexhq/prompt-
| engineering
|
| - OpenAI guidance:
| https://help.openai.com/en/articles/6654000-best-practices-f...
|
| - https://devblogs.microsoft.com/dotnet/gpt-prompt-engineering...
|
| - (great examples): https://www.deeplearning.ai/short-
| courses/chatgpt-prompt-eng...
|
| - (~hr) Karpathy talk:
| https://www.youtube.com/watch?v=bZQun8Y4L2A
|
| tl;dr:
|
| - Begin with the best and most forgiving model available,
| optimizing it later if necessary.
|
| - Write your prompts in a clear, specific, and detailed manner
|
| - Be mindful, however, of the tradeoff between including more
| detail and increasing latency or cost.
|
| - If you're doing any complex reasoning, ask the model to show
| its work, help "stretch out" computation over more tokens
|
| - "Escape" any included or quoted text properly to avoid
| confusing the model.
|
| - Keep track of your token budget, considering the context window
| in conversations and response length.
|
| - Employ an iterative process of measuring, adjusting, and
| improving your prompt engineering techniques.
| [deleted]
| kordlessagain wrote:
| It's maybe interesting to think about a simple chat history over
| time giving _some_ of the knowledge needed for improved
| interactions as related to the more complex patterns outlined
| here.
|
| Here's a similar paper I ran across this morning:
| https://arxiv.org/abs/2305.18323. Github is here:
| https://github.com/billxbf/ReWOO
|
| I indexed both documents with my own project, which uses semantic
| graphs to help with prompt assembly:
| https://github.com/FeatureBaseDB/DoctorGPT. DoctorGPT doesn't
| have dynamic prompt chaining yet, but I'm working on it. I
| hesitate posting any of the analysis of these papers using
| DoctorGPT here because it would be generated by the LLM, and not
| me...and some people seem to have an issue with that given this
| is a human forum.
|
| My sense is that SKGs are important in refining questions,
| offering alternative approaches, managing context, reflecting on
| LLM responses, and more.
| zapperdulchen wrote:
| I really like the troubleshooting advice for the installation
| of DoctorGPT:"These instructions are long, so ensure you follow
| them carefully. It is suggested you use ChatGPT to assist you
| with any errors. Simply paste in the entire content of this
| README into ChatGPT before asking your question about the
| install process."
| [deleted]
| fudged71 wrote:
| How much overlap is there currently between LLM prompt
| engineering and (for example) academic test/assignment question
| guidelines?
| Der_Einzige wrote:
| I think you shouldn't call it "prompt engineering" until you do
| some real "engineering". For example, if you use Guidance from
| microsoft, where you write a prompt "template" and let guidance
| filter the logits probabilities at the corresponding points for
| you, I claim that this is "engineering (the way we use it as SWEs
| anyway) because it's analogous to writing an actual program.
|
| I was actually upset about everyone claiming they're a "prompt
| engineer" without any real engineering that I wrote a snarky
| github gist about it that was on the front page for a big. It's
| mainly pointing out that NLP can't even do real prompt
| engineering like Stable Diffusion folks can because we didn't
| build the right tooling for it yet.
|
| https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
| tim_sw wrote:
| While somewhat useful - there are no systematic ablation and
| comparison studies, datasets, and quantitative evals here. It's
| all anecdotal, so should probably be a blogpost and not a "paper"
| gandalfgeek wrote:
| I think of this paper as being in the same category as the GoF
| Design Patterns book.
|
| Short video summary: https://youtu.be/ueRuMDb-cPo
| troelsSteegin wrote:
| "systematic ablation" -
| https://stats.stackexchange.com/questions/380040/what-is-
| an-.... How do you ablate a black box? You work with the
| inputs. I agree that this article lacks rigor, but I don't mind
| some structured thinking about what seems to work. I don't
| think the authors over-promised here.
| dangerwill wrote:
| I can't wait until there is a standardized ChatGPT Bible to guide
| this emerging machine-priest caste. And these posts will be the
| equivalent of the dead sea scrolls.
___________________________________________________________________
(page generated 2023-06-05 23:05 UTC)