[HN Gopher] SudoLang: A Powerful Pseudocode Programming Language...
___________________________________________________________________
SudoLang: A Powerful Pseudocode Programming Language for LLMs
Author : Michelangelo11
Score : 69 points
Date : 2023-04-03 15:02 UTC (7 hours ago)
(HTM) web link (medium.com)
(TXT) w3m dump (medium.com)
| anoy8888 wrote:
| Or just use a real language you know ?
| tgv wrote:
| It's an ad, self-promotion.
| tuchsen wrote:
| This feels like the future of programming to me. When trying to
| teach kids to code, a common complaint I've gotten is how rigid
| existing programming languages are. They mistype a little thing
| and then get some general error, and it immediately turns a
| creative and fun process into something that's frustrating for
| them.
|
| After things evolve a bit. Pseudocode programming languages are
| going to be a lot of young peoples first programming language. I
| know something like this integrated into Roblox Studio would
| instantly hook my 13 year old nephew.
| goldfeld wrote:
| I wonder, for Chinese speakers or even learners, we could do AI
| prompting as needed with ChinesePython[0] even though it is not
| pseudocode but actually a running environment.
|
| 0: https://chinesememe.substack.com/i/103754530/chinesepython
| groby_b wrote:
| So... you use a formal language to create a system that ~handles
| freeform language, which you then instruct to interpret a semi-
| formal language?
|
| This might well be useful, but I'd like to see one example of
| what you can do in SudoLang that you can't do equally well in
| natural language. Especially given the token cost of the prompt.
| m3kw9 wrote:
| Fibonacci program |> transpile(JavaScript):length=very concise
| const fibonacci = n => (n <= 1 ? n : fibonacci(n - 1) +
| fibonacci(n - 2));
|
| VS
|
| A concise Fibonacci function in js, don't explain. const
| fibonacci = n => n <= 1 ? n : fibonacci(n - 1) + fibonacci(n -
| 2);
| m3kw9 wrote:
| I see the point of this language as it may give you more
| consistent responses because everyone writes different quality
| prompts, but I rather just learn to write better prompts you
| know?
| waynenilsen wrote:
| prompt is here https://github.com/paralleldrive/sudolang-llm-
| support/blob/m...
| gamegoblin wrote:
| I've just been using Python "pseudocode" with good success. It's
| great for getting structured output from GPT3. Works less well
| with chat-fine tuned models, hopefully openai releases a simple
| instruct-tuned GPT4 instead of chat-tuned.
|
| For instance, this prompt: best_artists =
| search(year=2014, genre="hip-hop")
| assert(len(best_artists) == 10) for i, artist in
| enumerate(best_artists): print(f"{i}. {artist}")
| Stdout:
|
| Yields this output: 0. Kendrick Lamar
| 1. Drake 2. J. Cole 3. Logic 4. Childish
| Gambino 5. Joey Bada$$ 6. Chance the Rapper
| 7. Schoolboy Q 8. Vince Staples 9. Run the Jewels
|
| Note how the `assert` is used to constrain the output. The
| `search` method is not defined but what it does is obvious so the
| LLM hallucinates its output.
| groby_b wrote:
| Why? What is wrong with "10 best hip hop artists of 2014, as a
| numbered list, only names"
|
| I'm not saying your approach is wrong or anything, but I am
| trying to understand the value you get from it? I get almost
| the same output, modulo different ranking in the later ranks.
| (and that'll happening anyways, ChatGPT is deliberately not
| deterministic)
| gamegoblin wrote:
| 10 hip-hop artists is just an example here to demonstrate the
| general principle. You can do much more complicated stuff
| with nested loops, conditionals, recursion, etc that are very
| verbose or difficult to express unambiguously in English.
| groby_b wrote:
| I believe that, absolutely. I just can't see any examples
| that exercise this outside of toy examples. (And I have
| severe doubt LLMs will reliably evaluate those complex
| examples by themselves. You're better off adding memory and
| agentic behavior via a formal language wrapper)
| mncharity wrote:
| Consistency is apparently one current challenge.[1]
|
| [1] https://news.ycombinator.com/item?id=35416637
| btbuildem wrote:
| This seems misguided. Or at least, I can't see the value of it.
| It just seems like a classic JS community take - shape a thing so
| it fits with the familiar ecosystem, constrain it with structure
| and types and such, and keep grinding.
|
| I think the author misses the point by a million miles: part of
| the attractiveness of LLMs is that you don't have to bother with
| a formally structured language to achieve results. GPT
| effortlessly bridges the gap between programming and human
| languages - making some structured prompt language seems..
| pointless. Almost like an artifact of a mind that finds comfort
| in constraint, at a time when we're given freedom to roam.
| inductive_magic wrote:
| Language without formalization is unreliable; in any critical
| domain, clarity is not optional. Law, maths, _code_.
|
| The second you start incorporating LLMs in your product chain,
| you need the abilities to a) efficiently instruct them without
| wasting tokens and b) interpret their outputs according to the
| structure required downstream.
|
| When you're just playing, sure, you can talk to it like a
| toddler would, not like a lawyer/mathematician/programmer. When
| you actually want to create the kind of software that GPT
| suddenly enables, you need to come up with an interface between
| it and, well, other software. I'm not saying SudoLang is the
| adequate solution to that, but I wouldn't say that it "misses
| the point by a million miles". That's quite the statement
| there.
| [deleted]
| groby_b wrote:
| "Language without formalization is unreliable; "
|
| What makes you believe LLMs will stick to the rules of your
| formalized language? It's likely. There are only
| probabilistic guarantees, which kind of obviates the benefits
| of a formal spec.
|
| If you need formalism, wrap it into a formal language for the
| formal parts. But assuming that a formal language spec prompt
| avoids probabilistic outcomes? Yes, that _is_ missing it by a
| million miles. You 're missing the strength of LLMs. (Which
| is unstructured input->semi structured output)
| dustingetz wrote:
| You're both right
|
| a picture speaks a thousand words; a formula speaks a
| thousand pictures
|
| Electromagnetism is described with formulas. And yet, once
| you have the formulas, we use natural language to give
| meaning to the constituent semantic elements in the context
| of our reality. Plug the appliance into the 120V outlet. How
| much of what we do is just organizing pre-built components?
|
| Coding has a technical debt problem, but that's caused by
| human conflict of interests / principle agent problem. We'll
| have the Maxwell's Equations of CRUD apps soon enough and the
| Tesla/Edisons with the applications will follow shortly after
| that.
| thisoneworks wrote:
| Instead of relying on one big model (and all it's flaws),
| aren't you better off having separate smaller models? Better
| for auditing
| brushfoot wrote:
| > Language without formalization is unreliable; in any
| critical domain, clarity is not optional. Law, maths, code.
|
| That's the purpose of plugins/LangChain. For deterministic
| tasks, don't use an LLM.
|
| Whatever you use for process orchestration should be able to
| make use of the LLM where it's a good fit and something else
| where it's not.
|
| > you need the abilities to a) efficiently instruct them
| without wasting tokens
|
| Plain old brevity. There's no need for semicolons and
| parentheses and the level of punctiliousness that a formal
| language imposes.
|
| > b) interpret their outputs according to the structure
| required downstream.
|
| You can include downstream structure in your prompt. Still no
| need to design a language for it. LLMs understand imperatives
| like "structure your output like this."
| brushfoot wrote:
| This feels like inventing horseshoes for cars. The author asks
| ChatGPT for its advantages over natural language interactions,
| and the ever agreeable ChatGPT comes up with this:
|
| > Firstly, SudoLang provides a more structured and consistent
| syntax than free-form natural language interactions. This can
| make it easier to understand and modify code
|
| > Secondly, SudoLang is designed specifically for interacting
| with LLMs, which means it can take advantage of their unique
| capabilities, such as generating code, solving problems, and
| answering complex questions
|
| > Finally, SudoLang includes features like modifiers and template
| strings that allow for more precise control over the responses
| and outputs generated by the LLM
|
| But none of these is an actual advantage.
|
| 1. Formal languages' restrictiveness is a _disadvantage_ with an
| LLM. It artificially limits the LLM to acting like a traditional
| GPPL. With an LLM, when you need to express something that would
| break the mold of a formal language, you can simply express it:
| There 's no need to introduce a new version of the language spec
| or search Stack Overflow.
|
| 2. "Designed specifically for" isn't an advantage if the design
| is poor. LLMs were already designed specifically for interacting
| with natural language, which already lets the user "take
| advantage of [LLMs'] unique capabilities," as ChatGPT puts it.
|
| 3. Modifiers and template strings aren't useful because, again,
| LLMs don't have the constraints of formal languages. In fact, as
| ChatGPT points out in the article, "an LLM (large language model)
| does not need to be given the specification of SudoLang in order
| to interpret SudoLang code." In other words, my naive request to
| ChatGPT to "mail merge" a set of names into some text is as valid
| as SudoLang's template strings, without having to learn their
| syntax.
|
| It's an interesting experiment, but I don't think it makes sense
| beyond that.
| startupsfail wrote:
| To me this looks like an experiment that demonstrates primarily
| : GPT4 as an assistant would happily translate user's ideas
| into a lot of text.
| garyrob wrote:
| To the OP: you're getting a lot of discouraging comments here.
| But I had the same idea as you and just didn't take the time. I
| don't know if it's actually a useful idea or not, but I really do
| feel it has potential to be useful. As I see it, it's about
| creating a high-level language that is just concise enough to
| define what you're doing, without requiring more detail than
| necessary for the logic, and that can be transpired by GPT into
| any target language one would want. Recognizing that some fixing
| of the output may be necessary.
|
| So I'm glad you're doing it and I'll keep checking it out to see
| where you go with it (if indeed you do feel like continuing).
| Good luck.
___________________________________________________________________
(page generated 2023-04-03 23:02 UTC)