[HN Gopher] Testing Generative AI for Circuit Board Design
___________________________________________________________________
Testing Generative AI for Circuit Board Design
Author : DHaldane
Score : 333 points
Date : 2024-06-21 16:16 UTC (1 days ago)
(HTM) web link (blog.jitx.com)
(TXT) w3m dump (blog.jitx.com)
| bottlepalm wrote:
| It'd be interesting to see how Sonnet 3.5 does at this. I've
| found Sonnet a step change better than Opus, and for a fraction
| of the cost. Opus for me is already far better than GPT-4. And
| same as the poster found, GPT-4o is plain worse at reasoning.
|
| Edit: Better at chain of thought, long running agentic tasks,
| following rigid directions.
| stavros wrote:
| Opus is better than GPT-4? I've heard mixed experiences.
| imperio59 wrote:
| That's because the sample size is probably small and for
| niche prompts or topics.
|
| It's very hard to evaluate whether a model is better than
| another, especially doing it in a scientifically sound way is
| time consuming and hard.
|
| This is why I find these types of comments like "model X is
| so much better than model Y" to be about as useful as
| "chocolate ice cream is so much better than vanilla"
| r2_pilot wrote:
| And both flavors have a base flavor of excrement... Still,
| since I started using Claude 3 Opus (and now 3.5 Sonnet) a
| couple of months back, I don't see myself switching from
| them nor stopping use of LLM-based AI tech; it's just made
| me feel like the computer is actually working for and with
| me and even that alone can be enough to get me motivated
| and accomplish what I set out to do.
| skapadia wrote:
| "it's just made me feel like the computer is actually
| working for and with me and even that alone can be enough
| to get me motivated and accomplish what I set out to do."
|
| This is a great way to describe what I've been feeling /
| experiencing as well.
| r2_pilot wrote:
| Just an update on my initial impressions of Claude 3.5
| Sonnet. It's a better programmer than I am in Python;
| that's not saying much, but this is now two nights in a
| row I've been impressed with what I've created with it.
| stavros wrote:
| True, I just tried it for generating a book summary, and
| Sonnet 3.5 was very bad. GPT-4o is equally bad at that ,
| gpt-4-turbo is great.
| netsec_burn wrote:
| This more likely has to do with context length?
| stavros wrote:
| No, all the information is there, but gpt-4o tends to
| produce bullet points
| (https://www.thesummarist.net/summary/the-making-of-a-
| manager...), whereas gpt-4-turbo tends to produce much
| more readable prose (https://www.thesummarist.net/summary
| /supercommunicators/the-...).
| Obscurity4340 wrote:
| How is prose more readable than bullets?
| stavros wrote:
| * Clearer narrative
|
| * Connection between points
|
| * Flows better
|
| * Eyes don't start-stop as much
| DHaldane wrote:
| It really depends on the type of question, but generally I'm
| between Gemini and Claude these days for most things.
| DHaldane wrote:
| That's an interesting question - I'll take a few pokes at it
| now to see if there's improvement.
| DHaldane wrote:
| Update: Sonnet 3.5 is better than any other model for the
| circuit design and part finding tasks. Going to iterate a bit
| on the prompts to see how much I can push the new model on
| performance.
|
| Figures that any article written on LLM limits is immediately
| out of date. I'll write an update piece to summarize new
| findings.
| CamperBob2 wrote:
| That name threw me for a loop. 'Sonnet' already means
| something to EEs ( https://www.sonnetsoftware.com/ ).
| RF_Savage wrote:
| Yeah same here. Thought Sonnet had added some ML stuff
| into their EM simulator.
| cjk2 wrote:
| Ex EE here
|
| _> The AI generated circuit was three times the cost and size of
| the design created by that expert engineer at TI. It is also
| missing many of the necessary connections._
|
| Exactly what I expected.
|
| Edit: to clarify this is even below the expectations of a junior
| EE who had a heavy weekend on the vodka.
| shrimp_emoji wrote:
| It's like a generated image with an eye missing but for
| circuits. :D
| cjk2 wrote:
| AI proceeds to use 2n3904 as a thyristor.
|
| AI happy as it worked the first 10ns of the cycle.
| jeffreygoesto wrote:
| Every natural Intelligence knows that you need to reach out
| to a 2N3055 for heavy duty. ;)
| FourierEnvy wrote:
| Why do people think inserting an LLM into the mix will make it
| better than just an evolutionary or reinforcement model
| applied? Who cares if you can talk to it like a human?
| Terr_ wrote:
| Yeah, when the author was writing about that initial query
| about delay-per-unit-length, I'm thinking: "This doesn't tell
| us whether an LLM can apply the concepts, only whether
| relevant text was included in its training data."
|
| It's a distinction I fear many people will have trouble
| keeping in-mind, faced with the misleading eloquence of LLM
| output.
| Kuinox wrote:
| I think you are looking at the term generalizing and
| memorisation. It have been shown that LLM generalize, what
| is important to know is if they generalized it or memorized
| it.
| m-hilgendorf wrote:
| imo, it's the same reason that Grace Hopper designed COBOL to
| write programs instead of math notation.
|
| What natural language processing does is just make a much
| smarter (and dumber, in many ways) parser that can make an
| attempt to infer the _intent_ , as well as be instructed how
| to recover from mistakes.
|
| Personally I'm a skeptic since I've seen some hilariously bad
| hallucinations in generated code (and unlike a human engineer
| who will say "idk but I think this might work" instead of
| "yessir this is the solution!"). If you have to double check
| every output manually it's not _that_ much better than
| learning yourself. However, at least with programming tasks,
| LLMs are fantastic at giving wrong answers with the right
| vocabulary - which makes it possible to check and find a
| solution through authoritative sources and references instead
| of blindly analyzing a problem or paying a human a lot of
| money to tell you the answer to your query.
|
| For example, I don't use LLMs to give me answers. I use them
| to help explore a design space, particularly by giving me the
| vocabulary to ask better questions. And that's the real value
| of a conversational model today.
| thechao wrote:
| I think you've nailed a subtly -- and a major doubt -- I've
| been been trying to articulate about code helpers from LLMs
| from day one: the difficulty in programming is reducing a
| natural language problem to (essentially) a proof. I
| suspect LLM's are great at transferring _style_ between two
| sentences, but I don 't think that's the same as proof
| generation! I know work is being done I this area, but the
| results I've seen have been _weird_. Maybe transferring
| _style_ won 't work for math as easily as it does for
| spoken language.
| rzzzt wrote:
| I read an article on evolutionary algorithm-based designs a
| long time ago -- they are effectively indecipherable by humans
| and rely on the imperfections of the very FPGA that they are
| synthesized on, but work great otherwise.
|
| - https://www.damninteresting.com/on-the-origin-of-circuits/
|
| -
| https://www.sciencedirect.com/science/article/abs/pii/S03784...
| dindobre wrote:
| Using neural networks to solve combinatorial or discrete problems
| is a waste of time imo, but I'd be more than happy if somebody
| could convince me of the opposite.
| utkuumur wrote:
| There are recent papers based on diffusion that perform quite
| well. Here's an example of a recent paper
| https://arxiv.org/pdf/2406.01661. I am also working on ML-based
| CO. My approach has a close 1% gap on hard instances with
| 800-1200 nodes and less than 0.1% for 200-300 nodes on Maximum
| Cut, Minimum Independent Set, and Maximum Clique problems. I
| think these are very promising times for neural network-based
| discrete optimization.
| dindobre wrote:
| Thanks, will try to give it a read this weekend. Would you
| say that diffusion is the architectural change that opened up
| CO for neural nets? Haven't followed this particular niche in
| a while
| utkuumur wrote:
| I believe it helps but not the sole reason. Because there
| are also autoregressive models that perform slightly worse.
| Unsupervised learning + Diffusion + Neural Search is the
| way to go in my opinion. However, currently, the literature
| lacks efficient Neural search space exploration. The
| diffusion process is a good starting point for neural
| search space exploration, especially when it is used not
| just to create a solution from scratch but also as a local
| search method. Still, there is no clear exploration and
| exploration control in current papers. We need to
| incorporate more ideas from heuristic search paradigms to
| neural network CO pipelines to take it to the next step.
| HanClinto wrote:
| This feels like an excellent demonstration of the limitation of
| zero-shot LLMs. It feels like the wrong way to approach this.
|
| I'm no expert in the matter, but for "holistic" things (where
| there are a lot of cross-connections and inter-dependencies) it
| feels like a diffusion-based generative structure would be
| better-suited than next-token-prediction. I've felt this way
| about poetry-generation, and I feel like it might apply in these
| sorts of cases as well.
|
| Additionally, this is a highly-specialized field. From the
| conclusion of the article:
|
| > Overall we have some promising directions. Using LLMs for
| circuit board design looks a lot like using them for other
| complex tasks. They work well for pulling concrete data out of
| human-shaped data sources, they can do slightly more difficult
| tasks if they can solve that task by writing code, but eventually
| their capabilities break down in domains too far out of the
| training distribution.
|
| > We only tested the frontier models in this work, but I predict
| similar results from the open-source Llama or Mistral models.
| Some fine tuning on netlist creation would likely make the
| generation capabilities more useful.
|
| I agree with the authors here.
|
| While it's nice to imagine that AGI would be able to generalize
| skills to work competently in domain-specific tasks, I think this
| shows very clearly that we're not there yet, and if one wants to
| use LLMs in such an area, one would need to fine-tune for it.
| Would like to see round 2 of this made using a fine-tuning
| approach.
| DHaldane wrote:
| My gut agrees with you that LLMs shouldn't do this well on a
| specialty domain.
|
| But I think there's also the bitter lesson to be learned here:
| many times people say LLMs won't do well on a task, they are
| often surprised either immediately or a few months later.
|
| Overall not sure what to expect, but fine tuning experiments
| would be interesting regardless.
| cjk2 wrote:
| I doubt it'd work any better. Most of EE time I have spent is
| swearing at stuff that looked like it'd work on paper but
| didn't due to various nuances.
|
| I have my own library of nuances but how would you even fine
| tune anything to understand the black box abstraction of an
| IC to work out if a nuance applies or not between it and a
| load or what a transmission line or edge would look like
| between the IC and the load?
|
| This is where understanding trumps generative AI instantly.
| DHaldane wrote:
| I doubt it too, but I notice that I keep underestimating
| the models.
|
| Do you have a challenge task I can try? What's the easiest
| thing I could get an LLM to do for circuit board design
| that would surprise you?
| cjk2 wrote:
| Make two separate signals arrive at exactly the same time
| on two 50 ohm transmission lines that start and end next
| to each other and go around a right hand bend. At 3.8GHz.
|
| Edit: no VSWR constraint. Can add that later :)
|
| Edit 2: oh or design a board for a simple 100Mohm input
| instrumentation amplifier which knows what a guard ring
| is and how badly the solder mask will screw it up :)
| bmicraft wrote:
| It would seem to me that the majority of boards would be
| a lot more forgiving. Are you saying you wouldn't be
| impressed if it could do only say 70% of board designs
| completely?
| AdamH12113 wrote:
| Not the GP, but as an EE I can tell you that the majority
| of boards are not forgiving. One bad connection or one
| wrong component often means the circuit just doesn't
| work. One bad footprint often means the board is
| worthless.
|
| On top of that, making an AI that can regurgitate simple
| textbook circuits and connect them together in reasonable
| ways is only the first step towards a much more difficult
| goal. More subtle problems in electronics design are all
| about context-dependent interactions between systems.
| nurple wrote:
| I hate that this is true. I think ML itself could be
| applied to the problem to help you catch mistakes in
| realtime, like language servers in software eng.
|
| I have experience building boards in Altium and found it
| rather enjoyable; my own knowledge was often a constraint
| as I started out, but once I got proficient it just
| seemed to flow out onto the canvas.
|
| There are some design considerations that would be
| awesome to farm out to genai, but I think we are far from
| that. Like stable-diffusion is to images, the source data
| for text-to-PCB would need to be well-labeled in addition
| to being correllated with the physical PCB features
| themselves.
|
| The part where I think we lose a lot of data in pursuit
| of something like this, is all of the research and
| integration work that went on behind everything that
| eventually got put into the schematic and then laid out
| on a board. I think it would be really difficult to
| "diffuse" a finished PCB from an RFQ-level description.
| cjk2 wrote:
| No because it's hard enough picking up an experienced
| human's designs and work with them. A 70% done board is a
| headache to unwrap. I'd start again.
| nurple wrote:
| This is how I am with software. There's usually a reason
| I'm arriving at 70% done, and it's not often because it's
| well designed and documented...
| DHaldane wrote:
| Right - LLMs would be a bit silly for these cases. Both
| overkill and underkill. Current approach for length
| matching is throw it off to a domain specific solver.
| Example test-circuit:
| https://x.com/DuncanHaldane/status/1803210498009342191
|
| How exact is exactly the same time? Current solver
| matches to under 10fs, and I think at that level you'd
| have to fab it to see how close you get with fiber weave
| skew and all that.
|
| Do you have a test case for a schematic design task?
| cjk2 wrote:
| Yeah. But you need $200k worth of Keysight kit to test
| it.
|
| The point is there's a methodology to solve these
| problems already. Is this better? And can it use and
| apply it?
| LeifCarrotson wrote:
| Really? Most of the time?
|
| I find I spend an enormous amount of time on boring stuff
| like connecting VCC and ground with appropriate decoupling
| caps, tying output pins from one IC to the input pins on
| the other, creating library parts from data sheets, etc.
|
| There's a handful of interesting problems in any good
| project where the abstraction breaks down and you have to
| prove your worth. But a ton of time gets spent on the
| equivalent of boilerplate code.
|
| If I could tell an AI to generate a 100x100 prototype with
| such-and-such a microcontroller, this sensor and that
| sensor with those off-board connectors, with USB power, a
| regulator, a tag-connect header, a couple debug LEDs, and
| break out unused IO to a header...that would have huge
| value to my workflow, even if it gave up on anything analog
| or high-speed. Presumably you'd just take the first pass
| schematic/board file from the AI and begin work on anything
| with nuance.
|
| If generative AI can do equivalent work for PCBs as it can
| do for text programming languages, people wouldn't use it
| for transmission line design. They'd use it for the
| equivalent of parsing some JSON or making a new class with
| some imports, fields, and method templates.
| scld wrote:
| "Looks like you forgot pullups on your i2c lines" would
| be worth a big monthly subscription hahaha.
| oscillonoscope wrote:
| There are schematic analysis tools which do that now just
| based on the netlist
| makapuf wrote:
| This totally didnt happen to me _again_ recently. But
| next time I surely won 't forget those. (Cue to a few
| months from now...)
| DHaldane wrote:
| I've found that for speeding up design generation like
| that, most of the utility comes from the coding approach.
|
| AI can't do it itself (yet), and having it call the
| higher level functions doesn't save that much time...
| sweezyjeezy wrote:
| Some research to the contrary [1] - tldr is that they didn't
| find evidence that generative models really do zero shot well
| at all yet, if you show it something it literally hasn't seen
| before, it isn't "generally intelligent" enough to do it
| well. This isn't an issue for a lot of use-cases, but does
| seem to add some weight to the "giga-scale memorization"
| hypothesis.
|
| [1] https://arxiv.org/html/2404.04125v2
| HanClinto wrote:
| > But I think there's also the bitter lesson to be learned
| here: many times people say LLMs won't do well on a task,
| they are often surprised either immediately or a few months
| later.
|
| Heh. This is very true. I think perhaps the thing I'm most
| amazed by is that simple next-token prediction seems to work
| unreasonably well for a great many tasks.
|
| I just don't know how well that will scale into more complex
| tasks. With simple next-token prediction there is little
| mechanism for the model to iterate or to revise or refine as
| it goes.
|
| There have been some experiments with things like speculative
| generation (where multiple branches are evaluated in
| parallel) to give a bit of a lookahead effect and help avoid
| the LLM locking itself into dead-ends, but they don't seem
| super popular overall -- people just prefer to increase the
| power and accuracy of the base model and keep chugging
| forward.
|
| I can't help feeling like a fundamental shift something more
| akin to a diffusion-based approach would be helpful for such
| things. I just want some sort of mechanism where the model
| can "think" longer about harder problems. If you present a
| simple chess board to an LLM or a complex board to an LLM and
| ask it to generate the next move, it always responds in the
| same amount of time. That alone should tell us that LLMs are
| not intelligent, and they are not "thinking", and they will
| be insufficient for this going forward.
|
| I believe Yann LeCun is right -- simply scaling LLMs is not
| going to get us to AGI. We need a fundamental structural
| shift to something new, but until we stop seeing such insane
| advancements in the quality of generation with LLMs (looking
| at you, Claude!!), I don't think we will move beyond. We have
| to get bored with LLMs first.
| pton_xd wrote:
| > If you present a simple chess board to an LLM or a
| complex board to an LLM and ask it to generate the next
| move, it always responds in the same amount of time.
|
| Is that true, especially if you ask it to think step-by-
| step?
|
| I would think the model has certain associations for
| simple/common board states and different ones for
| complex/uncommon states, and when you ask it to think step-
| by-step it will explain the associations with a particular
| state. That "chattiness" may lead it to using more
| computation for complex boards.
| HanClinto wrote:
| > > If you present a simple chess board to an LLM or a
| complex board to an LLM and ask it to generate the next
| move, it always responds in the same amount of time.
|
| > Is that true, especially if you ask it to think step-
| by-step?
|
| That's fair -- there's a lot of room to grow in this
| area.
|
| If the LLM has been trained to operate with running
| internal-monologue, then I believe they will operate
| better. I think this definitely needs to be explored more
| -- from what little I understand of this research, the
| results are sporadically promising, but getting something
| like ReAct (or other, similar structures) to work
| consistently is something I don't think I've seen yet.
| visarga wrote:
| > I just want some sort of mechanism where the model can
| "think" longer about harder problems.
|
| There is such a mechanism - multiple rounds of prompting.
| You can implement diverse patterns (chains, networks) of
| prompts.
| anonymoushn wrote:
| We have 0 y/o/y progress on Advent of Code, for example.
| Maybe we'll have some progress 6 months from now :)
| https://www.themotte.org/post/797/chatgpt-vs-advent-of-code
| DHaldane wrote:
| Have you tried using more 4000x more samples?
|
| https://redwoodresearch.substack.com/p/getting-50-sota-on-
| ar...
| eimrine wrote:
| I like how you called it holistic, it is maybe the first time I
| see this word not in a "bad" context.
|
| What about the topic, it is impossible to synthesize STEM
| things not in the manner an engineer does this. I mean thou
| shalt to know some typical solutions and have all the
| calculations for all what's happening in the schematic being
| developed.
|
| Textbooks are not a joke and no matter who are you - a human or
| a device.
| omgJustTest wrote:
| I asked this question of Duncan Dec 22!
|
| If you are interested I highly recommend this + your favorite
| llm. It does not do everything but is far superior to some
| highly expensive tools, in flexibility and repeatability.
| https://github.com/devbisme/skidl
| HanClinto wrote:
| This tool looks really powerful, thanks for the link!
|
| One thing I've been personally really intrigued by is the
| possibility of using self-play and adversarial learning as a
| way to advance beyond our current stage of imitation-only
| LLMs.
|
| Having a strong rules-based framework to be able to be able
| to measure quality and correctness of solutions is necessary
| for any RL training setup to proceed. I think that skidl
| could be a really nice framework to be part of an RL-trained
| LLM's curriculum!
|
| I've written down a bunch of thoughts [1] on using games or
| code-generation in an adversarial training setup, but I could
| see circuit design being a good training ground as well!
|
| * [1] https://github.com/HanClinto/MENTAT
| hoosieree wrote:
| I agree diffusion makes more sense for optimizing code-like
| things. The tricky part is coming up with a reasonable set of
| "add noise" transformations.
| HanClinto wrote:
| > The tricky part is coming up with a reasonable set of "add
| noise" transformations.
|
| Yes, as well as dealing with a variable-length window.
|
| When generating images with diffusion, one specifies the
| image ahead-of-time. When generating text with diffusion,
| it's a bit more open-ended. How long do we want this
| paragraph to go? Well, that depends on what goes into it --
| so how do we adjust for that? Do we use a hierarchical tree-
| structure approach? Chunk it and do a chain of overlapping
| segments that are all of fixed-length (could possibly be
| combined with a transformer model)?
|
| Hard to say what would finally work in the end, but I think
| this is the sort of thing that YLC is talking about when he
| encourages students to look beyond LLMs. [1]
|
| * [1] https://x.com/ylecun/status/1793326904692428907
| surfingdino wrote:
| > This feels like an excellent demonstration of the limitation
| of zero-shot LLMs. It feels like the wrong way to approach
| this.
|
| There is one posted on HN every week. How many more do we need
| to accept the fact this tech is not what it is sold at and we
| are bored waiting for it get good? I am not say "get better",
| because it keeps getting better, but somehow doesn't get good.
| makk wrote:
| That's a perception and the problem isn't the AI it's human
| nature: 1. every time AI is able to do a thing we move the
| goalposts and say, yeah, but it can't do that other thing
| over there; 2. We are impatient, so our ability to get bored
| tends to outpace the rate of change.
| slg wrote:
| I don't think the problem is moving the goalposts, but
| rather there are no actual goalposts. Advocates for this
| technology imply it can do anything either because they
| believe it will be true in the near future or they just
| want others to believe it for a wide range of reasons
| including to get rich of it. Therefore the general public
| has no real idea what the ideal use cases are for this
| technology in its current state so they keep asking it to
| do stuff it can't do well. It is really no different than
| the blockchain in that regard.
| surfingdino wrote:
| One of the main issues I see amongst advocates of AI is
| that they cannot quantify the benefits and ignore
| provable failings of AI.
| goatlover wrote:
| The other side of this coin is everyone overhyping what AI
| can do, and when the inevitable criticism comes, they
| respond by claiming the goal posts are being moved.
| Perhaps, but you also told me it could do XYZ, when it can
| only do X and some Y, but not much Z, and it's still not
| general intelligence in the he broad sense.
| refulgentis wrote:
| I appreciate this comment because I think it really
| demonstrates the core problem with what I'll call the
| "get off my lawn >:|" argument, because it's avowedly
| about personal emotions.
|
| It's not "general intelligence", so it's over hyped, and
| They get so whiny about the inevitable criticism, and
| They are ignoring that it's so mindnumbingly boring to
| have people making the excuse that "designed a circuit
| board from scratch" wasn't something anyone thinks or
| claims an LLM should do.
|
| Who told you LLMs can design circuit boards?
|
| Who told you LLMs are [artificial] general intelligence?
|
| I get sick of it constantly being everywhere, but I don't
| feel the need to intellectualize it in a way that blames
| the nefarious ???
| ben_w wrote:
| > Who told you LLMs are [artificial] general
| intelligence?
|
| *waves*
|
| Everyone means a different thing by each letter of AGI,
| and sometimes also by the combination.
|
| I know my opinion is an unpopular one, but given how much
| more general-purpose they are than most other AI, I count
| LLMs as "general" AI; and I'm old enough to remember when
| AI didn't automatically mean "expert level or better",
| when it was a _surprise_ that Kasparov was beaten (let
| alone Lee Sedol).
|
| LLMs are (currently) the ultimate form of "Jack of all
| trades, master of none".
|
| I'm not surprised that it failed with these tests, even
| though it clearly knows more about electronics than me.
| (I once tried to buy a 220 k resistor, didn't have the
| skill to notice the shop had given me a 220 O resistor,
| the resistor caught fire).
|
| I'd still _like_ to call these things "AGI"... except
| for the fact that people don't agree on what the word
| means and keep objecting to my usage of the initials as
| is, so it would't really communicate anything for me to
| do so.
| derefr wrote:
| ML scientists will tell you it can do X and some Y but
| not much Z. But the public doesn't listen to ML
| scientists. Most of what the public hears about AI comes
| from businessmen trying to market a vision to investors
| -- a vision, specifically, of what _their business_ will
| be capable of _five years from bow_ given _predicted
| advancements in AI capabilities in the mean time_ ; which
| has roughly nothing to do with what current models can
| do.
| 127 wrote:
| What goals were achieved that I missed? Even for creative
| writing and image creation it still requires significant
| human guidance and correction.
| selestify wrote:
| This is a great example of goalposts shifting. Even
| having a model that can engage in coherent conversation
| and synthesize new information on the fly is
| revolutionary compared to just a few years ago. Now the
| bar has moved up to creativity without human
| intervention.
| doe_eyes wrote:
| But isn't this goalpost shifting actually _reasonable_?
|
| We discovered this nearly-magical technology. But now the
| novelty is wearing off, and the question is no longer
| "how awesome is this?". It's "what can I do with it for
| today?".
|
| And frustratingly, the apparent list of uses is
| shrinking, mostly because many serious applications come
| with a footnote of "yeah, it can do that, but unreliably
| and with failure modes that are hard for most users to
| spot and correct".
|
| So yes, adding "...but without making up dangerous
| nonsense" is moving the goalposts, but is it wrong?
| kenjackson wrote:
| There are a lot of things where being reliable isn't as
| important (or it's easier to be reliable).
|
| For example, we are using it to do meeting summaries and
| it is remarkably good at it. In fact, in comparison to
| humans we did A/B testing with - usually better.
|
| Another thing is new employee ramp. It is able to answer
| questions and guide new employees much faster than we've
| ever seen before.
|
| Another thing I've started toying with it with, but have
| gotten incredible results so far is email prioritization.
| Basically letting me know which emails I should read most
| urgently.
|
| Again, these were all things where the state of the art
| was basically useless 3 years ago.
| selestify wrote:
| IMO it's not wrong to want the next improvement ("...but
| without making up dangerous nonsense"), but it _is_
| disingenuous to pretend as if there hasn't already been a
| huge leap in capabilities. It's like being unimpressed
| with the Wright brothers' flight because nobody has
| figured out commercial air travel yet.
| surfingdino wrote:
| The leap has indeed been huge, but it's still not useful
| for any anything. The Wright brothers did not start a
| passenger airline after the first try.
| 127 wrote:
| No it's not. You can not shift goalposts that do not
| exist in the first place.
| XorNot wrote:
| No I'd say it's that people are very bad at knowing what
| they want, and worse at knowing how to get it.
|
| While it might be "moving the goal posts" the issue is that
| the goal posts were arbitrary to start with. In the context
| of the metaphor we put them on the field so there could be
| a game, despite the outcome literally not mattering
| anywhere else.
|
| This isn't limited to AI: anyone dealing with customers
| knows that the worst thing you can do is take what the
| customer says their problem is at face value, replete with
| the proposed solution. What the customer knows is they
| _have_ a problem, but it 's very unlikely they want the
| solution they think they do.
| jodrellblank wrote:
| > " _1. every time AI is able to do a thing we move the
| goalposts and say, yeah, but it can't do that other thing
| over there_ "
|
| So are you happy that a 1940s tic-tac-toe computer "is AI"?
| And that's going to be your bar for AI _forever_?
|
| " _Moving the goalposts is a metaphor, derived from goal-
| based sports such as football and hockey, that means to
| change the rule or criterion of a process or competition
| while it is still in progress, in such a way that the new
| goal offers one side an advantage or disadvantage._ " - and
| the important part about AI is that it be easy for
| developers to claim they have created AI, and if we move
| the goalposts then that's bad because ... it puts them at
| an unfair disadvantage? What is even wrong with "moving the
| goalposts" in this situation, claiming something is/isn't
| AI is not a goal-based sport. The metaphor is nonsensical
| whining.
| exe34 wrote:
| how long does it take for a child to start doing surgery?
| publishing novel theorems? how long has the humble
| transformer been around?
| ben_w wrote:
| Wall-clock or subjective time?
|
| I think it would take a human about 2.6 million (waking)
| years to actually _read_ Common Crawl[0]; though obviously
| faster if they simply absorb token streams as direct
| sensory input.
|
| The strength of computers is that transistors are
| (literally) faster than synapses to the degree to which
| marathon runners are faster than continental drift; the
| weakness is they need to, too -- current generation AI is
| only able to be this good due to this advantage allowing it
| to read far more than any human.
|
| How much this difference matters depends on the use-case:
| if AI were as good at learning as we are, Tesla's FSD would
| be level 5 autonomy years ago already, even with just
| optical input.
|
| [0] April 2024: 386 TiB; assuming 9.83 bits per word and
| 250 w.p.m: https://www.wolframalpha.com/input?i=386+TiB+%2F
| +9.83+bits+p...
| TeMPOraL wrote:
| Subjective time doesn't really matter unless something is
| experiencing it. It could be 2.6 million years, but if
| the wall-clock time is half a year, then great - we've
| managed to brute-force some degree of intelligence in
| half a year! And we're at the beginning of this journey;
| there surely are many things to optimize that will
| decrease both wall-clock and subjective training time.
|
| As the saying goes - "make it work, make it right, make
| it fast".
| surfingdino wrote:
| Nobody is telling an experienced heart surgeon to step
| aside and let a child plan an open heart surgery. And yet,
| AI and LLMs in particular are being sold as the tools that
| can do complex tasks like that. But let's leave complex
| tasks and have a look at marketing behind one of the tools
| that's aimed at business. The messaging of one of the ads
| I'm seeing promises that the tools in question can
| summarise a 150-page long document into a 5-slide
| presentation. Now, that sounds amazing, if we ignore the
| fact that a person who wrote a 150-page document has
| already prepared an outline and is perfectly capable of
| summarising each section of the document. Writing a
| 150-page document without a plan and not being able to
| organise would mean that people have evolved into content
| generators that need machines to help them write tables of
| contents and reformat them into a presentation. Coming back
| to your child analogy, why would a child be better at
| summarising content it knows nothing about that the person
| who wrote it?
| exe34 wrote:
| we do get consultants coming into companies and telling
| the experienced professionals how to screw up stuff all
| the time though. i think there are laws with teeth and of
| course the immediate body to get rid of that helps
| surgeons maintain the integrity of their profession. when
| the outcome is far removed from the decision, you do get
| people like ministers meddling in things they don't
| understand and leave the consequences for the next
| administration.
| echelon wrote:
| I'm in awe of the progress in AI images, music, and video.
| This is probably where AI shines the most.
|
| Soon everything you see and hear will be built up through a
| myriad of AI models and pipelines.
| slg wrote:
| > Soon everything you see and hear will be built up through
| a myriad of AI models and pipelines.
|
| It is so bizarre that some people view this as a positive
| outcome.
| echelon wrote:
| These are tools. Humans driving the tools have heart and
| soul and create things of value through their lens.
|
| Your argument rhymes with:
|
| - "Let's keep using horses. They're good enough."
|
| - "Photography lacks the artistic merit of portrait art."
|
| - "Electronic music isn't music."
|
| - "Vinyl is the only way to listen to music."
|
| - "Digital photography ruins photography."
|
| - "Digital illustration isn't real illustration and
| tablets are cheating."
|
| - "Video games aren't art."
|
| - "Javascript developers aren't real programmers."
|
| Though I'm paraphrasing, these are all things that have
| been said.
|
| I bet you my right kidney that people will use AI to
| produce incredible art that will one day (soon) garner
| widespread praise and accolade.
|
| It's just a tool.
| slg wrote:
| The specific phrase used was " _everything_ you see and
| hear " (emphasis mine). You weren't arguing this would be
| an optional tool that could be used in the creation of
| art. You were arguing that this will replace all other
| art. That isn't an argument that photography is an art
| equal to painting, it is an argument for it to replace
| painting.
| echelon wrote:
| > You were arguing that this will replace all other art.
|
| The population of people who want to create art is higher
| than the people who have the classical skills. By sheer
| volume, the former will dominate the latter. And
| eventually most artists will begin to use AI tools when
| they realize that's what they are -- tools.
| slg wrote:
| Now combine that with the photography and painting
| analogy that you made in the previous post. Photography
| was invented some 2 centuries ago. Do you think the world
| would be better if every painter of that era, including
| the likes of van Gogh and Picasso, picked up a camera
| instead of a paintbrush?
| vunderba wrote:
| Just to play devil's advocate - I'm surprised you (and
| many other people apparently) are unable to tell the
| operative difference between something like:
|
| 1. (real illustration vs digital illustration)
|
| 2. (composing on sheet music vs composing in a DAW)
|
| and
|
| 3. illustration vs Stable Diffusion
|
| 4. composing vs generative music models such as Suno
|
| What's different is the wide disparity between _input_
| and _output_. Generally, art has traditionally had a
| closer connection between the "creator" and the
| "creation". Generative models have married two
| conventionally highly disparate mediums together, e.g.
| text to image / text to audio.
|
| If you have zero artistic ability, you'd have about as
| much success using Photoshop as you would with
| traditional pencil and paper.
|
| Whereas any doofus can type in the description of
| something along with words like "3D", "trending on
| artstation", "hyper-realistic,", and "4K" and then
| proceed to generate thousands of images in automatic1111
| which they can flood DeviantArt with in a single day.
|
| The same applies to music composition whether you are
| laboriously notating with sheet music or dropping notes
| using a horizontal tracker in a DAW like Logic. If you're
| not a musician, the fanciest DAW in the world won't make
| you one.
| echelon wrote:
| I don't think you realize the sheer scale of people that
| are working their asses off to leverage AI in their work
| in creative ways, often times bending over backwards to
| get it to work.
|
| I spent 48 hours two weeks back (with only a few hours of
| sleep) making an AI film. I used motion capture,
| rotoscoping, and a whole host of other tools to
| accomplish this.
|
| I know people who have spent months making AI music
| videos. People who painstakingly mask and pose skeletons.
| People who design and comp shots between multiple
| workflows.
|
| These are tools.
| ziml77 wrote:
| Surely there's some point where it ceases being a tool
| though. We can't both be making AIs out to be comparable
| to humans while simultaneously calling them tools.
| Otherwise people who commission art would be considered
| artists using a tool.
| vasco wrote:
| Many many successful artists from the Renaissance until
| today are not actually artists but just rich people with
| a workshop full of actual artist they commission works
| from. The rich person curates.
|
| Many times this also happens with artists themselves.
| After a point, you are getting way more commissions than
| you can produce yourself, so you employ a small army of
| understudies that learn your techniques and make your
| pieces for you. So what you describe has existed for
| hundreds of years.
|
| A short list could include old ones like Rembrandt or
| Rubens and a new ones like Jeff Koons or Damien Hirst.
| zo1 wrote:
| What I find bizarre is people gatekeeping the process
| that helps get things from imagination onto canvas.
|
| Artists and "creative" people have long held a monopoly
| on this ability and are now finally paying the price now
| that we've automated them away and made their "valuable"
| skill a commodity.
| happypumpkin wrote:
| > Artists and "creative" people have long held a monopoly
| on this ability and are now finally paying the price
|
| I've seen a lot of schadenfreude towards artists
| recently, as if they're somehow gatekeeping art and
| stopping the rest of us from practicing it.
|
| I really struggle to understand it; the barrier of entry
| to art is basically just buying a paper and pencil and
| making time to practice. For most people the practice
| time could be spent on many things which would have
| better economic outcomes.
|
| > monopoly
|
| Doesn't this term imply an _absence_ of competition?
| There seems to be _a lot_ of competition. Anyone can be
| an artist, and anyone can attempt to make a living doing
| art. There is no certification, no educational
| requirements. I 'm sure proximity to wealth is helpful
| but this is true of approximately every career or hobby.
|
| Tangentially, there seem to be positive social benefits
| to everyone having different skills and depending on
| other people to get things done. It makes me feel good
| when people call me up asking for help with something I'm
| good at. I'm sure it feels the same for the neighborhood
| handyman when they fix someone's sink, the artist when
| they make profile pics for their friends, etc. I could be
| wrong but I don't think it'll be entirely good for people
| when they can just have an AI or a robot do everything
| for them.
| goatlover wrote:
| I sincerely hope not. Talk about a dystopian future. That's
| even worse than what social media has become.
| wewtyflakes wrote:
| Why would that be describing a dystopian future? A more
| generous framing might be to say that incredibly creative
| feats will be available to more people, and those who are
| particularly talented will create things that are now
| beyond our imagination using these tools. Who knows if
| that is how it will actually play out, but it also does
| not seem unreasonable to think that it might.
| ben_w wrote:
| They already are, when using the meaning of "AI" that I
| grew up with.
|
| The Facebook feed is AI; Google PageRank is AI; anti-spam
| filters are AI; A/B testing is AI; recommendation systems
| are AI.
|
| It's been a long time since computers took over from humans
| with designing transistor layouts in CPUs -- I was hearing
| about the software needing to account for quantum mechanics
| nearly a decade ago already.
| refulgentis wrote:
| There's this odd strain of thought that there's some general
| thing that will pop for hucksters and the unwashed masses,
| who are sheep led along by huckster wolves who won't admit
| LLMs aint ???, because they're profiting off it
|
| It's frustrating because it's infantalizing, it derails the
| potential of an interesting technical discussion (ex. Here,
| diffusion), and it misses the mark substantially.
|
| At the end of the day, it's useful in a thousand ways day to
| day, and the vast majority of people feel this way. The only
| people I see vehemently arguing the opposite seem to assume
| only things with 0 error rate are useful or are upset about
| money in some form.
|
| But is that really it? I'm all ears. I'm on a 5 hour flight.
| I'm genuinely unclear on whats going on that leads people to
| take this absolutist position that they're waiting for ??? to
| admit ??? about LLMs.
|
| Yes, the prose machine didnt nail circuit design, that
| doesn't mean whatever They you're imagining needs to give up
| and accept ???
| ben_w wrote:
| > But is that really it? I'm all ears. I'm on a 5 hour
| flight. I'm genuinely unclear on whats going on that leads
| people to take this absolutist position that they're
| waiting for ??? to admit ??? about LLMs.
|
| Irony: humans think in very black-and-white terms, one
| could even say boolean; conversely LLMs display subtly and
| nuance.
|
| When I was a kid, repeats of Trek had Spock and Kirk
| defeating robots with the liar's paradox, yet today it
| seems like humans are the ones who are broken by it while
| the machines are just going "I understood that reference!"
| refulgentis wrote:
| Excellent point, it really is what it comes down to.
| There's people getting hoodwinked and people hoodwinking
| and then me, the one who sees them for what they are.
| goatlover wrote:
| And yet we still don't have Data or the Holographic
| Doctor.
| ben_w wrote:
| You're demonstrating my point :)
|
| When we get to that level, we're all out of work.
|
| In the meantime, LLMs are already basically as good as
| the scriptwriters made the TNG-VOY era starship computers
| act.
| shermantanktop wrote:
| So what should we make of the presence of actual hucksters
| and actual senior execs who are acting like credulous
| sheep? I see this every day in my world.
|
| At the same time I do appreciate the actual performance and
| potential future promise of this tech. I have to remind
| myself that the wolf and sheep show is a side attraction,
| but for some people it's clearly the main attraction.
| wruza wrote:
| Why should we even?
|
| The problem with everything today is not only that it's
| hype-centric, but that that carries away those who were
| otherwise reasonable. AI isn't any special in this
| regard, it's just "crypto" of this decade.
|
| I see this trend everywhere, in tech, socio, markets.
| Everything is way too fake, screamy and blown out of
| proportion.
| refulgentis wrote:
| The wolves/sheep thing was to indicate how moralizing and
| infantalizing serves as a substitute for actually
| explaining what the problem is, because surely, it's not
| that the prose machine isn't doing circuit design.
|
| I'm sure you see it, I'd just love for someone to pause
| their internal passion play long enough to explain what
| they're seeing. Because I refuse to infantalize, I refuse
| to believe it's just grumbling because its not 100%
| accurate 100% of the time, and doesn't do 100% of
| everything.
| shermantanktop wrote:
| I am literally right now explaining to a senior exec why
| some PR hype numbers about developer productivity from
| genAI are not comparable to internal numbers, because he
| is hoping to say to his bosses that we're doing better
| than others. This is a smart, accomplished person, but he
| can read the tea leaves.
|
| The problem with hype is that it can become a
| pathological form of social proof.
| anoncareer0212 wrote:
| I see, I'm sorry that's happening :/ I was lucky enough
| to transition from college dropout waiter to tech startup
| on the back of the iPad, 6 years in, sold it and ended up
| at still-good 2016 Google. Left in 2023 because of some
| absolutely mindnumbingly banal-ly evil middle management.
| I'm honestly worried about myself because I cannot.
| stand. that. crap., Google was relatively okay _, and
| doubt I could ever work for someone else again. it was s
| t u n n i n g to see how easily people slip into
| confirmation bias when it involves pay / looking good.
|
| _ fwiw if someone's really into Google minutae: I'm not
| so sure it is relatively okay anymore, it's kinda freaky
| how many posts there are on Blind along the lines of "wow
| I left X for here, assumed i'd at least be _okay_ , but I
| am _deeply_ unhappy. its much worse than average-white-
| collar job I left "
| selestify wrote:
| Are there any write ups of the newly evil Google
| experience I can read about? When did things shift for
| you in the 2016 - 2023 timeframe?
| refulgentis wrote:
| No, my way of dealing with it is to whine on HN/twitter
| occasionally and otherwise don't say anything publicly.
| Feel free to reach out at jpohhhh@gmail, excuse the
| overly familiar invitation, paying it forward because I
| would have found talking about that sort of thing f a s c
| i n a t i n g.
|
| in general id recommend Ian Hickson's blog post on
| leaving. I can't remember the exact quote that hit hard,
| something like decisions moved from being X to Y to Z to
| being for peoples own benefit.
|
| I'd also add there was some odd corrupting effects from
| CS turning into something an aimless Ivy Leaguer would do
| if they didn't feel like finance.
| photonthug wrote:
| I'll play along. The thing that's annoying me lately is
| that session details leaking between chats has been
| enabled as a "feature", which is quickly making ChatGPT
| more like the search engine and social media echo
| chambers that I think lots of us want to escape. It's
| also harmful for the already slim chances of having
| reproducible / deterministic results, which is bad since
| we're using these things for code generation as well as
| rewriting emails and essays or whatever.
|
| Why? Is this naive engineering refusing to acknowledge
| the same old design flaws? Nefarious management fast
| tracking enshittification? Or do users actually want
| their write-a-naughty-limerick goofs to get mixed up with
| their serious effort to fast track circuit design? I
| wouldn't want to appear cynical but one of these
| explanations just makes more sense than the others!
|
| The core tech such as it is is fine, great even. But it's
| not hard to see many different ways that it's already
| spiraling out of control.
| refulgentis wrote:
| (thank you!) 100% cosign. It breaks my. goddamn. heart.
| that [REDACTED], the consummate boring boneheaded SV
| lackey is [REDACTED] of [REDACTED], and can't think
| outside 6 week sprints and never finishes launching. This
| is technology that should be _freeing_ us from random
| opaque algorithmic oppression and _enabling_ us to take
| charge if we want. I left Google to do the opposite, and
| I 'm honestly stunned that it's a year later and there's
| nothing on the market that challenges that. Buncha me-too
| nonsense doing all the shit I hate from the 2010s: bulk
| up on cash, buy users, do the recurring revenue thing and
| hope x > y, which inevitably, it won't be.
| Kiro wrote:
| This post supports your case way less than you think. I've
| sent it to several EE friends and none have expressed your
| discontent. The general consensus has been "amazing what AI
| can do nowadays", and I agree. This would have been complete
| science-fiction just a couple of years ago.
| yousif_123123 wrote:
| One downside for diffusion based systems (and I'm very noob in
| this) is that the model won't be able to see it's input and
| output in the same space, therefore wouldn't be able to do
| follow-up instructions to fix things or improve on it. Where as
| an LLM generating html could follow instructions to modify it
| as well. It's input and output are the same format.
| HanClinto wrote:
| Oh? I would think that the input prompt to drive generation
| is not lost during generation iterations -- but I also don't
| know much about the architectural details.
| guidoism wrote:
| This reminds me of my professor's (probably very poor)
| description of NP-complete problems where the computer would
| provide an answer that may or may not be correct and you just had
| to check that it was correct and you do test for correctness in
| polynomial time.
|
| It kind of grosses me out that we are entering a world where
| programming could be just testing (to me) random permutations of
| programs for correctness.
| moffkalast wrote:
| Well we had to keep increasing inefficiency somehow, right?
| Otherwise how would Wirth's law continue to hold?
| thechao wrote:
| Most of the HW engineers I work with consider the webstack to
| be far more efficient than the HW-synthesis stack; ie,
| there's more room for improvement in HW implementation than
| in SW optimization.
| cushychicken wrote:
| I'm terrified that JITX will get into the LLM / Generative AI for
| boards business. (Don't make me homeless, Duncan!)
|
| They are already far ahead of many others with respect to next
| generation EE CAD.
|
| Judicious application of AI would be a big win for them.
|
| Edit: adding "TL;DRN'T" to my vocabulary XD
| DHaldane wrote:
| I promise that we want to stay a software company that helps
| people design things!
|
| Adding Skynetn't to company charter...
| AdamH12113 wrote:
| The conclusions are very optimistic given the results. The LLMs:
|
| * Failed to properly understand and respond to the requirements
| for component selection, which were already pretty generic.
|
| * Succeeded in parsing the pinout for an IC but produced an
| incomplete footprint with incorrect dimensions.
|
| * Added extra components to a parsed reference schematic.
|
| * Produced very basic errors in a description of filter
| topologies and chose the wrong one given the requirements.
|
| * Generated utterly broken schematics for several simple
| circuits, with missing connections and aggressively-incorrect
| placement of decoupling capacitors.
|
| Any one of these failures, individually, would break the entire
| design. The article's conclusion for this section buries the lede
| slightly:
|
| > The AI generated circuit was three times the cost and size of
| the design created by that expert engineer at TI. It is also
| missing many of the necessary connections.
|
| Cost and size are irrelevant if the design doesn't work. LLMs
| aren't a third as good as a human at this task, they just fail.
|
| The LLMs do much better converting high-level requirements into
| (very) high-level source code. This make sense (it's
| fundamentally a language task), but also isn't very useful.
| Turning "I need an inverting amplifier with a gain of 20" into
| "amp = inverting_amplifier('amp1', gain=-20.0)" is pretty
| trivial.
|
| The fact that LLMs apparently perform better if you literally
| offer them a cookie is, uh... something.
| lemonlime0x3C33 wrote:
| thank you for summarizing the results, I feel much better about
| my job security. Now if AI could make a competent auto router
| for fine pitch BGA components that would be really nice :)
| neltnerb wrote:
| I think the only bit that looked handy in there would be if it
| could parse PDF datasheets and help you sort them by some
| hidden parameter. If I give it 100 datasheets for microphones
| it really should be able to sort them by mechanical height.
| Maybe I'm too optimistic.
|
| The number of times I've had to entirely redo a circuit because
| of one misplaced connection, yeah, none of those circuits
| worked for any price before I fixed every single error.
| DHaldane wrote:
| Agree that PDF digesting was the most useful.
|
| I think Gemini could definitely do that microphone study.
| Good test case! I remember spending 8 hours on DigiKey in the
| bad old times, looking for an audio jack that was 0.5mm
| shorter.
| hadlock wrote:
| As I understand it, PDF digestion/manipulation (and
| particularly translation) has long been a top request from
| businesses, based on conversations I've had with people
| selling the technology, so it doesn't surprise me that
| Gemini excels at this task.
| robxorb wrote:
| Anyone looking for an idea for something potentially
| valuable to make: ingest PDF datasheets and let us
| search/compare etc, across them. The PDF datasheet is
| possibly one of the biggest and most unecessary hurdles to
| electronics design efficiency.
| neltnerb wrote:
| Hah, you're not kidding. Literally my comment was inspired
| by a recent realization that it is not possible to search
| for a RF connector by footprint size.
|
| That's absurd to me, it took so long to figure out which
| random sequence of letters was the smallest in overall PCB
| footprint.
|
| Maybe we found it, we think it's the AYU2T-1B-GA-GA-ETY(HF)
| but sure would be nice if Digikey had a search by footprint
| dimensions.
|
| Yet strangely the physical ability of a device to fit into
| a location you need it is not in the list of things I can
| search. Takes ten seconds to find the numbers -- after I
| download and open the PDF file.
|
| https://www.digikey.com/en/products/filter/coaxial-
| connector...
|
| Just so strange, but so common. And digikey is heads and
| shoulders above average, McMaster might be the only better
| one I know of at it and they're very curated.
| doe_eyes wrote:
| Yes, this seemed pretty striking to me: the author clearly
| _wanted_ the LLM to perform well. They started with a problem
| for which solutions are pretty much readily available on the
| internet, and then provided a pretty favorable take on the
| model 's mistakes.
|
| But the bottom line is that it's a task that a novice could
| have solved with a Google search or two, and the LLM fumbled it
| in ways that'd be difficult for a non-expert to spot and
| rectify. LLMs are generally pretty good at information
| retrieval, so it's quite disappointing.
|
| The cookie thing... well, they learn statistical patterns.
| People on the internet often try harder if there is a quid-pro-
| quo, so the LLMs copy that, and it slips past RLHF because
| "performs as well with or without a cookie" is probably not one
| of the things they optimize for.
| oscillonoscope wrote:
| I don't know enough about LLMs to understand if its feasible or
| not but it seems like it would be useful to make certain tasks
| hard-coded or add some fundamental constraints on it. Like when
| making footprints, it should always check that the number of
| pads is never less than the number of schematic symbol pins.
| Otherwise, the AI just feels like your worst coworker
| sehugg wrote:
| How does this compare to Flux.ai?
| https://docs.flux.ai/tutorials/ai-for-hardware-design
| built_with_flux wrote:
| flux.ai founder here
|
| Agree with OP that the raw models aren't that useful for
| schematic/pcb design.
|
| It's why we build flux from the ground up to provide the models
| with the right context. The models are great moderators but
| poor sources of great knowledge.
|
| Here are some great use cases:
|
| https://www.youtube.com/watch?v=XdH075ClrYk
|
| https://www.youtube.com/watch?v=J0CHG_fPxzw&t=276s
|
| https://www.youtube.com/watch?v=iGJOzVf0o7o&t=2s
|
| and here a great example of levering AI to go from idea to full
| design https://x.com/BuildWithFlux/status/1804219703264706578
| shrubble wrote:
| Reminds me of this, an earlier expert-system method for CPU
| design, which was not used in subsequent designs for some reason:
| https://en.wikipedia.org/wiki/VAX_9000#SID_Scalar_and_Vector...
| surfingdino wrote:
| Look! You can design thousands of shit appliances at scale! /s
| Terr_ wrote:
| To recycle a rant, there's a whole bunch of hype and investor
| money riding on a very questionable idea here, namely:
|
| "If we make a _really really good_ specialty text-prediction
| engine, it could be able to productively mimic an imaginary
| general AI, and if it can do that then it can productively mimic
| _other_ specialty AIs, because it 's all just intelligence,
| right?"
| ai4ever wrote:
| investor money is seduced by the possibilities and many of the
| investors are in it for FOMO.
|
| few really understand what the limits of the tech are. and if
| it will even unlock the usecases for which it is being touted.
| seveibar wrote:
| I work on generative AI for circuit board design with tscircuit,
| IMO it's definitely going to be the dominant form of
| bootstrapping or combining circuit designs in the near future (<5
| years)
|
| Most people are wrong that AI won't be able to do this soon. The
| same way you can't expect an AI to generate a website in
| assembly, but you CAN expect it to generate a website with
| React/tailwind, you can't expect an AI to generate circuits
| without having strong functional blocks to work with.
|
| Great work from the author studying existing solutions/models-
| I'll post some of my findings soon as well! The more you play
| with it, the more inevitable it feels!
| HanClinto wrote:
| I'd be interested in reading more of your findings!
|
| Are you able to accomplish this with prompt-engineering, or are
| you doing fine-tuning of LLMs / custom-trained models?
| seveibar wrote:
| No fine tuning needed, as long as the target language/DSL is
| fairly natural, just give eg a couple examples of tscircuit
| React, atopile JotX etc and it can generate compliant
| circuits. It can hallucinate imports, but if you give it an
| import list you can improve that a lot.
| DHaldane wrote:
| I've found the same thing - a little syntax example, some
| counter examples and generative AI does well generating
| syntactically correct code for PCB design.
|
| A lot of the netlists are electrically nonsense when it's
| doing synthesis for me. Have you found otherwise?
| seveibar wrote:
| Netlists, footprint diagrams, constraint diagrams etc.
| are mostly nonsense. I'm working on finetuning Phi3 and
| I'm hopeful it'll get better. I'm also working on
| synthesized datasets and mini-DSLs to make that tuning
| possible eg https://text-to-footprint.tscircuit.com
|
| My impression is that synthetic datasets and finetuning
| will basically completely solve the problem, but
| eventually it'll be available in general purpose models-
| so it's not clear if its worth it to build a dedicated
| model.
|
| Overall the article's analysis is great. I'm very
| optimistic that this will be solved in the next 2 years.
| maccard wrote:
| > The same way you can't expect an AI to generate a website in
| assembly, but you CAN expect it to generate a website with
| React/tailwind
|
| Can you? Because last time I tried (probably about February) it
| still wasn't a thing
| mewpmewp2 wrote:
| Depends on the website, right. Because a single index.html
| can easily be a website which it cam generate.
| maccard wrote:
| I mean, yeah. But that's not exactly helpful. Technically a
| web server can serve plain text which your browser will
| render so that meets the definition for most people.
|
| I don't think pedantry helps here, it doesn't add to the
| conversation at all.
| jamesralph8555 wrote:
| I tried GPT-4o in May and had good results asking it to
| generate react+tailwind components for me. It might not get
| things right the first time but it is generally able to
| respond to feedback well.
| maccard wrote:
| That's not the same as generating a website though. You
| still need to iterate on the components, and use them.
|
| I agree that using llms for generating things like schemas,
| components, build scripts etc is a good use of the
| technology, but we're no closer to saying "make a saas
| landing page for X using vercel" and having it ready to
| deploy, then we were a year ago
| crote wrote:
| The problem is going to be getting those functional blocks in
| the first place.
|
| The industry does _not_ like sharing, and the openly available
| datasets are full of mistakes. As a junior EE you learn quite
| quickly to never trust third-party symbols and footprints - if
| you can find them at all. Even when they come directly from the
| manufacturer there 's a decent chance they don't 100% agree
| with the datasheet PDF. And good luck if that datasheet is
| locked behind a NDA!
|
| If we can't even get basic stuff like that done properly, I
| don't think we can reasonably expect manufacturers to provide
| ready-to-use "building blocks" any time soon. It would require
| the manufacturers to invest a _lot_ of engineer-hours into
| manually writing those, for essentially zero gain to them.
| After all, the information is already available to customers
| via the datasheet...
| seveibar wrote:
| This is why me and even some YC backed companies are working
| toward datasheet-to-component ai. We don't trust third party,
| but we do trust datasheets (at least, trust enough to test
| for a revision)
| blueyes wrote:
| See Quilter: https://www.quilter.ai
| kristopolous wrote:
| Just the other day I came up with an idea of doing a flatbed scan
| of a circuit board and then using machine learning and a bit of
| text promoting to get to a schematic
|
| I don't know how feasible it is. This would probably take low
| $millions or so of training, data collection and research to get
| not trash results.
|
| I'd certainly love it for trying to diagnose circuits.
|
| It's probably not really that possible even at higher end
| consumer grade 1200dpi.
| cmbuck wrote:
| This would be an interesting idea if you were able to solve the
| problem of inner layers. Currently to reverse engineer a board
| with more than 2 layers an x-ray machine is required to glean
| information about internal routing. Otherwise you're making
| inferences based on surface copper only.
| kristopolous wrote:
| Maybe not. I scanned a bluetooth aux transceiver yesterday as
| a test of how well a flatbed can pick up details. There's a
| bunch of these on the market and the cheap ones, they are
| more or less equivalent. It's a CSR 8365 based device, which
| you can read from the scan. The industry is generally
| convergent on the major design decisions for some hardware
| purpose for some given time period.
|
| And the devices, in this case, bluetooth aux transceivers,
| they all do the same things. They've even more or less
| converged on all being 3 buttons. When optimizing for cost
| reduction with the commodity chips that everyone is using to
| do the same things, the manufacturer variation isn't that
| vast.
|
| In the same way you can get 3d models from 2d photos because
| you can identify the object based on a database of samples
| and then guess the 3d contours, the hypothesis to test is
| whether with enough scans and schematics, a sufficiently
| large statistical model will be good enough to make decent
| guesses.
|
| If you've got say 40 devices with 80% of the same chips doing
| the same things for the same purpose, a 41st device might
| have lots of guessable things that you can't necessarily
| capture on a cheap flatbed
|
| This will _probably_ work but it 's a couple million away
| from becoming a reality. There's shortcuts that might make
| this a couple $100,000s project (essentially data contracts
| with bespoke chip printers) but I'd have to make those
| connections. And even then, it's just a hobbyist product. The
| chances of recouping that investment is probably zero
| although the tech would certainly be cool and useful. Just
| not "I'll pay you money" level useful.
| contingencies wrote:
| I think good RE houses have long since likely repurposed
| rapid PCB testing machines to determine common nets using
| flying CNC probes. The good ones probably don't need to
| depopulate to do it.
| catherd wrote:
| As long as you are OK with destructive methods,
| grinding/sanding the board down gives you all layers. "PCB
| delayering" is the search term.
| amelius wrote:
| Can we have an AI that reads datasheets and produces Spice
| circuits? With the goal of building a library of simulation
| components.
| klysm wrote:
| That's the kind of thing where verification is really hard, and
| things will look plausible even if incorrect.
| amelius wrote:
| The LLM can verify e.g. transistors by looking at the curves
| in the datasheet.
| klysm wrote:
| The LLM can't verify anything - it just generates what it
| thinks is plausible
| ncrmro wrote:
| I had it generate some opencad but never looked into it further.
| rkagerer wrote:
| Any discussion of evolved circuits would be incomplete without
| mentioning Dr. Adrian Thompson's pioneering work in the 90's:
|
| https://www.damninteresting.com/on-the-origin-of-circuits/
| djaouen wrote:
| Sure, this will end well lol
| al2o3cr wrote:
| TBH the LLM seems worse than useless for a lot of these tasks -
| entering a netlist from a datasheet is tedious, but CHECKING a
| netlist that's mostly correct (except for some hallucinated
| resistors) seems even more tedious.
| teleforce wrote:
| Too Lazy To Click (TLTC):
|
| TLDR: We test LLMs to figure out how helpful they are for
| designing a circuit board. We focus on utility of frontier models
| (GPT4o, Claude 3 Opus, Gemini 1.5) across a set of design tasks,
| to find where they are and are not useful. They look pretty good
| for building skills, writing code, and getting useful data out of
| datasheets.
|
| TLDRN'T: We do not explore any proprietary copilots, or how to
| apply a things like a diffusion model to the place and route
| problem.
| amelius wrote:
| The whole approach reminds me of:
|
| https://gpt-unicorn.adamkdean.co.uk/
| roody15 wrote:
| It makes me think of the saying "a jack of all trades a master of
| none".
|
| I cannot help but think there are some similarities between large
| model generative AI and human reasoning abilities.
|
| For example if I ask a physician with a really high IQ some
| general questions about say anything like fixing shocks on my
| mini van ... he may have some better ideas than me.
|
| However he may be wrong since he specialized in medicine,
| although he may have provided some good overall info.
|
| Let's take a lower IQ mechanic who has worked as a mechanic for
| 15 years. Despite this human having less IQ, less overall
| knowledge on general topics ... he gives a much better answer of
| fixing my shocks.
|
| So with LLM AI fine tuning looks to be key as it is with human
| beings. Large data sets that are filtered / summarized with
| specific fields as the focus.
| pylua wrote:
| That's not really reasoning, right ? Maybe humans rely
| disproportionate on association in general.
| MOARDONGZPLZ wrote:
| Author mentions prompting techniques to get better results,
| presumable "you are an expert EE" or "do this and you get a
| digital cookie" are among these. Can anyone point me to non-SEO
| article that outlines the latest and greatest in the promoting
| techniques domain?
___________________________________________________________________
(page generated 2024-06-22 23:01 UTC)