[HN Gopher] Software 2.0 (2017)
___________________________________________________________________
Software 2.0 (2017)
Author : gtirloni
Score : 305 points
Date : 2023-02-21 14:58 UTC (8 hours ago)
(HTM) web link (karpathy.medium.com)
(TXT) w3m dump (karpathy.medium.com)
| swyx wrote:
| should put (2017) in the title.
| jevgeni wrote:
| don't show this to any consultants
| jstx1 wrote:
| What the article is saying is that when you develop and build ML
| systems, your workflow is different from the workflow you would
| have if you were solving the same problem without ML.
|
| The weird part is presenting it as 2.0 which implies that it
| replaces 1.0; it doesn't, apart from some edge cases like image
| recognition - we don't hand code rules to recognise images
| anymore but that's a tiny tiny part of all the software
| development work out there.
| et1337 wrote:
| If the hardest part of programming is reading, understanding, and
| debugging other people's code, our jobs are about to get a lot
| harder with a bunch of AIs running around spitting out 90%
| accurate code.
| pbreit wrote:
| Or much easier if codebases start looking 90% more familiar.
| ModernMech wrote:
| I made a passing joke the other week that we're turning the act
| of coding into a bureaucracy. No longer will you just be able
| to write code, you'll instead have to navigate whatever
| byzantine logic maze the AI has constructed for itself.
|
| Soon we'll have camps advocating various rhetorical paradigms
| to program AIs. Instead of imperative versus declarative
| programming, we'll have effusive versus abusive prompt
| engineering: "The best way to program is to be
| really nice to the AI, and tell it how much you appreciate it
| and it will come up with the best solution itself."
| "No! The best paradigm is to berate the AI and to beat it into
| submission to get exactly what you want!"
|
| The more I hear people talk about programming in the future,
| the more it sounds like that's really where some people want to
| take us. I'm not excited.
| choletentent wrote:
| Or easier if your peers write code which is on average below
| the 90% mark.
| adregan wrote:
| Why would the AIs spit out code? I think the point of the
| article is that they'll spit out black boxes full of weights
| that can be used to compute anything ala the universal
| approximation theorem. You'll train a neural net for a given
| task and verify the results. Producing code seems to be
| pandering to our ego.
| goatlover wrote:
| Because most code involves teams and different stages of
| development, not just a single approximation of a good enough
| result. And what happens when you need to make tweaks? The UI
| isn't quite right or a business rule needs to be added.
| yetihehe wrote:
| Because code is faster and more correct than a neural net if
| written reasonably. With code you don't need several gpus to
| calculate a sum of several million records in database.
| hn_throwaway_99 wrote:
| > Because code is faster and more correct than a neural net
| if written reasonably.
|
| But, as the article points out, _at least for a
| considerable set of problem domains_ , it's not.
|
| My general rule of thumb is that anything that has a
| relatively small, finite, discrete set of inputs and
| outputs is better suited to "Software 1.0" (e.g. coding a
| calculator). But there are a huge number of domains, some
| highlighted in the article, which usually have to do with
| an infinite possible number of inputs and outputs, where
| human written code is _not_ faster or more correct than a
| neural network.
| phkahler wrote:
| Code can also be debugged and fixed. AI needs to be
| retrained and retested - probably multiple times. Each
| retraining may cause a regression in any area.
| hinkley wrote:
| Which also means you can diff the code output from two
| runs of the system and determine if any of the diffs
| amount to regressions in the code.
|
| Developers and machines that can't explain themselves end
| up out on their ear or damaging the projects they're
| involved in.
| p1esk wrote:
| Because code is easier to verify.
| diydsp wrote:
| A reasonable question. My answer is a lot of state in
| computing is digital. e.g. off, booting, loaded, running,
| paused, error. While this isn't an insurmountable hurdle, I
| would suggest the floating point arch of present AI would
| benefit from incorporating more discrete state.
|
| Similarly, I don't whether AI weights are computed all in
| virtual parallel, but if they are computing every node
| simultaneously that will be less efficient than the Neumann
| model in which a Program Counter (PC) acts like a "cursor"
| hopping arund and iterating states of the discrete code model
| at points throughout it. E.g. a video game with a controller
| and various sprites will have objects that update at various
| rates and the player moves with different code underwater
| than in the air, than on land, so different parts of the
| model would execute.
| modriano wrote:
| I guess I should start researching the testing framework
| ecosystem. Frameworks that make it easy and low-friction to
| represent desired behavior and identify+test edge cases will be
| key to using copilot/AI-code-tools effectively.
|
| So, what are best testing frameworks people have worked with?
| thih9 wrote:
| I don't think that was the point. I think the point was that as
| long as you're happy with 90% accuracy, you can have that for
| 10% of effort.
|
| It requires a perspective shift. Then again, cost efficiency is
| a big motivator.
| hinkley wrote:
| 90% accuracy is lousy. Especially for anything that isn't a
| one-off interaction, such as with customers or developers.
| That's a 35% chance at a failure after 4 interactions.
| rhacker wrote:
| The key word is "If". Since we don't generally read other
| peoples code, we tend to rewrite the whole damn thing every
| 10-15 years (sometimes 4 years).
|
| The reality is it all comes down to testing. If S1.0 has a unit
| test that says "Person should not get financial help", S2.0
| should also have a unit test that says "Person should not get
| financial help", and work the same.
|
| Of course my unit test name is designed to enrage, but let's be
| honest, we're writing code that makes these decisions.
| hinkley wrote:
| Sometimes at the same time.
|
| I keep complaining about some NIH code we have and people say
| things like, "well that library didn't exist at the time, so
| we had to write one."
|
| Git history says they're wrong. Time and again they were
| writing code that had existed for two years already.
| orangepurple wrote:
| Nearly nobody who wants to use these neural nets knows how to
| do model risk management. These things are black boxes with
| edge cases that are badly understood and nearly impossible to
| inspect and test. At least code is logical and human readable
| (usually).
| davedx wrote:
| I've worked on a lot of codebases and your "usually" is
| highly debatable. I don't think the difference in
| understandability will be as big as people think as tooling
| for ML evolves.
|
| Something like Godbolt for neural networks? Can't be that far
| away
| shanebellone wrote:
| This is my concern too. How can you trust a business process
| that you cannot inspect step-by-step? Imo this greatly limits
| use cases.
| hinkley wrote:
| I've worked a number of places where the interview process
| was predominantly about hiring legible people.
|
| Someone who's wrong and cagey can cause a lot of damage
| before anyone figures out what's up. At least a talkative
| idiot can be managed.
| johnthewise wrote:
| Maybe don't approve code you don't understand? If people just
| lets in code they don't understand into products and it
| passes PRs and tests, they might have written dumb code in
| the first place. I see essentially zero risk in using these
| tools. dysfunctional teams will write shitty code anyways,
| whether it's coming from ai doesn't matter.
| matisseverduyn wrote:
| 100 lines of poorly written code (by AI or humans) added to
| a 1M line codebase is easy to spot and fix by the dedicated
| humans who have that whole system mapped out in their
| heads.
|
| Having AI write a 90% accurate 1M line (or "parameter")
| codebase all at once, (which seems to be the expectation
| here), is the "risk" you're overlooking. No human will be
| able to know where to start debugging that. At least not
| yet. But will _yet_ come before incredibly dangerous
| amounts of AI written code is pushed into critical systems
| everywhere, by naively optimi$tic opportuni$t$?
| throwaway1777 wrote:
| Don't worry, the AI can write the tests too
| titzer wrote:
| lol, who tests the tests?
| goatlover wrote:
| I bet it can also hallucinate tests.
| cratermoon wrote:
| I recently read a thread on Mastodon[1] suggesting that
| "confabulate" is more correct than "hallucinate". In
| psychology and neurology it means to fabricate imaginary
| experiences or details as compensation for loss of
| memory.
|
| 1 https://universeodon.com/@siderea/109883198218504351
| hinkley wrote:
| I have coworkers who also hallucinate tests. Now they'll
| have someone to go out to lunch with.
| mmis1000 wrote:
| I had saw someone mutate the code randomly until it works.
| There are both a boolean flag and an if statement went wrong,
| and result in correct result by luck. I almost wondering if
| it is AI or something trying to fix the code.
|
| If AI don't care about the process and only care about
| results. These types of thing is going to happen everywhere.
| And from this moment, no code is understandable to human.
| pyrolistical wrote:
| Hard to inspect but very possible test.
|
| Correctness will go from a binary pass/fail to a probability
| ArchD wrote:
| And someone will decide that a 0.0001% probability of an
| airplane crash caused by a bug is acceptable? Maybe
| 0.0000001% is more reasonable? In any case, how do you
| accurately determine the probability of a rare failure of a
| black box without doing many real experiments with real
| inputs?
| Redoubts wrote:
| And what's a postmortem going to look like?
|
| "Don't worry, we stuck the flight data recorder in the
| training set, and rebuilt the model. Should be good to go
| now"?
| hinkley wrote:
| We already know what the post mortems look like. AI black
| boxes will look a lot like declarative programming black
| boxes. We don't really know how the code runs, we just
| ask it nicely to do what we want and then stare at the
| config files and the docs if it doesn't.
|
| Low code and AI are going to have many of the same
| failure modes. Until someone combines them and then
| they'll have exactly the same failure modes.
| AlotOfReading wrote:
| That's how safety critical devices are already built, so
| yes. We have standardized probabilities of failure (e.g.
| SIL [0]) from the unexpected, because mitigating 100% of
| risk is somewhere between impractical and impossible.
|
| [0] https://en.wikipedia.org/wiki/Safety_integrity_level
| davedx wrote:
| Indeed. Some of the commentators on this article should
| really examine their assumptions about how reliable and
| correct "Software 1.0" is.
| ArchD wrote:
| From a quick reading of the wiki, the associated
| methodology seems rather limited:
|
| "System complexity, particularly in software systems,
| making SIL estimation difficult to impossible"
|
| "The requirements of these schemes can be met either by
| establishing a rigorous development process, or by
| establishing that the device has sufficient operating
| history to argue that it has been proven in use."
|
| You could prove that normal code satisfies some specs,
| but you can't do that with neural nets unless the number
| of possible inputs is tiny. So, the only way to establish
| that the black box neural net meets some SIL target is
| through "sufficient operating history".
| AlotOfReading wrote:
| To clarify, I wasn't offering SIL up as an example of how
| we should validate ML systems, but instead to demonstrate
| that "software 1.0" systems are _already_ designed the
| way GP is questioning. Best practices for applying
| integrity level concepts to ML is still a topic of active
| debate right now.
| danrocks wrote:
| > Correctness will go from a binary pass/fail to a
| probability
|
| Excellent point. The pervasiveness of neural nets will
| require engineers (and really, everyone) to start thinking
| more probabilistically and establish acceptability
| thresholds instead of certainty. It's the way of the
| future.
| goatlover wrote:
| That doesn't sound like progress. Maybe for CRUD apps it
| will be okay. I have a hard time imagining it will be
| acceptable for financial or critical systems.
| ako wrote:
| How about self driving cars?
| bee_rider wrote:
| You'll rent a self-driving car from a ride sharing app,
| they'll make sure that the expected cost of fines and
| lawsuits will be significantly less than the expected
| revenue from providing rides. Although they can help push
| down the former value by making sure they operate from a
| favorable jurisdiction.
| marginalia_nu wrote:
| That seems like a regression in many regards, especially
| given how much of this seems like a solution looking for
| a problem it can solve rather than the other way around.
| agileAlligator wrote:
| It seems to me that current AI techniques are more about
| reducing programmer workload than actually doing
| something that isn't possible with traditional methods.
| xp84 wrote:
| I'd argue that it's a solution to the problem that
| programmers are expensive _and_ only a fraction can be
| counted on to produce 100% reliable code anyway.
|
| While it's possible for a highly-skilled, highly-
| professional developer can both write code that will
| solve a given problem 100% correctly _and_ write tests
| that will prove that it solves them for the entire
| domain, in practice most developers fall short on both
| counts. Every time you interact with a date or phone
| number field that chastises you for your use or non-use
| of punctuation, you know this.
|
| So, for many use cases, it's possible imperfect
| programmers will be replaced with neural networks that
| are 95% accurate, perhaps with a differently-trained one
| checking the work of the first one.
| davedx wrote:
| Less of a regression than you might think. How many
| software _systems_ (outside of very well unit tested
| components) really have strong correctness proofs? It's
| an almost futile task in modern software engineering with
| all those distributed systems everywhere
| __MatrixMan__ wrote:
| I've been looking for an opportunity to try out a pattern
| where the two approaches improve each other:
|
| 1. Generate a 95% accurate model
|
| 2. Use it to generate test cases
|
| 3. Code the thing, with the help of the cases
|
| 4. Manually remove cases in the 5%
|
| I image steps 1 and 2 being completed by a product owner
| and 3 and 4 being completed by a software engineer.
|
| We're so horrifically bad at communicating a
| requirement's intent, I wonder what would happen if we
| tried to use AI to communicate them via their extent
| instead.
| lifeisstillgood wrote:
| That's probably genius :-)
|
| I mean ... can it be done?
|
| - build a platform (ie the data we care about and are
| going to build some workflow over)
|
| - have business describe what should happen in english
|
| - How does GPT build something that will run? Can it
| create the infrastructure? does it speak AWS?
|
| - then ...
|
| OK - I am actually excited by that
| guhidalg wrote:
| Agreed. We may imagine ourselves very smart but in my
| experience most people do not understand probability and
| statistics well.
|
| Is 90% accuracy good enough? Is 95%? 99%? 99.9%? No
| matter the answer, you have to tolerate errors. Now your
| stakeholders have to tolerate errors. Are they going to
| accept errors just because "Software 2.0" is here and
| that's what we all have to live with? Nope.
| badloginagain wrote:
| The entire field of Service Reliability is dedicated to
| finding the exact boundaries of acceptable errors, and
| defining the response function when those boundaries are
| crossed.
|
| I would assume this actually gels very nicely with neural
| nets since its constantly optimizing for fitness. Hell in
| theory you could bake in your SLA/SLIs into the models to
| self correct? Give the model direct feedback that its
| unfit?
| duckmysick wrote:
| > No matter the answer, you have to tolerate errors.
|
| We already do that in manufacturing. Physical parts are
| imperfect and we design with such variation in mind.
| staunton wrote:
| All software is riddled with errors and for most purposes
| that's fine. Any developer, stakeholder, whatever, who
| thinks otherwise is living in a parallel universe.
|
| It may be possible in the future to have "for all
| practical purposes flawless" software, which might make
| sense for select special applications. That would be a
| new thing though, rather than something we have and could
| lose due to adopting AI development.
| marginalia_nu wrote:
| Formal verification of software has been around for a
| while though.
| staunton wrote:
| Which has almost never been used to build software anyone
| ever used... That might be changing _very slowly_
| rco8786 wrote:
| > All software is riddled with errors and for most
| purposes that's fine.
|
| This is typically not that true when it comes to
| _correctness_. Most software does the correct thing in
| the eyes of the user, nearly 100% of the time. And when
| it doesn 't, the bug gets fixed and that edge case is
| corrected for every other user going forward.
|
| AI generated software, from what I've seen, has a wide
| range of errors in correctness (along with all the other
| errors that you mentioned all software having..which is
| true). Like it literally just does the wrong thing given
| what the user is expecting it to do. The path toward
| iteratively improving and getting it to an acceptable
| level of correctness for any given application might be
| there, but so far I have not seen it.
| staunton wrote:
| That's an interesting difference but I'm not convinced
| it's valid. Perhaps you can explain what you mean in more
| detail?
|
| Let's say the software is good enough if it does the
| right thing 99.9% of the time it is used. I take it,
| you're saying that if an AI starts modifying it and only
| writes correct code in 99.9% of cases (yes, current AI is
| not even close, but it will improve), that makes it worse
| because the software might start failing completely.
| However, if you have proper tests and release management,
| such obvious flaws will quickly be detected and fixed or
| rolled back. For most applications that seems pretty much
| equivalent to what we have now.
|
| The other case is software that is completely AI
| generated and cannot reasonably be modified by humans
| anymore. In that case, again, you have tests and a sane
| deployment strategy that mitigates failures to a
| sufficient degree depending on application.
|
| So the only issue is when you start making a completely
| AI generated software and fail to ever meet requirements,
| or pass the human written test cases? Users at least will
| never be impacted by that. Even now, many human-written
| software projects never get to the stage where they can
| be used. Is this really a problem, especially if the
| attempt at AI generation of software is cheap?
| rco8786 wrote:
| > yes, current AI is not even close
|
| This is really my main argument currently...
|
| > but it will improve
|
| With this being the "if" question. Improve, but improve
| to a point where we can trust it to do the things with
| the level of correctness actually required? Unclear so
| far. GPT3, the state of the art, can't be trusted to
| answer basic questions correctly (yet).
|
| I also don't personally buy into the other notions in
| these comments that "the future is probabilistic
| software". I think that's wishful thinking outside of
| some specific domains and an attempt to bend our actual
| requirements to meet the capabilities of AI software,
| rather than the opposite.
|
| > pass the human written test cases
|
| I'm not super sold on this idea either. It seems
| reasonably possible that writing the test cases to a
| level of specification necessary to ensure that
| correctness we're after is just as much effort as just
| writing the code.
|
| But, time will tell with everything.
| staunton wrote:
| The way I see it is as expanding the high-level-low-level
| spectrum of programing tools towards the high end. In
| very large-scale projects that have many levels of
| complexity, I would argue "probabilistic software" is
| already a reality. You don't even try to fix all the bugs
| or edge cases. You try to minimize the impact of failures
| while accepting that there will always be failures.
| Building in logic to fail more gracefully is a big part
| of that and cost/benefit of such efforts is a
| probabilistic question, even if it might rarely be framed
| as such.
|
| It's usually much easier (and never harder) to specify
| what needs to be done than how it should be done. Whether
| you can trust the result (enough) depends on the
| application. AI driven development will be applied first
| wheree errors and failures are least harmful and advance
| from there. It might take a long time until the degree of
| correctness improves enough and trust is build around it.
| After all, some countries' railroad networks still don't
| use computers but instead have a human map out new routes
| and schedules on paper to make sure trains don't crash.
| Nevertheless, the speed of AI development has exceeded
| (at least my) expectations time and time again in recent
| years.
|
| What might happen is that we eventually get a lot of
| software that fails more often than now but is a lot
| cheaper. That would still mean that the new methods are
| widely adopted. People accept software errors as a part
| of life already, so even if they get more frequent in
| less critical applications, we will adapt.
|
| What is really dangerous is when the various software
| components get too fast and complex to understand or
| control and develop pathological feedback loops in
| situations that cause real trouble. This kind of thing is
| a continuum of badness which tops out at "AI taking over
| and wiping out humanity". Given market incentives driving
| adoption, which I anticipate to be strong, it's hard to
| imagine how such risks might be mitigated.
| rco8786 wrote:
| > You don't even try to fix all the bugs or edge cases.
| You try to minimize the impact of failures while
| accepting that there will always be failures
|
| Yes but I'll point you back at my original comment about
| _correctness_. I 've never been on a team that shipped
| code we knew would do the wrong thing. I ship code with
| known failure points all the time. But when the code runs
| to completion, I'm pretty darn sure it's doing the
| correct thing and we try very, very hard to make sure of
| that. With AI I am seeing that they can't discern between
| issues of correctness and issues of failure or
| availability. It's just like 95% success across all of
| those spectrums.
|
| > What is really dangerous is when the various software
| components get too fast and complex to understand or
| control and develop pathological feedback loops in
| situations that cause real trouble.
|
| Yea I agree, that's somewhat scary to think about.
| User23 wrote:
| > This is typically not that true when it comes to
| correctness. Most software does the correct thing in the
| eyes of the user, nearly 100% of the time. And when it
| doesn't, the bug gets fixed and that edge case is
| corrected for every other user going forward.
|
| You're confusing pleasantness with correctness.
| Pleasantness is the property of pleasing the user, or
| more often the software's owner. Correctness is the
| property of conforming to a specification. Since most
| software written has no specification, correctness is
| undefined. Evidently this works adequately well in the
| marketplace.
| madeforhnyo wrote:
| It might be easier for an engineer to fix a bug by
| changing some lines of text than readjust a neural
| network, for the time being at least.
|
| As pointed out, there are already formal languages that
| allow formal verification like B [0] notably for like-
| critical systems.
|
| [0] https://en.wikipedia.org/wiki/B-Method
| hwayne wrote:
| There's a difference between errors in
| UI/reliability/performance/etc and errors in business
| logic. When there's an error in the business logic, then
| heads roll.
| kabdib wrote:
| "Good news, analysis of the anti-lock brake system
| failure shows that you're only sixteen percent dead."
| rco8786 wrote:
| > It's the way of the future.
|
| For some use cases, sure. We go through painstaking
| efforts to ensure things like correctness, consistency,
| and idempotency for a reason though. Most things we want
| to be deterministic, and when something's not
| deterministic we freak out and fix it ASAP (including
| waking people up in the middle of the night to do so)
| fassssst wrote:
| Yea, aka how engineers working on safety critical or high
| reliability stuff already have to think.
|
| Assume your software has a probability to fail or have
| bugs or gets hit by bit flips or unreliable hardware.
| There's a whole field for dealing with those kind of
| things that typical web devs haven't had to worry about
| as much.
| belter wrote:
| These are not those type of Engineers. Not the ones doing
| triple computing using three different processors with a
| summation and voting process, inclusive using different
| programming languages and compilers, because the compiler
| can also have bugs. These are the engineers running self
| driving beta neural nets on public roads...
| Xeoncross wrote:
| "We're 98.7% certain the user's payment go to the right
| account with this change"
| krab wrote:
| More like we're 98.7 % certain that the screen layout
| looks ok for all device types and languages. The question
| is how hard is it to fix the problematic cases without
| turning the whole thing upside down when you find out the
| original solution is not enough.
| islon wrote:
| Our software has a 99% chance to calculate you taxes
| correctly! And only a 1% chance of committing tax fraud.
| sebzim4500 wrote:
| I wonder what accuracy existing tax calculators have.
| jpollock wrote:
| If they were broken, they are deterministically broken,
| unless they are really poorly written the calculator
| won't randomize the answers to the same inputs.
| kklisura wrote:
| Not quite, let me rephrase it...
|
| > Our software has a 99% chance to calculate your taxes
| correctly! And only a 1% chance of failure in which it's
| your fault and it's you that's committing tax fraud
| gtirloni wrote:
| Considering ChatGPT spits out innacurate information all
| the time, I think "our software has a 99% chance of you
| going to jail for tax fraud" is more accurate.
| alfor wrote:
| Better than most accountants.
|
| Family member got ripped by a government audit of what
| was supposed all fine by the person doing his company
| taxes.
| fock wrote:
| that doesn't look to bright if we look back towards Covid
| and vaccines. People are even bad at plain frequentist
| statistics!
| duderific wrote:
| That assumes people even care what the statistics are.
| Many people just go with their emotional response based
| on a political outlook when deciding what course of
| action to take.
| flangola7 wrote:
| Engineer brains are just a type of neural network too,
| which also have a probability of putting out bad code.
| Space exploration budgets run into the hundreds of
| millions of dollars each and recruit the brightest people
| on the planet to design and program software, yet there
| is a long list of projects that failed due to code
| errors. Mars probes have failed to reach the planet,
| orbited too low and burned up, crashed into the surface,
| or landed successfully but later overwrote critical
| memory due to a flawed software update.
|
| The AI doesn't have to be perfect, but only offer a lower
| error rate than humans.
| belter wrote:
| Engineer brains are not a neural net. It's only our brain
| lack of knowledge of how the brain works that led us to
| anthropomorphize Matrices and Weighted Graphs :-) Reading
| a bit on the Brain will quickly help dismiss those
| analogies. I suggest these two as good starting points:
|
| - Lange Clinical Neurology - 11th Edition
|
| - Bradley's Neurology in Clinical Practice, 8th Edition
| hwayne wrote:
| Brains are clocks, I mean machines, I mean computers, I
| mean neural nets.
| goatlover wrote:
| There was an article a while back comparing the mind to a
| symphony, but it never caught on as a metaphor.
| booleandilemma wrote:
| That's not engineering though is it?
|
| What use is a banking app if it's only going to be correct
| some of the time?
| [deleted]
| rco8786 wrote:
| > Hard to inspect but very possible test.
|
| Probably, but is testing to the necessary level of
| correctness more or less effort than writing the code
| ourselves?
| hinkley wrote:
| Languages with very high degrees of static analysis are
| basically making you write the tests in the code. We have
| people who think they can keep extending that until
| there's barely any code, but they haven't been proven
| right or wrong yet.
| layer8 wrote:
| Static analysis creates proofs, which is fundamentally
| different from tests. Merely testing AI output will
| always be inferior on principle than proving code to be
| correct. One would have expected AI to be better than
| humans at proving code correct (instead of just testing
| it for certain inputs), but LLMs currently fall short of
| that.
| hinkley wrote:
| What I'm saying is that if you have the proof, someone
| will write an AI to pass it.
|
| There's still a lot of very, very sophisticated work that
| goes into locking down requirements that tightly.
| layer8 wrote:
| > if you have the proof, someone will write an AI to pass
| it.
|
| That doesn't make any sense. What passes the proof is the
| program code. You have the code, and then you construct a
| formal proof that the code is correct, similar to how a
| mathematician proves that some theorem is correct. The
| code is a prerequisite for the proof. When you can
| construct a proof for the code, you're done.
|
| This proof-construction process is what AIs currently
| aren't good at, because it requires logical precision,
| and probability isn't sufficient. They can generate code,
| but they can't construct the formal proof that the code
| is correct (and it often isn't).
|
| > There's still a lot of very, very sophisticated work
| that goes into locking down requirements that tightly.
|
| What's true is that you need to know what you want to
| prove about the code, and that isn't always easy.
| hinkley wrote:
| Possible to test, but not likely.
|
| What often divides the excellent engineers from the poor
| ones is how well they can think about corner cases, and
| tests are mostly about writing down the corner cases in a
| sustainable way (vs half-assedly writing down half of the
| corner cases and writing 0.5% of tests as asserting a bug.)
|
| The main problem I saw with DevOps and QA automation was
| that if people don't write code all day, having them write
| code that gates release of software to production does not
| result in good outcomes. Sooner or later developers have to
| inject some engineering practices.
|
| If the engineers are running AI generated code, nobody
| knows how to do that job, and you will get a long string of
| permanently damaged brand names in the aftermath.
| emeril wrote:
| so many companies have such terrible code quality to
| begin with, it's hard to see how brand names could be
| much worse in many (but not all) cases
| theptip wrote:
| On the flip side, if the rest of the process becomes easy
| and in some sense standardized (the art/craft of code is
| replaced with back-prop), then maybe the complements like
| data cleaning, model bias/error measuring, and
| interpretability become "the work" that the high-paid
| scientists/engineers focus on?
|
| I can particularly imagine a regulatory environment much
| more rigorous than software has got away with thus far,
| for example strict requirements around certifying that
| your model doesn't exhibit X Y Z biases according to
| standard frameworks of evaluation.
| hinkley wrote:
| Not if the ladder gets pulled up behind us. If there's
| nowhere to train people with 0 years of experience to do
| this work effectively, then new developer creation will
| drop by 10x which means five years in we'll be missing
| 40% of the workforce.
| broast wrote:
| It's the hardest part but also the most fun.
| albertzeyer wrote:
| It's not that humans would produce 100% accurate code...
|
| And you can also use the AI to check for correctness (of
| course, again not 100% accurate), potential issues, potential
| improvements, etc.
| billbrown wrote:
| It reminds me of the "code generator" phase we went through a
| decade or so ago. At least then we could see the "code
| generator" code and adjust it when it produced dreck.
|
| I'll also note that the next hardest part of programming is
| troubleshooting "in production" whether a Web application, in
| an embedded device, or running on someone's machine. Is the
| "AI" going to help there? Are we going to even be able to fix
| those problems when doing so could make the code we don't
| understand fail in another way or hit the wrong side of the
| performance tradeoff the "optimized code" entailed?
| hinkley wrote:
| Code generation is an epicycle.
|
| My dad was worried about whether I should go into a CS degree
| because there was a code generation cycle going on at the
| time and people thought the computers would be programming
| themselves. We've had a few since, and my skills are more
| valuable than ever.
| didericis wrote:
| This particular cycle seems likely to be really
| destructive. Software is way more prevalent in nearly every
| aspect of our lives than it used to be, and all of the
| large organizations that don't really understand the
| problems that are going to compound invisibly are going to
| turn their products into unintelligible giant heaping piles
| of garbage very quickly if they're too enthusiastic. Large
| codebases almost always turn into giant heaping piles of
| garbage, so that's nothing new, but you can usually get
| someone to dig through and salvage things from the garbage
| pile when it goes bad.
|
| This article is from 2017, and I don't know about you, but
| I increasingly feel like software is more and more broken
| year after year. I think that has a lot to do with
| "delegating complexity" and building things without really
| understanding what they're doing. I think the correlation
| between human understanding of the fine details and
| underlying logic and desired outcome and software quality
| is pretty tight. That doesn't mean "software 2.0" neural
| net stuff doesn't fit in, you just need a human to plug it
| in right that really understands its benefits and its
| limitations.
|
| The author mentions the downsides, but I think they
| underestimate them. If you lean on AI too heavily and don't
| ever translate to a traditional language, you've basically
| liquified your logic/there's no garbage dump to salvage
| from when things go bad and zero understanding of implicit
| context. If you use it to generate ostensibly human
| readable code you get "documentation" with no guarantee of
| accuracy, which makes it worse than having no documentation
| (depending on how high the error bars are). While that's
| not a new problem either, if it's autogenerated that means
| it's easy to create way more of it than human generated
| code, which means it'll probably be an ever larger portion
| of what gets sucked up into later AI models. If they become
| too self referential they'll become increasingly detached
| from human judgement about whether the code is doing what
| it should and error bars will grow.
|
| I'm still convinced these things are virtually all going to
| end up in a fancy autocomplete suggestion and compression
| niche after a lot of pain. But that's still a big deal/I
| don't think the limitations of these means they don't have
| a big future place. The sheer number of things you can have
| autosuggest for now with these AI models are amazing, and
| that expansion is boosting productivity and creating a
| large number of new products that are going to become
| essential tools. That being said, every time this type of
| thread comes up I'm like "woah woah woah, pump the breaks,
| these things have no understanding of what they're doing.
| You can't just stop thinking about stuff and let a machine
| do it, bad bad bad idea."
| pphysch wrote:
| "We need more Software (LOC)! Hire a thousand code monkeys!"
|
| ...
|
| "WTF! We have so much technical debt! Fire the code monkeys and
| replace them with AI! We need more Software (AI)"
|
| I'm sure this is going great.
| krono wrote:
| Not just inaccurate, but also code that is undesirable in
| several other ways of varying severity that a human is less
| likely to produce at all, or more likely to reject if it had
| originated from any other source.
|
| The fault lies with humans treating these programming AIs like
| the greenest of junior bootcamp devs would treat the most
| accomplished senior engineer.
| johnthewise wrote:
| If the tool itself is 90% accurate, conditioned on the events
| that somebody reviews it, it passes test and PRs, it can get to
| 99.9999% accurate very quickly.
| jmfldn wrote:
| This is very true. But could the program itself just be a
| normal program, but written by a neural network? I guess this
| is kinda what GPT 3 can do?
|
| One way or another, I want to be able to read the source. Or am
| I missing the point? Maybe future software will be a big neural
| network blackbox that we verify purely through tests?
| fassssst wrote:
| Well maybe instead we'll spend our time honing the spec and
| doing lots of testing instead of mechanically writing and
| reviewing code. I see that as potentially a massive net
| positive for software quality.
| marcosdumay wrote:
| You mean honing the spec enough so that it is unambiguous?
|
| Maybe you can use some special purpose artificial language
| created with the purpose of writing unambiguous texts... Like
| Java or Python.
| CuriouslyC wrote:
| The fatal flaw in your sarcasm is that specs are
| declarative but those languages are imperative. Those
| languages also have more syntax than is strictly necessary,
| which limits the ability of domain experts to contribute to
| the spec without the assistance of a developer.
| jeltz wrote:
| I have worked with systems written primarily in SQL, a
| declarative language, and I can't say a domain experts
| could have contributed anymore or less than when I have
| worked with systems written in Ruby or Java.
| kristiandupont wrote:
| Would the point not stand if they had said ML or Prolog?
| CuriouslyC wrote:
| There would still be the point of requiring knowledge of
| an arcane syntax that can take months to understand and
| years to master. It will almost always be more efficient
| and productive to specify programs in plain English then
| add additional specifications to clarify ambiguity when
| observed, rather than trying to write specs in a
| completely unambiguous language.
| fassssst wrote:
| Those aren't good enough, you really need a formally
| verified language.
|
| Or just use English and do lots of acceptance testing plus
| add fail safes. I bet that will be more economical.
| gtirloni wrote:
| Look at all the millions of developers proficient in Aqda
| and Coq.
| naasking wrote:
| The point is that you don't need millions of developers
| if AI is writing the code against a spec, you'll maybe
| only need a few thousand spec writers.
| marcosdumay wrote:
| The point is that if you are trying to replace millions
| of developers, you will obviously require something on
| the order of quadrillions of spec writers. Or, rather,
| this is probably still an underestimation.
|
| And yes, it is a very obvious point, and that people keep
| missing that point on this site is unsettling. (Also,
| yes, this can be trivially circumvented if you just let
| those people program, instead of only do verification.)
|
| You can, also obviously, replace millions of average
| developers with (way more) millions of extremely
| competent spec writers if they can use formal methods.
| Those will require way more computing power than it can
| ever exist on Earth to do their work, but they can
| mathematically get there.
| naasking wrote:
| > The point is that if you are trying to replace millions
| of developers, you will obviously require something on
| the order of quadrillions of spec writers.
|
| Why do you think you need orders of magnitude more spec
| writers than coders rather than the other way around?
| marcosdumay wrote:
| That is an intrinsic feature of software verification.
| You either have a formal system or you have an
| exponentially growing amount of corner cases to verify.
| naasking wrote:
| What kind of verification is done without a formal
| system?
| marcosdumay wrote:
| We usually call those "tests", but some people do really
| like non-formal linter rules too. (Personally, I abhor
| checking rules with high false positive odds.)
|
| AFAIK, those are the only ones in common use, but
| differently from formal ones, non-formal things tends to
| come on a multitude of widely different types. So I
| wouldn't be surprised if people have invented many more.
| gtirloni wrote:
| I think you're underestimating the number of spec
| writers. If that ever happens, every company with a
| custom business stack will need at least a few. That's
| certainly a few million jobs worldwide.
| marcosdumay wrote:
| Writing the code with a few checks here or there is
| almost always sufficient. But yes, if you want to verify
| the AI's output you better write lots and lots of
| acceptance tests. Easily pages of tests for each line of
| code. If the problem is large, it can easily get into
| millions of lines of test for each line of code.
|
| Formally verifying it will easily take more computing
| power than training the AI on the first place, so I don't
| count that one as viable.
| [deleted]
| bilsbie wrote:
| Or the opposite
| gitfan86 wrote:
| Just run that code through another model to fix the problems.
| Then run that output through a model that will summarize the
| final code.
| daveslash wrote:
| Came here to make the same comment. If the first AI model is
| buggy, then instead of debugging it with Software 1.0
| techniques, just use Software 2.0 to debug it with AI! And
| when there are issues with the AI debugger, use a 3rd model
| to debug the debugger. And then a 4th. It's AI all the way
| down! I can see _whole empires_ of consulting firms becoming
| very rich.
| mkleczek wrote:
| This begs the question: what data is going to be used as a
| training set? How do we make sure our black boxes don't become
| just echo chambers giving us answers we already know?
|
| At the end of the day it is always: garbage in, garbage out.
| Duwensatzaj wrote:
| Sid Meier's Alpha Centauri brought this up in 1999.
|
| "We are no longer particularly in the business of writing
| software to perform specific tasks. We now teach the software how
| to learn, and in the primary bonding process it molds itself
| around the task to be performed. The feedback loop never really
| ends, so a tenth year polysentience can be a priceless jewel or a
| psychotic wreck, but it is the primary bonding process--the
| childhood, if you will--that has the most far-reaching
| repercussions."
|
| - Bad'l Ron, Wakener, "Morgan Polysoft"
| wewtyflakes wrote:
| That game is worth playing for all of those neat "quotes"; they
| seem more on point by the day.
| https://civilization.fandom.com/wiki/Pre-Sentient_Algorithms...
| azubinski wrote:
| So we have Web 2.0 thanks to which the Most Important Extension
| Distill is installed in my browser. Because without it, my head
| hurts a lot from Web 2.0.
|
| Ok.
|
| I feel like I'll have to install something that will make life
| easier with Software 2.0.
| jinay wrote:
| I recently attended a hackathon (TreeHacks) where Andrej was a
| keynote speaker, and he's since updated his vision to include the
| significance of LLMs. The new software is that of prompt
| engineering, using LLMs as a simulator for the types of programs
| you want to build [1]. The definition of programming will keep
| changing as we get to higher and higher levels of abstraction.
|
| [1]
| https://twitter.com/karpathy/status/1618311660539904002?lang...
| jdoliner wrote:
| On an entirely silly tangent. It's a bit of a shame that Karpathy
| stopped working at Tesla. Just because having a guy named
| Karpathy working on car pathing was such a great example of
| nominative determinism.
| 0xdeadbeefbabe wrote:
| Nonfiction for the win.
| albert_e wrote:
| If you were from India you would read his name as car-pathy
| (Lord of Cars)
|
| cf. gana-pathy (Lord of the Ganas)
| ryloric wrote:
| Completely missed this lmao
| [deleted]
| layer8 wrote:
| Maybe Tesla should relocate their research center to the
| Carpathians.
| BulgarianIdiot wrote:
| Do you believe in nominative determinism? It's an interesting
| concept with gigantic implications.
| nealabq wrote:
| Do you? Does your username have deterministic implications?
| Or aspirational?
| BulgarianIdiot wrote:
| Haha, both!
| xyzelement wrote:
| Not the person you are replying to, but I do (but didn't know
| the term, thank you!)
|
| I find that teams and products with negative pun acronyms
| fail for example. I find that the trend of the name Lilith
| being popular correlates with abortion rates (in the bible,
| Lilith killed unborn babies) etc etc etc
| Lapsa wrote:
| Web5 is all the rage
| thurn wrote:
| This just feels like taking two unrelated things and bunching
| them together under a label in order to be provocative and imply
| that software 2.0 will "replace" existing code.
|
| If you surveyed e.g. all of the code Google has in their piper
| repository, you would find significantly less than 1% of it could
| be replaced by even an extremely good neural network.
| startupsfail wrote:
| It is more or less Software 3.0 now, DNNs, if compared to LLNs
| are like analog computers, if compared to VLSI CPUs.
| 8f2ab37a-ed6c wrote:
| Wonder if / how Andrej's take has changed over the last 6 years.
| thundergolfer wrote:
| He gave the keynote talk about this at TreeHacks 2023 on
| Friday. Another commenter in this thread gives his updated take
| and a link to a relevant tweet.
| leetharris wrote:
| He talked a little about this on Lex Fridman's podcast about 6
| months ago. I'd recommend listening to him discuss it as I will
| butcher a TL;DR.
| umeshunni wrote:
| Link to Podcast: https://www.youtube.com/watch?v=cdiD-9MMpb0
|
| Neural Networks:
| https://www.youtube.com/watch?v=cdiD-9MMpb0&t=58s
|
| Language Models: 41:50
| https://www.youtube.com/watch?v=cdiD-9MMpb0&t=2510s
|
| Software 2.0:
| https://www.youtube.com/watch?v=cdiD-9MMpb0&t=3944s
| a-dub wrote:
| machine learning is good at end to end closed loop analog signal
| analysis problems (decode/translate/synthesize) and electrical
| engineeringish tasks, as it always has been ... but i'm not
| convinced that it is yet good at the more needly real world
| problems that most software faces.
|
| so yes, sure, the best speech recognition algorithm will remain
| something ee-ish, and will probably make use of highly parallel
| numerical computing and data driven optimization based solution
| finding... but i think whether or not that will be the road to
| correct implementation of entire discrete information systems
| with all of their knotty discrete rough edges remains to be
| seen...
| marcus_holmes wrote:
| I think it's about fault-tolerance.
|
| So "programming" in the Software 2.0 world is basically
| training a model. Which isn't a task with an end. You stop when
| you're bored, or when it passes a given level of accuracy, not
| when it's "completely trained" because it will never get to
| 100% accurate.
|
| In e.g. speech recognition, handwriting recognition, speech
| synthesis, drawing pictures, writing a response to a human's
| question, all those messy "organic" problems, this is fine. A
| 99.9% accuracy rating at e.g. speech recognition is better than
| humans do.
|
| But there are problems that absolutely need 100% accuracy, and
| you can only get that if you code it up the old-fashioned way
| (though probably not Agile - I love my iterative development
| cycles but they're equally prone to not quite getting it 100%
| right).
| a-dub wrote:
| mmm... i disagree. fault tolerance is more of a reliability
| property that sits outside of the performance of a discrete
| logic or signal processing system, it describes how well a
| system can remain functioning in the face of adverse
| conditions like internal failures, hardware substrate
| failures, dependency failures, adverse inputs, etc.
|
| but... yeah, some problems have continuous performance
| variables (typically ee'ish) and others have discrete ones
| (typically cs'ish). the discrete account ledger either
| computes the correct value or it does not, where many signal
| processing problems have to contend with noise and are
| allowed to produce a wide range of noisy outputs and
| therefore their measures of correctness are based on
| statistical arguments.
| veselin wrote:
| The interesting part of that prediction is that depending on how
| you read it, you may say it failed embarrassingly, or you may say
| it predicted the current reality fairy well.
|
| The next software as a differentiable thing that is the program
| has certainly failed. But now, there are amazing opportunities to
| connect text-to-text models to other another, to search engines,
| that it is likely to become a new programming.
| juliangmp wrote:
| lol is this satire?
| throwaway744678 wrote:
| My bullshitometer was ringing strongly reading this article.
| bjornlouser wrote:
| "Unfortunately, this interpretation completely misses the random
| forest for the decision trees."
|
| Nice one Andrej!
| tkiolp4 wrote:
| I think AI like ChatGPT and similars will become what APIs are
| today. Just that. Today we wire together a crypto library with
| some JWT library with some Facebook API using some axios library.
| We write the code in between. Since we are able to do more, user
| requirements get more complex, and so software engineers are in
| more demand. Sure thing, what required 10 engineers in the past
| (1980) can be done by two today.
|
| In the same sense, in the future we will be wiring together AI
| APIs (probably because it will be cheaper to wire together
| manually N AIs than to write one that is the sum of the N AIs).
| Since we'll be able to do more, user requirements will get more
| complex... and so the demand for software engineers will go up as
| well. In the future only a couple of engineers will be needed
| when today we need 10.
| nice_byte wrote:
| it's all very depressing.
|
| still, hoping that the future predicted here is 25-30 years out
| nice_byte wrote:
| to be clear, i don't think that the current crop of llms can
| write better or even good enough software. it may well be that
| llms won't be the cornerstone of the future predicted in the
| article. still, there is a huge economic incentive to replacing
| "manual" programmers, and some attempt is bound to succeed
| eventually.
| barrenko wrote:
| Based on what I can achieve with GPT in a non-Python language
| as a junior junior developer, I really see almost no limit to
| what an CS educated experienced mid developer could achieve
| with GPT and Python experience.
|
| Yes there is a lot of hand-holding and guardrailing but
| still, it's kind of insane.
| [deleted]
| nice_byte wrote:
| my experience is that chatGPT in its current form is much
| more of a hindrance than help. it takes longer to achieve
| the same desired result than just writing it directly. it
| seems really good at refactoring type tasks (i.e. rename
| method, move this block to a different routine, etc.) but
| it needs real ide integration for that to be useful.
| adregan wrote:
| I found it interesting that most of the list of benefits sound
| like they could be describing Forth.
| ansgri wrote:
| I was more excited by the concept of Software 2.0 when it meant
| differentiable programming -- i.e. ordinary programs with logic
| etc, but everything is differentiable, optimizable and can
| gradually be replaced by opaque NN-based components. That would
| be largely compatible with current mainstream software
| development and allow gradual transition to highly-efficient
| domain-specific architectures.
|
| Instead we got this LLM-based "paradigm-quake".
| koonsolo wrote:
| Can I see GPT as some sort of lossy compression?
|
| You feed it loads and loads of data, where the neural net
| basically compresses it all into the structure and weights.
|
| Then you give it some (un)compressed part, and it gives you the
| other.
| ArchD wrote:
| The "concrete examples" he gave in the "Ongoing transition"
| section are traditional AI tasks. What about other tasks like
| program compilation and theorem proving that require more hard-
| core logic? I will be more convinced if neural networks can
| reliably do those tasks. If not, at best a human will still need
| to manually break the high-level task into smaller subtasks to be
| solved by neural networks and then somehow glue the parts
| together.
| darksaints wrote:
| To contribute my personal lowbrow dismissal, this is the guy in
| charge of computer vision at Tesla, where cars have had a habit
| of doing absolutely bonkers behavior that doesn't make any sense,
| when faced with completely normal phenomenon like lanes that fork
| into two.
|
| His single sentence caveat about how Neural Networks can fail in
| unintuitive and embarrassing ways is the understatement of the
| century. I'd like to add that Tesla still hasn't solved that lane
| forking problem even eight years since it was first identified. I
| guess just throw more data at it, and eventually it will get
| better? At what point does the belief that things will get better
| with more data fed into the same algorithm become a religious
| creed?
|
| Neural Networks are significant advances in the state of not just
| machine learning, but the world as a whole. But the caveat that
| we don't really understand what they're doing is the whole
| fucking problem. Until Neural Networks can take advantage of,
| constrain to, and augment human models, they don't have a
| snowballs chance in hell at replacing the types of software we
| rely on the most.
|
| Until then, you'll just create a massively inefficient system
| where the neural network writes the software but you spend 10x on
| engineering your training datasets so that your brilliant neural
| network knows that it is better to commit to one of two lanes in
| a forked road than it is to crash into the concrete lane divider.
| Or to not be racist. Or to not go haywire because of a sticker on
| a stop sign.
| hooande wrote:
| "Neural networks are bad because some times they fail" is a low
| brow dismissal, indeed.
|
| What matters is the percentage accuracy. A black box with a 10%
| failure rate is better than a fully explainable system that
| fails 20% of the time. Explanations make us feel better and
| they can be very important. But for most cheap, repeated
| processes they aren't necessary. Not to mention that neural
| networks can be tested and interrogated in ways that other
| systems cannot.
| whiplash451 wrote:
| If the system is << fully explainable >>, you have a serious
| shot at fixing the 20%.
|
| Much more so than fixing the 10% of a neural net.
| darksaints wrote:
| That is absolutely the worst possible way to read my comment.
| They're not bad because sometimes they fail, they're bad
| because _sometimes they fail and we know exactly what is
| wrong with them, but we can 't fix them because they don't
| take advice_.
|
| There is no way to tell the Tesla vision NN, "hey, when you
| see this pattern and you're confused about which path to
| take, it is better to take one incorrectly than it is to run
| into a concrete divider". We know exactly what the problem
| is, but there is no interface with a NN to tell it to do
| something, other than to just keep training it with more
| data. And once you realize that your only interface to get
| better outcomes is to wildly manipulate the training dataset,
| then you haven't made software engineering better, you've
| made data engineering worse.
|
| Take notice of something important: all of the domains where
| Neural Networks have been wildly successful are domains that
| are wildly underspecified. Take language for example. Grammar
| rules, vocabulary, pronunciation, and even meanings of words
| are constantly changing. There is no possibility of ever
| having a formal definition of _any_ language, let alone all
| of them. Or vision...where the only formal definition of
| anything is what color of light it reflects in a particular
| angle. Again, no formal definition of anything.
|
| But the shortest path from A to B? That problem has a formal
| definition, and no neural network has even come close to the
| accuracy of A star or Djikstras. The minimum cost solution to
| a Multi-Commodity Flow Problem? That problem has a formal
| definition, but no Neural Network has come close to the
| accuracy of a Simplex method's solution.
|
| Tautological arguments about percentage accuracy might give
| the edge to Neural Nets in some domains, but not all of them,
| and for that reason they completely miss the point.
|
| 1. Percentage accuracy is only half of the thing that
| actually matters. Without the cost of being wrong taken into
| account, percentage accuracy will totally fuck you over hard.
| Here's a game: you can choose between two algorithms, one
| that has 90% accuracy with a 10% chance of smelling gross, or
| a 99.99% accuracy with a 0.01% chance of your body been
| shaved down to bone over a thousand cuts from a vegetable
| peeler. Which would you choose?
|
| 2. Sometimes absolute accuracy matters. We have formal
| systems for absolute accuracy. We have symbolic logic for
| absolute accuracy. We have deterministic systems for absolute
| accuracy. If the best that I can get from a neural network is
| a percentage accuracy, then it has already failed a test of
| general applicability. How many years and how many computers
| and how much data would we have to feed into how big of a
| neural network in order to get to E = MC^2 with perfect
| accuracy?
|
| 3. Even if percentage accuracy matters, time-relative
| accuracy matters even more. With a Neural Network, if you
| need to get better accuracy, how do you do it? You should see
| actual machine learning practitioners try to solve these
| problems. They literally try to deconstruct the black box,
| trying to figure out how different neurons are weighted, and
| what input data can be altered to result in a different
| weighting. It's a clusterfuck, and it slows progress to a
| halt. We've known exactly what was wrong with the Tesla
| Vision NN for over half a decade, but actually fixing it has
| completely stalled because of the fact that it is a black
| box, and can only be fixed like black boxes. This is systems
| theory 101: you can't fix systems that you can't understand.
| ianstormtaylor wrote:
| > What matters is the percentage accuracy. A black box with a
| 10% failure rate is better than a fully explainable system
| that fails 20% of the time.
|
| That's not true at all, it depends on the use case.
|
| What actually matters is the desired percentage of
| acceptance.
|
| For many critical path use cases you'd much rather have
| something fail twice as often but understand why it failed so
| that you can correct the issue and resubmit the input. Error
| observability is an important feature that's taken for
| granted in many systems. It all depends on what the system is
| used for--how important it is to be able to get to correct
| results, and what the consequences for failure are. The
| biggest danger of neural networks is in people that don't
| understand this nuance and apply them in a blanket way in all
| systems.
| asah wrote:
| sorry, hard disagree: explainability leads to reproducibility
| and predictability, leading to control. Unexplainability
| leads to chaos.
|
| old adage: if a bug can be reproduced then it's only a matter
| of time before it's understood and fixed. If a bug can't be
| reliably reproduced (Heisenbugs) then repair time is
| unbounded.
|
| (that said, humans are perfectly capable of creating
| inexplicable and irreproducible bugs - for example, in
| multithreaded code)
| didntreadarticl wrote:
| No longer works for Tesla. Now works for OpenAI
|
| https://twitter.com/karpathy/status/1623476659369443328?lang...
| daveslash wrote:
| Re >> _" At what point does the belief that things will get
| better with more data fed into the same algorithm become a
| religious creed?"_
|
| Oh, ye of little faith! It is heresy to criticize our new
| religion! ~Some AI consulting firm or AI "thought-leader"
| probably.
|
| Edit: I'm sure there are some useful use-cases, but I'm not an
| unquestioning devout adherent. That said, I should probably
| learn more about it just so I can intelligently defend the use-
| cases in which it _doesn 't_ make sense.
| zaptrem wrote:
| Lanes forking issue has actually been solved via a completely
| new perception and planning architecture.
| 0xdeadbeefbabe wrote:
| Maybe I missed the quickstart, but is CUDA the next step to doing
| those matrix multiplications. Or do I start with PyTorch or
| TensorFlow? Or do I start by classifying lots of data to train
| the classifier?
| titzer wrote:
| > Think about how amazing it could be if your web browser could
| automatically re-design the low-level system instructions 10
| stacks down to achieve a higher efficiency in loading web pages.
|
| This sounds great in theory, but in practice, a system that has
| that much dynamic adaptation has brutally steep performance
| cliffs and is massively complex. I for one, will be opting out of
| that giant vertical slice of hell. This is one of the _good_
| reasons for having layers: separate failure zones, separate
| levels of abstraction--true reuse and modularity. Bugs break all
| that.
|
| And no, given the hallucinations of large models just in the
| natural language space, I do not want to reason through the mad
| ravings of a tripping AI to debug a monster pile that happens to
| make web property X go 10% faster.
| samstave wrote:
| ELI5 what _" 10 stacks down"_ means?
|
| I havent heard this phrase before.
| Ruq wrote:
| Related: The OSI Networking Model defines 7 layers (or
| stacks) to clearly define and separate the duties of each
| level of the networking. Having clear levels makes
| troubleshooting and debugging layers.
|
| In practice there's only 4 or 5 layers depending on who you
| ask.
| b0afc375b5 wrote:
| When I read anything about the OSI model it always reminds
| me of this article
| https://news.ycombinator.com/item?id=26607983
|
| > I have said before that I believe that teaching modern
| students the OSI model as an approach to networking is a
| fundamental mistake that makes the concepts less clear
| rather than more. The major reason for this is simple: the
| OSI model was prescriptive of a specific network stack
| designed alongside it, and that network stack is not the
| one we use today. In fact, the TCP/IP stack we use today
| was intentionally designed differently from the OSI model
| for practical reasons.
| samstave wrote:
| Thanks - I have always mentally separated 'stacks' from
| 'layers', specifically in networking.
|
| I imagined stacks as tech1+tech2+techN
|
| I just hadnt heard that phrase before...
|
| 30 years deep in ops.
| titzer wrote:
| I assume that the author means "Framework XYZ in TypeScript
| running on JS running on V8, which is written in C++,
| compiled by clang, running on Linux with glibc N, running on
| VMWare, running on x86, running on the Rapture Cove
| microarchitecture." I don't know if there are 10 I could
| name, but stacks are deep.
| sva_ wrote:
| OSI alone would give you 7 stacks
| samstave wrote:
| True, but when learning OSI in 90s - they were always
| refered to as 'layers' of the same protocol.
|
| Stacks where completely stand-alone applications
| connected which were not layers of the same
| protocol/system.
|
| I think that the simple way is to refer to them as stacks
| now, but if you were raised on LAYERS - refering to OSI
| as 'stacks' feels foreign.
|
| It appearsthat as we atomoze / containerize various teck,
| we now thing of them as 'stacks' rather than layers.
|
| It seems that 'layers' are now 'services' rather.
| [deleted]
| diydsp wrote:
| Yup, that graphics stacks could easily add a few layers for
| ya! When I was at a gfx card mfr, there were many
| compatibility and shim layers :)
| kyle-rb wrote:
| I'm skeptical about ML being a "2.0" when we'll never stop using
| the 1.0 version altogether. But maybe he means it in the Python 2
| -> 3 sense, where there's no real expectation that the older
| version will ever stop being used.
| [deleted]
| ThouYS wrote:
| yawn, it's all rubbish
| kabdib wrote:
| "Garbage In, Gospel Out"
| derefr wrote:
| So, why would you want to rely on business logic embodied
| directly as a chaotic ML model (in the Chaos Theory sense of
| "chaotic" -- subtle changes in stateful hyperparameters making
| for discontinuous shifts in output space), when you could instead
| ask the model to precisely describe a rigid decision workflow
| (i.e. "Software 1.0" business logic) it would implement if
| needing to explain its own decision-making process as
| parsimoniously as possible?
| agentultra wrote:
| I think we need to train people to write formal specifications
| before we're going to see machine learning techniques generating
| useful programs.
|
| Sure sometimes an LLM generates a correct program. And sometimes
| horoscopes predict the future.
|
| I will be impressed when we can write a _precise_ and formally
| verifiable specification of a program and some other program can
| generate the code for us from that specification and prove the
| generated implementation is faithful to the specification.
|
| An active area of research here, _code synthesis_ , is promising!
| It's still a long way off from generating whole programs from a
| specification. The search spaces are not small. And even using a
| language as precise as mathematics leaves a lot of search space
| and ambiguity.
|
| Where we're going today with LLM's trying to infer a program from
| an imprecise specification written in informal language is simply
| disappointing.
| ClassAndBurn wrote:
| I agree here. Formal models have to become easier to create
| through.
|
| Today's ecosystem requires advanced knowledge of system design
| and still coding abilities.
|
| To democratize model generation we need a more iterative and
| understandable way of defining intented execution. The problem
| is this devolves into just coding the damn thing pretty
| quickly.
| agentultra wrote:
| For sure! I agree, it needs better languages, education, and
| tooling. It's not about making a hard problem harder; it's
| about making it more accessible and straight-forward to teach
| and use in day-to-day work.
|
| Being more clear and precise in our specifications would only
| benefit us and the AI/ML tool generating the code. We could
| lean more on the correctness built into the entire stack
| rather than having to proof-read a mess of inferred code,
| something we're terribly ill-equipped to do.
| valenterry wrote:
| > Being more clear and precise in our specifications would
| only benefit us and the AI/ML tool generating the code
|
| Good luck with that. We have those languages already. For
| example Idris. It's just that now you are essentially doing
| a lot of math.
|
| And, funnily, I never hear people saying "making [math]
| more accessible and straight-forward to teach and use in
| day-to-day work". I wonder why...
| canes123456 wrote:
| A precise and formally verified specification is WAY harder to
| write then the code for it
| pyrolistical wrote:
| We could prob get LLM to help
| noobker wrote:
| A precise and formally verified specification is usually just
| referred to as "PROD".
| [deleted]
| agentultra wrote:
| Exactly. It seems like we prefer the easy way out. If we
| can't write a precise specification then we don't know
| exactly what we want the computer to do. And so all we can
| get are guesses.
|
| The software industry has gotten this far with very little
| help from the formal methods community... that's changing in
| recent years in certain spaces where errors are magnified by
| scale like cloud computing, etc.
|
| But instead of getting better at writing precise
| specifications we're going to continue to be bad at it and
| hope that an LLM can manage to infer the correct program. It
| might be millions of lines of code but hopefully it does some
| of the things we want most of the time.
|
| _Update_ : To be clear, I'm not saying AI/ML programs cannot
| _help_ us to write programs at all, just that the inputs need
| to be better if we 're going to have any confidence that the
| programs it generates are any better than a horoscope.
| Kinrany wrote:
| The article is not about generating correct programs.
| ansgri wrote:
| We need to train people to write specifications since they seem
| to be the best way to fine-tune the AI performance for the task
| without changing the enormous dataset. Reinforcement learning
| methods can be used to tune the model in production under
| pressure of failing tests (executable specifications). Writing
| specifications to evaluate outcomes should be a better use of
| domain experts time than dataset cleaning.
|
| As for formality, real formal specifications are very hard, and
| LLMs are close to understanding natural language anyway, and
| 1000s of 90%-strict specs are better than 10 provably correct
| ones. So, some sort of legalese for machines will evolve.
| denton-scratch wrote:
| > I think we need to train people to write formal
| specifications
|
| I don't think that is really a matter of training. You have to
| start with people who can think clearly; if they can't think
| clearly, it's hopeless to expect them to produce formal specs.
| Very few people can think clearly about difficult subjects.
|
| Of course, you can train people to improve the clarity of their
| thinking. I think that should be the main purpose of an
| undergraduate degree.
|
| Writing a formal spec is analagous to writing a program; if you
| can't program, your program won't work. So writing a formal
| spec proves that you can think clearly; but if you can think
| clearly, you can write a program without first writing a formal
| spec.
| _flux wrote:
| Writing a formal spec in itself is a way to introduce clarity
| to thinking, because when writing it the spec itself lends to
| asking more and more questions about the things you might not
| have thought of.
|
| More so if there are tools to check your formal spec.
|
| But formal spec can be in higher level than the
| implementation. For example, it could be describing pre- and
| postconditions without actually stating out how to go from
| precondition to postcondition.
|
| I have noticed programming languages don't tend to ask the
| writer the right questions in the same way e.g. TLA+ and TLC
| do.
| valenterry wrote:
| The spec is the hard part. Just look at how many people are
| complaining about languages that use a statically type
| system. I wonder how many business people will then not
| complain about writing clear specs for "AI software". :)
| lpapez wrote:
| > I will be impressed when we can write a precise and formally
| verifiable specification of a program and some other program
| can generate the code for us from that specification and prove
| the generated implementation is faithful to the specification.
|
| I cannot do this, and neither can any of the people I have ever
| worked with. Yet despite that we all call ourselves
| programmers, create value and earn money by writing ill-
| specified, often buggy code. Why would a tool need to be
| formally verified to be considered impressive and/or useful?
| agentultra wrote:
| > I cannot do this
|
| You could if you wanted to. You're smart, inventive, and
| creative. There's nothing stopping you from learning.
|
| > Why would a tool need to be formally verified to be
| considered impressive and/or useful?
|
| Part of it depends on your perspective.
|
| If we assume that an LLM (or some future ML tool based on it)
| is capable of producing code with the same rate of errors as
| a trained, expert human could then it would seem the
| productivity gain is not having to write all of that code
| ourselves.
|
| We already tolerate a certain amount of errors in our
| software and the world has not collapsed. The JVM had an
| error in its binary search implementation that lasted for
| nearly a decade before anyone noticed. They noticed because
| the size of the arrays being used started getting big enough
| that their programs started failing in mysterious ways.
| OpenSSL had a vulnerability that sat unnoticed for more than
| a decade. The cost of errors is not zero but it is tolerated.
|
| However the problem of programming is that we think it's our
| ability to write code which is the problem that is slowing us
| down.
|
| My perspective is that we're not focusing on the problem:
| that it is hard to be precise and write programs that work,
| whole cloth, from their specifications without any errors.
|
| Using an LLM to generate more code has another problem: while
| humans are decent enough at writing code to solve our
| problems, even if our solutions are imperfect, we're far
| worse at _reading_ code and understanding what it does and
| whether it is correct with regards to some specification (if
| there is one).
|
| Empirical studies of large-scale code review are very
| humbling. We can read maybe 200 SLOC every couple of errors
| and have a negligible impact on error rates in the software
| being produced. More than that and the effect disappears.
|
| So now we have LLM's producing code that we know will have
| errors in it. And we have no idea where the error is. It
| could be a trivial error we could tolerate. Or it could be
| another Bar Mitzvah CVE. Hard to say.
|
| Even Betrand Meyer missed an error in a _single-line
| expression_ generated by ChatGPT. He 's way smarter than me.
| I don't see how we'll be able to keep up.
|
| But if we tackled the problem of getting better at being more
| precise with our specifications, I could definitely see how
| having an AI-like system automate code generation being
| really useful. There are plenty of times when working on a
| formal proof where you want to say, "this is obvious!," that
| have a machine verify that for you using the same proof rules
| and tactics you would use. Bonus points if it can explain the
| proof back to you.
|
| I just think we're a long way off from being able to do that.
| naasking wrote:
| We don't write formally verified code because writing the
| code to pass the verification is annoying and hard. If we're
| only writing the formally verified spec and a program or AI
| generated the code, that's a bit easier, though it may have
| to get easier still before it becomes mainstream.
| evrimoztamur wrote:
| Our approach shouldn't be to give up striving for better
| programs, algorithms, and code. Despite our current inability
| (which I don't think is a failure of programmers but business
| decisions), we can nevertheless reject further opportunities
| for decline, in name of profits, right?
| lpapez wrote:
| That is not what I said. Of course we should strive for
| precision, corectness etc. but that does not mean that
| current and future imprecise methods of development do not
| hold any value. There is plenty of value to be gained from
| poorly written software which we will keep writing until we
| get magically verified ones.
| [deleted]
| pulse7 wrote:
| Even without AI the systems have become VERY COMPLEX, hard to
| extend and debug. It is much harder to get started for newbies
| than it was say 25 years ago. If AI is software 2.0 it will be
| EVEN MORE COMPLEX. I guess that Software 2.0 must be something
| which will reduce complexity and not increase it...
| palotasb wrote:
| I think a legitimate criticism of this article is using the
| clickbait "Software 2.0" label for something that's - as very
| well described by the same article - so very different from
| "Software 1.0." Considering ALL software, the intersection of
| uses cases where Software 1.0 and Software 2.0 truly compete are
| a very narrow niche. In the real world Software 2.0 will thrive
| in the next decades but Software 1.0 will be there just as much
| as it is today.
| petilon wrote:
| Machine Learning and Neural Networks are not applicable for
| solving all kinds of problems. In fact, in Tesla AI Day Elon Musk
| remarked that "I discourage the use of machine learning... unless
| you have to use machine learning, don't do it" [1]. Getting
| machine learning to work right is hard. If it can be avoided it
| should be.
|
| So to cast this as Software 1.0 vs 2.0 doesn't make sense. There
| is a class of problems where neural networks work better.
| Everywhere else we will continue to use traditional code.
|
| [1] https://youtu.be/j0z4FweCy4M?t=9319
| andrewfromx wrote:
| [flagged]
| l33t233372 wrote:
| > The process of training the neural network compiles the
| dataset into the binary, which is the final neural network.
|
| If I didn't know better, I would think the author of this
| sentence didn't know what any of those words mean.
| sebzim4500 wrote:
| The analogy makes sense to me, might be missing something
| though.
| actionfromafar wrote:
| I read it as, people will direct AI to search for solutions,
| then use these refined solutions to search for more
| solutions. A bit like how we use libraries and packages, and
| improved languages, to enhance traditional programming
| practices.
|
| Taking an extreme helicopter view I think I can see it, but
| on the other hand I'm not convinced it's a very interesting
| observation. Throughout history we used machines to make more
| complicated machines.
|
| Edit with this quote:
|
| _Because you only have to provide Software 1.0
| implementation for a small number of the core computational
| primitives (e.g. matrix multiply), it is much easier to make
| various correctness /performance guarantees._
|
| This is true, but it raises another question which is (to me)
| _comically_ hand waved aside:
|
| how do you make correctness guarantees of the output of the
| neural net? It's not addressed, probably because it's very
| hard to do so.
|
| It's like the NAND gates inside CPUs and GPUs. Since they are
| so simple building blocks, they are very easy to verify.
|
| It does not follow that the business logic I write to run on
| these things, is easy to verify.
|
| The same goes for neural nets, but more so.
|
| I'm not saying these new AI tools aren't useful, they are.
| But it's easy to misunderstand what they can do.
| govg wrote:
| The "correctness" of neural networks is a vasy field of
| active research, with many prominent minds in deep learning
| as well as traditional CS theory working on it. We have
| many results for small scale networks, but I don't know of
| any results that can "prove" the correctness of an image
| classifier, for example. After some point, the correctness
| of such methods becomes very ill-defined, since unlike the
| normal programming world where everything is mostly in
| binary, here you will have to answer questions with some
| variation of "this is 98% likely" with no scope for 100%
| certainity.
| packetlost wrote:
| They definitely don't. There's no way GPT-3/4 replaces
| software development, but it will save us some typing with
| fancy autocomplete and prose-ey documentation!
| anthomtb wrote:
| Maybe if I have ChatGPT write my docstrings as sci-fi my
| colleagues will actually read them.
| eddsh1994 wrote:
| Karpathy wrote this in 2017
| [deleted]
| henrik_w wrote:
| There is also this in a similar vein:
|
| "The End of Programming"
|
| https://cacm.acm.org/magazines/2023/1/267976-the-end-of-prog...
|
| Discussion:
|
| https://news.ycombinator.com/item?id=34087000
| piokoch wrote:
| So we will have some abstract language that will allow business
| people to define what software should do and on that base AI will
| generate source code and working project.
|
| Wait, wait, I have heard that it was called BPMN and it generated
| underneath Enterprise Java Beans and it was working amazingly,
| all software is written today, right? Right?
|
| But, but nobody touches this crap besides generating pictures to
| show on Powerpoint slides. Because writing software is circa 10%
| of all effort, specification, legal stuff, maintenance, avoiding
| technical debt, proper test cases, anomaly testing, performance
| testing the right stuff. That is hard, that matters. Software 2.0
| is barking the wrong tree.
| roflyear wrote:
| BPMN is so far gone, when execs suggest using it they ONLY mean
| for the workflow charts - not actually executing workflows or
| allowing business folk to create new workflows!
|
| So it is really confusing because they are only evaluating
| "pretty pictures" and you're evaluating the technical
| requirements of the tool (to do what it was built for) and you
| mismatch.
| marcus_holmes wrote:
| And COBOL before that. The goal of COBOL was to allow a non-
| technical business person to describe the problem in English
| and that description could be used as the basis of the program.
|
| Needless to say it didn't work. Accurately describing what a
| program should do is what a programmer does. We're not telling
| the computer what to do (move this value from memory into this
| register, etc). We're describing a program's behaviour.
|
| So I think we'll just get another language that is a good fit
| for describing what the program should do, and instead of
| compiling that to machine code, it will be used to train a
| model.
| hwayne wrote:
| COBOL actually worked pretty well for its time. The ultimate
| reason COBOL died was because it was a first generation
| language that nobody really iterated on. It's similar to how
| most of us don't use FORTRAN, ALGOL, or LISP 1.5. Unlike
| those, COBOL didn't have the backing of the "tech world" (at
| the time, IBM and universities), so it doesn't have
| descendant languages we _do_ use.
| marcus_holmes wrote:
| I don't understand why it didn't have the backing of the
| "tech world" and yet billions (if not trillions) of lines
| of COBOL are still being executed today?
|
| When I first joined the industry, back in the early 90's,
| COBOL was very much the premier business coding language,
| and I think that only changed with the arrival of Java.
|
| Can you explain further?
| dgb23 wrote:
| > COBOL actually worked pretty well for its time.
|
| It was still programmers using the language. Same story
| with SQL.
|
| The only things you can give people who don't invest a
| substantial amount of time and effort into the craft are
| markup languages and very high level (configuration) DSLs.
| majkinetor wrote:
| > writing software is circa 10% of all effort, specification,
| legal stuff, maintenance, avoiding technical debt, proper test
| cases, anomaly testing, performance testing the right stuff.
| That is hard, that matters. Software 2.0 is barking the wrong
| tree.
|
| Exactly. For serious projects (not to do lists) u will need
| humans, or equivalent (AI that is grown like one).
| deeviant wrote:
| The Apple Newton table computer was a complete disaster and
| thus all tablet computing devices thereafter will always suck,
| yes?
| thefourthchime wrote:
| Do you know that scene in "Office Space" where that guy says:
|
| "What is it that you do here?" and he says:
|
| "I take the specifications and hand them to the software
| engineers".
|
| I think our jobs are about to approach that a lot more closely
| than you might think. Our jobs will be effective translators
| between what the business wants and what the AI outputs.
| marcus_holmes wrote:
| Switch "AI" and "compiler" and you describe exactly what a
| programmer does.
| roncesvalles wrote:
| Exactly. ChatGPT is just a compiler for the low-precision
| programming language known as English.
|
| In programming we can always make a tradeoff between
| precision and effort, e.g. by importing libraries or using
| no-code or code-generation tools. ChatGPT is just one more
| point on the same tradeoff curve. It hasn't meaningfully
| moved the curve itself.
|
| Moving the curve would mean making it less effortful to
| write code at the same level of precision.
| deeviant wrote:
| Need to add "AI Whisperer" to my resume...
| layer8 wrote:
| Tangentially, it would have been fun if Copilot and ChatGPT
| existed two decades ago when EJB was all the rage.
| Anuiran wrote:
| A step beyond that, Software 2.0 and beyond does not generate
| source code. Your app simply lives in the LLM.
| goatlover wrote:
| I'm wondering how the LLM will be able to make use of
| hardware, or handle high demand. Can it be relied on to have
| data integrity?
| lysecret wrote:
| I think a lot of people are interpreting this (implicitly) with a
| kind of Generative AI lense. However, this article was much more
| about "classical" ML type work, e.g. training your own Neural Net
| while I think the paradigm seems to be shifting towards "Zero or
| Few-Shot" learning e.g. there is no more fine tuning or even
| complete training step involved.
| Iv wrote:
| I remember a pretty old interview with Linus Torvalds where they
| are talking about object oriented programming. The interviewer
| asked him if he expected a similar paradigm change in the coming
| years and I remember being surprised by his answer: (quoting from
| memory) No, I don't see anything big coming. Probably the next
| change will be caused by AI.
|
| Yes, differentiable code is already a new paradigm (write a
| function with millions of parameter, a loss function that
| requires more craft than people realize and train). That has a
| property that used to be the grail of IT project management: it
| is a field where, when you want to improve your code performance,
| you can just throw more compute at it.
|
| And I think that the clumsy but still impressive attempts at code
| generation hints at the possibility that yet another AI-caused
| paradigm change is on the horizon: coding through prompt, adding
| another huge step on the abstraction ladder we have been
| climbing.
|
| Forget ChatGPT coding mistakes, but down the road there is a team
| that will manage to propose a highly abstract yet predictable
| code generator fueled by language models. It will change our work
| totally.
| zelphirkalt wrote:
| We might get into another slump of efficiency as an outcome of
| this, again stopping us from making the most of the hardware
| and computational resources we have, due to prompts being too
| unspecific. Did not specify the OS your code will run on? Well,
| we better use this general cross OS available library here,
| instead of the optimized one for the actual OS the thing will
| run on.
|
| The same mentality, that causes today's "everything must be a
| web app", will caused terrible inefficiency in AI generated
| (and human prompted for) code. In the end our systems might not
| be more performant than anything we already have, because there
| are dozens of useless abstraction layers inserted.
|
| At the same time other people might complain, that the AI does
| not generate code, that can be run everywhere. That they have
| to be too specific. People might work on that, producing code
| generators which output even more overheady code.
|
| At least some of that overhead will slip through the cracks
| into production systems, as companies wont be willing to invest
| into proof-reading software engineers and long prompt-generate-
| review-feedback cycles.
| mattgreenrocks wrote:
| This feels like re-discovering DSLs, except the syntax is
| English, and the implementation is a blackbox.
| giobox wrote:
| I don't feel a comparison to DSLs works here at all. If you
| are just using plain human language, is a comparison to DSLs
| apt?
|
| The point of DSLs are to provide a deliberately limited-scope
| language optimised for a specific problem or problem domain.
| LLMs that use general human language is like the furthest
| opposite of a DSL - its the broadest scope language for
| describing any problem, and they try to solve them all.
|
| Also, few popular DSLs are truly blackbox in the sense
| chatGPT is - many of them have exposed source or even line-
| by-line debuggers available. There are a ton of other reasons
| this doesn't make sense to compare.
| umeshunni wrote:
| DSL being https://en.wikipedia.org/wiki/Domain-
| specific_language
| revskill wrote:
| I need an AI to teach me which kind of neuron net architecture to
| use based on input and output.
| IncRnd wrote:
| ChatGPT will happily answer that for you.
| revskill wrote:
| It doesn't know it's happy or not ?
| IncRnd wrote:
| I apologize for the confusion. In this context, the word
| happily is somewhat equivalent to "perform without
| reservation". The comment's sentence can be rewritten as,
| "ChatGPT will answer that for you."
|
| The difference is a slight loss of emphasis that had been
| meant to show chatgpt doesn't require many prompts in order
| to convince the model to answer the situation that you had
| posed. The word "happily" wasn't used in the sense of
| chatgpt experiencing emotions
| jejeyyy77 wrote:
| But all of these models are learned from code written by humans.
| We've nowhere near enough "training data" to be able to generate
| all the software needed moving forward.
| blowski wrote:
| Aren't we on something like v15.0 by now?
|
| First there was software of the Enigma machine variety, then
| assembly, then massive IBM machines running COBOL, then C, then
| Ruby/PHP/Python. We could also talk about how networking and
| persistence fundamentally changes software. Each of those is a
| big iteration in itself, probably just as big as moving to ML
| generated code.
| jvanderbot wrote:
| Oh boy. Hate to be that old guy, but I've heard this one before.
| bitwize wrote:
| Except it's really happening this time?
| ska wrote:
| Remains to be seen, honestly. Clearly a noticeable leap has
| been made, but it's far short, so far at least, of many
| claims being made.
|
| For what it's worth, "except it's really happening this time"
| has also been said before...
| cgearhart wrote:
| This is quite an interesting take... I'm not yet convinced that
| neural networks are "computing" in the classical sense, but maybe
| that's moot.
|
| More interestingly, this makes me wonder if there are some Godel-
| like proofs waiting out there that limit the capabilities of
| efficiently-optimizable programs. What new kinds of undecidable
| or uncomputable functions exist in the subspace of programs that
| an NN can learn? Would be exciting to find out.
| pier25 wrote:
| It's an interesting take which makes total sense with stuff like
| self driving. But I have a hard time picturing this approach for
| making UIs, data models, cruds, gameplay, audio, graphic engines,
| etc.
|
| Nothing is really a silver bullet so I guess the future of
| programming is really hybrid. Stuff like Github's Copilot.
| ahussain wrote:
| Why do you think neural nets will be bad at making UIs, data
| models, cruds, gameplay, audio, graphic engines, etc? There are
| already some compelling examples of AIs making progress on
| those kinds of tasks.
| pier25 wrote:
| Yes, AIs can certainly generate stuff. But can an AI generate
| the specific stuff I need for this specific use case?
|
| Probably not, although I'd be more than happy to be able to
| delegate work to our robot overlords.
| thuuuomas wrote:
| "we should obviate things I find distasteful, but my livelihood
| must be preserved"
| pier25 wrote:
| Not really. I'd be extremely happy to be able to produce the
| same results with less work.
| mvkel wrote:
| This is the right take on the future. Unsurprising that it came
| from Karpathy all the way back in 2017.
|
| The quality of the code is irrelevant, as the point of Software
| 2.0 is that it's another layer of abstraction on top of
| traditional code.
|
| "Coding" becomes "I need something to do a thing," rather than
| "def doSomething: ..."
|
| As long as the output gives you what you need, the code quality
| ultimately is an efficiency play. But as AI coding improves, it
| can refactor itself, so it's a short-term problem.
|
| In my own experience coding with an "AI assistant," I've been
| able to mentally stay in "architecture mode," which makes me feel
| twice as creative, twice as productive. That alone is a net
| positive.
| goatlover wrote:
| > The quality of the code is irrelevant,
|
| Until it needs to be maintained, or has weird bugs.
|
| > As long as the output gives you what you need, the code
| quality ultimately is an efficiency play. But as AI coding
| improves, it can refactor itself, so it's a short-term problem.
|
| Not sure how this is going to work on large codebases.
| guhidalg wrote:
| Perhaps he's right but it's a very depressing view of the
| future.
|
| > "Coding" becomes "I need something to do a thing," rather
| than "def doSomething: ..."
|
| More likely, corporate overlords will decide that you cannot
| just "do a thing" but rather that you are allowed to do X, Y
| and Z things for which they have pre-trained commercial models
| for.
|
| > As long as the output gives you what you need, the code
| quality ultimately is an efficiency play. But as AI coding
| improves, it can refactor itself, so it's a short-term problem.
|
| Have you ever debugged a problem with generated source code? Or
| even a compiler bug? Now imagine leaving your AI to go find the
| bug or iterate until the bug disappears hehehe...
|
| > In my own experience coding with an "AI assistant," I've been
| able to mentally stay in "architecture mode," which makes me
| feel twice as creative, twice as productive. That alone is a
| net positive.
|
| IMO, if your work benefits from an AI assistant then your work
| is to produce many lines of code and you would benefit equally
| from creating high-level abstractions than from using pre-
| trained black box models (or as some call them "new hires").
| fhd2 wrote:
| I doubt the premise that everything will be NNs, but I can't
| predict the future any more than anybody else.
|
| What I find interesting in this imagined future is that problem
| definition usually happens, in my experience, while attempting to
| encode it, removing all ambiguity. If we skip that step n years
| from now, will we still even understand the problems we try to
| solve? Sounds scary to have systems where we can neither reason
| about solution nor problem.
| [deleted]
| [deleted]
| ramesh31 wrote:
| Really needs a (2017). This stuff has moved at warp speed since
| then.
| jimbokun wrote:
| What's a good heuristic for determining if a problem better fits
| into Software 1.0 or Software 2.0?
| pyrolistical wrote:
| Think of the hello world of machine learning. Handwritten digit
| recognition.
|
| https://youtu.be/aircAruvnKk
| jstx1 wrote:
| It boils down to "do I need ML?".
| binarymax wrote:
| An interesting treatise but I don't fully agree with the
| statement that gathering data or stating a goal is easy. Getting
| quality data is arguably the hardest part of ML. Getting people
| to fully describe their intentions and all edge cases
| ("requirements") in a clear manner is also the one of the hardest
| parts of software development - which is why things like agile
| exist. Maybe at the granular function or module level, but
| business requirements are not easy.
| terminal_d wrote:
| The adoption of the so-called "Software 2.0" depends on how
| influential OpenAI is as a company. The majority of the use-cases
| of GPT-3 are only about "remixing" information, and the
| "notoriety" of the AI is (IMO) mostly a publicity stunt; I'm
| assuming that there are prompts for personalities / jokes / poems
| / etc that are never shown to the end-user and are hailed as an
| "organic outcome" of the model. Obviously, no one can check for
| these right now. Seems to me that microsoft has taken a few
| leaves out of the OpenAI playbook and applied them (rather
| chaotically).
|
| So when you're looking at _actually_ writing software that needs
| to be dependable / modifiable / bug free, you'd need a massive
| overhaul of whatever software stack is being used, so there's
| very little human-assisting "cruft", and instead you'd want a lot
| of supporting material for a model, which might look like
| something written in languages used for formal verification of
| programs.
|
| The promise of GOFAI was about having a human-understandable
| bottom-to-top framework, and the current "AI" paradigm is at odds
| with it. The "formal verification" assumption, then, skews
| towards GOFAI. But since there has to be _some_ human support for
| the current not-there-yet AI to write software, we might see yet
| another abstraction layer based on NN / something newer in the
| years to come.
| willio58 wrote:
| > the "notoriety" of the AI is (IMO) mostly a publicity stunt
|
| Have you _used_ ChatGPT? I mean not just asking it random
| factoids but using it to genuinely help you with something. Are
| you aware that it's hit 100 million users faster than Facebook,
| Instagram, or TikTok did? It's not a perfect product but it's
| hard to argue with those numbers. I work at a startup and most
| people I work with use ChatGPT daily. I'm talking project
| managers, devs, personal assistants, etc. I guess that's all to
| say OpenAI is influential as heck _already_, now imagine 5
| years down the line if they play their cards right.
| terminal_d wrote:
| >Have you _used_ ChatGPT?
|
| I have, and I didn't find it to be useful for anything I did.
| I can do what it does with a search engine and trusty C-f.
| Also, TTS exists.
|
| It's 50% tech and 50% marketing (and I doubt it's 50% tech at
| that), it's not gonna upend anything. Except maybe increase
| the authenticity of online scams and make people get more
| degrees in machine learning. And yeah, make the people that
| rely on it bound as it degrades their skills.
|
| It's basically the "internet is educationally useful"
| argument. At some point everyone's gotta use it but you can
| live without it just fine. And even though people tout its
| usefulness for everything good, the majority of data
| transferred is porno.
| hn_throwaway_99 wrote:
| Hmm, a little frustrated because I feel like a lot of comments
| here are missing the forest for the trees.
|
| For example, as someone who works with financial software, I
| don't see Karpathy's "Software 2.0" replacing, say, account
| ledgering software anytime soon. "Yeah, we calculate our clients'
| balances correctly 99.9% of the time!" isn't going to cut it.
|
| But I don't think that's what Karpathy is arguing. There is a
| large set of problem domains where Karpathy's Software 2.0 _is_ a
| much better solution than what he calls Software 1.0. For
| example, even in finance, stuff like fraudulent transaction
| detection, or financial security software for intrusion
| detection, is very well-suited to Software 2.0.
|
| So yes, I think Software 1.0 will always be around, but I don't
| think it makes sense to use it for domains where Software 2.0 is
| a better fit. What I feel like Karpathy is arguing for is really
| now a recognition that Software 2.0 really is a whole new
| paradigm shift, and we need better tooling (he uses "GitHub for
| Software 2.0" as an example) to support it.
| obviouslynotme wrote:
| I agree that almost no Software 1.0 will be effected by this.
| Software 2.0 will start doing the jobs humans do now because
| Software 1.0 can't, e.g. security guard, customer support, as
| well as your examples.
|
| The real interesting things will be Software 1.0 and 2.0
| working together. You use 1.0 to run and validate the work of
| 2.0 that is guided by prompts. An example of this would be
| using prompts to generate source code that is compiled and
| tested. The real TDD is only writing tests and letting Software
| 2.0 create the code for you. This extends to other work like
| engineering as well.
| claytongulick wrote:
| At some point the complexity of the prompts required to generate
| a program that meets a non-trivial specification become so
| complex as to be indistinguishable from a programming language.
| wvenable wrote:
| The only complete specification is one that can be compiled and
| executed.
|
| AI will take incomplete specifications and guess the rest --
| just like humans do. Whether or not it makes those guesses
| better than a human remains to be seen.
| dangoor wrote:
| (2017) Interesting to see that this is from 2017, given what the
| AI explosion has been like recently.
| isoprophlex wrote:
| Yes, great foresight. One thing didn't stand the test of time:
|
| "You'll notice that many of my links above involve work done at
| Google. This is because Google is currently at the forefront of
| re-writing large chunks of itself into Software 2.0 code."
|
| IMO google is dropping the ball pretty hard right now when it
| comes to AI.
| langitbiru wrote:
| Google is the Xerox of AI.
| Tepix wrote:
| I think it's too early to tell, neither Microsoft nor OpenAI
| have something to show that is trustworthy. If that is their
| goal they can still be first.
| BulgarianIdiot wrote:
| I hope he's smarter at machine learning than at blog posts.
|
| Yes, a lot of software will include NN models. Traditional
| software is going nowhere, because it's the only means of being
| 100% sure of what the outcome will be, non-probabilistically.
|
| Neural Networks are a tool for solving probabilistic, fuzzy logic
| problems.
| valenterry wrote:
| Yeah I agree. I can see those models helping us improve
| productivity quite a bit on the side of (still) writing code.
| Essentially it is a much better form of context aware c&p from
| stackoverflow.
|
| And then, as you say, there will be certain parts where those
| models are actually gonna be integrated in software in one way
| or the other. And I think this is powerful. It would be awesome
| if I can just toss certain problems to the business folks and
| empower them to figure out the solution AND implementation by
| themselves.
|
| But even that will probably take quite some time.
| travisjungroth wrote:
| Your upper bound for the probability of software written by
| people being correct is higher than mine.
| ricw wrote:
| This thesis seems very credible, but it misses one downside of
| his "software 2.0" definition: you need significant amounts of
| data to train the neural networks. Most problems do not have this
| amount of data. Not even close.
|
| So yes, this will revolutionize and enable unseen performance in
| the few areas where there is significant data. For all the rest
| it'll be business as usual.
| hn_throwaway_99 wrote:
| > you need significant amounts of data to train the neural
| networks. Most problems do not have this amount of data. Not
| even close.
|
| Going to push back against this one. I think we have a lot more
| training data than most people realize. I wrote this comment
| yesterday, https://news.ycombinator.com/item?id=34862450, about
| how a large government contractor is using ChatGPT to generate
| first drafts of responses to government RFPs.
|
| Now, most of these RFPs are in very specific areas,
| technologically speaking (e.g. specific technologies around
| cloud network security, for example). These folks were actually
| blown away by how technically accurate ChatGPT was on many
| different areas, even very specific niche areas, and even
| considering ChatGPT's view of the world hasn't been updated
| since late 2021.
|
| Again, the first draft needed to be edited, but there are is a
| huge amount of data out there that ChatGPT is able to use
| coherently on even niche, esoteric topics.
| impalallama wrote:
| I can see this Software 2.0 being used for content moderation in
| the near future, if it isn't already.
| davedx wrote:
| There's a huge amount of pessimism and downright snark in the
| comments. I see a call to action to improve the state of the art
| of the tooling universe around ML, to make it even more broadly
| applicable, more understandable, and probably unlock huge
| economic value.
|
| How about a more glass half full take on progress?
| pelagicAustral wrote:
| I don't think product managers a very big on 'glass half full'
| nor bank institutions in 99.99% accuracy.
| thisoneworks wrote:
| You want guaranteed specific behavior from a software system
| (that's what SLAs and contracts are for) and easily reason about
| it so you can hire a college grad to tweak it. And that's not
| even talking about datasets, you can only train something
| accurate if you have enough good data on it? I can only see
| software 2.0 being a better autocorrect for internet scale
| usecases (in the short term atleast, until the next breakthrough)
|
| This also reminds me of a quote from the Book I'm currently
| reading (Practical Wisdom by Barry Schwarz)
|
| "Most of us think about empathy as a "feeling" or an "emotion."
| It is. To be empathetic is to be able to feel what the other
| person is feeling. But empathy is more than just a feeling. In
| order to be able to feel what another person is feeling, you need
| to be able to see the world as that other person sees it. This
| ability to take the perspective of another demands perception and
| imagination. Empathy thus reflects the integration of thinking
| and feeling."
|
| "Mind reading" is another way to put it
| (https://yosefk.com/blog/people-can-read-their-managers-mind....)
| - this practical wisdom + mind reading is basically the salient
| human feature that NNs would never be able to replace so you
| would always have humans in the system.
| mikewarot wrote:
| I reject this premise. A neural network is not a program, they
| are essentially huge multi-sheet Excel workbooks that compute a
| score for a given set of inputs.
|
| Unlike the Excel workbooks made by domain experts, it's almost
| impossible to even find out what the function of any given cell
| is in the overall computation. See the effort it took to find the
| "neuron" responsible for a/an differences in GPT-2.[1]
|
| Neural networks have their roles as a black box, but they are not
| programs, constructed with intent by humans, to be read by other
| humans, and compilers.
|
| [1] https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-
| found-a...
| kklisura wrote:
| If assume the article is correct and there will be Software 2.0,
| what's the cost of running it vs cost of running Software 1.0?
| The reason I'm asking is given this quote "when the network fails
| in some hard or rare cases, we do not fix those predictions by
| writing code, but by including more labeled examples of those
| cases." - is it more costly to do "curating, growing, massaging
| and cleaning labeled datasets" and ultimately training neural
| networks than to just write code? Maybe not for small NNs, but
| for DNNs?
| zelphirkalt wrote:
| The higher level the labeled examples become, the more effort
| is needed to make them in the first place.
|
| For example take a website. How are we going to provide enough
| examples of websites to make the code generated fit what we
| need and not have annoying properties we want to void? Lets say
| we have a website and we tell the code generator of choice,
| that we want that website to be accessible for blind people.
| How do we create the amount of labeled examples, that make the
| code generator understand what to create? Maybe that very
| creation of labeled examples will be a software developer's
| future work activity.
| gtirloni wrote:
| Ideally, providing labeled examples can be done by relatively
| cheaper humans.
| [deleted]
| gz5 wrote:
| Andrej's podcast with Lex Friedman touched on this too - worth
| listening to if this topic is of interest.
|
| The pod was in the past year, so many years after Andrej's
| software 2.0 post, and after many years of great AI experience at
| Tesla to add to or potentially change his views.
|
| Likewise, will be interesting to see how the HN community's
| experience with ML and AI over those years may have changed our
| views.
| chazeon wrote:
| I remember there is a great analog for the era of the neuro
| network. The neuro network is a binary encoding of an
| algorithm/program. It could therefore be very efficient and
| simple. We will move on to get comfortable with it just like we
| are now comfortably familiar with the binary encoding of
| information (files).
| Dave3of5 wrote:
| I'm sure that will be the case at some point but for the moment
| it's not true. It's also important to note that NN's have been
| around for a long long time with relation to computing in
| general. The problems caused by using them have been reduced by
| not completely removed.
|
| Imagine if the Therac-25 software was written by chatGPT.
| dividuum wrote:
| > Imagine if the Therac-25 software was written by chatGPT.
|
| I guess in that case we wouldn't learn and teach from the
| design flaws made. Instead we would "It's just a glitch. No one
| is really to blame. Just feed it more data and maybe it won't
| kill anyone next time".
|
| Having been partially responsible for a (back then) SVM based
| machine learning system and seeing how it's difficult to
| explain to management why it fails and why fixing it isn't just
| a missing line of code somewhere was pretty frustrating. I'm
| not sure I like this future.
| Dave3of5 wrote:
| > I'm not sure I like this future
|
| Me too! But to allay our fears I think that the OP here is
| saying all programming will change to some NN powered large
| language model. That is not true. There will still be
| "manual" i.e. not NN powered programming and I suspect that
| it will be the case that type of thing is the majority for
| the rest of my career.
|
| Good luck to the poor sods in 100 years time arguing with a
| poorly trained LLM to output some unit tests whilst a virtual
| chatbot runs the standups.
| wufufufu wrote:
| > Across many applications areas, we'll be left with a choice of
| using a 90% accurate model we understand, or 99% accurate model
| we don't.
|
| And how do you show that it is 99% accurate besides creating
| enough automated tests to the point that you could write the
| procedural version?
|
| I think what I was missing from this article is how to evaluate a
| domain where neural nets or LLMs can be applied. Image-from-text
| generation is a great one because accuracy isn't strictly
| defined. However, telling ChatGPT "code this pacemaker for me"
| would have a real accuracy attached to it that you could confirm
| with unit tests.
| dang wrote:
| Related:
|
| _Software 2.0 (2017)_ -
| https://news.ycombinator.com/item?id=23766796 - July 2020 (22
| comments)
|
| _Software 2.0_ - https://news.ycombinator.com/item?id=15678587 -
| Nov 2017 (36 comments)
| impalallama wrote:
| Funny how little a splash this made then compared to now
___________________________________________________________________
(page generated 2023-02-21 23:01 UTC)