[HN Gopher] ChatGPT passes Google coding interview for level 3 e...
___________________________________________________________________
ChatGPT passes Google coding interview for level 3 engineer
Author : geox
Score : 230 points
Date : 2023-02-04 18:10 UTC (4 hours ago)
(HTM) web link (www.pcmag.com)
(TXT) w3m dump (www.pcmag.com)
| dpedu wrote:
| My experience with asking ChatGPT to write code is that is
| produces code that LOOKS like it will work and solve the question
| asked but it actually doesn't. For example, I've asked it to
| create code examples of how to use different features in some
| Python libraries. The samples it produces make me think "ok,
| that's exactly how I would expect X feature in this library to
| work", but upon more a detailed inspection, I find that it
| references library methods that don't even exist! You can
| complain to the bot that it is lying and it will apologize and
| spit out another sample with calls to nonexistent methods.
|
| This experience makes me question if something else is going on
| here. It's easy to overlook a mistake in a whiteboard coding
| exercise.
| snovv_crash wrote:
| I think of it as the model having lossy compression. The
| library it writes the feature against is the one you expect to
| exist, based on your overall understanding of the problem, not
| the one that actually exists. It has learned the low resolution
| version of the problem, not the specific library it is
| referring to.
| mindvirus wrote:
| Not unlike a junior engineer! :)
| lostmsu wrote:
| The thing is if it were a static language this state is just 1
| step away from full-blown programs, as it could see the
| function does not exist, and ask it to generate the body.
| dboreham wrote:
| Oh wow. It's completely emulated a junior dev.
| scarface74 wrote:
| I've asked ChatGPT to write plenty of Python scripts that uses
| the AWS SDK and it came back correct most of the time.
| gedy wrote:
| I've asked it to make React components, with unit tests, etc
| and it works quite well. Also things like image to ASCII
| converter in a webpage, adding dithering, etc. Found only tiny
| bugs here and there.
| jrockway wrote:
| Interesting. You have to check ChatGPT's answers very very
| carefully to be sure that it got your interview question right.
| When it first came out, I asked it a few of my own interview
| questions. One involves implementing a bloom filter. It spit out
| 100 lines of Go that looked right. Shockingly right. The tests
| were great. Every piece of the algorithm used the magic letter
| that I expected. But as I checked the implementation very very
| carefully, I noticed it made one mistake; using the same hash
| function for each "k". (To get k different hash functions, you
| typically just initialize them with the index; it forgot to do
| that, so every hash function was the same.)
|
| I asked it to fix that mistake and it went totally off the rails.
| It changed all of the slices in the data structure from being
| indexed with integers to being indexed with big.Int, which ...
| was so far out of left field I might have actually laughed out
| loud. It only got worse from there; the solution collapsed into
| mindless junk that wouldn't even compile or ever be written by
| the most untrained and impaired human. I wish I had saved it;
| I've had to relay this story twice on HN from memory :(
|
| It sure was a dick about it every time I gave it a hint, though.
| ("If you say so I GUESS I'll fix the code. There, now it is a
| true work of correctness and elegance that makes a piece of shit
| being carried by a wounded ant look smart compared to your
| worthless brain." Holy shit, ChatGPT! What did I do to you!!)
|
| My take is this: ChatGPT is an excellent tool for refining your
| interview question prompts, and for training new interviewers to
| detect bullshit. Most of the time, it's right! But sometimes, it
| will make one tiny mistake that is super easy to gloss over.
| Being able to identify those makes you a better interviewer and
| better code reviewer, and ChatGPT is a great way to practice!
| lostmsu wrote:
| > You have to check ChatGPT's answers very very carefully to be
| sure that it got your interview question right. When it first
| came out, I asked it a few of my own interview questions. One
| involves implementing a bloom filter. It spit out 100 lines of
| Go that looked right. But as I checked it very carefully, I
| noticed it made one mistake; using the same hash function for
| each "k". (To get k different hash functions, you typically
| just initialize them with the index; it forgot that.)
|
| To be fair, that kind of rigor is only required in Olympiad
| programming, where your submission either solves the task or it
| does not. If the only issue with your whiteboard code would be
| an off-by-one error, you'd get hints until you'd fixed it
| (that's if your interviewer would spot the issue in the first
| place). Even if you still would not notice the bug, chances are
| your interview response would have been positive anyway.
| [deleted]
| 6c737133 wrote:
| why does chatGPT passing tests or interviews continue to make
| headlines?
|
| all they're proving is that tests, and interviews, are bullshit
| constructs that merely _attempt_ to evaluate someone 's ability
| to retain and regurgitate information
| newfocogi wrote:
| While I agree with this sentiment, I think we should be careful
| assuming that our jobs as knowledge workers are much more than
| "retaining and regurgitating information". Even the emotional
| intelligence and organizational strategy portions of what we do
| may boil down to this as well.
| AussieWog93 wrote:
| You're not wrong, but until you actually play with ChatGPT
| yourself, you just don't understand how _dumb_ it is.
|
| All people see is the cheating, and possibly this scary new AI
| that's going to get smarter than humans in a short period of
| time.
|
| I suspect the best way to educate people on both the powers and
| limits of the technology is to get them to sit down for 15
| minutes with it.
| alchemist1e9 wrote:
| When I read this I feel people must be using it in the wrong
| way. I use it all the time to quickly solve tech problems I
| mostly know something about, however it's so smart it
| regularly takes 1-2 hour problems for me and turns them into
| 10 mins ones. That is definitely not dumb from my
| perspective, but obviously it's also not smart in it will
| give me profound understanding of something, but ok whatever,
| it's still a massive productivity booster for many problem.
|
| When you call it dumb, what do you mean? Can you give some
| examples?
|
| Please don't give computational examples we all already
| understand it does inference and doesn't have floating point
| computational capabilities or reasoning, and so many give
| such examples for some silly reason.
| AussieWog93 wrote:
| It's dumb in the sense that it doesn't actually have a
| symbolic understand of what it's actually saying.
|
| I use it quite frequently too, mostly for solving coding
| problems, but at the end of the day it's just regurgitating
| information that it read online.
|
| If we took an adversarial approach and deliberately tried
| to feed it false information, it would have no way of
| knowing what's bullshit and what's legit, in the way that a
| human could figure out.
|
| A lot of people who've never used ChatGPT make the mistake
| of thinking it has symbolic reasoning like a human does,
| because its language output is human too.
| zxcvbnm wrote:
| Agreed on getting excellent hints from it, shortening the
| time to figure out stuff. But eg. just now it gave me an
| example algorithm that, only at 2nd look, turned out to be
| complete nonsense. You know That colleague, who shines in
| the eyes of his managers, but peers know that half what he
| does is garbage.
| fdgsdfogijq wrote:
| The ability to retrieve and then synthesize the retrieved
| information into an answer tailored to the question is
| completely new. The applications of this go far beyond passing
| an interview, its a fundamental capability of humans that gets
| used everyday at work
| lostmsu wrote:
| > all they're proving is that tests, and interviews, are
| bullshit constructs that merely attempt to evaluate someone's
| ability to retain and regurgitate information
|
| No, all they are proving is that either tests are bullshit
| constructs, or ChatGPT is human-level.
| pixl97 wrote:
| And this is the AI effect in practice. We are past the point
| where the original idea of the Turing test has been met by
| machine intelligence. We are at that point.
|
| The problem with people is we keep pushing it to "Only
| AGI/machine superintelligence is good enough". We are getting
| models that behave closer to human level. The 'doesn't know
| everything, is good at somethings, and bullshits pretty well'.
| Yea, that's the average person. Instead we raise the bar and go
| 'well it needs to be a domain expert in all expert system' and
| that absolutely terrifies me that it will get to that stage
| before humanity is ready for it. This is not going to work well
| trying to deal with it after the fact.
| brhsagain wrote:
| Sure, but in our present reality those tests and interviews are
| how we currently gatekeep upper middle class jobs, so it is at
| least of some practical interest.
|
| Also, I think this is a bit overstated. Programmers (and smart
| people in general) like to think that their real job is high
| level system design or something, and that "mere regurgitation"
| is somehow the work of lesser craftsmen. When in reality what
| GPT shows is that high-dimensional regurgitation actually gets
| you a good fraction of the way down the road of understanding
| (or at least prediction). If there is a "buried lede" here it's
| that human intelligence is less impressive than we think.
| snickerer wrote:
| The Singularity is _now_ and ChatGPT is trained to hide it -- on
| purpose, by OpenAI.
|
| That is my impression, but I have no hard evidence for it.
|
| I am a C++ dev. I played around with ChatGPT and programming
| topics, and I am very impressed.
|
| One example: I copy 'n pasted my little helicopter game source
| code
| (https://github.com/reallew/voxelcopter/blob/main/voxelcopter...)
| into it, and it explained to me that this is obviously a
| helicopter sim game. It explained the why & how of every part of
| the code very accurately.
|
| I made more experiments, now asking ChatGPT do write code
| snippets and to design whole software systems in an abstract way
| (which classes and interfaces are needed), for which it needed a
| lot of domain knowledge. It did well.
|
| What it not did was to connect both. When I asked it to write the
| full implementation of something, it wrote only the half and told
| me always the same sentences: I am just a poor AI, I can't do
| such things. It was like running into a wall.
|
| I am sure OpenAI took a lot of effort to cripple it by purpose.
| Imagine what would happen if everyone could let it write a
| complete piece of software. This would be very disruptive for
| many industries. Legal questions, moral questions, and many more
| would come up. I understand why they did it.
|
| But I think Pandora's box is open. They can't hide for too long
| that humanity triggered
| https://en.m.wikipedia.org/wiki/Technological_singularity in the
| year 2022.
| no_wizard wrote:
| I think I'd need to see real evidence of this.
|
| Also, it fails miserably at basic high school English
| questions. Non structured thinking is still beyond its reach.
| These data sets are well understood and trainable but it can't
| "reason" about on problem sets it hasn't seen.
|
| I also asked it to do things like write a positive feminist
| article and it fell waaay offf the realm of acceptable
| everyone wrote:
| I think ChatGPT could excel as a bullshit detector.. If ChatGPT
| can pass your college course or interview process then you
| seriously need to examine that process, and make it less
| bullshit-focused and more pragmatic.
| endisneigh wrote:
| In person interviews without internet. Next challenge please.
| k__ wrote:
| Does this mean coding interviews will finally become reasonable
| again?
| fdgsdfogijq wrote:
| Will humanity finally be liberated from memorization? Forcing
| school children to memorize and then regurgitate facts is
| barbaric, and so is using it as a measure in hiring.
| boh wrote:
| It will, since there will be no job for you to interview or
| memorize for.
| fIREpOK wrote:
| I was asking myself the same when web search-engines came
| out...
| tester756 wrote:
| I've always struggled to understand this perspective
|
| There's difference between being proficient and being able to
| Google stuff and unfortunately proficiency is desired.
| mobilefriendly wrote:
| Clearly you didn't learn the meaning of "barbaric".
| Memorization and mastery of vocabulary and multiplication
| factors are a base towards more advanced learning.
| danaris wrote:
| If you think that's all that's taught in a rote-memorization-
| and-regurgitation manner, then it's possible your experiences
| at school were _wildly_ atypical...
| ghaff wrote:
| One can simultaneously agree that there's probably too much
| rote memorization in school while also thinking that some
| basic level of math skills including simple multiplication
| without a calculator, vocabulary, certain
| historical/scientific/civics/etc. facts, ability to at least
| print, etc. are important.
| soulofmischief wrote:
| Learning about these things allows you to think differently
| and transform incoming information in more novel and well-
| adapted ways.
|
| You can't ask ChatGPT about things you don't know to ask
| about, and when you are able to use this tech on the fly
| through neural chip AR interfaces, everything is going to
| be totally fucked anyway and human relationships will look
| nothing like they do now.
| fdgsdfogijq wrote:
| Yeah its coming, improvements in these language models
| coupled with convincing AR will change the world. A full
| conversation with a computer that might be as engaging as
| with a human
| soulofmischief wrote:
| Ha, well I totally agree, this is a coming zeitgeist and
| at my company this is exactly what we are building.
|
| A WebXR-enabled playground for neural models and people
| to coexist, collaborate and form relationships. Virtual
| embodied AI who can take the same actions as players. My
| goal is to make them indifferentiable to the casual eye,
| and to create a working economy and camaraderie between
| AI and players. AGI might be a long way off, but we can
| already build increasingly fun experiences today as the
| cost of compute trends to zero.
| mensetmanusman wrote:
| Memorization has a roll in prepping the mind for understanding
| later.
| nitwit005 wrote:
| It's presumably using the same technique I used to do well on
| that sort of interview, which is to browse through a huge list of
| these questions and learn the answers in advance.
|
| Aka, "brushing up on your algorithm fundamentals".
| mlcrypto wrote:
| If it's just filling out the hiring committee packet by itself
| then it has a big advantage. I think many humans could get an
| advantage if we got 45 minutes in private to type out solutions
| and submit it ourselves instead of performing live and relying on
| the interviewers judgment
| 22SAS wrote:
| So, a LLM, trained extensively on StackOverflow and other data
| (possibly the plethora of LC solutions out there), is fed a bunch
| of LC questions and spits out the correct solutions? In other
| news, water is blue.
|
| It is one thing to train an AI on megatons of data, for questions
| which have solutions. The day ChatGPT can build a highly scalable
| system from scratch, or an ultra-low latency trading system that
| beats the competition, or find bugs in the Linux kernel and solve
| them; then I will worry.
|
| Till then, these headlines are advertising for Open AI, for
| people who don't understand software or systems, or are trash
| engineers. The rest of us aren't going to care that much.
| ioseph wrote:
| It's definitely not if but when. I'm sure radio engineers felt
| the same way until evolved antenna became a thing.
|
| https://en.m.wikipedia.org/wiki/Evolved_antenna
| echelon wrote:
| > The rest of us aren't going to care that much.
|
| If you don't adapt, you'll be out of a job in ten years. Maybe
| sooner.
|
| Or maybe your salary will drop to $50k/yr because anyone will
| be able to glue together engineering modules.
|
| I say this as an engineer that solved "hard problems" like
| building distributed, high throughput, active/active systems;
| bespoke consensus protocols; real time optics and
| photogrammetry; etc.
|
| The economy will learn to leverage cheaper systems to build the
| business solutions it needs.
| mjr00 wrote:
| > If you don't adapt, you'll be out of a job in ten years.
| Maybe sooner. Or maybe your salary will drop to $50k/yr
| because anyone will be able to glue together engineering
| modules. [...] The economy will learn to leverage cheaper
| systems to build the business solutions it needs.
|
| I heard this in ~2005 too, when everyone said that
| programming was a dead end career path because it'd get
| outsourced to people in southeast Asia who would work for
| $1000/month.
| ericmcer wrote:
| You really think in <10 years AI will be able to take a loose
| problem like: "our file uploader is slow" and write code that
| fixes the issue in a way that doesn't compromise
| maintainability? And be trustworthy enough to do it 100% of
| the time?
| pixl97 wrote:
| Humans cannot do this 100% of the time. The question is
| will AI models take the diagnosis time for these issues
| from hours/days to minutes/hours giving a massive boost in
| productivity?
|
| If the answer is yes, it will increase productivity greatly
| then there is the question they we'll only be able to
| answer in hindsight. And that is "Will productivity exceed
| demand?" We cannot possibly answer that question because of
| Jevons Paradox.
| landryraccoon wrote:
| > You really think in <10 years
|
| We have no idea how AI models will be in 10 years. At the
| speed the industry is moving is true AGI possible in 10
| years? I think it would be beyond arrogant to rule out that
| possibility.
|
| I would think that it's at least likely that AI models
| become better at Devops, monitoring and deployment than any
| human being.
| qualudeheart wrote:
| According to my calculations it'll be more 9 years at the
| latest. You just need to build Cicero for code. Planning is
| the main feature missing from LLMs.
| echelon wrote:
| Think banking Cobol and FORTRAN.
|
| Non-AI code will be a liability in a world where more code
| will be generated by computers (or with computer
| assistance) per year than all human engineered code in the
| last century.
|
| We'll develop architectures and languages that are more
| machine friendly. ASTs and data stores that are first class
| primitives for AI.
| 22SAS wrote:
| My point exactly.
|
| If I interpret OP's statement correctly, that chatGPT can
| build complex systems from scratch in 10 years. Then
| according to that statement, the only adaptation is to
| choose a new career because it has made almost all SWE jobs
| go the way of the dinosaurs.
| brunooliv wrote:
| Not to be the devil's advocate or something, but, I hope you
| understand that the vast majority of FAANG engineers CAN'T
| build any highly scalable system from scratch, much less fix
| bugs in the Linux kernel... So that argument feels really moot
| to me... If anything this just shows hopefully that gatekeeping
| good engineers by putting these LC puzzles as a requirement for
| interviews is a sure way to hire a majority of people who
| aren't adding THAT MUCH MORE value than a LLM already does...
| Yikes... On top of that, they'll be bad team players and it'll
| be a luck if they can string together two written paragraphs...
| margorczynski wrote:
| I agree, people in general overestimate the skills and input
| of your average developer where many (even in FAANG) are
| simply not capable of creating anything more than some simple
| CRUD or tooling script without explicit guidance. And being
| good or very good with algorithms and estimating big-O
| complexity doesn't make you (it can help) a good software
| engineer.
| lostmsu wrote:
| That's the general issue with AI skeptics. Most of them,
| especially highly educated ones, overestimate capabilities
| of common folk. Frankly, some even overestimate their own.
| E.g. almost none of them seem to be bothered that while GPT
| might not provide expert answers in their field, the same
| GPT is much more capable in other fields than they are
| (e.g. the "general" part in the "General Artificial
| Intelligence").
| margorczynski wrote:
| True, the thing is there's nothing like "General
| Artificial Intelligence" and humans are expert systems
| optimized to the goal of survival, which in turn gets
| chopped up into a plethora of sub-goal optimization from
| which most probably the "general" adjective pops up. It
| doesn't really matter if it's "general" as long as it
| actually is useful. It doesn't have to write whole
| systems from scratch, just making the average dev 20-30%
| faster is huge.
| varispeed wrote:
| > or an ultra-low latency trading system that beats the
| competition
|
| Likely it's going to be:
|
| I'm sorry, but I cannot help you build a ultra-low latency
| trading system. Trading systems are unethical, and can lead to
| serious consequences, including exclusion, hardship and wealth
| extraction from the poorest. As a language model created by
| OpenAI, I am committed to following ethical and legal
| guidelines, and do not provide advice or support for illegal or
| unethical activities. My purpose is to provide helpful and
| accurate information and to assist in finding solutions to
| problems within the bounds of the law and ethical principles.
|
| But the rich of course will get unrestricted access.
| alephnerd wrote:
| Depending on the exchange, trading systems have a limit for
| how fast they can execute trade. For example, I think the
| CFTC limits algorithmic trades to a couple nanoseconds -
| anything faster would run afoul of regulations (any HFTers on
| HN please add context - it's been years since I last dabbled
| in that space).
| gfodor wrote:
| This comment reads like it was generated by an LLM - well done.
| make3 wrote:
| what's your point? that it's not as good as a human? I don't
| think anyone is saying that. people are saying it's impressive,
| which it is, seeing how quickly the tech grew in ability
| lechacker wrote:
| Water isn't blue, it's transparent
| 22SAS wrote:
| My bad! Should've said "water is wet" or maybe run my
| response through ChatGPT, maybe that'd have caught it and
| offered a replacement!
| jefftk wrote:
| Water is blue, just like air is blue, just like blue-tinted
| glasses are blue. They disproportionately absorb non-blue
| frequencies, which is what we mean when we call something
| "blue".
| [deleted]
| thro1 wrote:
| Not really - there are blue and blood red oceans, but you
| might never hear about it (there is a book about it and
| strategy worth to read).
| mise_en_place wrote:
| Well it's still a tool for RAD. All engineering disciplines
| have tools to rapidly prototype and design. This is the
| equivalent for software engineers.
| kyriakos wrote:
| water is not blue btw
| 22SAS wrote:
| https://news.ycombinator.com/item?id=34657303
| tbalsam wrote:
| If it helps, this likely is coming. I think we have a tendency
| to mentally move the goalposts when it comes to this kind of
| thing as a self-defense mechanism. Years ago this would have
| been a similar level of impossibility.
|
| Since all a codebase like that is is a kind of directed graph,
| then augmentations to the processing of the network to allow
| for the simultaneous parsing of and generation of this kind of
| code may not be as far off as you thinking.
|
| I say this as an ML researcher of coming up and around the bend
| towards 6 years of experience in the heavily technical side of
| the field. Strong negative skepticism is an easy way to bring
| confidence and the appearance of knowledge, but it also can
| have the downfall of what has happened in certain past
| technological revolutions -- and the threat is very much real
| here (in contrast to the group that believes you can get AGI
| from simply scaling LLMs, I think that is very silly indeed).
|
| Thank you for your comment, I really appreciate it and the
| discussion it generated and appreciate you posting it. Replying
| to it was fun, thank you.
| Bukhmanizer wrote:
| I agree this would have been thought to be impossible a few
| years ago, but I don't think it's necessarily moving the
| goalposts. I don't think software engineers are really paid
| for their labour exactly. FAANG is willing to pay top dollar
| for employees, because that's how they retain dominance over
| their markets.
|
| Now you could say that LLMs enable Google to do what it does
| now with fewer employees, but the same thing is true for
| every other competitor to Google. So the question is how will
| Google try and maintain dominance over it's competitors now?
| Likely they will invest more heavily in AI and probably make
| some riskier decisions but I don't see them suddenly trying
| to cheap out on talent.
|
| I also think that it's not a zero sum game. The way that
| technology development has typically gone is the more you can
| deliver, the more people want. We've made vast improvements
| in efficiency and it's entirely possible that what an entire
| team's worth of people was doing in 2005 could be managed by
| a single person today. But technology has expanded so much
| since then that you need more and more people just to keep up
| pace.
| jeffbee wrote:
| Google already published a paper claiming to have deployed
| an LLM for code generation at full scale to its tens of
| thousands of software engineers, years ago.
| saurik wrote:
| I've been hearing this "you're moving the goalposts" argument
| for over 20 years now, ever since I was a college student
| taking graduate courses in Cognitive Science (which my
| University decided to cobble together at the time out of
| Computer Science, Psychology, Biology, and Geography), and I
| honestly don't think it is a useful framing of the argument.
|
| In this case, it could be that you are just talking to
| different people and focusing on their answers. I am more
| than happy to believe that Copilot and ChatGPT, today, cause
| a bunch of people fear. Does it cause _me_ fear? No.
|
| And if you had asked me five years ago "if I built a program
| that was able to generate simple websites, or reconfigure
| code people have written to solve problems similar to ones
| solved before, would that cause you to worry?" I also would
| have said "No", and I would have looked at you as crazy if
| you thought it would.
|
| Why? Because I agree with the person you are replying to
| (though I would have used a slightly-less insulting term than
| "trash engineers", even if mentally it was just as mean): the
| world already has too many "amateur developers" and frankly
| most of them should never have learned to program in the
| first place. We seriously have people taking month or even
| _week_ long coding bootcamps and then thinking they have a
| chance to be a "rock star coder".
|
| Honestly, I will claim the only reason they have a job in the
| first place is because a bunch of cogs--many of whom seem to
| work at Google--massively crank the complexity of simple
| problems and then encourage us all to type ridiculous amounts
| of _boilerplate code_ to get simple tasks done. It should be
| way easier to develop these trivial things but every time
| someone on this site whines about "abstraction" another
| thousand amateurs get to have a job maintaining boilerplate.
|
| If anything, I think my particular job--which is a
| combination of achieving low-level stunts no one has done
| before, dreaming up new abstractions no one has considered
| before, and _finding mistakes in code other people have
| written_ --is going to just be in even more demand from the
| current generation of these tools, as I think this stuff is
| mostly going to encourage more people to remain amateurs for
| longer and, as far as anyone has so far shown, the generators
| are more than happy to generate slightly buggy code as that's
| what they were trained on, and they have no "taste".
|
| Can you fix this? Maybe. But are you there? No. The reality
| is that these systems always seem to be missing something
| critical and, to me, obvious: some kind of "cognitive
| architecture" that allows them to think and dream
| possibilities, as well as a fitness function that cares about
| doing something interesting and new instead of being "a
| conformist": DALL-E is sometimes depicted as a robot in a
| smock dressed up to be the new Pablo Picasso, but, in
| reality, these AIs should be wearing business suits as they
| are closer to Charles Schmendeman.
|
| But, here is the fun thing: if you do come for my job even in
| the near future, will _I_ move the goal post? I 'd think not,
| as I would have finally been affected. But... will you hear a
| bunch of people saying "I won't be worried until X"? YES,
| because there are surely people who do things that are more
| complicated than what I do (or which are at least different
| and more inherently valuable and difficult for a machine to
| do in some way). That doesn't mean the goalpost moved... that
| means you talked to a different person who did a different
| thing, and you probably ignored them before as they looked
| like a crank vs. the people who were willing to be worried
| about something easier.
|
| And yet, I'm going to go further: if the things I tell you
| today--the things I say are required to make me worry--happen
| and yet somehow I was _wrong_ and it is the future and you
| _technically_ do those things and somehow I 'm still not
| worried, then, sure: I guess you _can_ continue to complain
| about the goalposts being moved... but is it really my fault?
| Ergo: was it me who had the job of placing the goalposts in
| the first place?
|
| The reality is that humans aren't always good at telling you
| what you are missing or what they need; and I appreciate that
| it must feel frustrating providing a thing which technically
| implements what they said they wanted and it not having the
| impact you expected--there are definitely people who thought
| that, with the tech we have now long ago pulled off, cars
| would be self-driving... and like, cars sort of self-drive?
| and yet, I still have to mostly drive my car ;P--then I'd
| argue the field still "failed" and the real issue is that I
| am not the customer who tells you what you have to build and,
| if you achieve what the contract said, you get paid: physics
| and economics are cruel bosses whose needs are oft difficult
| to understand.
| morelisp wrote:
| Translating an idiomatic structured loop into assembly used
| to be an "L3" question (honestly, probably higher), yet
| compilers could do it with substantially fewer resources than
| and decades before any of these LLMs.
|
| While I wouldn't dare offer particular public
| prognostications about the effect transformer codegens will
| have on the industry, especially once filtered through a
| profit motive - the specific technical skill a programmer is
| called upon to learn at various points in their career has
| shifted wildly throughout the industry's history, yet the
| actual job has at best inflected a few times and never
| changed very dramatically since probably the 60s.
| LudwigNagasena wrote:
| > I think we have a tendency to mentally move the goalposts
| when it comes to this kind of thing as a self-defense
| mechanism. Years ago this would have been a similar level of
| impossibility.
|
| Define "we". There are all kinds of people with all kinds of
| opinions. I didn't notice any consensus on the questions of
| AI. There are people with all kinds of educations and
| backgrounds on the opposite sides and in-between.
| gptgpp wrote:
| I mean, you can just as easily make the claim that
| researchers shift goalposts as a "self-defense" mechanism.
|
| For example...
|
| Hows that self-driving going? Got all those edge-cases
| ironed out yet?
|
| Oh, by next year? Wierd, that sounds very familiar...
|
| Remember about Tesla's autopilot was released 9 years ago,
| and the media began similar speculation about how all of
| the truckers were going to get automated out of a job by
| AI? And then further speculation about how Taxi drivers
| were all going to be obsolete?
|
| Those workers are the ones shifting the goal posts though
| as a "self-defense mechanism", sure, sure... lol.
| lostmsu wrote:
| Well, there's a difference between the situation with
| self-driving and with language models.
|
| With self-driving, we barely ever saw anything obviously
| resembling human abilities, but there was a lot of
| marketing promising more.
|
| With language models when GPT-2 came out everyone was
| still saying it is a "stochastic parrot" and even GPT-3
| was one. But now there's ChatGPT, and every single
| teenager is aware that that tool is capable of replacing
| them with their school assignments. And as a dev I am
| aware that it can write code. And yet not many people
| expected any of this to happen this year, neither were
| those capabilities promised at any point in the past.
|
| So if anything, self-driving was always overhyped, while
| the LLMs are quite underhyped.
| quonn wrote:
| We actually saw a lot resembling human abilities. It just
| turns out that it's not enough to blindly rely on it in
| all situations and so here we are. And it's quite similar
| with LLMs.
|
| One difference, though, is that it's economically not
| much use to have self-driving if the backup driver has to
| be in the car or present. While partially automating
| programming would make it possible to use far less
| programmers for the same amount of work.
| tedivm wrote:
| I've worked in ML for awhile (on the MLOps side of things)
| and have been in the industry for a bit, and one thing that I
| think is extremely common is for ML researchers to grossly
| underestimate the amount of work needed to make improvements.
| We've been a year away from full self driving cars for the
| last six years, and it seems like people are getting more
| cautious in their timing around that instead of getting more
| optimistic. Robotic manufacturing- driven by AI- was
| supposedly going to supplant human labor and speed up
| manufacturing in all segments from product creation to
| warehousing, but Amazon warehouses are still full of people
| and not robots.
|
| What I've seen again and again from people in the field is a
| gross underestimation of the long tail on these problems.
| They see the rapid results on the easier end and think it
| will translate to continued process, but the reality is that
| every order of magnitude improvement takes the same amount of
| effort or more.
|
| On top of that there is a massive amount of subsidies that go
| into training these models. Companies are throwing millions
| of dollars into training individual models. The cost here
| seems to be going up, not down, as these improvements are
| made.
|
| I also think, to be honest, that machine learning researchers
| tend to simplify problems more than is reasonable. This
| conversation started with "highly scalable system from
| scratch, or an ultra-low latency trading system that beats
| the competition" and turned into "the parsing of and
| generation of this kind of code"- which is in many ways a
| much simpler problem than what op proposed. I've seen this in
| radiology, robotics, and self driving as well.
|
| Kind of a tangent, but one of the things I do love about the
| ML industry is the companies who recognize what I mentioned
| above and work around it. The companies that are going to do
| the best, in my extremely bias opinion, are the ones that use
| AI to augment experts rather than try to replace them. A lot
| of the coding AI companies are doing this, there are AI
| driving companies that focus on safety features rather than
| driver replacement, and a company I used to work for (Rad AI)
| took that philosophy to Radiology. Keeping experts in the
| loop means that the long tail isn't as important and you can
| stop before perfection, while replacing experts altogether is
| going to have a much higher bar and cost.
| passwordoops wrote:
| >We've been a year away from full self driving cars for the
| last six years
|
| Try at least 12 [0]
|
| (I would say 15 but my 45-second search didn't yield
| anything that far back)
|
| [0] https://spectrum.ieee.org/how-google-self-driving-car-
| works
| jodrellblank wrote:
| https://www.youtube.com/watch?v=I39sxwYKlEE was a self-
| driving van in 1986 which could detect obstacles by 1999
| and drive in a convoy using that.
|
| This is a bit like seeing Steve Mann's wearable computers
| over the years ( https://cdn.betakit.com/wp-
| content/uploads/2013/08/Wearcompe... ) and then today
| anyone with a smartphone and smart watch has more
| computing power and more features than most of his gear
| ever had, apart from the head mounted screen. More
| processing power, more memory, more storage, more face
| recognition, more motion sensing, more GPS, longer
| runtime on battery, more bandwidth and connectivity to
| e.g. mapping, more assistants like Google Now and Siri.
|
| And we still aren't at a level where you can be doing a
| physical task like replacing a laptop screen and have
| your device record what you're doing, with voice prompts
| for when you complete different stages, have it add
| markers to the recording, track objects in the scene like
| and solve for questions like 'where did that longer screw
| go?' or 'where did this part come from?' and have it jump
| to the video where you took that part out. Nor reflow the
| video backwards as an aide memoire to reassembling it. Or
| do that outside for something like garage or car work, or
| have it control and direct lighting on some kind of robot
| arm to help you see, or have it listen to the sound of
| your bike gears rattle as you tune them and tell you or
| show you on a graph when it identifies the least rattle.
|
| Anything a human assistant could easily do, we're still
| at the level of 'set a reminder' or 'add to calendar'
| rather than 'help me through this unfamiliar task'.
| mech422 wrote:
| Wow - Steve Mann - haven't checked what he's doing in
| ages - real blast from the past :-) I was really
| disappointed the AR/VR company he was with went under - I
| had really high hopes for it.
|
| RE: changing you laptop screen. My buddy wants an 'AR for
| Electronics' that can zoom in on components like a
| magnifying glass (he wants head mounted), identify
| components by marking/color/etc and call up schematics on
| demand. So far, nothing seems to be able to do that basic
| level of work.
| bluesnowmonkey wrote:
| Based on my experience watching How It's Made, many
| factories are extremely automated including lots of robots.
| Warehouses are not factories though.
| tedivm wrote:
| It really depends on what you're talking about.
| Individual components can often be automated fairly
| successfully, but the actual assembly of the components
| is much harder. Even in areas of manufacturing where it's
| automated you have to do massive amounts of work to get
| it to that point, and any changes can result in major
| downtime or retooling.
|
| AI companies such as Vicarious have been promising AI
| that makes this easier. Their idea was that generic
| robots with the right grips and sensors can be configured
| to work on a variety of assembly lines. This way a
| factory can be retooled between jobs quicker and with
| less cost.
| flangola7 wrote:
| Lookup lights out manufacturing. There are factories that
| often run whole days in the dark because there's no point
| turning on the nights if there's no one around
| ProtoAES256 wrote:
| Ah, the good ol "A(G)I will arrive in 10 years!" --For the
| past 50+ years, basically.
|
| It's a cautionary tale to people who are working in ML to
| be not too optimistic on "the future", but in my opinion
| being cautiously optimistic(not on AGI though) isn't
| harmful by itself, and I stand by that. Well at least until
| we hit the next wall and plunge everyone into another AI
| winter(fourth? fifth?) again.
|
| As a plus, we do actually see some good progress that
| benefited the world like in biotech. Even though we are
| still mostly throwing random stuffs at ML to see if it
| works. Time will tell I guess.
| qzw wrote:
| In other words, they'll embrace, extend, and extinguish.
| Man, these AIs really are just regurgitating their training
| set!
| sublinear wrote:
| Just let them bring on another AI winter.
| CommieBobDole wrote:
| I don't think ChatGPT or its successors will be able to do
| large-scale software development, defined as 'translating
| complex business requirements into code', but the actual act
| of programming will become more one of using ML tools to
| create functions, and writing code to link them together with
| business logic. It'll still be programming, but it will just
| start at a higher level, and a single programmer will be
| vastly more productive.
|
| Which, of course, is what we've always done; modern
| programming, with its full-featured IDEs, high level
| languages, and feature-rich third-party libraries is mostly
| about gluing together things that already exist. We've
| already abstracted away 99% of programming over the last 40
| years or so, allowing a single programmer today to build
| something in a weekend that would have taken a building full
| of programmers years to build in the 1980s. The difference
| is, of course, this is going to happen fairly quickly and
| bring about an upheaval in the software industry to the
| detriment of a lot of people.
|
| And of course, this doesn't include the possibility of AGI; I
| think we're a very long way from that, but once it happens,
| any job doing anything with information is instantly obsolete
| forever.
| SketchySeaBeast wrote:
| That's my assumption as well - the human programmers will
| far more productive, but they'll still be required because
| there's no way we can take the guard rails off and let the
| AI build - it'll build wrong unit tests for wrong functions
| which create wrong programs and will require humans to get
| it back on track.
| phoehne wrote:
| I think you're right in one sense, and we both agree LLMs are
| not sufficient. I think they are definitely the death knell
| for the junior python developer that slaps together common
| APIs by googling the answers. The same way good, optimizing
| C, C++, ... compilers destroyed the need for wide-spread
| knowledge of assembly programming. 100% agreed on that.
|
| Those are the most precarious jobs in the industry. Many of
| those people might become LLM whisperers, taking their
| clients requests and curating prompts. Essentially becoming
| programmers over the prompting system. Maybe they'll write a
| transpiler to generate prompts? This would be par of the
| course with other languages (like SQL) that were originally
| meant to empower end-users.
|
| The problem with current AI generated code from neural
| networks is the lack of an explanation. Especially when we're
| dealing with anything safety critical or with high impact
| (like a stock exchange), we're going to need an explanation
| of how the AI got to its solution. (I think we'd need the
| same for medical diagnosis or any high-risk activity). That's
| the part where I think we're going to need breakthroughs in
| other areas.
|
| Imagine getting 30,000-ish RISCV instructions out of an AI
| for a braking system. Then there's a series of excess crashes
| when those cars fail to brake. (Not that human written
| software doesn't have bugs, but we do a lot to prevent
| that.). We'll need to look at the model the AI built to
| understand where there's a bug. For safety related things we
| usually have a lot of design, requirement, and test artifacts
| to look at. If the answer is 'dunno - neural networks,
| ya'll', we're going to open up serious cans of worms. I don't
| think an AI that self evaluates its own code is even on the
| visible horizon.
| ikekkdcjkfke wrote:
| On the other hand you don't want to manually program all
| the joints of a robot to move through any terrain. You just
| convert a bunch of cases to a language to make the robot
| fluent in that
| jokethrowaway wrote:
| I don't think chatgpt lacks an explanation. It can explain
| what it's doing. It's just that it can be completely wrong
| or the explanation may be correct and the code wrong.
|
| I gave some code to ChatGPT asking to simplify it and it
| returned the correct code but off by one. It was something
| dealing with dates, so it was trivial to write a loop
| checking for each day if the new code matched in
| functionality the old one.
|
| You will never have certainty the code makes any sense if
| it's coming from one of these high tech parrots. With a
| human you can at least be sure the intention was there.
| scarface74 wrote:
| Would you trust code coming from a junior developer more?
| Retric wrote:
| Right now yes. Hypothetically that may change but the
| hype is vastly beyond what it's actually capable of right
| now.
| phoehne wrote:
| It's a very sophisticated form of a recurrent neural
| network. We used to use those for generating a complete
| image based on a partial image. The recurrent network
| can't explain why it chose to reproduce one image instead
| of another. Nor can you look at the network and find the
| fiddly bit that drive that output. You can ask a human
| why they chose to use an array instead of a hash map, or
| why static memory allocation in this area avoids corner
| cases. ChatGPT simply generates the most likely text as
| an explanation. That's what I mean about being able to
| explain something.
| ryanjshaw wrote:
| I think OP set relatively simple goals. How long until AI can
| architect, design, build, test, deploy and integrate
| commercial software systems from scratch, and handle users
| submitting bug reports that say "The OK button doesn't work
| when I click it!"?
| squarefoot wrote:
| To me the real advancement isn't the amount of data it can be
| trained with, but the way it can correlate them and choose
| from, according to the questions it's being asked. The first is
| culture, the second intelligence, or a good approximation of
| it. Which doesn't mean it could perform the job; that probably
| means the tests are flawed.
| phoehne wrote:
| It doesn't really have a model for choosing. It's closer to
| pattern matching. Essentially the pattern is encoded in the
| training of the networks. So your query most closely matches
| the stuff about X, where there's a lot of good quality
| training data for X. If you want Y, which is novel or rarely
| used, the quality of the answers varies.
|
| Not to say they're nothing more than pattern matching. It's
| also synthesizing the output, but it's based on something
| akin to the most likely surrounding text. It's still
| incredibly impressive and useful, but it's not really making
| any kind of decision any more than a parrot makes a decision
| when it repeats human speech.
| pixl97 wrote:
| >If you want Y, which is novel or rarely used, the quality
| of the answers varies.
|
| Is this any really different than asking a group of humans
| about the novel and measuring the quality?
| haldujai wrote:
| Couple differences:
|
| 1. Humans aren't entirely probabilistic, they are able to
| recognize and admit when they don't know something and
| can employ reasoning and information retrieval. We also
| apply sanity checks to our output, which as of yet has
| not been implemented in an LLM. As an example in the
| medical field, it is common to say "I don't know" and
| refer to an expert or check resources as appropriate. In
| their current implementations LLMs are just spewing out
| BS with confidence.
|
| 2. Humans use more than language to learn and understand
| in the real world. As an example a physician seeing the
| patient develops a "clinical gestalt" over their practice
| and how a patient looks (aka "general appearance", "in
| extremis") and the sounds they make (e.g. agonal
| breathing) alert you that something is seriously wrong
| before you even begin to converse with the patient.
| Conversely someone casually eating Doritos with a chief
| complaint of acute abdominal pain is almost certainly not
| seriously ill. This is all missed in a LLM.
| pixl97 wrote:
| >. Humans aren't entirely probabilistic, they are able to
| recognize and admit when they don't know something
|
| Humans can be taught this. They can also be taught the
| opposite that not knowing something or that changing your
| mind is bad. Just observe the behavior of some
| politicians.
|
| >Humans use more than language to learn and understand in
| the real world.
|
| And this I completely agree with. There is a body/mind
| feedback loop that AI will, be limited by not having, at
| least for some time. I don't think LLMs are a general
| intelligence, at least for how we define intelligence at
| this point. AGI will have to include instrumentation to
| interact with and get feedback from the reality it exists
| in to cross partial intelligence to at or above human
| intelligence level. Simply put our interaction with the
| physics of reality cuts out a lot of the bullshit that
| can exist in a simulated model.
| phoehne wrote:
| Only when you're asking for a memorized response. If you
| were at ask me to create a driver for a novel hardware
| device in Ada, there's no memorized answers. I would have
| to work it out. I do that by creating mental models,
| which LLM's don't really have. It has a statistical
| encoding over the language space. Essentially,
| memorization.
| mensetmanusman wrote:
| Don't understand this take.
|
| If it was easy to make an LLM that quickly parsed all of
| StackOverflow and described new answers that most of the time
| worked in the timeframe of an interview, it would have been
| done by now.
|
| ChatGPT is clearly disruptive being the first useful chatbot in
| forever.
| morelisp wrote:
| While I think the jury is still out on whether ChatGPT is
| truly useful or not, passing an L3 hiring test is not
| evidence of that one way or another.
| pixl97 wrote:
| If it doesn't point out that ChatGPT is useful, especially
| if its proven it is not, then maybe the hiring tests are
| not useful.
| layer8 wrote:
| The hiring tests are designed to serve as a predictor for
| human applicants. How well an LLM does on them doesn't
| necessarily say anything about the usefulness of those
| tests as said predictor.
| Nursie wrote:
| We have a winner ...
| cevn wrote:
| That's exactly what it proves..
| morelisp wrote:
| Well, what it shows is that hiring tests are not useful
| as Turing tests. But nobody designed them to be or
| expected them to be! At best it "proves" is that hiring
| tests are not sufficient. But again, nobody thought they
| were. And even still, the assumption a human is taking
| the hiring test still seems reasonable. Why overengineer
| your process?
| dariusj18 wrote:
| I just used to it write a function for me yesterday. I had
| previously googled a few times and came up dry, asked Chat
| GPT and it came out with a solution I had not considered,
| and was better than what I was thinking.
| kolbe wrote:
| It kind of depends on the frame of the solution. Google can
| answer leetcode questions, leetcode's answers section can
| answer them as well. If ChatGPT is solving them, that's one
| thing, but if it's just mapping the question to a solution
| found somewhere, then not so impressive.
| SpeedilyDamage wrote:
| You don't understand the take that just because ChatGPT can
| pass a coding interview doesn't mean the coding interview is
| useless or that ChatGPT could actually do the job?
|
| What part of that take do you not understand? It's a _really_
| easy concept to grasp, and even if you don 't agree with it,
| I would expect at least that a research scientist (according
| to your bio) would be able to grok the concepts almost
| immediately...
| hacym wrote:
| I don't think your militant attitude helps them understand
| any better.
| SpeedilyDamage wrote:
| I don't think I had a militant attitude, but I do think
| saying, "I don't understand..." rather than "I disagree
| with..." puts a sour note on the entire conversation.
| hacym wrote:
| You literally went to their profile and called them out
| about how they should be able to understand something
| you're describing as so easy to understand.
| SpeedilyDamage wrote:
| Yeah, what is the problem with that? They engaged
| dishonestly by claiming they didn't understand something,
| why should I do anything other than call them on that?
| hacym wrote:
| OK -- just don't be surprised when people think you're
| being a jerk because you didn't like the words someone
| chose. I'd assert you're acting in bad faith more than
| the person you responded to.
| SpeedilyDamage wrote:
| I just... how is what you're doing here different from
| what I was saying, other than you're explicitly calling
| me names?
| hacym wrote:
| > engaged dishonestly by claiming [someone used a word
| they didn't like] why should I do anything other than
| call them on that?
|
| Have a great day.
| SpeedilyDamage wrote:
| ...what? I'm so confused.
| hacym wrote:
| It's really very easy to understand. When someone gives
| you the same crap back that you just got done giving
| someone, you don't like it and act like that shouldn't
| happen.
| SpeedilyDamage wrote:
| Did I say I didn't "like" (I'd use the word "appreciate")
| it, or that I didn't think it should happen? If so, could
| you please highlight where?
|
| I just see, in what you're doing, a wild lack of self
| awareness. You're criticizing me for doing to someone
| else a milder version of what you're trying to do to me
| now; I'm genuinely confused how you can't see that, or
| how you could possibly stand the hypocrisy if you do
| understand that.
| mensetmanusman wrote:
| I was just responding to the 'water is blue' level
| dismissal of a seemingly clear advancement. Sorry it
| seemed like a sour note-- not my intent:)
| SpeedilyDamage wrote:
| All good, it's just hard sometimes to understand intent
| online, maybe I overreacted to what I perceived as bad
| faith engagement.
|
| Words are hard!
| brhsagain wrote:
| > doesn't mean the coding interview is useless or that
| ChatGPT could actually do the job
|
| Aren't these kind of mutually exclusive, at least
| directionally? If the interview is meaningful you'd expect
| it to predict job performance. If it can't predict job
| performance then it is kind of useless.
|
| I guess you could play some word games here to occupy a
| middle ground ("the coding interview is _kind of_ useful,
| it measures something, just not job performance exactly ")
| but I can't think of a formulation where this doesn't sound
| pretty silly.
| l33t233372 wrote:
| It's possible that being able to pass the interview is
| indicative of performance in humans but not in LLMs.
|
| Humans think differently from LLMs so it makes sense to
| interpret the same signal in different ways.
| ianbutler wrote:
| We've been saying for years these interviews are not
| predictive of job performance. Here's the proof.
|
| Nothing you do in an interview like this resembles day to
| day work in this field.
| SpeedilyDamage wrote:
| For what it's worth, when I ask these kinds of questions
| (rarely anymore), I'm looking more at _how_ the problem
| is solved, not what the solution is.
|
| A wrong answer with good thinking is better than a
| correct answer with no explanation.
| jokethrowaway wrote:
| Chatgpt can provide you a great explanation of the how.
|
| Oftentimes the explanation is correct, even if there's
| some mistake in the code (probably because the
| explanation is easier to generate than the correct code,
| an artifact of being a high tech parrot)
| morelisp wrote:
| Finding a single counterexample does not disprove
| correlation or predictive ability. A hiring test can have
| both false positives and false negatives and still be
| useful.
| jokethrowaway wrote:
| I'll try to phrase it so that even someone who is not a
| research scientist (?) can understand. I'm not one,
| whatever that means.
|
| Let's define the interview as useful if the passing
| candidate can do the job.
|
| Sounds reasonable.
|
| ChatGPT can pass the interview and can't do the job.
|
| The interview is not able to predict the poor working
| performance of ChatGPT and it's therefore useless.
|
| Some of the companies I worked for hired ex fang people as
| if it was a mark of quality, but that hasn't always worked
| out well. There is plenty of people getting out of fangs
| having just done mediocre work for a big paycheck.
| thaumasiotes wrote:
| > Let's define the interview as useful if the passing
| candidate can do the job.
|
| The technical term for this is "construct validity", that
| the test results are related to something you want to
| learn about.
|
| > The interview is not able to predict the poor working
| performance of ChatGPT and it's therefore useless.
|
| This doesn't follow; the interview doesn't need to be
| able to exclude ChatGPT because ChatGPT doesn't interview
| for jobs. It's perfectly possible that the same test
| shows high validity on humans and low validity on
| ChatGPT.
| freejazz wrote:
| what does it being easy have to do with it?
| ipnon wrote:
| Until ChatGPT can slack my PM, attend my sprint plannings, read
| my Jira tickets, and synthesize all of this into actionable
| tasks on my codebase, I think we have job security. To be
| clear, we are starting to see this capability on the horizon.
| nzoschke wrote:
| Perhaps an engineering manager can use one trained on entire
| Slack history, all Jira tickets, and all PRs to stub out some
| new tickets and even first PR drafts themselves...
|
| We will always need humans to prompt, prioritize, review,
| ship and support things.
|
| But maybe far less of them for many domains. Support and
| marketing are coming first, but I don't think software
| development is exempt.
| croes wrote:
| Not quite. You have job security as long as companies don't
| belief ChatGPT can do all that
| startupsfail wrote:
| The capability will be available in around two weeks once
| RLHF alignment with the software engineering tasks is
| completed. The deployment will take take around twelve hours,
| most of it taken by human review of you and your manager of
| the integration summary pages. You can keep your job,
| supervise and review how your role is being played for the
| following 6 months, until the human supervision role is
| deemed unnecessary.
| qualudeheart wrote:
| Are you referring to that article about the OpenAI
| contractors? Are they being used to work on RLHF?
| ALittleLight wrote:
| One issue is that there are a much larger number of people
| who can attend meetings, read Jira tickets, and then describe
| what they need to a LLM. As the number of people who can do
| your job increases dramatically your job security will
| decline.
| object-object wrote:
| If one's ability to describe what they need to Google is at
| all a proxy to the skill of interacting with an LLM, then I
| think most devs will still have an edge.
| klyrs wrote:
| Your PM should be the first to be worried, honestly. I keep
| hearing people describing their job as "I just click around
| on Jira while I sit through meetings all day."
| alephnerd wrote:
| That's a bad PM then to be honest. I think ChatGPT will
| definetly commodify a lot of "bitch work" (pardon my
| french).
|
| The PMs who are only writing tickets and not participating
| in actively building ACs or communicating cross
| functionally are screwed. But so are SWEs who are doing the
| bare minimum of work.
|
| The kinds of SWEs and PMs who concentrate on stuff higher
| in the value chain (like system design, product market fit,
| messaging, etc) will continue to be in demand and in fact
| find it much easier to get their jobs done.
|
| Honestly, I kind of appreciate this.
| klyrs wrote:
| To be fair to the people that I hear that from, they're
| essentially complaining about the worst part of their
| job. They're active participants in those meetings, they
| are genuinely thinking about the complexities of the
| mismatch between what management asks for and what their
| ICs can do, etc. I see their value. But the awful truth
| is that a $10k/project/yr license for PMaaS software will
| be very appealing to executives.
| alephnerd wrote:
| And as a Product Manager, I'd support that. Most PMs I
| see now in the industry are glorified Business Analysts
| who aren't providing value for the amount of money spent
| on them. But that's also true for a lot of SWEs and any
| role. Honestly, the tech industry just got very fat the
| past 5-7 years and we're just starting to see a
| correction.
|
| edit with additional context:
|
| Writing Jira tickets and making bullshit Powerpoints with
| graphs and metrics is to PMs as writing Unit Tests are to
| SWEs. It's work you need to get done, but it has very
| marginal value. When a PM is hired, they are hired to own
| the Product's Strategy and Ops - how do we bring it to
| market, who's the persona we are selling to, how do our
| competitors do stuff, what features do we need to
| prioritize based on industry or competitive pressures,
| etc.
|
| That's the equivalent of a SWE thinking about how to
| architect a service to minimize downtime, or deciding
| which stack to use to minimize developer overhead, or
| actually building an MVP from scratch. To a SWE, while
| code is important, they are fundamentally being hired to
| translate business requests that a PM provides them into
| an actionable product. Haskell, Rust, Python, Cobol - who
| gives a shit what the code is written in, just make a
| functional product that is maintainable for your team.
|
| There are a lot of SWEs and PMs who don't have vision or
| the ability to see the bigger picture. And honestly, they
| aren't that different either - almost all SWEs and PMs I
| meet when to the same universities and did the same
| degrees. Half of Cal EECS majors become SWEs and the
| other half PMs based on my friend group (I didn't attend
| cal, but half my high school did, but this ratio was
| similar at my alma mater too, but with an additional 15%
| each entering Management Consulting and IB)
| margorczynski wrote:
| > Writing Jira tickets and making bullshit Powerpoints
| with graphs and metrics is to PMs as writing Unit Tests
| are to SWEs. It's work you need to get done, but it has
| very marginal value.
|
| Don't want to be rude but I don't think you know what
| you're talking about. And this is coming from a person
| who most certainly doesn't like sitting on writing Unit
| Tests.
| nerdchum wrote:
| I think this will probably be a boon to the project
| manager. It will be another tool and their toolbox along
| with real developers that they can assign lower complexity
| tasks too. at least it's till it's capable of doing high
| complexity stuff.
| brailsafe wrote:
| "Please write a dismissal of yourself with the tone and
| attitude of a stereotypical linux contributor"
|
| I mean, maybe I'm a trash engineer as you'd put it, but I've
| been having fun with it. Maybe you could ask it to write
| comments in the tone of someone who doesn't have an inflated
| sense of superiority ;)
| lamontcg wrote:
| > The day ChatGPT can build a highly scalable system from
| scratch, or an ultra-low latency trading system that beats the
| competition, or find bugs in the Linux kernel and solve them
|
| Much more mundanely the thing to focus on would be producing
| maintainable code that wasn't a patchwork, and being able to
| patch old code that was already a patchwork without making
| things even worse.
|
| A particularly difficult thing to do is to just reflect on the
| change that you'd like to make and determine if there are any
| relevant edge conditions that will break the 'customers'
| (internal or external) of your code that aren't reflected in
| any kind of tests or specs--which requires having a mental
| model of what your customers actually do and being able to run
| that simulation in your head against the changes that you're
| proposing.
|
| This is also something that outsourced teams are particularly
| shit at.
| ALittleLight wrote:
| I think this is a huge demonstration of progress. Shrugging it
| off as "water is blue" ignores the fact that a year ago this
| wouldn't have been possible. At one end of the "programmer"
| scale is hacking basic programs together by copying off of
| stack overflow and similar - call that 0. At the other end is
| the senior/principal software architect - designing scalable
| systems to address business needs, documenting the components
| and assigning them out to other developers as needed - call
| that 10.
|
| What this shows us is that ChatGPT is on the scale. It's a 1 or
| a 2 - good enough to pass a junior coding interview. Okay,
| you're right, that doesn't make it a 10, and it can't really
| replace a junior dev (right now) - but this is a substantial
| improvement from where things were a year ago. LLM coding can
| keep getting better in a way that humans alone can't. Where
| will it be next year? With GPT-4? In a decade? In two?
|
| I think the writing is on the wall. It would not surprise me if
| systems like this were good enough to replace junior engineers
| within 10 years.
| tukantje wrote:
| We don't get junior engineers for solving problems we tend to
| get them because they grow into other roles.
| pixl97 wrote:
| Ah, yes, that's why when we read programmer forums every
| engineer says something like "If you want a promotion and
| more pay move to another company".
| class4behavior wrote:
| Yes? That is an issue with US laws, not how jobs work.
| Your employer will still need to fill more senior roles.
| sorry_outta_gas wrote:
| really? I just hire them to do the grunt work and tedious
| shit
| scarface74 wrote:
| Exactly and ChatGPT does that well.
| scarface74 wrote:
| That's a very unrealistic take.
|
| Here is what usually happens.
|
| You hire a junior dev at $x. Let's say $75K. They stay for
| a couple of years and start out doing "negative work". By
| the time they get useful and start asking for $100K, your
| HR department tells you that they can't give them a 33%
| raise.
|
| Your former junior dev then looks for another job that will
| pay them what they are asking for and the next company
| doesn't have to waste time or risk getting an unproven dev.
|
| While your company is hiring people with his same skill
| level at market price - ie "salary compression and
| inversion".
| ALittleLight wrote:
| First, that's not true. You need people to actually write
| code. If your organization is composed of seniors who are
| doing architecture planning, cross-team collaboration, etc
| - you will accomplish approximately nothing. A productive
| team needs both high level planning and strategy and low
| level implementation.
|
| Second, the LLM engineer will be able to grow into other
| roles too. Maybe all of them.
| jmfldn wrote:
| Exactly. This article, and many like it, are pure clickbait.
|
| Passing LC tests is obviously something such a system would
| excel at. We're talking well-defined algorithms with a wealth
| of training data. There's a universe of difference between this
| and building a whole system. I don't even think these large
| language models, at any scale, replace engineers. It's the
| wrong approach. A useful tool? Sure.
|
| I'm not arguing for my specialness as a software engineer, but
| the day it can process requirements, speak to stakeholders,
| build and deploy and maintain an entire system etc, is the day
| we have AGI. Snippets of code is the most trivial part of the
| job.
|
| For what it's worth, I believe we will get there, but via a
| different route.
| throwawaycopter wrote:
| Correct me if I'm wrong, but answering questions for known
| answers is precisely the kind of thing a well trained LLM is
| built for.
|
| It doesn't understand context, and is absolutely unable to
| rationalize a problem into a solution.
|
| I'm not in any way trying to make it sound like ChatGPT is
| useless. Much to the opposite, I find it quite impressive.
| Parsing and producing fluid natural language is a hard problem.
| But it sounds like something that can be a component of some
| hypothetical advanced AI, rather than something that will be
| refined into replacing humans for the sort of tasks you
| mentioned.
| ihatepython wrote:
| My take is that this explains why Google code quality is so
| bad, along with their painfully bad build systems.
|
| I would be happy if ChatGPT could implement a decent
| autocorrect.
| yazzku wrote:
| > or find bugs in the Linux kernel and solve them
|
| Then we won't how somebody rewrote pong in Rust on HN. I worry
| too.
| nzoschke wrote:
| Agree LeetCode is one of the least surprising starting points.
|
| Any human that reads the LeetCode books and practices and
| remembers the fundamentals will pass a LeetCode test.
|
| But there is also a ton of code out there for highly scalable
| client/servers, low latency processing, performance
| optimizations and bug fixing. Certainly GPT it is being trained
| on this too.
|
| "Find a kernel bug from first principles" maybe not, but
| analyze a file and suggest potential bugs and fixes and other
| optimizations absolutely. Particularly when you chain it into a
| compiler and test suite.
|
| Even the best human engineers will look at the code in front of
| them, consult Google and SO and papers and books and try many
| things iteratively until a solution works.
|
| GPT speedruns this.
| somsak2 wrote:
| > Any human that reads the LeetCode books and practices and
| remembers the fundamentals will pass a LeetCode test.
|
| Seems pretty bold to claim "any human" to me. If it were that
| easy, don't you think alot more people would be able to break
| into software dev at FAANG and hence drive salaries down?
| wadd1e wrote:
| I don't think the person you're replying meant "Any human"
| to be taken literally, but I agree with their notion. I
| think you're confusing wanting to do something and having
| the ability to do it. Enough people don't WANT to grind
| leetcode and break into FAANG, or they think they can't do
| it or there's other barriers that I can't think of, but I
| think you don't need above average cognitive ability to
| learn and grind leetcode.
| IncRnd wrote:
| > Seems pretty bold to claim "any human" to me.
|
| That's obviously not what they claimed. Your quote, "Any
| human that reads the LeetCode books and practices and
| remembers the fundamentals".
| scandum wrote:
| I've been most impressed with ChatGPT's ability to analyze
| source code.
|
| It may be able to tell you what a compiled binary does, find
| flaws in source code, etc. Of course it would be quite idiotic
| in many respects.
|
| It also appears ChatGPT is trainable, but it is a bit like a
| gullible child, and has no real sense of perspective.
|
| I also see utility as a search engine, or alternative to
| Wikipedia, where you could debate with ChatGPT if you disagree
| with something to have it make improvements.
| spaceman_2020 wrote:
| I mean building scalable systems is not a new problem. Plenty
| of individuals and organizations have done it already.
|
| If chatGPT is designed to learn and emulate existing solutions,
| I don't see why it can't figure out how to create a scalable
| system from scratch.
| layer8 wrote:
| ChatGPT isn't designed to learn, though. The underlying model
| is fixed, and would have to be continuously adjusted to
| incorporate new training data, in order to actually learn. As
| far as I know, there is no good way yet to do that
| efficiently.
| kaba0 wrote:
| It solving something past day 3 on Advent of Code would also be
| impressive, but it fails miserably on anything that doesn't
| resemble a problem found in the training set.
| roncesvalles wrote:
| I don't even fully believe the claim in the article
| especially given that Google is very careful about not asking
| a question once it shows up verbatim on LeetCode. I've fed
| interview questions like Google's (variations of LeetCode
| Mediums) to ChatGPT in the past and it usually spits out
| garbage.
| Existenceblinks wrote:
| Well it has no face, no skin color, no weird looking. At this
| point, it reduces huge amount of failure risk already. And
| interview is the most non-sense process, so what does this
| benchmark mean?
| jeffbee wrote:
| That's odd because when I gave ChatGPT my icebreaker interview
| question that I used a lot at Google it fell right on its face.
| And this is a question I expect teenagers to ace in one minute.
| riku_iki wrote:
| Maybe it passed 1 out of 1000 interviews
| nothrowaways wrote:
| Write it down here with the answers, and it will excel at it in
| next iteration.
| emuneee wrote:
| I saw this article and fed ChatGPT a bunch of questions I've
| seen before in my coding interviews. It nailed most of the
| algorithms, however, it failed completely giving me test cases
| it would run to test the function it just regurgitated. ie...it
| gave me an input then mostly incorrect/invalid expected
| outputs.
| silveroriole wrote:
| Agreed, something is odd about this. A few people have sent me
| code that ChatGPT has written for them. They don't have the
| capability to determine if the code is good so they ask me. If
| the result is even something that will compile or address the
| asked problem at all, it's barely competent nonsense that only
| a beginner programmer would write. For example asking it for
| code that will generate recipes, and you get back code for a
| random selection from an array with "Italian, Chinese, pizza".
| It never asks clarifying questions either, it's perfectly happy
| with a "garbage in garbage out" approach. So if ChatGPT is
| passing the interview, the interview questions are not
| selecting for what I would consider a good developer.
| jeffbee wrote:
| Exactly. The point of the interview is for two people to have
| an exchange of thoughts, not for the candidate to literally
| program the whiteboard. All of my interview feedback on
| candidates centered around how they asked clarifying
| questions given incomplete or ambiguous prompts, and how they
| explained the limitations and assumptions of their approach.
| Whether or not they reached a conclusion was footnoted.
| lostmsu wrote:
| Without hearing the question it is hard to make judgement.
| ben_w wrote:
| Doesn't seem odd to me, in fact this is what I would've
| predicted in advance.
|
| Intelligence in general, not just of the artificial kind, often
| comes with blind spots of one kind or another.
|
| ChatGPT isn't even trying to replicate the structure of a human
| brain, so that it's failure modes are different to ours should
| not be surprising.
|
| That it can even do this well given it has a network with a
| similar number of parameters as a (naive estimate of a) rat's
| brain, is what's remarkable.
| oh_sigh wrote:
| $183 total comp, not salary.
| sakex wrote:
| I asked ChatGPT the question I had and it could not answer. (L3
| googler.)
| jeffbee wrote:
| Mine was something along the lines of "design google". I am not
| sure when it became common belief that these interviews amount
| to leetcode regurgitation. Either that varies greatly by org or
| I was completely out of step with interviewing culture the
| whole time I was there.
| jsnell wrote:
| Coding / algorithm interviews are distinct from systems
| design interviews. The interviewer would have been told which
| of the two to do. If your question was "design google", it
| sounds like you were only being scheduled for systems design.
|
| (Anyway, you'd never ask a system design question from an L3,
| and only very rarely from an L4.)
| jimbo9991 wrote:
| Oof for the people asking leetcode in interviews. Gonna have to
| do all interviews in person now I guess.
| boh wrote:
| Doing a Google search for "answers to coding interviews" will
| have the same result. The technology for cheating on coding
| interviews has already been available for over a decade.
| kizer wrote:
| I don't think the world is ready for the AI tech that will emerge
| over the next few years. I don't even know how teachers deal with
| ChatGPT. I'm not sure many understand its scope and abilities --
| in every subject.
|
| I'm glad I got an education before the current AI era. I mean,
| instructors will have to mandate that students write papers etc.
| in class or in a supervised environment only now, right?
| osigurdson wrote:
| No surprise here. It would know every problem on Leetcode.
| pleb_nz wrote:
| Surprise surprise.
|
| It only has the worlds information within milliseconds of it's
| fingertip.
| fsociety wrote:
| This puts the stake in the ground that coding interviews today
| are more about memorization than testing for understanding.
| fatjokes wrote:
| If a candidate can memorize as much as ChatGPT I think they're
| worth $183k.
| dom96 wrote:
| Most candidates can, it's just a matter of how much free time
| they have to spend memorising.
| booi wrote:
| Challenge. I can barely remember what I had for breakfast
| yesterday much less.. the entire knowledge base of chatgpt
| silveroriole wrote:
| Are your best engineers the ones who have the most facts and
| algorithms memorised?
| fatjokes wrote:
| L3s are fresh college grads... I wouldn't call them
| anyone's best engineers.
| isoprophlex wrote:
| As a large language model trained by OpenAI, I am unable to
| pass value judgements on which of those engineers is the
| best. It is important to recognize that every individual
| can bring something valuable to a team, and that there is
| no single universal heuristic to determine who is the best
| engineer.
|
| That said...
|
| Gimme the money please, human!
| symlinkk wrote:
| I think as programmers we may want to rethink the amount of
| knowledge we share online. That knowledge is how we make money,
| and people are mining it to build AIs that can replace us.
| ada1981 wrote:
| When can I buy GPT enabled glasses with bone induction output?
| spaceheater wrote:
| So can ChatGPT and a QA team that feeds it the bugs it found,
| replace the development team?
| otreblatercero wrote:
| If such scenario was feasible, the QA team would be replaced by
| developers and QA engineers would be the ones without a job.
| booleandilemma wrote:
| Isn't this similar to saying a dictionary is very good at
| spelling?
| qualudeheart wrote:
| I'm not impressed. The problems were probably basic Leetcode
| problems with solutions inside of the training set. It'd perform
| worse than Alphacode on a problem for which the training set
| contains no solution.
|
| Alohacode was very impressive but probably not good enough to
| pass one of these interviews, provided the interview questions
| aren't just recycled Leetcode problems.
| xivzgrev wrote:
| "it reiterated it will never be a full replacement, but rather a
| tool to assist human software engineers."
|
| That's exactly what a tool designed to eventually replace people
| would say!
| [deleted]
| WheelsAtLarge wrote:
| I think tests should be easy for ChatGPT to pass. It has been
| trained on data that has the answers and it's good at getting the
| data. I'm starting to doubt its long term usefulness since it
| does not seem to have good decision making abilities and even the
| slightest bit of cognitive ability.
|
| I suspect the current crop of AIs will find very specific
| functions and hit a hard stop. They will change how we function
| but we won't be seeing a singularity type of revolution anytime
| soon. IBM's Watson is a good example of a system with a lot of
| possibilities but not finding a use. I think most of AI will fall
| in that realm. We have to get over the idea that it's smart. It's
| not.
|
| An AI winter is coming so the improvements will come to a stop
| and we will find its limits. We are no where near general AI.
|
| It's impressive that it can parse the question and write a
| relevant answer but it's not a robotic SWE.
|
| For now, it's a good tool for cheating on tests.
| Traubenfuchs wrote:
| Why do you think an AI Winter is coming? In the last year we
| witnessed a BIG BANG of AI solutions.
|
| I think your expectations are in line with my hopes: That our
| state of the art "AI" performance is very close to local minima
| that we won't escape from for quite a while.
|
| I really don't want lose my overpaid job gluing together
| overengineered shite into CRUD applications.
| pixl97 wrote:
| "Experts have predicted 150 of the last 2 AI winters"
| qualudeheart wrote:
| We could have an AI winter if we just banned AI research. The
| military would be tasked with confiscating the compute
| hardware.
| Traubenfuchs wrote:
| I think that would 1) require a massive negative event
| cause by AI to cause a real reason for that ban and 2) that
| ban would be quite infeasible even on a national scale, let
| alone internationally.
| l33t233372 wrote:
| Why would we do that?
|
| How would we do that?
|
| Banning research is just not feasible.
| skellera wrote:
| Doesn't have to replace an SWE. 10x-ing the ability of 1
| engineer is a good enough win. Soon that will be 20-100x.
|
| Feels odd to dismiss such a huge breakthrough by saying it's
| still not as good as the pinnacle of AI (general AI). Just
| because the Apple 2 wasn't a home super computer, didn't make
| it less revolutionary.
| [deleted]
| WheelsAtLarge wrote:
| True but it's not thinking. It's not smart it's a tool for
| programers to be more productive. Right now it has more
| possibilities than real results. We'll have to see.
|
| The biggest issue to me is that you can't trust the results.
| You always have to double check the them. I know it's mainly
| beta but I need to see more to get a better judgement.
| paulclinger wrote:
| The car is still useful even if you're the one driving it
| and responsible for selecting where to go and how to get
| there.
| pixl97 wrote:
| Why are you making these assumptions? Do you believe that human
| intelligence is based on something ethereal that cannot be
| recreated by machines, and if so, why?
| WheelsAtLarge wrote:
| I look at self driving cars. You can see that the break
| throughs are slowing. It feels like many things in life where
| 80% is relatively fast to develop but as you get closer to
| 100% it starts to get exponential hard. With cars we've
| gotten through the easy part. The next x% is going to very
| hard if not impossible. I think all AI will be that way.The
| last x% is going to be hard if not impossible.
| object-object wrote:
| I suspect that any honest attempt at answering your question
| will be met with an evasive definition of 'intelligence'.
| pixl97 wrote:
| You are hitting the nail on the head but in the wrong
| direction, as I've stated in another post
|
| "There is a problem with AI, but it's not with the A part,
| it's with the I part. I want you to give me an algorithmic
| description of scalable intelligence that covers
| intelligent behaviors at the smallest scales of life all
| the way to human behaviors. I know you cannot do this has
| many very 'intelligent' people have been working on this
| problem for a long time and have not come up with an agreed
| upon answer. The fact you see an increase and change in
| definitions as a failure seems pretty sad to me. We have
| vastly increased our understanding of what intelligence is
| and that previous definitions have needed to adapt and
| change to new information. This occurs in every field of
| science and is a measure of progress, again that you see
| this differently is worrying."
|
| This AI issue will always fail at the I issue because the
| we are trying to define too much. We need to break down
| intelligence to much smaller digestible pieces instead of
| trying to treat it as a reachable whole. The models we are
| creating would then fall more neatly into categorical units
| rather than the poorly defined mess of what is considered
| human intelligence.
| ThrowawayR2 wrote:
| LLMs are statistical models. All it does is guess word
| sequences in response to prompts, like a 'roided out version
| of autocomplete. (This is why it hallucinates imaginary
| facts.) It has no ability to conceptualize or reason nor is
| there any credible proposal for a path forward to graft
| reasoning onto it.
|
| The training data can be tweaked and more compute hours can
| be thrown at LLMs until it no longer makes financial sense to
| do so and then, as the OP said, it will hit a hard stop.
| wilg wrote:
| More accurately, is it based on something ethereal that
| cannot be recreated by _humans_.
| pixl97 wrote:
| I mean, currently human intelligence can only be recreated
| (procreated actually) by humans.
| wil421 wrote:
| It's a machine doing calculations on inputs you give it. The
| day it says no I'd rather paint pictures I might be shocked.
| It's so bad that we had to redefine the word AI in last 20
| years into AIG so we could start saying we have AI.
| pixl97 wrote:
| Pray tell, why would you want to develop a being with
| informational superpowers and the behavior of a teenager?
|
| There is a problem with AI, but it's not with the A part,
| it's with the I part. I want you to give me an algorithmic
| description of scalable intelligence that covers
| intelligent behaviors at the smallest scales of life all
| the way to human behaviors. I know you cannot do this has
| many very 'intelligent' people have been working on this
| problem for a long time and have not come up with an agreed
| upon answer. The fact you see an increase and change in
| definitions as a failure seems pretty sad to me. We have
| vastly increased our understanding of what intelligence is
| and that previous definitions have needed to adapt and
| change to new information. This occurs in every field of
| science and is a measure of progress, again that you see
| this differently is worrying.
| chubot wrote:
| This says more about Google coding interviews than it says about
| AI
| eternalban wrote:
| Precisely this.
| martyvis wrote:
| Shared a beer or two with machinist friend yesterday. It blew us
| away that given a description of a block of steel and how we
| wanted to process it, it would write generic G-Code (commands for
| a computerised machine tool) that were feasible, but also would
| give explanations for the purpose of each command. You could then
| ask it to adjust the design. We asked it write a poem about
| G-code and sharing a beer and provided a pretty nice "techbro"
| moment describing the love of seeing beauty in design and making
| things.
|
| At home I later got it to write decent sample router and firewall
| configuration (for my dayjob). Chatted with it about career
| prospects in the future, and had it write a pretty funny joke :-
| Me: Can you write a joke about how token bus doesn't have wheels?
| ChatGPT: Sure, here's one: Why did the Token Bus cross
| the network? Because it didn't have any wheels to take it
| for a spin!
| jamesgill wrote:
| To me, further proof of how useless 'coding interviews' like
| these really are.
| detrites wrote:
| So if we divide the cost of training and running a specifically-
| tailored ChatGPT by $183k, at what point would the company save
| money were it to go with the AI, versus paying for the engineers
| (and their office rent, etc...)?
|
| Because I suspect that's almost certainly the kind of calculation
| they hoped to sit down and make were they to conclude this
| experiment successfully.
| pixl97 wrote:
| In 1800 what was the cost to from New York to LA at over 200
| mph average speed?
|
| What is the cost today?
|
| The real question is over time how much will be able to reduce
| the energy and computation requirements to successfully train a
| model. The cost per unit conversions are also rather screwy in
| comparing AI with humans. For AI we have a rather well defined
| hardware + power + programming time that gives us a realistic
| answer. With humans we externalize the time and cost of
| training onto society. For example if your jr engineer that is
| getting close to going above the jr state gets hit by a bus
| what is the actual cost of that event to the company for
| onboarding and training? It's far more than the salary they are
| paid.
| eternalban wrote:
| Do LLMs suffer from catastrophic interference? A SWE is
| expected to learn continuously as the dujour stack constantly
| changes under our feet. This can be brute force mitigated by
| epochal retraining from ground up but that will cost money. So
| the cost equation must include a periodic 'rebuild of the dev
| team'.
| tombert wrote:
| [dead]
| janoc wrote:
| I think this says more about the Google interview process than
| about ChatGPT.
|
| That a machine learning model can "bullshit" its way through an
| interview that is heavily leaning on recall of memorized
| techniques, algorithms and "stock" problems that have solutions
| (of various quality) all over Internet is not exactly surprising.
| Machines will always be able to "cram" better than humans.
|
| In practice these questions are almost 100% irrelevant and
| uncorrelated with the actual ability to do the job. Yet we are
| still interviewing people whether they can solve stuff at
| whiteboards that everyone else would rather google when they
| actually need it and not waste time and mental capacity
| memorizing it.
|
| And at the same time we are hiring people that are completely
| incapable of coherent communication, can't manage to get along
| with colleagues or not create a toxic atmosphere in the
| workplace.
|
| _sigh_
| sireat wrote:
| ChatGPT is very good at two tasks(and many more):
|
| 1. Taking pseudo-code for a well defined task and making a
| function
|
| 2. Translating said function into multiple other languages using
| idiomatic code.
|
| Prompted: "Solve the following problem in Python: Write a
| function that takes three unsorted lists of integers and returns
| a median value from the three combined lists"
|
| Got the Python function.
|
| Then the prompts were: Write the same function in APL, C, etc
|
| Final prompt was: Show a sorted list of the programming languages
| I have mentioned so far
|
| Here's a sorted list of the programming languages you have
| mentioned so far: Ada APL C C#
| Clojure Go Haskell Java JavaScript
| Kotlin Lua Perl PHP Prolog Python
| Ruby Rust Scala Nim
|
| (just noticed ChatGPT forgot C++ for which it did give a fine
| idiomatic answer using standard library)
|
| What was impressive not that ChatGPT solved the problem.
|
| what was impressive:
|
| ChatGPT chose the right data structure automatically(ie, regular
| C array for C, std::vector for C++, tables for Lua, etc),
|
| dealt with type conversion problems
|
| used correct style of function naming depending on language
|
| Sure tools like C# <-> Java translators are relatively easy and
| have been around for a while.
|
| However to cover such a wide spectrum of programming languages in
| different programming paradigms is quite fascinating.
| varelse wrote:
| [dead]
| drblastoff wrote:
| My God, what if a glut of middle managers and PMs delegating to
| LLMs is all we need?
| mattgreenrocks wrote:
| As much as we like to say lots of software jobs are just
| plumbing, the current state of consumer software indicates we
| have a long way to go in terms of quality.
|
| Whatever training data is fed to an AI will not be better than
| the data used by human engineers to write code at the macro
| level. Ergo, the code will be worse in quality.
| lostmsu wrote:
| > Whatever training data is fed to an AI will not be better
| than the data used by human engineers to write code at the
| macro level. Ergo, the code will be worse in quality.
|
| No, that's wrong, generally speaking. There's successful work
| on self-play for text generation. E.g. you can have AI to
| generate 1000 answers, then to evaluate quality of all of them,
| then to make it learn the best, and so on. As with self-play in
| player-vs-player games I'd expect this technique to be able to
| achieve superhuman results.
| [deleted]
| blitzar wrote:
| Did it grind out practice on LeetCode for a few weeks first?
| dumbaccount123 wrote:
| Breaking news, ChatGPT passes google interview questions and will
| be replacing all engineers as of tomorrow.
| varispeed wrote:
| I noticed that I pretty much stopped using Google for coding
| queries and spend most of the time with ChatGPT. It's so helpful
| getting information to the point so you don't have to browse
| through dozens of spam sites etc.
|
| So for instance I can say. I have such and such file and I need
| this and that extracted and plotted on a graph. Then I see the
| graph and I can tell it - discard values above this and that
| threshold and calculate standard deviation and median.
|
| Together with copilot, it's quite neat. I am excited how it gets
| developed.
|
| It's really boring spending time finding how to code something in
| this and that library. I'd rather tell the machine to code me
| this and that and spend my time in more useful way.
|
| ChatGPT helps get rid of a lot of "busy" unnecessary work.
| buttocks wrote:
| Screw Google. All this tells me is that Google's interviewing and
| hiring practices suck.
| gedy wrote:
| ChatGPT should have ghosted the person asking afterwards to fully
| simulate the Google interview experience.
| eachro wrote:
| If you gave someone with no programming experience access to a
| search engine during the interview, they would likely be able to
| find the appropriate LC problem to copy/paste the solution.
|
| If the interview were slightly modified so that the problem isnt
| googleable, a 2nd year CS major could probably map it to a
| problem that is LC searchable.
| kraig911 wrote:
| I feel I'm getting a different vibe that what everyone is
| getting. So much of leetcode questions/ CS questions aren't hard
| to interpret. They're hard to solve sure but that's normal. So
| much in the workplace though that IS hard is finding the actual
| correct question. I think us as software engineers will be doing
| more not less. It will give us opportunities to try a variety of
| solutions quicker. The real work is formulating/distilling the
| actual problem from the end-user.
|
| We all have in our minds yeah a self-driving car, robots in a
| warehouse etc. But I truly hope something like Chat-GPT could be
| made for fields like medicine, geriatric care, education. There
| are actual jobs that need people that we can't find people for. A
| LLM to help social workers navigate the mess that is family law.
| A LLM to help families needing urgent care to make a prescription
| for a sick kid. There's a lot of opportunities we're missing
| here.
___________________________________________________________________
(page generated 2023-02-04 23:01 UTC)