[HN Gopher] How much would it have cost if GPT-4 had written you...
___________________________________________________________________
How much would it have cost if GPT-4 had written your code
Author : yodon
Score : 37 points
Date : 2023-05-29 20:39 UTC (2 hours ago)
(HTM) web link (pypi.org)
(TXT) w3m dump (pypi.org)
| BeefySwain wrote:
| Come on everyone, we all know that "most of the work that goes
| into software engineering isn't writing code", that "it's
| impossible to create a working system in one shot", etc etc etc,
| and therefore this tool will not give a realistic idea of what it
| would cost to build a system with nothing but GPT-4.
|
| No one here (including the author) is under the impression that
| you can just ask ChatGPT to "write a Unix-like kernel" and get
| Linux spit out the other side.
|
| That said, I am really curious what it would cost, as a
| theoretical lower bound, to pay OpenAI to write the Linux Kernel.
| 10 grand? 100 grand? I have no clue, but I'm glad someone wrote a
| tool to make it easier to find out!
| londons_explore wrote:
| > No one here (including the author) is under the impression
| that you can just ask Chat GPT to "write a Unix-like kernel"
| and get Linux spit out the other side.
|
| I really don't see such a thing being far off. GPT-4 is already
| pretty good at writing small modules ("write a disk IO queue
| for my custom kernel in C"). With a little more work in
| allowing GPT-4 to test out code it has written and iteratively
| make changes, allowing it to use debuggers, benchmarks, and
| sanitizers, allowing it to write its own tools, and then put
| modules together, I think we could very soon be asking it to
| "write me a Unix-like kernel".
| gwoolhurme wrote:
| It's that good in your experience? Enough that it is close to
| writing something as nice as the Linux Kernel we have now?
| How close is close? Reading comments like these makes me feel
| like I'm swallowing crazy pills. It legitimately makes me
| feel like I'm using it wrong. I do Android and iOS native
| development and some IoT and it gives me really wrong code
| very often. To the point that I don't see much difference
| between chatgpt and gpt4. But you say it's close to just
| "write me a unix-like kernel"
| jakear wrote:
| Personally I couldn't even get it to draw me a regular
| pentagon in CSS. It was happy to draw hexagons and call
| them pentagons. It was even happy to go back and fix it
| mistakes when I informed it that it was making hexagons,
| not pentagons. It, of course, readily accepted that I was
| correct and it had indeed drawn a hexagon, but this time
| it'd be different. This time, it'd draw a pentagon. And...
| repeat.
| starbugs wrote:
| It's not close and I also see no big difference between GPT
| 3.5 and 4. Don't get hyped. I am sure it will eventually
| happen, but calling it close is very optimistic.
| londons_explore wrote:
| Not close to as in "it can nearly write it correctly", but
| close to as in "I believe within a small number of years,
| gpt-4-like tools would be able to write you a unix-like
| kernel from scratch and have it actually work, with no
| human input".
|
| I think we already have most of the pieces in place:
|
| * big language models that sometimes get the right answer.
|
| * language models with the ability to write instructions
| for other language models (ie. writing a project plan, and
| then completing each item of the plan, and then putting the
| results together).
|
| * language models with the ability to use tools (ie. 'run
| valgrind, tell the model what it says, and then the model
| will modify the code to fix the valgrind error')
|
| * language models with the ability to summarize large
| things to small.
|
| * language models with the ability to review existing work
| and see what needs changing to meet a goal, including
| chucking out work that isn't right/fit for purpose.
|
| With all these pieces, it really seems that with enough
| compute/budget, we are awfully close...
| gwoolhurme wrote:
| Is this that hard of a tool? Why is it a package? Seems like
| something someone with a few lines of code could do. OpenAI is
| transparent and fairly open about pricing per token... it seems
| like it's just adding bloat.
| dathinab wrote:
| all your mony, as you never would have reached a complient
| program
|
| maybe with GPT-6 or so, we will see, but not with GPT-4
| gwoolhurme wrote:
| The entire premise of this thing is wrong too because what if you
| need to regenerate something? Which even with the mighty gpt4?
| Still happens a lot... It's almost akin to me asking a co-worker
| how fast he types and calculating how much of his salary it is
| for him to generate code based on that without any other factors.
| Also pointed out but why is this even a package? It seems like
| the definition of bloat when this should have just been a blog
| post tool or something. It's all completely unnecessary and seems
| to try and cash in on hype.
| jumploops wrote:
| It cost me $7.61 to build this library[0] using GPT-4, much more
| than the direct cost of tokens required to reimplement the
| library.
|
| The majority of the cost was the large context windows being
| proxied back and forth to OpenAI from my local machine, rather
| than just the additional code with each new message. Also, I had
| no idea how to do what I was doing, so even if I could tell GPT-4
| exactly what to build, I wasn't even sure it was possible!
|
| [0] https://github.com/jumploops/magic
| tr33house wrote:
| now, was this written ... by GPT-4?
| wild_pointer wrote:
| It's even cheaper to get a monkey to type that code for you. Will
| probably cost a banana. Chances it will type the code you expect?
| Similar to GPT-4 for anything remotely non-trivial.
| planetpluta wrote:
| If a banana costs ~60C/ (depending where you get it!) and GPT-4
| is 6C/ per 1k tokens... you're better off using a monkey for
| everything > 10k tokens!
| KMnO4 wrote:
| What's the purpose of putting this on the package index instead
| of just a GitHub repository?
| elbigbad wrote:
| Better discoverability and so people could very easily install
| and run. Git adds more overhead.
| Mistletoe wrote:
| Can anyone give me some estimates instead of defending your jobs
| as coders?
| gwoolhurme wrote:
| What? Nobody is defending their jobs here. It's explaining how
| non-trivial this is. What is the point of your comment? If you
| wanted pure token to money amount and are too lazy to calculate
| it your self use the tool in the link. That's not how the real
| world even with an LLM will work. Let's assume the LLM writes
| everything, even gpt4 won't get everything in 1 go now.
| Requirements of software are hard to put into pure English...
| why write this snarky message. Adds nothing to the
| conversation.
| swyx wrote:
| its obviously an impossible question to answer with confidence
| but i attempted to put some fermi estimates here
| https://twitter.com/swyx/status/1657603239251181570
|
| and based on kiinda conservative assumptions, ai coders start
| being competitive in 2028
| gwoolhurme wrote:
| This seems sloppy too. Is this based on anything
| scientifically rigorous? Where I live even startups will take
| on less profit for a quarter to train new graduates who have
| never touched an IDE on purpose. The point of training a
| junior isn't to be immediately competitive. It's to take over
| your job as you move on to other roles.
| Mistletoe wrote:
| Thank you!
| Jeff_Brown wrote:
| How much money could a restaurant save by replacing the food with
| dirt.
| ldoughty wrote:
| Isn't this akin to: "The cost of an iphone is $500" ? Sure, it's
| $500 in parts if you just quantify the final working result...
| but it ignores all prototyping, overhead, and manual work to link
| together code (e.g. assembling the phone)...
|
| Maybe I'm misunderstanding just how good GPT-4 is at making
| entire repos of code... or exactly how this tool functions?
| starbugs wrote:
| You aren't misunderstanding it. This is not a valid way to
| calculate the cost of your code.
| SketchySeaBeast wrote:
| Well, that's a fun exercise, but it's not the volume of code that
| has value, it's how it's put together.
| taneq wrote:
| What, you don't pay your devs per kilogram of code? ;)
| krick wrote:
| Volume is measured in liters or m3.
| magic_hamster wrote:
| It's how much it would cost to produce the text files in your
| code. I think that most software engineers will agree that
| writing the code is the easiest part of the job. The hard parts
| are figuring out the problem, figuring out your data modeling,
| what is going to work and what it would cost, considering
| tradeoffs in speed, scalability, maintainability, etc. And, you
| know, getting the thing to actually run.
|
| There's no doubt that AI is going to be a huge factor in coding.
| But we will have to change a lot of things for it to outright
| replace humans. I'm sure some people are already working on those
| changes...
| mackopes wrote:
| I could imagine an LLM that brainstorms all the edge cases,
| data modelling, algorithm, etc... The final decision will still
| have to be done by a human, but having a companion LLM that
| gives you ideas, plays devils advocate and looks for problems
| would definitely speed up the decision process.
| starbugs wrote:
| Unless it leads you astray and that costs more time than
| coming up with the code on your own in the first place.
| grogenaut wrote:
| Wait, sounds like you've rolled out SAP before, I'd love to
| pick your brain for tips other than "don't".
| starbugs wrote:
| > don't
|
| Just don't
| webnrrd2k wrote:
| Even all that, difficult though it is, doesn't include the cost
| if understanding the actual problem, knowing the field well
| enough to propose a good solution, and coordinating and
| communicating with everyone the whole way through.
|
| "Every problem is a people problem", and people-solutions are
| something that AI still doesn't solve.
|
| I'm not saying that AI doesn't have it's uses, it certainly
| does, but I'm trying to highlight that purely technical
| solutions very often aren't good solutions to a lot if real-
| world problems.
| worldsayshi wrote:
| When AI truly becomes helpful in the workplace developer jobs
| will probably change a lot but I think the biggest change
| will be in the size of the business scope. Most projects I've
| worked in could easily grow ten times if developers were ten
| times as efficient.
|
| At the same time I've always wondered why IT problems haven't
| all been solved comprehensively already because so many
| problems seem so similar.
|
| I can't really fit those two narratives together.
| lumost wrote:
| This... is not what people would have said 10 years ago.
| Remember the idea of the 10x developer? Even at a design heavy
| firm, 80% of engineers _despise_ writing the design docs.
| 0xr0kk3r wrote:
| The hardest part is supporting the platform for 7~10 years.
| With those lifetimes, writing code is like 5% of the effort,
| the rest is supporting it for years and handling all the nasty
| bugs that show up when thousands of engineers pound on it 24/7.
| They don't teach you about that in college, and I wish they
| did, but I doubt anyone would take a class on how to support
| software even though it's the most important part.
| midasuni wrote:
| I've got code barely touched in 15 years still in day to day
| production use.
| taneq wrote:
| Isn't it nice when requirements don't change! :)
| theWreckluse wrote:
| Let's see how it does when it's constantly worked on to add
| new features, etc.
| polishdude20 wrote:
| And every few months someone wants to switch to a new
| framework or module for one specific reason and then the
| codebase is riddled with 10 different ways to do the same
| thing.
| layer8 wrote:
| I'll add that 7-10 years is rather on the short side compared
| to the systems I've been involved with in my career. 20 years
| isn't that rare, and the more interesting challenges tend to
| happen after the 10-year mark.
| BrissyCoder wrote:
| The hard part is figuring out other peoples code.
| pmontra wrote:
| A lot of conversation is needed to create that code. I
| guesstimate that the actual tokens are at least 100 times than
| the ones in the repository, even if one can convince ChatGPT not
| to explain every choice it makes.
| starbugs wrote:
| > convince ChatGPT not to explain every choice it makes
|
| I assume this is a major issue, because it tends to be more
| talky than my ex which is a real feat.
| jbverschoor wrote:
| Let's just all print $100 bills.. it can be done for under $5
| starbugs wrote:
| This seems to assume that GPT4 can actually generate the code of
| your repo. Good luck with that assumption.
| mackopes wrote:
| It seems that the package calculates the lower bound of how much
| would it cost to write the same code with GPT-4. It does not take
| into consideration that even GPT-4 cannot write code at god mode
| and write it correctly on the first try.
|
| However, it gets interesting when you realise that if writing the
| code with an LLM is dirt cheap, then 1000 iterations of writing
| the same code with the guidance of a skilled software engineer
| would still be cheap and probably faster. I can imagine a world
| where whole engineering teams are replaced by just one engineer
| with a code-generating LLM.
___________________________________________________________________
(page generated 2023-05-29 23:01 UTC)