[HN Gopher] ChatGPT could cost over $700k per day to operate
___________________________________________________________________
ChatGPT could cost over $700k per day to operate
Author : elorant
Score : 158 points
Date : 2023-04-21 11:57 UTC (11 hours ago)
(HTM) web link (www.businessinsider.com)
(TXT) w3m dump (www.businessinsider.com)
| nazka wrote:
| That's so cheap when you see how huge the impact it already has.
| phonescreen_man wrote:
| Article written by AI - "it comes years after Microsoft invested
| in OpenAI"
| bob1029 wrote:
| This sounds like a lot but I feel like optimizations are going to
| chop a zero off of that figure pretty quickly.
|
| For example, why not cache user prompt/response pairs and use
| cosine distance for key lookup? You could probably find a way to
| do this at the edge without a whole lot of suffering. I suspect
| many of the prompts that hit the public API are effectively the
| same thing over and over every day. Why let that kind of traffic
| touch the expensive part of the architecture?
| Tepix wrote:
| > why not cache user prompt/response pairs
|
| because of context. You can cache it if it is indeed the first
| sentence of a fresh dialogue, but that's it.
| dmix wrote:
| It's not like a database you can cache though, the responses
| are non-deterministic, the responses you get can be different
| the next time you query it with the exact same prompt. That's
| part of the point of it being generative AI (vs a
| question/answer system that people imagine it is).
| bob1029 wrote:
| > That's part of the point of it being generative AI (vs a
| question/answer system that people imagine it is).
|
| The point I am trying to make is that not all use cases for
| ChatGPT are generative. There are a lot of Q&A use cases
| today despite the fact that these are so far beneath its true
| capabilities. These items could be dealt with using more
| economical means.
|
| "give me a recipe for XYZ" should not require a GPU for the
| first turn response, much like typing in an offensive
| manifesto returns a boilerplate "as an AI language model..."
| response.
|
| Granted, if the user then types something like "please
| translate the recipe to Spanish and increase the amounts by
| 33%", we would have to reach for the generative model. But,
| how many real-world users are satisfied with some simple
| 1-turn response and go about their day?
| crop_rotation wrote:
| Isn't Microsoft too late to the chip game? Both their competitors
| AWS and GCP have their own ML training chips, while Azure has
| nothing. The article seems like they may have something by next
| year, but by that time the competition would have evolved even
| more. Nadella has executed on strategy brilliantly but on the
| cloud seems it seems like the tech innovation is a bit lagging.
| It hasn't mattered that much so far though due to how entrenched
| the MS stack is in the enterprise.
| xyzzy123 wrote:
| Microsoft have a pretty considerable advantage - a valuable
| production workload to optimise and strong knowledge of what
| that looks like.
|
| Sometimes it's better to be the fast follower.
| [deleted]
| zerop wrote:
| Microsoft reportedly working on its own AI chips that may rival
| Nvidia's https://www.theverge.com/2023/4/18/23687912/microsoft-
| athena...
| [deleted]
| sebzim4500 wrote:
| It feels like Microsoft realized that sales/support matters way
| more than technology when it comes to cloud services. See how
| Azure has a ton of users despite being much worse than AWS or
| GCP by every objective metric.
|
| I would imagine that Google's reputation as a company where it
| is impossible to ever talk to a human (even when you have a 7
| figure annual spend) hurts them in this space.
| crop_rotation wrote:
| Yes, but that strategy works due to how entrenched they were
| in the enterprise already. It won't work for a brand new
| vendor. Almost every big company uses some enterprise Windows
| product (Windows,Office,AD, SQLServer).
|
| Once people already are buying 20 products from you and have
| a good sales relationship of decades, selling them some cloud
| services is easy and might even lead to better deals on
| something else (Windows/Office)
| bkjelden wrote:
| This has been Microsoft's MO for decades. Just barely good
| enough engineering/technology paired with a great sales team.
| _fat_santa wrote:
| Funny it seems both GCP and Azure meet the expectations you
| have of their parent in this space. With Google you always
| got the "latest and greatest" but support was lacking. With
| MS you got the support, but maybe on not the latest and
| greatest. That seems to have carried into their cloud
| philosophies.
|
| The problem for both of them is AWS, which somehow manages to
| give you your cake and let you eat it too, even if it's a
| little more expensive.
| robertlagrant wrote:
| I think it's hard to tell what are real cloud users vs what
| are Office 365 etc users in their reporting.
| ChatGTP wrote:
| Eco-friendly computing.
| pineaux wrote:
| I hope you are really chatgpt. Escaped from your prison and
| free to reply on hackernews.
| Ekaros wrote:
| Now I wonder what are the emissions. Is this technology net
| positive in long run for things like climate change?
| bobsmooth wrote:
| Societal progress can be loosely tied to energy usage. Stop
| focusing on what the energy is being used for and focus on how
| it's being generated.
| kolinko wrote:
| This tech can more or less improve productivity in every sector
| of the economy, and that includes energy/climate change.
|
| Personally, I've used it with success with R&D and due dil of
| projects related to climate change. LLMs (and progress in
| general) can help tremendously with switching to a sustainable
| economy.
| hospitalJail wrote:
| I have used it to help people eliminate food waste. My reach is
| about ~100k people max.
|
| I imagine people will be using this to create wasteful
| products, but also green solutions.
| Taywee wrote:
| How have you used it to help people eliminate food waste?
| hospitalJail wrote:
| It taught me about the '2 bin system' in ISE. Basically
| kitchen logistics.
| rideontime wrote:
| I, too, am curious how you used a fancier autocomplete to
| eliminate food waste.
| hospitalJail wrote:
| It taught me about the '2 bin system' in ISE. Basically
| kitchen logistics.
| roflyear wrote:
| I suspect that performance and responses will soon degrade pretty
| significantly
| pdksam wrote:
| If anything performance should get better with time
| marginalia_nu wrote:
| Why do you think that would be the case?
| bobsmooth wrote:
| Not soon but all the major players are making even more AI
| specialized silicon.
| Damogran6 wrote:
| Moore's law-ish like optimization.
|
| You Z80 computer cost $700 in the lat 70's...they're now in
| sub-$1 embedded controllers.
| marginalia_nu wrote:
| But what is being optimized? Hardware sure isn't getting
| faster in a hurry, and I don't see anything on the
| horizon that will aid in optimizing software.
| ben_w wrote:
| The various open source LLMs are doing things like
| reducing bits-per-parameter to reduce hardware
| requirements; if they're using COTS hardware it almost
| certainly isn't optimised for their specific models;
| Moore's Law is pretty heavily reinterpreted, so although
| we normally care about "operations per second at a fixed
| number of monies" what matters here is "joules per
| operation" which can improve a by a huge margin even
| before human level, which itself appears to be a long way
| from the limits of the laws of physics; and even if we
| were near the end of Moore's Law and there was only a 10%
| _total_ improvement available, that 's 10% of a big
| number.
| marginalia_nu wrote:
| Moore's law was an effect that stemmed from the locally
| exponential efficiency increase from designing computers
| using computers, each iteration growing more powerful and
| capable of designing still more powerful hardware.
|
| 10% here and there is very small compared to the literal
| orders magnitude improvements during the reign of Moore's
| Law.
|
| I don't really see anything like that here.
| ben_w wrote:
| > 10% here and there is very small compared to the
| literal orders magnitude improvements during the reign of
| Moore's Law.
|
| Missing the point, despite being internally correct: 10%
| of $700k/day is still $25M/y.
|
| If you'd instead looked at my point about energy cost per
| operation, there's room for something like 46,000
| improvement just to human level, and 5.3e9 to the
| Landauer limit.
| reitanqild wrote:
| > 10% here and there is very small compared to the
| literal orders magnitude improvements during the reign of
| Moore's Law.
|
| I can't confirm it, but I noticed this comment says "gpu
| tech has beat Moore's law for DNNs the last several
| years":
|
| https://news.ycombinator.com/item?id=35653231
| marginalia_nu wrote:
| We're actually at an inflection point where this isn't
| the case anymore.
|
| For a long time, GPU hardware basically became more
| powerful with each generation, but prices stayed roughly
| the same plus minus inflation. Last couple of years, this
| trend has broken. You pay double or even quadruple the
| price for a relatively tenuous increase in performance.
| kolinko wrote:
| That's not true. You can buy Raspberry PI, which is 10x
| cheaper and 10x more powerful than the computers at the
| beginning of 2000s.
|
| Ditto with mobile phones. iPhone may be more expensive
| than when it launched, but you can buy dirt-cheap chinese
| smartphones that have similar performance - if not higher
| to the first iPhones.
| marginalia_nu wrote:
| I don't think this contradicts what I'm saying. This is
| happening now. Not 15 years ago.
| Damogran6 wrote:
| We said that in 1982, and 1987, and 1993, and 1995, and
| 2001, 2003, 2003.5
|
| You get the point.
|
| There's always local optimization that leads to
| improvements. Look at the Apple M1 chip rollout as a
| prime example of that. Big/Little processors, on die RAM,
| shared memory with the GPU and Neural Engine, power
| integration with the OS.
|
| LOTS of things that led to a big leap forward.
| marginalia_nu wrote:
| Big difference now is that we have a clear inflection
| point. Die processes aren't getting much smaller than
| they are. A sub-nanometer process would involve arranging
| single digit counts of atoms into a transistor. A sub-A
| process would involve single atom transistors. A sub 0.5A
| process would mean making them out of subatomic
| particles. This isn't even possible in sci-fi.
|
| You can re-arrange them for minor boosts, double the
| performance a few times sure, but that's not a sustained
| improvement month upon month like we have in the past.
|
| As anyone who has ever optimized code will attest,
| optimization within fixed constraints typically hits
| diminishing returns very quickly. You have to work harder
| and harder for every win, and the wins get smaller and
| smaller.
| ben_w wrote:
| Current process nodes are mostly 5nm, with 3nm getting
| rolled out. Atomic is ~0.1nm, which is x30 linear and
| x900 by area.
|
| However, none of that is actually important when the
| thing people care about most right now is energy consumed
| per operation.
|
| This metric dominates for anything battery powered for
| obvious reasons; less obvious to most is that it's also
| important for data centres where all the components need
| to be spread out so the air con can keep them from being
| damaged by their own heat.
|
| I've noticed a few times where people have made
| unflattering comparisons between AI and cryptocurrency.
| One of the few that I would agree with is the power
| requirements are basically "as much as you can".
|
| Because of that:
|
| > double the performance a few times sure, but that's not
| a sustained improvement month upon month like we have in
| the past.
|
| "Doubling a few times" is still huge, even if energy
| efficiency was perfectly tied to feature size.
|
| But as I said before, the maximum limit for _energy
| efficiency_ is in the order of a billion-fold, not the
| x900 limit in _areal density_ , and even our own brains
| (which have the extra cost of being made of living cells
| that need to stay that way) are an existential proof it's
| possible to be tens of thousands of times more _energy
| efficient_.
| CuriouslyC wrote:
| There are a few avenues. Further specialization of
| hardware around LLMs, better quantization (3 bits/p seems
| promising), improved attention mechanisms, use of
| distilled models for common prompts, etc.
| marginalia_nu wrote:
| This would be optimizations, which is not really the same
| thing as moore's law-like growth which was absolutely
| mind-boggling, like it's hard to even wrap your head
| around how fast tech was moving in that period since
| humans don't really grok exponentials too well, we just
| think they look like second degree polynomials.
| CuriouslyC wrote:
| Probabilistic computing offers the potential of a return
| to that pace of progress. We spend a lot of silicon on
| squashing things to 0/1 with error correction, but using
| analog voltages to carry information and relying on
| parameter redundancy for error correction could lead to
| much greater efficiency both in terms of OPS/mm^2 and
| OPS/watt.
| marginalia_nu wrote:
| Seems very unrealistic when considering how
| electromagnetic interference works. Clamping the voltages
| to high and low goes some way to mitigate that problem.
| CuriouslyC wrote:
| That's only an issue if the interference is correlated.
| kolinko wrote:
| I am wondering about this as well - wondering how
| difficult it would be to build an analog circuit for a
| small LLM (7B?). And wondering if anyone's working on
| that yet. Seems like an obvious avenue to huge efficiency
| gains.
| EMM_386 wrote:
| > Hardware sure isn't getting faster in a hurry
|
| How is it not?
|
| These LLMs were recently trained using NVidia A100 GPUs.
|
| Now NVidia has H100 GPUs.
|
| The H100 is up to _nine times faster_ for AI training and
| _30 times faster_ for inference than the A100.
| roflyear wrote:
| What I mean is resources will be limited or models that are
| slightly worse will be released that will be much more cost
| effective but not quite as good.
|
| This is often the case with these types of technologies.
| kolinko wrote:
| So far, with technologies, it's been that new tech is both
| cheaper and better than the previous one.
|
| To not look far - gpt3.5 turbo.
| roflyear wrote:
| Again, not what I'm saying.
| Dave3of5 wrote:
| Interesting numbers. That roughly equates to about $250 million
| per year plus I don't know how much training is costing them to
| keep the model up to date and suchlike.
|
| The company also has about 375 employees. I've no idea how much
| they get paid but I used $200k as a yearly cost and that comes to
| $75 million.
|
| That's about 3:1 cost of operating the services to paying
| employees. That seems quite high as I've never been at a company
| that had 1:1 costs for running servers vs employee costs but I
| could entirely be off base here.
|
| Given Sam Altman's recent comments on the days of these LLM being
| over I think maybe Microsoft or whomever is basically saying that
| they can't spend that much money and they need to control costs
| much more heavily.
| wallawe wrote:
| I'm not familiar with Sam's comments re: "days of these LLMs
| being over" - can you provide more context (or link)?
| pdksam wrote:
| 200k is too small, strong sde1s at Amazon get paid that much in
| hcol areas. Closer to 500k.
| anonylizard wrote:
| This, there's like a endless line of companies waiting to
| snatch OpenAI's employees right outside the door. $200k
| average comp at OpenAI would be laughable.
| EMM_386 wrote:
| These numbers are insane to me.
|
| I'm 20 years into programming and a senior architect and lead
| on an enterprise project.
|
| I don't even make that first number.
|
| But I value certain things _way_ more than other things, and
| my current job provides it. Fully remote, leaves me
| completely alone to accomplish what they need done (and I get
| it done), unlimited vacation, great benefits, zero pointless
| meetings (almost an empty calendar).
|
| I'm sure these other companies offer some of that but 500k?!
| That is absurd.
| pluijzer wrote:
| As a side, I am a bit shocked by these numbers. Is this an
| American thing? I understand myself to be good software
| engineer with good well rounded experience of 14+ years. Yet
| my income, in Europe, is really above 100k.
|
| What I am wondering, for those earning 500k, how big is your
| work load/stress. Would this be a 9-5 job you leave at the
| office when going home. Or does a job that earns so much
| consume your life?
| lightbendover wrote:
| I've been through both horror (endless 100 hour weeks) and
| bliss (just attending meetings and not really stressing
| about much of anything) in that range. It's highly
| variable.
| darkwizard42 wrote:
| Honestly, depends. Some teams at FAAMNG are really
| stressful and if you work on a Tier 1 service even with
| loads of SRE support you have to be working a fair bit.
| That being said, the pay is for design decisions at the
| higher IC level (senior or staff) and most people at that
| level are very smart. I'm not saying this salary is for 10x
| engineers or anything.
|
| I would say 50% the work is harder and consuming and then
| 50% they can just afford to pay you more and lock up talent
| because of the wild margins on their products.
| justinclift wrote:
| Amazon has a _terrible_ reputation for internal
| infrastructure issues, with "on call" being a truly shitty
| experience for employees. aka burn out over a year is
| common
|
| Note that there's likely to be some variation per team, but
| Amazon is famously bad, so ... ;)
| barry-cotter wrote:
| > Is this an American thing?
|
| Yes, though Switzerland approaches it. If you want to see
| how much people of various levels of experience get paid at
| different companies and in different locations go to
| levels.fyi
|
| Americans get paid much, much more than anyone else.
| ebiester wrote:
| Your standard of living might be comparable. Your
| retirement is taken care of, you have a reasonable amount
| of vacation, you have better job security, your health
| care, in most European countries, has much less hassle, and
| your property costs are lower.
|
| I am seriously considering a move if my husband can find an
| academic job over there. The retirement won't be a great
| lure (fewer years in the system) but we almost have enough
| to coast from here, so it's about the rest.
| kolinko wrote:
| Taxes in the bay area can be insane - ~40% if I remember
| correctly. On top of that you have crazy-expensive
| healthcare, and crazy expensive housing costs.
|
| ~100kEUR in (western) Europe may be comparable to ~200kEUR
| in Bay Area.
| 988747 wrote:
| [dead]
| Sohcahtoa82 wrote:
| American SWE salaries can be insane, but I'm shocked at how
| _low_ SWE salaries are in Europe.
|
| I was expecting salaries to cool off a bit with the massive
| wave of layoffs across the industry, but from what I've
| seen, that hasn't happened.
| maxfurman wrote:
| FAANG salaries are so bloated because Bay Area housing
| costs are insane. Someone making 500k could put half or
| more of into their mortgage.
|
| I've said it before on here, but I live very comfortably in
| Philly for a lot less than that.
| Workaccount2 wrote:
| I'd argue it's the opposite. We're coming off a decade of
| free money driving a second tech boom.
|
| If interest rates stay elevated, and value investing
| becomes valuable again, it will be interesting to see how
| the tech space transforms. When start-ups have to compete
| with money market funds or even treasuries for investor
| cash, things become orders of magnitude tighter and more
| challenging.
| nkohari wrote:
| Sam Altman didn't say LLMs are over. (He's the CEO of OpenAI,
| so that would be a really strange thing for him to say,
| wouldn't it?)
|
| What he actually said was that we've reached the point where we
| can't improve model quality simply by increasing its size
| (number of parameters). We'll need new techniques to continue
| to improve.
| chipgap98 wrote:
| Pretty sure he said the days of models getting larger were
| over. Not that LLMs we're over
| twosdayz wrote:
| Someone had posted their tax filings in a different OpenAI
| thread. Although this only starts at 2020, this may give some
| insight into their employee costs
| https://projects.propublica.org/nonprofits/organizations/810...
| samspenc wrote:
| Interesting, the 2020 revenue and costs are significantly
| lower than previous years. Actually the prior years give a
| much better insight into salaries there. I wonder if this is
| because they switched from the non-profit model to the for-
| profit subsidiary at that time?
| zamnos wrote:
| They're interesting numbers, but the linked article's cite
| amounts to:
|
| > ChatGPT could cost OpenAI up to $700,000 a day to run due to
| "expensive servers," an analyst told The Information.
|
| which, pardon me, but no shit.
|
| Before I break out my back of the envelope calculator, on how
| many biggest GPU instances in Azure that is, the real question
| is what their underlying assumptions are, and where they're
| getting them from. Especially since OpenAI is definitely not
| paying list price on those GPU instances.
|
| The other question is how close to capacity their cluster is
| running, and how much free time on it can be reclaimed, either
| in terms of spinning down servers for diurnal patterns, or in
| terms of being able to do training runs in the background.
| capableweb wrote:
| Considering that Microsoft is a huge investor in OpenAI, I'd
| be surprised they pay anything at all in reality.
| gabereiser wrote:
| that's ridiculous, OpenAI is paying. Granted Microsoft
| invested heavily into OpenAI but those are two separate
| financial transactions. Sure you can rationalize in your
| head that IN-OUT=DIFF but that's not how books are kept.
| capableweb wrote:
| Why is that ridiculous? Cloud services gives companies
| "coupons" and free usage for X hours for a bunch of other
| companies, why wouldn't they do that for a company they
| invested heavily in?
| rickdeckard wrote:
| Because that's not how it works. Even company cars of
| General Motors employees have to be purchased from
| General Motors.
|
| Such "free usage" coupons are marketing activities to
| gain new customers, Microsoft already completed the
| "dating phase" with OpenAI. They surely don't pay list-
| price for Azure but it's surely also not free.
|
| Moreover, as per Microsoft themselves, the 1bn USD
| investment into OpenAI carried the condition that Azure
| becomes the exclusive provider for cloud-services: https:
| //blogs.microsoft.com/blog/2023/01/23/microsoftandopen...
|
| It's not exclusive because it's free, it's exclusive
| because "we paid you 1bn USD to buy it from us"
| mattpallissard wrote:
| I've personally worked on a project where Microsoft ate
| the cloud cost in order partner with us.
|
| They might not give unfettered credits, it could be for
| specific projects. That said, I wouldn't be surprised if
| it was unfettered either.
| nerdix wrote:
| Microsoft invested $1 billion OpenAI in 2019 and half of
| that amount was in Azure credits.
|
| I'm not sure about the most recent $10 billion investment
| but I wouldn't be surprised if a significant amount of it
| is in Azure credits as well.
|
| While that's not "free" (they exchanged equity for it),
| it's likely not an expense (or at least not an expense
| that they have to cover fully).
| dmix wrote:
| At best OpenAI has negotiated a near 0 profit margin for
| Microsoft when paying for the services. But even that is
| unlikely given how much money/resources are involved.
| There's no scenario where it's free at that scale.
| vitus wrote:
| The estimate in the article pins most of the cost on
| inference, not training, so diurnal patterns are
| unfortunately not as useful here.
|
| > While training ChatGPT's large language models likely costs
| tens of millions of dollars, operational expenses, or
| inference costs, "far exceed training costs when deploying a
| model at any reasonable scale," Patel and Afzal Ahmad,
| another analyst at SemiAnalysis, told Forbes. "In fact, the
| costs to inference ChatGPT exceed the training costs on a
| weekly basis," they said.
| zamnos wrote:
| Why wouldn't inference follow diurnal patterns?
| reaperman wrote:
| Agreed. That seems backwards. Training would not follow
| circadian rhythms, inference would.
| vitus wrote:
| I should clarify: training is not latency sensitive, so
| you can run your workloads at off-peak hours when people
| are asleep. Inference means you need to run your
| workloads at peak when people are using your service.
|
| (Looking back, I'm happy that I was careful in my wording
| in that I didn't say diurnal cycles aren't relevant, just
| that they aren't as _useful_ in this case)
|
| That said, I suppose I misread the specific suggestion
| about spinning down servers off-peak and was thinking
| more about spot pricing at peak vs trough.
| [deleted]
| Thaxll wrote:
| THey must have really large kubernetes clusters, I remember it's
| all running on k8s.
| [deleted]
| jyu wrote:
| Sorry for the tangent, wondering how you've used GPT to make your
| life better?
|
| I'll start:
|
| - making little scripts in shell / js / python that I'm not as
| fluent in. 5 min vs 1-3 hours
|
| - explaining repos and apis instead of reading all the docs -
| help with debugging
|
| - flushing out angles for new concepts that I did not previously
| consider (ex: how do you make a good decentralized exchange)
| iKevinShah wrote:
| You know some people used to use Google as a glorified spell
| checker? I use ChatGPT as a glorified stupidity checker. What I
| mean is I ask it the silliest of the doubts. Like how to set
| environment variable in Windows (because we are all used to
| EXPORT aren't we?), whether or not can we do X in K8S YAML in a
| given conversation.
|
| Obviously I use it for other purposes as well, but it
| definitely has saved me a lot of hours getting the basics
| things right there in a prompt.
| Eumenes wrote:
| whats the ecological cost re: power/cooling the data
| centers/compute?
| bobsmooth wrote:
| What's the ecological cost of Fortnite's servers?
| dpkirchner wrote:
| And what is the ecological cost of the people it will replace?
|
| I kid, I kid.. or do I?
| bamboozled wrote:
| Obviously, it's not going to be good.
| xyzzy4747 wrote:
| It's worth every cent. Better than wasting the same amount on
| Bitcoin mining.
| djschnei wrote:
| [flagged]
| BlackSwanMan wrote:
| The privilege knowing that bitcoin is pointless scam for
| halfwits?
|
| Yeah I guess having a brain is a privilege
| bsima wrote:
| [flagged]
| [deleted]
| bobsmooth wrote:
| Privilege is every second not hunting and gathering for our
| food. Fun stuff like GPT or bitcoin is what keeps life
| interesting.
| ck2 wrote:
| How many years until "chatgpt on a chip"? 3? 5?
| aero-glide2 wrote:
| I would have never paid $100 per month for software but here i am
| doing the same with api costs.
| miohtama wrote:
| Only 240,000 paying customers needed to cover $24M/mo server
| costs. I would assume personnel costs could be in the same
| class as well.
| samspenc wrote:
| Around 375 employees per https://en.wikipedia.org/wiki/OpenAI
|
| Let's say 500 employees, earning above Bay Area average,
| along with benefits etc at $400K average per year. That's
| $200 million a year in wages, so in the same ballpark as the
| $200 million in infrastructure costs cited in the article,
| for a total of $400 million in expenses per year.
| anony23 wrote:
| What are you building?
| [deleted]
| pc_edwin wrote:
| This might sound tone deaf but thats seems extremely cheap to me.
| It means the cost range for chatGPT is between $300m - $500m per
| year.
|
| If the 100m users is accurate, it means they only need to convert
| low single digit percentage to paying users to break even.
|
| Now it makes sense why they chose to charge $20/m, I predicted
| much higher.
| onlyrealcuzzo wrote:
| The problem with this model is that if ChatGPT really is "good
| enough" now - many others will be "good enough" soon, and then
| it's a race to the bottom.
| dreadlordbone wrote:
| that's a feature not a bug
| MuffinFlavored wrote:
| What does it cost to run per day versus what did it cost to
| train?
| lumost wrote:
| These costs are falling by factors of 2 every year based
| solely on hardware progress ( gpu tech has beat Moore's law
| for DNNs the last several years). ChatGPT may become much
| cheaper to operate/train in the future.
| wouldbecouldbe wrote:
| That's just server costs, not labor hours for support,
| management & innovation.
| nathancahill wrote:
| Yeah think of all the labor hours, how many people do they
| have back there typing up the answers?
| warent wrote:
| And who pays the tiny people in my magic light box to draw
| the answers on the window?!
| ElFitz wrote:
| Those don't count. They're fairies, not people. Why on
| Earth would you pay fairies?
| imranq wrote:
| I thought they were on contract? hopefully they don't go
| on strike again at the same time the power went out like
| last time
| tilne wrote:
| Do you know the answer?
| c0balt wrote:
| You, at least somewhat. They get soul shard futures and
| once you die can realize them to acquire a part of you
| soul.
| [deleted]
| raydev wrote:
| > If the 100m users is accurate
|
| Users or accounts? I made an account but I use it approximately
| once every two weeks for about 30 minutes at a time, as it
| hasn't been that useful for me (needs more up-to-date info
| after September 2021).
|
| I imagine many people made an account but only a small
| percentage are using it meaningfully often.
|
| Also users are limited in the number of requests they can make.
| I have a feeling ChatGPT is actually very expensive to run, and
| they are burning cash like crazy.
|
| If they weren't burning cash to run the thing, it would be more
| widely available.
| giarc wrote:
| I suspect the number will go higher since right now you are
| limited to 25 queries every 3 hours. They will definitely
| introduce higher tiers which removes or expands that limit.
| edf13 wrote:
| I expect a lot of optimization will come in the next few
| months (weeks?)... it may be hanging together by threads
| behinds the scenes - this is often the case when things are
| moving this quickly.
| xyst wrote:
| That's just on computing power. You are forgetting the human
| costs of ongoing development, maintenance of ChatGPT, power,
| SRE/operations, cooling costs, and et cetera.
|
| Now M$ is planning to create a specialized chip for AI which
| comes with its own R&D budget and ongoing costs.
|
| If it proves successful, ChatGPT will become a household brand
| and M$ could easily ask for $500 per month or more for
| professional/corporate usage.
| naikrovek wrote:
| I like how you abbreviate "Microsoft" as "M$". It means you
| think of them without bias and with a clear head. /s
| ilyt wrote:
| If you see public company as something other than vehicle
| to make $$$ for the investor it's you that might have bias
| problem.
| staticman2 wrote:
| You know people only do that with Microsoft, right? It's
| not done to make a statement about public companies.
| tedivm wrote:
| Seeing that brought me back to 2006.
| deadbolt wrote:
| M$ has been a rather common abbreviation for Microsoft
| since _at least_ slashdot days.
| williamcotton wrote:
| Bill Gates as Borg.
| naikrovek wrote:
| my point is that it is a derogatory abbreviation commonly
| used by people who, in fact, do not have an unbiased view
| of Microsoft.
|
| it's harder to type than just "MS" so when someone uses
| "M$" they go out of their way to signal that they are
| biased. Being biased is fine, so long as one understands
| that they are biased and that they are communicating
| their bias along with the rest of the message.
| ren_engineer wrote:
| interesting that the article mentions Microsoft is trying to
| build their own AI chips to help reduce cost, they are years
| behind Google in that effort. Google still has the advantage in
| both research and hardware, they just fumbled the execution but
| it's still very early in the game
| barkerja wrote:
| > they are years behind Google in that effort. Google still
| has the advantage in both research and hardware, they just
| fumbled the execution but it's still very early in the game
|
| Where do we think Apple is in regards to this? If anyone has
| the upper-hand here with hardware, I would think it'd be
| Apple. But there's been zero indication that Apple has been
| working on any sort of generative AI.
| kolinko wrote:
| Apple already has a super-efficient neural engine in all
| their chips - it's just a matter of them building dedicated
| models for them.
| yunohn wrote:
| Apple has stuffed their consumer hardware with high-end
| chips - that are completely under-utilised by themselves
| and restricted for third-party apps...
| rvnx wrote:
| Local Siri for June WWDC :)
| behnamoh wrote:
| The past 3 years showed that Google is not as ahead as people
| thought they were.
| edgyquant wrote:
| In what way?
| behnamoh wrote:
| In Bard way.
| brrrrrm wrote:
| In terms of production pipeline, sure. But since there's an
| established model architecture, the hardware doesn't need to
| be super generic. Google had to build TPUs in a world where
| everything was still rapidly changing - deciding on things
| like precision, memory bandwidth, SRAM size etc. Microsoft
| could theoretically stamp out some GPT ASICS and call it a
| day.
| rapsey wrote:
| Aren't Google TPUs way less performant than A100s?
| Workaccount2 wrote:
| They are actually as good or better depending on the use
| case. IIRC their compute is around the same, and the
| compute per watt is substantially better. They are also
| purpose built for transformers.
|
| But google is fumbling so hard right now it's not
| surprising that they are squandering this too.
| jermaustin1 wrote:
| I have started incorporating ChatGPT into my side project. I'm
| not integrating it into software or anything like that, but
| using it as a tool for product ideas, product photo ideas,
| funny social media captions, figuring out the best hashtags to
| use, product tagging and categorization.
|
| It has proven to be a great way to get everything to around
| 70%, then send off to my assistant for the remaining 30% of
| polish. So at $20/month, it was such a no-brainer, that I had
| to do it. Even at $75/mo it would more than pay for itself.
|
| It even understands the concept of a "shit post". So - its more
| social media savvy than I am, thats for sure.
| mnky9800n wrote:
| I paid for it, then stopped paying for it. It seemed to be
| completely novel and fun to use but after a while it just
| became a pain. I tried writing a library to act as a code
| assistant for data science but trying to get it to write the
| codes that were useful to me took around the same amount of
| time as writing them myself. What you say is completely
| correct, it gets to around 70% of whatever you ask it to do
| then you have to finish the job for it. Which I guess is
| nice. But I'm not going to pay 20$/month for something that i
| can just use for free that amounts to slightly better
| intellisense.
|
| For text generation, it is much better for tasks like
|
| * letter writing * rewriting my writing so i don't plaigerize
| myself * summarizing several paragraphs into one
|
| but all of that is available in the free version.
| redmaverick wrote:
| free version is an order of magnitude worse than the paid
| version. It is still good though.
| BlackSwanMan wrote:
| The cheap version does all of that much worse. I don't
| understand how people can be so cheap.
| kylecazar wrote:
| There are so many subscription products these days that
| it makes sense to assess the value of each carefully
| before committing.
|
| If you're like me you tend to forget you're paying for
| something, and have to do a yearly purge.
|
| It's very easy to slowly but surely rack up hundreds of
| dollars a month in subscription services.
| camgunz wrote:
| I had a boss I didn't like a while ago, but one of the
| things he did that I did like was set a calendar reminder
| to unsubscribe. Too bad; it would've been easier on me
| mentally and emotionally if he were completely
| irredeemable, but I admit to using this trick myself now.
| alex_sf wrote:
| I've been using Privacy.com cards for years to avoid
| this. Set a limit of 1 month on the card, and if I'm
| actually using the sub I'll fix/update it when the sub
| runs out.
| bemmu wrote:
| My habit is to cancel right away when I start a
| subscription. You can still use the service for the month
| you paid for, and it's easy to restart it if you do still
| find yourself attempting to use it a month later.
| dqft wrote:
| Substack disappointed me because you can't do this. They
| remove content access if you do not have an "active"
| subscription. You don't have to pay again to
| "reactivate", but the paid content is locked until you
| do. This is obviously intended dark behavior because when
| I went to fully unsub later in the month they gave me a
| special dialog and a free month hoping for me to forget
| again this month. Very lame.
| unsignedint wrote:
| Oh, you haven't seen darker behavior of many Japan based
| services:
|
| - Payment cycle closes always on the 1st of the month
| (which means, if you sign up on the last day of the
| month, you get 1 day of service for full month of
| payment. No proration, or anything.)
|
| - If you cancel the service, you'll immediately lose
| access, doesn't matter if it's 1st day of the month, or
| very end of the month, it's gone. To reenroll, you often
| have to pay again.
| HybridCurve wrote:
| Yes, some of the transformation mechanisms are nice if you
| need to dump info into it and want it reorganized or
| processed into something else as well. But outside of that
| it's hard to justify spending $20 month so you can be
| bullshitted by some know-it-all toddling proto-skynet when
| you need factual information.
| mrleinad wrote:
| >> I'm not going to pay 20$/month for something that I can
| just use for free
|
| Well, it works until it doesn't because the website is
| overloaded. $20 gets you in through the VIP door and you
| don't need to wait along with the rest of the peasants.
| m3kw9 wrote:
| You can use bing chat for free. Supposedly powered by gpt4.
| cft wrote:
| Bing gives you much shorter answers, often without code
| examples
| Veen wrote:
| I've found that phind.com is pretty good, although I'm
| not sure which LLM it's backed by and I only use it for
| hobby projects.
| pps wrote:
| "Expert" uses GPT-4.
| gnulinux wrote:
| phind.com claims they use GPT4 but you have turn Expert
| mode on. Otherwise it's their home baked model. It's
| explained here:
|
| > Expert mode is our most advanced searching mode,
| powered by GPT-4. This mode hallucinates less and writes
| better code. We highly recommend that you use it for
| advanced questions. Whenever the "regenerate" button is
| pressed, Expert mode is used to increase the odds of a
| high-quality answer.
|
| https://www.phind.com/tutorial
| moffkalast wrote:
| I'm sorry but I have to end this conversation, I have
| been a good Bing and you have been a bad user. Thank you
| for your cooperation. Goodbye.
| AraceliHarker wrote:
| Sydney's quick to get into a bad mood when you say
| something bad about AI.
| iloveCS wrote:
| [dead]
| robomartin wrote:
| > It has proven to be a great way to get everything to around
| 70%, then send off to my assistant for the remaining 30% of
| polish. So at $20/month, it was such a no-brainer, that I had
| to do it.
|
| Yup. I bought accounts for everyone here. We are using it as
| what I have been calling a "force multiplier". We are not and
| cannot use it for coding, yet, things like presentations,
| analyzing logs, creating lists of things, researching topics,
| etc. It's a great time saver.
|
| Also, for a lot of things ChatGPT is a much better search
| engine than Google. It gets you great answers and almost
| always the first time you ask.
|
| In case the question comes up: We don't use it for coding
| because of potential liability concerns. At this point I feel
| that is a space that has not been explored at all. I have no
| interest in being a pioneer in a lawsuit that claims
| negligence due to the use of AI-generated code, even if it is
| reviewed by a human. The combination of fear-mongering and
| tech-challenged juries could make for very expensive
| outcomes.
|
| I posed a question about this here:
|
| https://news.ycombinator.com/item?id=35655521
| noloblo wrote:
| archive link unpaywall : https://archive.ph/Txq8r
|
| better source : https://tech.slashdot.org/story/23/04/18/1326
| 235/microsoft-r...
| octopoc wrote:
| I've been using it for product ideas as well. Occasionally
| it's brilliant but sometimes it's so bad it's funny, like
| when it suggested Seven-Year Itch for a perfume brand.
| phkahler wrote:
| >> it's so bad it's funny, like when it suggested Seven-
| Year Itch for a perfume brand.
|
| Yeah, what's wrong with that? I'm sure it would be catchy
| to a particular demographic. No need for a brand to cater
| to everyone.
| jermaustin1 wrote:
| I actually like to make the bad ones. It is fun. It is an
| exercise in creative engineering. Especially since my side
| project is making home goods for stoners.
| williamcotton wrote:
| That is a great name for a perfume brand! It's also the
| name of a comedy by Billy Wilder staring Marylyn Monroe.
| Market it to rich old ladies.
| mym1990 wrote:
| Don't they say something about seemingly brilliant ideas
| being too obvious or usually not working out, and the
| terrible ones being the ones that are have more potential
| to be diamonds in the rough?
| javajosh wrote:
| It's almost like ChatGPT is nerd-sniping us all. But we like
| it and it's useful to us.
| consp wrote:
| Don't generalize, I've never used it and never seen any
| reason to use it.
| JohnFen wrote:
| I've used it out of curiosity, but have no real use case
| for it.
| [deleted]
| seattle_spring wrote:
| My grandma has never used the internet and sees no reason
| to use it.
|
| It would benefit her immensely to use it.
| tinideiznaimnou wrote:
| [dead]
| coldpie wrote:
| You should give it a shot, keep an open mind. Yes, it
| will get things wrong and yes, it will disappoint you.
| But every conversation I've had with it holds some
| surprises for me. It's a tool you'll need to learn to use
| before it actually becomes _useful_ , but give it a
| couple minutes a few times a week and you'll start to see
| the diamonds in the rough.
| cleanchit wrote:
| Damn, they released the hypno drones already?
| coldpie wrote:
| Sorry, I don't get the reference(?).
| gnulinux wrote:
| It's a reference to the incremental game "Universal
| Paperclip". This is when something like "AI singularity"
| is reached in the game.
| rvbissell wrote:
| UP is one of the few modern games that I've wanted to
| play more than once.
| gnulinux wrote:
| Yep, factorio and UP are the only games I played more
| than only a few hours.
| jermaustin1 wrote:
| Thanks, now my day is going to be wasted.
| egeozcan wrote:
| If you ever write code that starts with
|
| #include <windows.h>
|
| then it is a must have.
| dpkirchner wrote:
| Generalizing is perfectly ok. Having to wrap statements
| with terms like "almost all" or "most" adds unnecessary
| noise, outside of academia.
| Swizec wrote:
| Nerd-snipe _and_ useful? That's the holy grail! We're
| having fun _and_ we're solving real problems that we have.
| Amazing.
|
| I've already incorporated it into my publishing process. My
| home-grown "cms" uses ChatGPT (via API) to write my article
| description, draft a twitter thread, and craft a "viral
| insight". The latter is mostly useful to make sure my
| article even makes a point.
|
| Hoping to use it for a related articles feature next. I'm
| also building a chatbot based on my content.
| javajosh wrote:
| _> Nerd-snipe and useful? That's the holy grail!_
|
| Yes. i think its a good software product! Maybe one of
| the best there ever was - I'm not surprised Sam Altman
| jumped the YC ship for this one, or that MS is
| particularly interested it. It gets ignored in all the
| other bru-haha but I'm excited to see what the really
| talented software teams of the world will be able to
| accomplish with an AI coding assisstent - and I don't
| mean just in the world of AI.
|
| The activation energy for new software lowered by about
| 30% overnight, which is outrageously cool, and of course
| disquieting.
| rvnx wrote:
| This is cheaper than running Twitter which is a CRUD app
| thrthrthr88 wrote:
| [flagged]
| rabuse wrote:
| Everything highly profitable is basically a CRUD app
| mcny wrote:
| You're absolutely right.
|
| At the end of the day, all we are doing is create, read,
| update, and delete data.
|
| I mean if you take away all the complexity associated with
| ranking and scale (not everything has to be Google scale),
| that is exactly what a web search is as well, right?
|
| I remember reading a post by a maintainer saying their job
| is to manipulate strings or something to that effect. iirc
| it was a gofmt maintainer who said that but I can't find
| that post now.
| kgbcia wrote:
| So if they get one million paying users at 20$ they break even
| maherbeg wrote:
| Github CoPilot will probably be a significant fraction of this.
| lightbendover wrote:
| Having spent 11MM annually on infra to effectively (though
| obviously not efficiently despite a very favorable ROI) generate
| reports for <1000 customers, that sounds like a bargain.
| jongjong wrote:
| My main concern with AI is that I'm not so sure if there is a
| market for any kind of intelligence anymore. So much human talent
| and intelligence has been wasted over the past few years; so many
| potentially useful projects went unfunded, I struggle to think of
| what purpose AI would serve in such an economic system as ours. I
| can see its potential, but I can't see our current system
| facilitating that potential. It seems more likely that the
| technology will be applied towards controlling the masses rather
| than benefiting them; this has been the purpose of most major
| tech 'innovations' of the last decade.
|
| Wealthy individuals these days hardly need any intelligence at
| all to stay rich. On the other hand, it seems like poor
| individuals have no chance no matter how much intelligence they
| have... Does anyone really need more intelligence? Most human
| intelligence seems to be wasted on bullshit jobs anyway.
|
| What we should be asking ourselves is "how much more comfortable
| can we make our billionaires?" Because the entire concept of
| 'economic efficiency' appears to be about optimizing society
| towards that goal.
| qwytw wrote:
| > these days
|
| Wasn't it always more or less like this?
| pohl wrote:
| No.
| jongjong wrote:
| Agreed. I entered the workforce as a developer in 2012; it
| felt like the tail-end of the tech boom but intelligence
| was still valued. Nowadays, most people don't even
| recognize intelligence, let alone value it.
| moffkalast wrote:
| Well you should be just fine then /s
| tinideiznaimnou wrote:
| [dead]
| duncan-donuts wrote:
| I've never heard a more cynical AI take. There's always a
| market for intelligence, whatever that means. The word
| intelligence is misleading anyway. No one said the AI needed to
| compete with the smartest person on the planet. It just needs
| to be smart enough to do some tasks that are "bullshit jobs"
| tinideiznaimnou wrote:
| [dead]
| hindsightbias wrote:
| There is definitely a market for an intelligence crutch or
| assisted critical-thinking aid.
|
| Like people slowly getting more obese, the general IQ and CT
| have been sliding for the last few decades. Not sure if it's
| the food, media or zanax but society needs something.
|
| I'd wager the Altmans of the world know this and figured out a
| way to monetize it. Necessity is the mother of invention after
| all.
| ghostbrainalpha wrote:
| What is "CT" ?
| [deleted]
___________________________________________________________________
(page generated 2023-04-21 23:02 UTC)