https://old.reddit.com/r/MachineLearning/comments/11rizyb/d_anyone_else_witnessing_a_panic_inside_nlp_orgs/ jump to content my subreddits edit subscriptions * popular * -all * -random * -users | * AskReddit * -funny * -movies * -pics * -news * -worldnews * -mildlyinteresting * -gaming * -todayilearned * -explainlikeimfive * -videos * -Jokes * -tifu * -science * -aww * -dataisbeautiful * -OldSchoolCool * -TwoXChromosomes * -LifeProTips * -IAmA * -Music * -books * -askscience * -Futurology * -space * -nottheonion * -Showerthoughts * -Art * -gifs * -food * -DIY * -photoshopbattles * -nosleep * -UpliftingNews * -gadgets * -InternetIsBeautiful * -Documentaries * -history * -sports * -GetMotivated * -WritingPrompts * -EarthPorn * -philosophy * -announcements * -creepy * -listentothis * -blog more >> MachineLearning MachineLearning * comments * other discussions (3) Want to join? Log in or sign up in seconds.| * English [ ][] [ ]limit my search to r/MachineLearning use the following search parameters to narrow your results: subreddit:subreddit find submissions in "subreddit" author:username find submissions by "username" site:example.com find submissions from "example.com" url:text search for "text" in url selftext:text search for "text" in self post contents self:yes (or self:no) include (or exclude) self posts nsfw:yes (or nsfw:no) include (or exclude) results marked as NSFW e.g. subreddit:aww site:imgur.com dog see the search faq for details. advanced search: by author, subreddit... this post was submitted on 15 Mar 2023 1,078 points (98% upvoted) shortlink: [https://redd.it/11ri] [ ][ ] [ ]remember mereset password login Submit a new link Submit a new text post Get an ad-free experience with special benefits, and directly support Reddit. get reddit premium MachineLearning joinleave2,600,342 readers 2,671 users here now Rules For Posts +Research +Discussion +Project +News @slashML on Twitter Chat with us on Slack Beginners: Please have a look at our FAQ and Link-Collection Metacademy is a great resource which compiles lesson plans on popular machine learning topics. For Beginner questions please try /r/LearnMachineLearning , /r/ MLQuestions or http://stackoverflow.com/ For career related questions, visit /r/cscareerquestions/ --------------------------------------------------------------------- Advanced Courses (2016) Advanced Courses (2020) --------------------------------------------------------------------- AMAs: Pluribus Poker AI Team 7/19/2019 DeepMind AlphaStar team (1/24//2019) Libratus Poker AI Team (12/18/2017) DeepMind AlphaGo Team (10/19/2017) Google Brain Team (9/17/2017) Google Brain Team (8/11/2016) The MalariaSpot Team (2/6/2016) OpenAI Research Team (1/9/2016) Nando de Freitas (12/26/2015) Andrew Ng and Adam Coates (4/15/2015) Jurgen Schmidhuber (3/4/2015) Geoffrey Hinton (11/10/2014) Michael Jordan (9/10/2014) Yann LeCun (5/15/2014) Yoshua Bengio (2/27/2014) --------------------------------------------------------------------- Related Subreddit : * LearnMachineLearning * Statistics * Computer Vision * Compressive Sensing * NLP * ML Questions * /r/MLjobs and /r/BigDataJobs * /r/datacleaning * /r/DataScience * /r/scientificresearch * /r/artificial a community for 13 years MODERATORS * message the mods discussions in r/MachineLearning <> X 74 * 8 comments [hG5KGru-] [P] nanoT5 - Inspired by Jonas Geiping's Cramming and Andrej Karpathy's nanoGPT, we fill the gap of a repository for pre-training T5-style "LLMs" under a limited budget in PyTorch 2057 * 333 comments [D] Our community must get serious about opposing OpenAI 41 * 12 comments [RsmwON-A] [R] Memory Augmented Large Language Models are Computationally Universal 193 * 24 comments [N] A $250k contest to read ancient Roman papyrus scrolls with ML 10 * 4 comments [D] Creating an open platform for collecting corrective feedback on conversational ML products & projects 12 [N] bloomz.cpp: Run any BLOOM-like model in pure C++ 184 * 29 comments [N] PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever 609 * 161 comments [D] What do people think about OpenAI not releasing its research but benefiting from others' research? Should google meta enforce its patents against them? 4 * 12 comments [D] Comparison of the Model prediction uncertainty of two different models 4 * 4 comments [D] paperspace or another cloud service Welcome to Reddit, the front page of the internet. Become a Redditor and join one of thousands of communities. x 1077 1078 1079 Discussion[D] Anyone else witnessing a panic inside NLP orgs of big tech companies? (self.MachineLearning) submitted 1 day ago by thrwsitaway4321[Illuminati] I'm in a big tech company working along side a science team for a product you've all probably used. We have these year long initiatives to productionalize "state of the art NLP models" that are now completely obsolete in the face of GPT-4. I think at first the science orgs were quiet/in denial. But now it's very obvious we are basically working on worthless technology. And by "we", I mean a large organization with scores of teams. Anyone else seeing this? What is the long term effect on science careers that get disrupted like this? Whats even more odd is the ego's of some of these science people Clearly the model is not a catch all, but still * 410 comments * share * save * hide * report top 200 commentsshow all 410 sorted by: best topnewcontroversialoldrandomq&alive (beta) [ ] Want to add to the discussion? Post a comment! Create an account [-]uchi__mata 243 points244 points245 points 1 day ago (34 children) My (HR tech) startup has quite intentionally chosen not to try and create competitive advantage via bespoke ML algorithms for exactly this reason. It's just too hard to keep up with the handful of cutting edge players. That said, being able to quickly find use cases for and incorporate things like emergent LLMs into your own platform is still something that can provide differentiation in the market and is a piece of what I'm working on now. But I do feel for scientists who have devoted careers to working on NLP who feel like it was a total dead end in the face of GPT etc, though that's always a risk you run when you make bets on truly innovative research (that it might all be for naught). * permalink * embed * save * report * give award * reply [-]undone_function 110 points111 points112 points 1 day ago* (20 children) I wish I could find the actual link, but a well respected, old school CS person talked about how we're in the infancy of AI. Basically the equivalent of when home computers started to exist, and I think that's accurate. At that time, I'm sure there were a ton of engineers who felt the same anxiety as the OP, and for good reasons. They were all working on competing platforms and it can feel like a zero sum game. But as we know now, use cases expand, implementation expands, and everything changes rapidly. Obviously I don't know the future, but there's still a life beyond being the first to build a thing that is a big deal at the time. And you're right, most businesses are looking for a tool they can use. Be it a framework, a language, or an NLP model. As a run of the mill software engineer that's been around for 20 years, I definitely worry about the future and what it means for me and my family, particularly the impact that AI will have. But I like to remain optimistic that humans will find new and innovative ways to use these tools that don't cut everyone out or crown a singular "winner" (to shorthand OPs post). Who knows what will happen? At least we can look at human history and hope for the best for all of us, and remember that failing to make the history books in one instance doesn't mean that greatness or at least happiness and fulfillment isn't still in our collective future. Edit: I found the post I mentioned in my browser's history: https:// steveblank.com/2022/05/17/ artificial-intelligence-and-machine-learning-explained/ * permalink * embed * save * parent * report * give award * reply [-]uchi__mata 20 points21 points22 points 1 day ago (2 children) While the companies that make the the best AI tools will definitely (and deservedly) get very rich, like with any new tech it's as important how other firms find ways to use that tech to enhance their operations. I absolutely do not think we're entering a future where only AI firms make money and everyone else is picking up scraps, but I do think that at least in tech that there will be a bifurcation between companies that are able to quickly find use cases and adopt emerging AI tech to enhance their existing business vs. those that can't do so. The potential productivity gains from AI are so large that I think we are only a few years from an 'adopt of die' scenario for firms across industries. * permalink * embed * save * parent * report * give award * reply [-]bohreffect 7 points8 points9 points 1 day ago (1 child) You can already see it now where ChatGPT acts as like a kind of oracle for bespoke services. I would guess that the major tech companies working directly on their own very powerful AI just expose the endpoints as just another, albeit incredibly useful, product in their cloud offerings. Like, the antithesis of what DeepMind does. * permalink * embed * save * parent * report * give award * reply [-]pyepyepie 7 points8 points9 points 1 day ago (0 children) Well, what DeepMind do is amazing as well. They work on issues that deserve attention but don't get it. For example, cooperative AI and nowdays RL. They also look from a holistic point of view. I would be very surprised if a company like Meta or OpenAI will work on a better mechanism for things we need, but DeepMind do it. We are lucky Google spend money on DeepMind and that they publish their research, LLMs are not the start and end of life. As you say, they publish research. * permalink * embed * save * parent * report * give award * reply [-]AntAgile 2 points3 points4 points 22 hours ago (0 children) Sounds like this Tom Scott video maybe? * permalink * embed * save * parent * report * give award * reply load more comments (16 replies) [-]EnsignElessar 28 points29 points30 points 1 day ago (11 children) I still think its worth it to invest in NLP for at least one reason. We probably don't want to be sending all our business data to companies like open ai who are going to use our data to give advantage to themselves and others. * permalink * embed * save * parent * report * give award * reply [-]uchi__mata 8 points9 points10 points 1 day ago (9 children) Yeah, if you have to send data to another company that's a problem. But I don't assume that'll be the model on a long term basis. I think too many firms would revolt against it. * permalink * embed * save * parent * report * give award * reply [-]EnsignElessar 19 points20 points21 points 1 day ago (8 children) How do you revolt though? You decide not to use the api tools and your company tanks as all your competitors gobble up the new tech without thinking much about it * permalink * embed * save * parent * report * give award * reply [-]uchi__mata 15 points16 points17 points 1 day ago (5 children) In my specific industry of HR tech I think that's unlikely simply because of the high financial and reputational risk of sending PII outside our own systems. And generally that's not how companies operate at this point, I think firms know the value of their data and aren't quick to give it away in a way that wasn't true 10-20 years ago. I don't know, I could be totally wrong, but I don't see companies (especially those that have massive data that would be really valuable for Open AI et al to refine their models) just giving it away at this point. Besides, the converse point is also true: AI might end up an oligopoly but it's not a monopoly, if Open AI insists on taking your data for its own use and Google doesn't then Google will dominate them in the marketplace. * permalink * embed * save * parent * report * give award * reply [-]pyepyepie 4 points5 points6 points 1 day ago (0 children) The question is, though, if it would not become a game of deploying and fine-tuning easily LLMs. Seems like it is, in 2 years you will have amazing open LLMs which are just as good as GPT-4. * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) load more comments (2 replies) [-]eterevsky 2 points3 points4 points 3 hours ago (0 children) There are open models like Llama which are only a bit weaker than GPT 3.5. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]lacker 475 points476 points477 points 1 day ago (73 children) It is common for big divisions of successful tech companies to work for a long time on products that ultimately fail. Remember the Facebook Platform? The Windows Phone? And many products don't even get that far. If you're working on a worthless technology, go work on something else! Your smart coworkers will be the first ones to leave. * permalink * embed * save * report * give award * reply [-]thrwsitaway4321[S] 94 points95 points96 points 1 day ago (12 children) Good point :) * permalink * embed * save * parent * report * give award * reply [-]SunshineDeliveries 32 points33 points34 points 1 day ago (9 children) Maybe I'm missing something here, aren't LLMs just a subset of NLP models? Why does LLMs' success jeopardize the work being done in the parent field? * permalink * embed * save * parent * report * give award * reply [-]PuddyComb 37 points38 points39 points 1 day ago (4 children) Because compromising means specializing your NLP for a very specific task so that you can package and sell it. You had a SwissArmy Knife and now you have a fork, and now you have to make it a 'special shiny' fork for just one company; that MIGHT be a startup that doesn't make it./Or your team incurrs paycuts before exiting the project. There's a lot of uncertainty, and if your competitor is EVERYWHERE, it's demoralizing. I always say "there was merit to reinventing the wheel; we had stone, then wood spokes, now we use steel and rubber injected with air." But as you know, you wouldn't want to have to reinvent the wheel of today- thousands of engineers have picked it over, for all terrains, it's the best design we collectively know of. If we know we have a NLP that works nearly every time, why put years and millions of dollars into another one? That's the existential question. * permalink * embed * save * parent * report * give award * reply [-]stiffitydoodah 24 points25 points26 points 1 day ago (1 child) OK, but to elaborate on your wheel analogy: we have snow tires, all-weather tires, racing slicks, tractor tires, etc.; the tire companies are still developing new materials and designs for longer tire life and increased suitability to different conditions; electric motors have recently affected the mechanical side of wheel designs; and so on. My point is, just because you have one general purpose model does not mean you've met every specialized need. And you certainly don't have a generally intelligent agent, meaning there's still plenty of work to be done. * permalink * embed * save * parent * report * give award * reply [-]PuddyComb 1 point2 points3 points 1 day ago (0 children) Right but I already have a hypothetical working tire. So now your market is slightly smaller. It might not be all terrain but it works for my needs. You see the point I am making. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]Smallpaul 5 points6 points7 points 1 day ago (3 children) Isn't NLP a problem area and LLM a potential solution? What are the NLP problems this LLM's are weak at? * permalink * embed * save * parent * report * give award * reply [-]camyok 2 points3 points4 points 16 hours ago (1 child) Long term memory, guaranteed factual accuracy and, even if you're so goddamn rich you don't care about the time and hardware needed to train it, inference speed. * permalink * embed * save * parent * report * give award * reply [-]Smallpaul 1 point2 points3 points 12 hours ago (0 children) Those three all seem like problems that will likely be solved for AI in general and not for NLP in particular. Long term memory for speech is probably not that different than long-term memory for vision. The bitter pill is that NLP doesn't exist as a field anymore. It's just an application of general technologies. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]lmericle 1 point2 points3 points 1 day ago (1 child) Is there a reason the consensus isn't "go the route of Alpaca and do knowledge distillation / fine-tuning"? * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]ktpr 43 points44 points45 points 1 day ago (0 children) The last sentence is gold. A lot of folks will overlook that but it's one of the first signs that you too should pivot away from a company. If you're suddenly the smartest person in the room it's already very late to get out * permalink * embed * save * parent * report * give award * reply [-]sanity 13 points14 points15 points 1 day ago (3 children) Google+ was even worse, they shut down popular apps like Google Reader so they wouldn't distract attention from it. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) [-]SpookyTardigrade 26 points27 points28 points 1 day ago (5 children) What was Facebook Platform? * permalink * embed * save * parent * report * give award * reply [-]IshKebab 77 points78 points79 points 1 day ago (4 children) API for Facebook (with ridiculously lax permissions). IIRC when it was first introduced lots of people pointed out that the default permissions were that any app could access lots of data about anyone that used the app and their friends. Obviously insane but nobody cared at the time. Years later it transpired that, surprise surprise some apps harvested all that data and sold it. Cambridge Analytica is one company that used the data, and oh look maybe people should have paid attention when the insane security model was pointed out at the time! (By the time of the Cambridge Analytica scandal the Facebook Platform was long dead; they were using old data.) I think a lot of people still think it was only Cambridge Analytica using this data, and they somehow hacked Facebook or something. * permalink * embed * save * parent * report * give award * reply [-]TrefoilHat 1 point2 points3 points 24 minutes ago (0 children) I think a lot of people still think it was only Cambridge Analytica using this data, and they somehow hacked Facebook or something. Most people I talk to think Facebook sold the data directly to Cambridge Analytica and continues to sell customer data today. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) [-]Zondartul 45 points46 points47 points 1 day ago (26 children) This might sound cynical, but does it even matter if your product succeeds or fails as long as your salary is paid? * permalink * embed * save * parent * report * give award * reply [-]jewelry_wolf 31 points32 points33 points 1 day ago (17 children) Not sure if you are aware but tech worker has a big portion of their compensation as stock. So if the company is screwed, they only get the basic pay, like only 160k or something a bit more. * permalink * embed * save * parent * report * give award * reply [-]hughperman 65 points66 points67 points 1 day ago (16 children) Only * permalink * embed * save * parent * report * give award * reply [-]22vortex22 4 points5 points6 points 1 day ago (12 children) Good chunk of that goes to taxes and rent. Especially in the bay area where those are some of the highest in the US * permalink * embed * save * parent * report * give award * reply [-]vanguard_SSBN 4 points5 points6 points 1 day ago (0 children) Same with any job. * permalink * embed * save * parent * report * give award * reply [-]sdmat -5 points-4 points-3 points 1 day ago (9 children) Exactly, after taxes and retirement contributions $160K barely covers a lower middle class lifestyle in the bay area. It's ridiculous. * permalink * embed * save * parent * report * give award * reply [-]Trotskyist 19 points20 points21 points 1 day ago (7 children) The median household income in SF is $126,000, so I don't think that's accurate. Cost of living may be high, but that doesn't mean a salary of $160,000 is "lower middle class." Though psychologically it certainly can feel that way if you only meaningfully interact with people who make at or above that level. * permalink * embed * save * parent * report * give award * reply [-]professorlust 9 points10 points11 points 1 day ago (0 children) As a random aside, thanks for using median income and not average income: It's a far superior metric * permalink * embed * save * parent * report * give award * reply [-]new_name_who_dis_ 2 points3 points4 points 1 day ago (0 children) It's the old idea of everyone thinking they are middle class haha. With good stock options you make 250-300k total you feel like you're middle class, then once you're down to 160k total, you feel like you're lower middle class. * permalink * embed * save * parent * report * give award * reply load more comments (5 replies) load more comments (1 reply) load more comments (1 reply) load more comments (3 replies) [-]goodTypeOfCancer 12 points13 points14 points 1 day ago (7 children) That would be sad AF. I'm glad that people will be using the products I made for decades. * permalink * embed * save * parent * report * give award * reply [-]mongoosefist 16 points17 points18 points 1 day ago (3 children) Nobody wants to work on something that fails, but I think /u/ Zondartul is correct in the fact that people shouldn't internalize those failures. It's pretty cool to get paid to do something experimental or risky enough that there is no guarantee that it will work. That's usually the environment where I learn and grow the most. Making something that people end up using for decades is just a bonus. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) load more comments (3 replies) [-]AlmightySnoo 16 points17 points18 points 1 day ago (0 children) Given the current climate of layoffs at bigtechs, you usually don't want to be part of a team that worked on a failed product. So the anxiety is justified. * permalink * embed * save * parent * report * give award * reply [-]Swolnerman 18 points19 points20 points 1 day ago (13 children) I had a friend working in Google fiber for a while... * permalink * embed * save * parent * report * give award * reply [-]GreatBigBagOfNope 29 points30 points31 points 1 day ago (1 child) At least they deployed some worthwhile infrastructure that, correct me if I'm wrong, is actually in use and on sale in its limited areas? * permalink * embed * save * parent * report * give award * reply [-]giritrobbins 1 point2 points3 points 1 day ago (0 children) I was in Huntsville AL for work and it's supposed to be growing in coverage according to the ads. * permalink * embed * save * parent * report * give award * reply [-]ktpr 12 points13 points14 points 1 day ago (0 children) Google fiber is really cool where it still exists and gave monopolies the (sheets). Good on them * permalink * embed * save * parent * report * give award * reply [-]aCleverGroupofAnts 7 points8 points9 points 1 day ago (9 children) What happened to Google Fiber? * permalink * embed * save * parent * report * give award * reply [-]Swolnerman 2 points3 points4 points 1 day ago (8 children) Shut down in it's prime like most other cool Google groups, I believe there's a list of abandoned Google products somewhere online * permalink * embed * save * parent * report * give award * reply [-]aCleverGroupofAnts 8 points9 points10 points 1 day ago (4 children) But Google Fiber still exists, so what specifically got shut down? On Wikipedia it says they halted expansions in 2016, but in 2022 they announced plans to expand to more areas again. * permalink * embed * save * parent * report * give award * reply [-]Swolnerman 2 points3 points4 points 1 day ago (2 children) Hmm now I'm unsure if I'm misremembering the team he was on, or if this just happened in 2016 and he got screwed over * permalink * embed * save * parent * report * give award * reply [-]OcelotRL 13 points14 points15 points 1 day ago (0 children) no you're correct, while not fully shuttered they laid off (in a very nice way, giving people a lot of time to find new roles internally) most of the Google Fiber eng/ops teams, and abandoned the plan to meaningfully compete at scale with comcast. (I say as a former gfiber employee) * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]ianitic 2 points3 points4 points 1 day ago (0 children) I know they had some unique deployment method to try out and legal battles in my city (I think with AT&T). The deployment method had a lot of issues, and eventually lost the legal battle I believe. They then withdrew completely from the city and expansions halted. * permalink * embed * save * parent * report * give award * reply [-]rainnz 1 point2 points3 points 19 hours ago (1 child) From a month ago: Google Fiber continues to grow - Next Up: Westminster, Colorado, and Chandler, Arizona! * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) load more comments (1 reply) load more comments (7 replies) [-]redlow0992 289 points290 points291 points 1 day ago (97 children) Panic inside NLP orgs of big tech companies? What about the panic at NLP departments in universities? I have witnessed my friends putting their work on PhDs go into despair after ChatGPT and now GPT-4. Quite literally, majority of the research topics in NLP are slowly becoming obsolete in front of our eyes. * permalink * embed * save * report * give award * reply [-]RobbinDeBank 222 points223 points224 points 1 day ago (15 children) Can't go into PhD in NLP anymore when apparently "scale is all you need" in this subfield. * permalink * embed * save * parent * report * give award * reply [-]starfries 148 points149 points150 points 1 day ago (2 children) You get a bitter lesson, you get a bitter lesson, everybody gets a bitter lesson! * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]CommunismDoesntWork 5 points6 points7 points 1 day ago (6 children) People need to learn to work on toy datasets where you can test different architectures while using far less parameters. If the gpt architecture is the best, it should be the best at all levels of parameters. * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush 39 points40 points41 points 1 day ago (0 children) If the gpt architecture is the best, it should be the best at all levels of parameters. There is no such rule. * permalink * embed * save * parent * report * give award * reply [-]octopusdna 9 points10 points11 points 1 day ago (4 children) It would be amazingly useful if we could test architectures using small models, but it appears to be just not the case with transformers. * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) load more comments (5 replies) [-]The_color_in_a_dream 146 points147 points148 points 1 day ago (18 children) Absolutely agree. Turns out just scaling up the parameter space worked to solve the challenges that people had spent years coming up with complex systems to tackle. * permalink * embed * save * parent * report * give award * reply [-]new_name_who_dis_ 29 points30 points31 points 1 day ago (5 children) I kind of am skeptical that GPT-4 is just scaling up. GPT-3 was 175B params, which is already absurd, and that recent FAIR paper showed that after a certain point better data is better than more params. Also considering they didn't release the parameter count, it might be to mislead competitors and have them try many absurd parameter counts, while they figured out some other better tricks. They did mention that it was the most stable train they had out of all the GPT models, which I feel like means it's more than just scaling up of the model. But it's just a hypothesis. * permalink * embed * save * parent * report * give award * reply [-]jdsalaro 1 point2 points3 points 20 hours ago (1 child) that recent FAIR paper showed that after a certain point better data is better than more params. Can you please link it ? * permalink * embed * save * parent * report * give award * reply [-]new_name_who_dis_ 1 point2 points3 points 6 hours ago (0 children) I believe it was the LLama paper that argued that. They claimed to have achieved comparable performance to gpt3 with a much smaller parameter space but with better data. * permalink * embed * save * parent * report * give award * reply [-]MysteryInc152 1 point2 points3 points 1 hour ago (1 child) Scale isn't just param size. Data is also part of the scale equation. If they trained GPT-4 on 4x as much data without changing param size for example, that's still scaling up. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) load more comments (1 reply) [-]suflaj 20 points21 points22 points 1 day ago (4 children) Hey, there's another side to the story, some people are still in denial and reference the NFL theorem despite empirical evidence. * permalink * embed * save * parent * report * give award * reply [-]qalis 9 points10 points11 points 1 day ago (2 children) Well, the theorem is true, but it is in the limit, for all possible problems. Which has never been practical, just widely misunderstood. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]Nameless1995 5 points6 points7 points 1 day ago (0 children) NFL is just problem of induction and applies to humans as well. * permalink * embed * save * parent * report * give award * reply [-]pyepyepie 4 points5 points6 points 1 day ago (5 children) Dude, it makes me want to cry. I really like linguistics inspired (e.g. people who work on parsers) and clever ideas (e.g. combining language models with knowledge bases or cool ways to use embeddings for NMT), but it seems like what works is predicting the next word. Elegant for sure, but man what a dead-end for other cool research directions. We have to learn to value ideas, not only performance - it's an art, really, not only practical. Can anyone argue GANs are not beautiful? Who cares if diffusion "works better"? * permalink * embed * save * parent * report * give award * reply [-]CreationBlues 11 points12 points13 points 21 hours ago (3 children) lmao, we've barely scratched the surface of model architectures. There's plenty of gold in them hills. Also, just because we have these artifacts doesn't mean we actually understand them. Working on interpretation is gonna be a project people work on for the next decade. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) [-]lnplum 1 point2 points3 points 10 hours ago (0 children) If it's any consolation, when I was studying computational linguistics in the mid-2000s the department was still heavily invested in a vision of converting a sentence from one language into an abstract syntax tree to compile it back down to another language as long as you just got all the parts of speech tagging right. And I don't think that has changed in the years/decades since. Google Translated already existed at the time and was eating its lunch. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]PinusPinea 81 points82 points83 points 1 day ago (23 children) Aren't most PhDs working on more nuanced things than just developing performant systems? I can see industry researchers being undercut by this, but I figured PhD students would be working on more fundamental questions. * permalink * embed * save * parent * report * give award * reply [-]nullbyte420 24 points25 points26 points 1 day ago (2 children) Hopefully yes. I had a plan to do research on NLP and electronic health journals, that's all still perfectly valid and possible, and hampered by data regulations. * permalink * embed * save * parent * report * give award * reply [-]Jurph 20 points21 points22 points 1 day ago (1 child) There are tons of use cases like this, where privacy or secrecy concerns require performant locally-operated models rather than a gargantuan pay-per-token model that is at best privy to your data, and at worst saving/selling it. I hope you'll keep pushing in this direction, even if you have to adapt to things like using FlexGen to localize/distill large models. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]redlow0992 71 points72 points73 points 1 day ago (6 children) It was the case in the past but, unfortunately, in past 5 or so years nuanced research has become very hard to publish. If you don't beat SOTA, well, good luck graduating. * permalink * embed * save * parent * report * give award * reply [-]Jurph 102 points103 points104 points 1 day ago (3 children) It's a damn shame you can't build a Ph.D. career on reproducing (or failing to reproduce) weak-ass SOTA papers that get their performance gains with "stochastic grad student descent" -- that is, have the grad researcher try lots of runs until one randomly hits SOTA, save the seed, and publish. Just grab the last 20 supposedly-SOTA papers in the field, set up their conditions, run their same experiment on 100 random seeds, and publish the variance of their models. Call your paper Cherry picking is all you need and tell 'em you want the NeurIPS keynote. * permalink * embed * save * parent * report * give award * reply [-]CactusOnFire 30 points31 points32 points 1 day ago (1 child) 'Grad Student Descent' has me snickering. * permalink * embed * save * parent * report * give award * reply [-]RobbinDeBank 17 points18 points19 points 1 day ago (0 children) Already published, one of the author works for ClosedAI * permalink * embed * save * parent * report * give award * reply [-]womenrespecter-69 11 points12 points13 points 1 day ago (0 children) Some google researchers tried reproducing a few transformer model variations a few years ago and the results were pretty depressing: https://arxiv.org/pdf/2102.11972 * permalink * embed * save * parent * report * give award * reply [-]ktpr 14 points15 points16 points 1 day ago (0 children) The way around this is to come up with new novel problems to solve, convince reviewers why it's an important problem by referring to theory from other disciplines, and then the work is SOTA on the new problem * permalink * embed * save * parent * report * give award * reply [-]Sunapr1 1 point2 points3 points 1 day ago (0 children) Isn't performing working on let's say computational social science would be better * permalink * embed * save * parent * report * give award * reply [-]needlzorProfessor 19 points20 points21 points 1 day ago (10 children) You are correct. There is still plenty to do. If those absurdly large language models kill benchmark-driven NLP research that will be a net gain, and people can focus on solving interesting problems instead. * permalink * embed * save * parent * report * give award * reply [-]respeckKnuckles 5 points6 points7 points 1 day ago (9 children) LLMs aren't going to kill benchmark-driven NLP research. They're going to exacerbate it by requiring that you use GPT-4 (or whatever newest AI-as-a-service model is out) to even begin to compete. * permalink * embed * save * parent * report * give award * reply [-]needlzorProfessor 8 points9 points10 points 1 day ago (8 children) That's just as good as killing it. A lot of universities won't shell out the dough to use those services, so people will have to find more interesting research questions than advancing the SOTA by 0.1%. For those who do, they 'll still have to find research topics which are not likely to be obsoleted by the release of GPT N+1. In any case I am glad I pivoted out of core ML a while ago. * permalink * embed * save * parent * report * give award * reply load more comments (8 replies) load more comments (2 replies) [-]needlzorProfessor 43 points44 points45 points 1 day ago (6 children) Good. Maybe it will kill benchmark-driven ML research and we can go back to being scientists. * permalink * embed * save * parent * report * give award * reply [-]milesper 20 points21 points22 points 1 day ago (5 children) Agreed 100%. If I never have to read another paper where they tried a million variations of a model and found one that did 0.1% better than SOTA, I'm happy. Also, GPT-4 (and arguably previous GPT models) aren't really relevant in a science setting, since we don't know the architecture or training data. There's already been some evidence that GPT-4's training dataset is contaminated with some of its benchmark datasets. If, for instance, a company tried to present a black box physics model that only they could access as "solving physics", they would be laughed out of the conference. * permalink * embed * save * parent * report * give award * reply load more comments (5 replies) [-]ChatumTannin 12 points13 points14 points 1 day ago (24 children) Could someone knowledgeable explain this to me? Why isn't it an exciting new basis for further research, rather than a dead end? * permalink * embed * save * parent * report * give award * reply [-]500_Shames 93 points94 points95 points 1 day ago[gold_48] (12 children) Because if you don't have access to the same resources that OpenAI has, you can't compete. The best metaphor I can come up with is that we're all spending years to practice and refine the perfect combat art. "New state of the art punching performance achieved with this slight modification to our stance. By planting the foot very carefully and turning while striking, we can break 8 boards rather than just 7, as was the limit of the previous gold standard." Quickly we graduated to swords, so everyone had to get good at smelting and blacksmithing at home. Still accessible, but now a lot of people had to redirect their research from biomechanics to metallurgy. Anyone with a GPU or two could iterate on some aspects of the status quo at home, try to find minor modifications or make a breakthrough. Dropout is a really cool, groundbreaking approach to address overfitting that anyone could have come up with, apply, and publish a paper on if they had the idea and skill to implement on consumer hardware. Then we started scaling. Scaling hard. Think of this as introducing guns, vehicles, and mass production to the equation. Again, you can try to make iterative improvements, but now you need much bigger capital investments to make this happen. Several years ago, to try and push limits in NLP often meant having access to a supercluster at a university. Still doable, but the groundbreaking katana design you were working that would be 5% sharper than the previous gold standard is sorta irrelevant now that we have armor piercing rounds that get the job done through brute force. Now you need to figure out how to push the envelope once again. Last week, we were working on very nuanced challenges in armor penetration. Why does the bullet go through these materials, but not these? Even if we can't build a new gun altogether, we can still push for iterative improvements. If you worked on the biomechanics of punching, then biomechanics of swinging a sword, you could still do proper firing stance research. Yesterday, they revealed they had achieved nuclear fission and GPT-4 is the atom bomb. All of the problems we were working on were rendered irrelevant by the sheer size and power of GPT-4. This is exciting as a giant leap forward, but concerning in that it makes going any other direction far harder. No one cares about work on armor piercing bullets when the state of the art is vaporizing a city block. We worry that internal inefficiencies don't matter if you have enough data and computing power to make it so big and strong to compensate. Now if we want to "iterate" on this new gold standard, we have to ask OpenAI nicely to use their tool. If we want to try anything new, it will be with the knowledge that there's no way we will come close to the performance of GPT-4, not because our approach is wrong, but because we lack the same resources. NLP journals will likely be "The Journal of GPT-4" for the next few years. I'm being hyperbolic here, but I hope the concept I'm trying to explain makes sense. * permalink * embed * save * parent * report * give award * reply [-]zero_for_effort 5 points6 points7 points 1 day ago (0 children) I found this informative, cheers. * permalink * embed * save * parent * report * give award * reply [-]User48562 6 points7 points8 points 1 day ago (0 children) You build a ICBM, they build Skynet that builds time travel and terminators - I can see how this could get out of hand. * permalink * embed * save * parent * report * give award * reply [-]jmhobrien 1 point2 points3 points 11 hours ago (0 children) Ah damn, if only we'd been collectively computing a public LLM instead of mining pointless crypto for the last 10+ years. Oh well. * permalink * embed * save * parent * report * give award * reply [-]ninjasaid13 0 points1 point2 points 1 day ago (6 children) Because if you don't have access to the same resources that OpenAI has, you can't compete. I'm not knowledgeable on anything but isn't there multiple ways to skin a cat or is scaling the only way or the low hanging fruit? * permalink * embed * save * parent * report * give award * reply [-]currentscurrents 11 points12 points13 points 1 day ago (0 children) Scaling seems to be a fundamental law. It's probably possible to build smaller, more efficient algorithms but they'd still work better with more resources. This is a good thing though! It turns an impossible problem (figuring out intelligence) into a relatively easy one (building more powerful computers.) * permalink * embed * save * parent * report * give award * reply [-]spudmix 2 points3 points4 points 1 day ago (0 children) This is a controversial topic known as "the bitter lesson" in AI research, or with slightly less edge as the "scaling hypothesis", with the core idea being that some things scale better than others but scaling computation is ultimately the way forward. You can view AI research as a series of leaps forward, with gradual progress being made as humans try to encode our knowledge of the world in a model and then a "leap" occurring when we forget all that and just make a bigger model with fewer assumptions. It happened with chess, with voice and then image recognition, with reinforcement learning for games in general and then other tasks, and OpenAI are doing it to language processing right now. People will refine and distill the current progress, iterating and improving and making it more efficient, but Inductively it seems that scale really is the way to go. * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) load more comments (2 replies) [-]terath 25 points26 points27 points 1 day ago (3 children) Because very few companies and zero universities are able to spend the millions out dollars it takes to train these large models. That said this thread is rather alarmist. There is still plenty of work going on trying to reduce the parameter space without sacrificing accuracy, more efficiently training models, new architectures, and so on. The performance and cost issues with these big models also apply to corporate labs. I'd hardly call smaller models and other techniques obsolete. Productization advice really hasn't changed. You still want the smallest and simplest model that achieves your task. Both from a cost perspective and reducing surprise edge case errors. * permalink * embed * save * parent * report * give award * reply [-]deepspacespice 2 points3 points4 points 1 day ago (2 children) Sure, but for a Ph.D student it can be depressing, there is no clever idea that will perform better that sheer computation power. Or maybe but empirically that was never the case for all the history of AI. Working on improving models is indeed needed and there is still room for very large improvement, for example making LLM usable on a personal device would be a game changer but that's maybe not as exiting as discovering a new clever SOTA method. * permalink * embed * save * parent * report * give award * reply [-]terath 5 points6 points7 points 1 day ago (0 children) The history of AI is built on small improvements! Adding processing power has helped neural networks improve yes, but you can't divorce the success from the many many incremental improvements to training algorithms or architectural improvements like transformers. These large models in no way prevent PhD students working on the core machinery. These huge successes are generally not "one clever idea" but are built off of a history of many many small ideas. It's an unfortunate fact that often one or two people who put on the last improvement are the one that get all the fame, but that's true of all of computer science. * permalink * embed * save * parent * report * give award * reply [-]milesper 4 points5 points6 points 1 day ago (0 children) That was never the case for all the history of AI If this were true, we would all be using massive FFNs on every task. We would never have invented things like CNNs, LSTMs, dropout, layer norm, etc. SOTA on the big tasks is overrated. So often it comes down to the ML equivalent of p-hacking where out of hundreds of similar attempts, one model happens to get a 0.01% improvement. If the only goal of your PhD was trying to beat the SOTA, I'm dubious that your work was all that interesting. * permalink * embed * save * parent * report * give award * reply [-]salgat 8 points9 points10 points 1 day ago (1 child) To get competitive results you'd need to blow through your entire department's budget training the model. * permalink * embed * save * parent * report * give award * reply [-]el_chaquiste 6 points7 points8 points 1 day ago (0 children) And why for? Because it will most likely still be worse than current GPT, and rendered even more obsolete by the OpenAI steamroller in a year or so. We are witnessing a technology escape scenario. * permalink * embed * save * parent * report * give award * reply [-]deepspacespice 4 points5 points6 points 1 day ago (0 children) To provide another metaphor than the armor/weapon. Imagine you're working on improving racing car aerodynamic but you have to compete with RedBull F1 SOTA, they have supercomputer clusters to simulate every shapes and situations. Sure you can came up with clever ideas but they would probably be worse than their brute force solutions. This is known as the Bitter Lesson of AI : sadly human knowledge is not as effective in the long run as leveraging computation power. * permalink * embed * save * parent * report * give award * reply [-]marr75 1 point2 points3 points 1 day ago (3 children) OpenAI's GPT-3 and 4 series of models are closed source, closed weights, and cost upwards of $5M just in computation resources to train (to say nothing of the expert labor needed to oversee that infrastructure). Generally, researchers, startups, and even blue chip public companies can't hope to marshall resources like that. There are some opportunities to work with the OpenAI team but it will be on their terms. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) [-]tango_telephone 1 point2 points3 points 1 day ago (0 children) Yes...slowly... * permalink * embed * save * parent * report * give award * reply load more comments (5 replies) [-]sciehigh 169 points170 points171 points 1 day ago (4 children) Just an intern, but my previous work on NLP sentiment analysis is 100% obsolete. My previous seniors are very worried that most of their work is now just implementing GPT APIs. If I was on a larger team and more senior [and cared about keeping the job and not just jumping ship], I would be looking for a way to establish myself as the workplace expert with the new tools (pledge yourself to the new overlords). * permalink * embed * save * report * give award * reply [-]pitruchaML Engineer 58 points59 points60 points 1 day ago (1 child) What I found to work well: * text extremely hard to classify -> GPT3.5 API with a prompt that explains the task in detail. * text not THAT hard to classify + ability to deploy sufficiently large transformer -> train your own model. Lacking data? GPT3.5 till you collect enough training data. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]EnsignElessar 10 points11 points12 points 1 day ago (1 child) Your work is still invaluable imho. Because if I were a business owner I would not my orgs data being used in anyone's training set. * permalink * embed * save * parent * report * give award * reply [-]simonw 9 points10 points11 points 1 day ago (0 children) You can run decent large language models on your own hardware now: https://simonwillison.net/2023/Mar/11/llama/ * permalink * embed * save * parent * report * give award * reply [-]Fidodo 51 points52 points53 points 1 day ago (5 children) The writing was already on the wall when GPT-3 came out. We've been moving everyone involved in NLP to LLM projects at my company. There's still plenty of problems to figure out. Programming has always been about figuring out how to make code more structured, predictable, and reliable and while LLMs have the amazing ability to help us gain insight from unstructured data, it totally breaks all those other rules and there are a lot of problems to solve to make it predictable and reliable again. They said, it's more about data flow than mathematics now that the NLP side is being commoditized. I do think people working on a NLP will be in a good position to have insight into how that data flow should work, but frankly, they will need to learn new skills to work in the changing landscape, but those skills can potentially work really well with their existing skills in concert. * permalink * embed * save * report * give award * reply [-]uchi__mata 17 points18 points19 points 1 day ago (2 children) it's more about data flow than mathematics now that the NLP side is being commoditized. Absolutely this. I don't see it as fundamentally different than when XG Boost came out and all your different methods for creating classification models on relatively small tabular data sets became largely unnecessary. Data science for most companies will become (if it isn't already) not about the algorithm but rather about identifying use cases and having strong ML ops capabilities to integrate predictive tools created by third parties. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]hexagonshogun 7 points8 points9 points 1 day ago (1 child) Language models are black boxes. There's still value in knowing how something is parsed. * permalink * embed * save * parent * report * give award * reply [-]Fidodo 1 point2 points3 points 1 day ago (0 children) I think I know what you mean, but could you give an example? * permalink * embed * save * parent * report * give award * reply [-]andreichiffaResearcher 36 points37 points38 points 1 day ago (0 children) If Bing chat trials are any indication, there is a lot of space to fill by other solutions, if it is not just through alignment and debiasing to avoid lawsuits. Realistically though, it looks like a major management mishap and tech awareness issue. Sam Altman is not know to play around and a total monopoly in the ML space, starting with NLP, was the only outcome OpenAI could have gone for in principle. If it is really that major of a team and no one was allowed to shade InstructGPT/SeeKeR/... papers, or no one on the team wanted to, they would would have been boned in other ways. * permalink * embed * save * report * give award * reply [-]Direct-Suggestion-81 65 points66 points67 points 1 day ago (4 children) I think it will be a game of catch up between the open-source models and OpenAI's models with the open-source models lagging 6-8 months behind. Personally, I'm thinking of contributing to the OpenAI Eval project to improve my understanding of their system. Simultaneously I've been working on integrating LLMs into projects using Langchain. It would be great if the science orgs pivoted to complex LLM use-cases instead of just focusing on building the LLM itself. * permalink * embed * save * report * give award * reply [-]cajmorgans 23 points24 points25 points 1 day ago (0 children) Yep, open source is extremely important and I'm thinking to do something similar when I have more experience * permalink * embed * save * parent * report * give award * reply [-]Necessary-Meringue-1 6 points7 points8 points 1 day ago (0 children) Personally, I'm thinking of contributing to the OpenAI Eval project to improve my understanding of their system. I love that they refuse to reveal any internals of their model, while at the same time asking the community to helping them evaluate their model. No hate on you, I think it's great you want to do that. But I think it's pretty cynical of them to be honest. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]maxio-mario 32 points33 points34 points 1 day ago (4 children) Taking a wild guess of your organization... Amazon Alexa? My previous supervisor got out of academics and got on the Alexa team and ... things have been pretty stressful for him. * permalink * embed * save * report * give award * reply [-]Remper 1 point2 points3 points 21 hours ago (3 children) Why would big companies ever be stressful about this? They can afford to build a GPT-4 of their own. It's small/mid-size NLP orgs and some benchmark-driven researchers that are screwed. The thing to understand is that GPT-4 does what it does because of scale, not because of some proprietary algorithm that is hard to replicate. * permalink * embed * save * parent * report * give award * reply [-]RemarkableGuidance44 3 points4 points5 points 20 hours ago (0 children) Exactly, its scale and they know it. That's why they are hiding what they are doing now. Comp going to crush them. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]suflaj 29 points30 points31 points 1 day ago (15 children) Not sure what classifies as a big tech company but realize this: * Google is not out of the game at all * Microsoft == OpenAI * Amazon has been out of the game for a long while, but they never focused on LLMs, really * permalink * embed * save * report * give award * reply [-]xjE4644Eyc 36 points37 points38 points 1 day ago (2 children) Amazon is selling the pick axes to the gold prospectors. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]bis_g 2 points3 points4 points 1 day ago (2 children) anecdotal but from my observation ,amazon ml research has largely focussed on Graph Neural network and it application of late * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) load more comments (9 replies) [-]PassingTumbleweed 100 points101 points102 points 1 day ago* (7 children) I see the same. The panic is entirely predictable. These models are seen as an existential threat to the things they have been working on the last few years. But there's a reason that productionization of those NLP models has taken years (quality, cost, and latency) and I don't think gpt is magically going to fix those problems. In many cases gpt is going to be worse than an in-house model because you can't control cost or latency, address quality issues, and now you have a dependency on an external API. Gpt will be disruptive in some areas but I don't think anyone really knows which use cases will become billion dollar profit machines and which ones are just money pit tech fever dreams. For my money I think data is way more important than models and open AI has been very smart about data collection but I see no reason others can't catch up * permalink * embed * save * report * give award * reply [-]CardboardDreams 4 points5 points6 points 1 day ago (6 children) I'm skeptical of that last line. To say that data is more important than models implies that the agent can actually get the relevant data. ChatGPT can only solve a real world problem if someone has already written something like it down somewhere in the data. Its like an efficient search engine. Any deviation from this and its output is tenuous. When I ask it solve problems for which something like a readymade solution isn't available it breaks down terribly. E.g. I've asked it to solve three software problems, two of which were novel, apparently, because its solution was ridiculous. The third one had a common answer (a popular library was already available), and in that case it was immensely helpful. It gave a result that even Google couldn't find for me. But it can't discover something new about the world, a new solution that hasn't been found before, because it can't test that it would actually work. And it's upfront about that. with all its "As an AI language model..." It can't observe and learn on its own. That would require a new model and new sources of data, not just more data. Finally, data doesn't just "come from the universe". It is a human interpretation of our experiences - e.g. we chose to call this set of experiences a "house" or "the Renaissance". It bases its output on our human interpretations and labels. To create such labels itself from experiences would again require a new model. * permalink * embed * save * parent * report * give award * reply [-]PassingTumbleweed 2 points3 points4 points 1 day ago (5 children) It's not that modeling techniques aren't important, it's that modeling techniques don't confer as much of a competitive advantage as having a hoarde of high quality does. Modeling technique is basically just IP and therefore easy to steal or recreate. Years of accumulated data is much harder to come by. * permalink * embed * save * parent * report * give award * reply load more comments (5 replies) [-]itanorchi 17 points18 points19 points 1 day ago (0 children) Not where I work as I don't work at an NLP org, but this has definitely has happened at parts of a big tech firm from what I've heard from friends there. Their NLP team essentially got sidelined by the openai models. The leadership there apparently totally undervalued their team and the right advocates had left before. Absolutely sucks because I loved some of the work they were doing. My friends are considering leaving the company and going elsewhere now. Their CEO basically brought in openai to do the work and they will ignore most of all the work that has been done by the NLP team for the past decade. Bonkers. * permalink * embed * save * report * give award * reply [-]roselan 47 points48 points49 points 1 day ago (2 children) My take is a bit different. As chinchilla and quantization have shown, costs can be reduced by an order of magnitude over a couple of months. I won't be surprised if within a year it would be possible to train or at least fine tune a model at the fraction of the cost of what openai was able to do. Gpt-4 sure is shiny, but it's only a start. * permalink * embed * save * report * give award * reply [-]mepster 4 points5 points6 points 21 hours ago (1 child) Yeah... but unfortunately, you may need the big models to find those optimizations. Chinchilla paper says, "Due to the cost of training large models, we only have two comparable training runs at large scale (Chinchilla and Gopher), and we do not have additional tests at intermediate scales." Even DeepMind were limited by cost! And the main conclusion of the Chinchilla paper was that you also need to scale the data... but the big companies have the big datasets too! :-( So they use their 10x performance gain, spend the same $100m to train, and get a new level of emergent behaviors in a couple of months. Too bad the era of sharing deep learning frameworks / methods / architectures / datasets / models is coming to a close. Fun while it lasted! * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]Oswald_Hydrabot 96 points97 points98 points 1 day ago* (11 children) GPT doesn't have business-specific knowledge. So at the very least, it requires finetuning for certain things that it has never seen. I am unsure of your current role; web-based chatbot development is certainly a bad market to compete against them in but there are plenty of markets that they will never touch, nor are they at all immune to competition, much of what they have is hype. Also, it really is just an LLM. It can't do everything, and it isn't unlikely that it will eventually become obsolete. GPT is a walled garden, sort of like Apple products. They may be great products but Linux excels at certain things because you have complete, comprehensive control over what it can do. GPT won't make NSFW content. It won't assist in running automated profiling for political advertising on Facebook. It won't help you use semantic analysis to help track violent terrorists online. These are some pretty lightweight examples but you are highly underestimating how artificially limited that OpenAI is making their own products, and how many other opportunities there are to outcompete them. There are plenty of things that GPT cannot be used for simply because of the nature of OpenAI's business practices. Optimization is highly undervalued; lightweight models that run on cheap hardware for computer vision remain incredibly valuable for example, and there will come a time where GPT stops being affordable as OpenAI continues their campaign to monopolize and lobby. The value of their product is limited, they have no push to optimize for external hosting or for a product that runs within resource constraints. There is opportunity in what they are too greedy to do. Worse comes to worse, leave and join the dark side of computer vision. We still value optimization and control of the products we develop in this space; my entire job is figuring out how to make big/ performant things happen on junk, and there is so much more disruption to be made than people may realize in that regard. The architecture of agoraphobia will bite OpenAI in the ass and cause GPT to lose value over time as smaller models improve in contexts of scalability that require sharing them openly/fully. * permalink * embed * save * report * give award * reply [-]nullbyte420 24 points25 points26 points 1 day ago (3 children) Yes it does you can pay a small sum to fine tune it, which includes adding company knowledge * permalink * embed * save * parent * report * give award * reply [-]GitGudOrGetGot 16 points17 points18 points 1 day ago (0 children) A lot wrong with the post you're replying to, most fundamentally though it focuses on one companies implementation Its not a walled garden at all if other disruptors (e.g meta) choose to democratize the compute they've invested Not to mention the precedent this recent breakthrough presents. Even if there are things gpt4 can't do, 2 more years could vastly increase the number of use cases unlocked in such a short space of time, the fine tuning you mentioned being one of them. Maybe they're in it for a lot more than writing poems all day * permalink * embed * save * parent * report * give award * reply [-]gwern 2 points3 points4 points 1 day ago (1 child) And this is what companies like Harvey (law) are already doing, just like people were finetuning GPT-3 through the API before. (Most (in) famously, AI Dungeon.) * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]ginsunuva 5 points6 points7 points 1 day ago (0 children) NSFW startups are probably the way to go tbh * permalink * embed * save * parent * report * give award * reply [-]EnjoyableGamer 1 point2 points3 points 1 day ago (1 child) This misses OP's point I feel, all these extensions/ improvements are not fundamental research. Still interesting, but not as exciting * permalink * embed * save * parent * report * give award * reply [-]Oswald_Hydrabot 1 point2 points3 points 1 day ago* (0 children) Well if having a job is exciting then researching something of interest to business should be too. Implementation isn't an "extension". It is funny to me how academia and research loses touch with it's purpose sometimes. Implications don't build themselves, if you are feeling helpless then prove the value of some of those implications; not all of them turn out to be practical or even true, but when you find one that can be proven and you do that through a production ready implementation it is quite exciting. I am an engineer, if you make something valuable to me then you make something valuable to the business. I can't tell you how to get excited about that, but there is a lot of room to be disruptive on weak/small hardware. Getting away from supply dependencies drives this, but the results that are already emerging have me excited on the engineering side, I can't imagine the research required for recent breakthroughs in optimization was not at least partially a work of passion by the authors. * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) [-]_Arsenie_Boca_ 12 points13 points14 points 1 day ago (6 children) Are you talking about engineering or science? For an engineer, a new powerful technology should be great, it makes their job easier. If past work is thrown out the window because its worse and more complicated, so be it. For scientiets, this might seem like a threat of course, but only if you are trying to compete with big tech in creating the biggest, most general pretrained model. There is lots of interesting research directions where either ChatGPT is not sufficient or where you can build on top of it. No reason to panic * permalink * embed * save * report * give award * reply load more comments (6 replies) [-]Evening_Emotion_ 12 points13 points14 points 1 day ago (1 child) One day I was called in a closed room , it was dark my arms were sweating and heartbeats getting faster. I saw manager coming in, I asked him what is this all about. He started by saying, they don't need Data Scientist anymore , but probably will keep some resources for tinkering. I was told I was safe, but my entire team was laid off . I have become less productive and lesser focused. A bit suicidal too * permalink * embed * save * report * give award * reply [-]smyja 2 points3 points4 points 1 day ago (0 children) Wow, sorry. * permalink * embed * save * parent * report * give award * reply [-]currentscurrents 77 points78 points79 points 1 day ago (19 children) We have these year long initiatives to productionalize "state of the art NLP models" that are now completely obsolete in the face of GPT-4. You're in a big tech company, you have the resources to train large models; why can't you match GPT-4? But I get the feeling of panic. I'm in the tech department of a non-tech company and we're panicking about LLMs too. It's clearly going to be a disruptive technology across a wide range of industries. * permalink * embed * save * report * give award * reply [-]thrwsitaway4321[S] 52 points53 points54 points 1 day ago (10 children) They are absolutely working on it somewhere in the company. But not in my org, im not sure all these people can just pivot that fast. Regardless, its not realistic to continue down the same path. Things that seemed innovative now seem old * permalink * embed * save * parent * report * give award * reply [-]currentscurrents 23 points24 points25 points 1 day ago (0 children) Talk to your management about your concerns. If they're smart, they'll listen; if they don't, that's your sign to find a job with better management. * permalink * embed * save * parent * report * give award * reply load more comments (9 replies) [-]Jurph 33 points34 points35 points 1 day ago (5 children) You're in a big tech company, you have the resources to train large models There are five American companies -- Amazon, NVIDIA, Meta, Google, and Microsoft -- who have the resources to train capital-L Large models, scale-breaking behemoths like GPT-4. The next tier down can fine-tune on their industry-specific datasets, provided they can find-and-pay an ML Ph.D. who wants to lead a second-tier effort rather than pull an oar at a top-tier program. My company is in "fast follow" mode. We're watching the research emerge, talking to academia and industry, and making smart bets on where our customer domain expertise will synergize with our understanding of the top-tier tech. We're able to get prototypes to customers, tailored for their use cases, within a month or two of a big model being open-sourced. * permalink * embed * save * parent * report * give award * reply [-]ktpr 9 points10 points11 points 1 day ago (2 children) Governments will start to train their own language models soon. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) load more comments (2 replies) [-]TheTerrasque 14 points15 points16 points 1 day ago (0 children) why can't you match GPT-4? For one, cost. It cost a lot training such a model, even if you know exactly how you should do it. It makes sense for OpenAI, since they have their whole business revolving around delivering state-of-the-art AI. For a company's "personal" use? The math looks very different. * permalink * embed * save * parent * report * give award * reply [-]Hyper1on 1 point2 points3 points 1 day ago (0 children) Even big tech companies balk at spending in the high tens of millions for a single training run. And that's not counting the staff of over 100 very expensive devs and researchers it took to train and evaluate GPT-4. That said, big tech can absolutely catch up, they've just been asleep at the wheel for 3 years and OpenAI have a big lead. * permalink * embed * save * parent * report * give award * reply [-]thecity2 17 points18 points19 points 1 day ago (17 children) It's not just NLP either. * permalink * embed * save * report * give award * reply [-]thrwsitaway4321[S] 4 points5 points6 points 1 day ago (16 children) Where are you seeing it? * permalink * embed * save * parent * report * give award * reply [-]thecity2 31 points32 points33 points 1 day ago (15 children) I think it's everything. My company is doing risk models and our founder keeps bugging me about ChatGPT. Luckily for me so far it doesn't do math well lol. But I mean GPT4 is multimodal so it will probably disrupt a lot more than just NLP. Nothing is safe! * permalink * embed * save * parent * report * give award * reply [-]RobbinDeBank 9 points10 points11 points 1 day ago (0 children) LLM isn't really an expert at niche topics tho. I'm curious why would GPT with hundreds of billions of params be better for a specific task at your company. Wouldn't your own sub-1B params model much more efficient? * permalink * embed * save * parent * report * give award * reply [-]VVindrunner 10 points11 points12 points 1 day ago (11 children) GPT4 is much better at math XD * permalink * embed * save * parent * report * give award * reply [-]T10- 4 points5 points6 points 1 day ago (5 children) Wait really? What have you tested it with? * permalink * embed * save * parent * report * give award * reply [-]GitGudOrGetGot 11 points12 points13 points 1 day ago (4 children) Yesterday's demo had it solve a tax calculation given this year's tax code as prompt, but I'm not convinced of its robustness compared to an actual calculator just yet * permalink * embed * save * parent * report * give award * reply [-][deleted] 1 day ago (3 children) [deleted] [-]smyja 1 point2 points3 points 1 day ago (0 children) Just add an agent to it. Langchain has some of them. This isn't an issue at all. * permalink * embed * save * report * give award * reply load more comments (2 replies) [-]sprcow 1 point2 points3 points 1 day ago (0 children) It's also much better at chess than chatGPT. It can keep track of pieces better and play with much higher accuracy, and do a better job of coming up with correct explanations for why each move is made. It's not perfect, or even great, but it's pretty good, and extremely impressive for 'just a language model'. * permalink * embed * save * parent * report * give award * reply [-]nirehtylsotstniop 0 points1 point2 points 1 day ago (2 children) Wait really? What have you tested it with? please god let this be true. there is literally no good high level math resources out there. i need a tutor at the entry to graduate level. * permalink * embed * save * parent * report * give award * reply [-]VVindrunner 8 points9 points10 points 1 day ago (1 child) Better, yes, still not trustworthy. It will still sometimes write out believable looking proof of a completely false answer. * permalink * embed * save * parent * report * give award * reply [-]nirehtylsotstniop 5 points6 points7 points 1 day ago (0 children) I tested it a bit tonight. It still cannot be used in this way, especially for more theoretical stuff. It feels like it's not close either. It is significantly better though. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) load more comments (2 replies) [-]uniklas 7 points8 points9 points 1 day ago (0 children) Depends on the task. If your goal is to compete with LLMs directly then yea, but most use cases are not only about how smart the model is, but also its efficiency. If you need to do some specific and efficient inference on huge dataset then logistic regression with years worth of task specific feature engineering might as well be state of the art. But there is no denying that LLMs are a lot more general, so it's all about the goal. * permalink * embed * save * report * give award * reply [-]bubudumbdumb 6 points7 points8 points 1 day ago (2 children) Yeah The key is that AI engineering MGMT has to proactively figure out and prioritize LLM opportunities before product managers try to throw gasoline into the fire of carefully planned understaffed roadmaps. Surfing the wave of the hype cycle, from here it can only get worse * permalink * embed * save * report * give award * reply load more comments (2 replies) [-]tripple13 6 points7 points8 points 1 day ago (0 children) If you are from the organisation I think you are, you could still position yourself as the goto-platform for fine-tuning SoTA models. Maybe that will also evidently go away, but the main contribution of OpenAI - Which I don't want to discount, its incredible - Is their multi-year effort in high quality human labelling. A lot of organisations sit on a treasure trove of data, the key is to activate it. * permalink * embed * save * report * give award * reply [-]z_fi 47 points48 points49 points 1 day ago* (1 child) Welcome to R&D. It's quite wonderful that you're working for a company that can fund research or innovation projects. Those kinds of projects must be managed quite differently than operations or development Realize that the business need is always to productize or commercialize work, that is, to turn a profit. So the question you'll have to answer is how to do that. Or you'll have to hope your boss has the answer Do you really care about vague "long term science careers" or hot and bothered about other peoples egos? What's your actual question here? * permalink * embed * save * report * give award * reply [-]thrwsitaway4321[S] 26 points27 points28 points 1 day ago (0 children) Just an observation, but also it feels like this disruption is more drastic than something like from n-gram models to word embeddings * permalink * embed * save * parent * report * give award * reply [-]_Repeats_ 55 points56 points57 points 1 day ago (8 children) OpenAI is a startup that has no reputation to lose if their models start telling people to cut their wrists or leave their wives. Big tech absolutely has customers to answer if that crap happens. It is why both Microsoft and Google abandoned their 1st generation chat bots within days. They started spewing out Hail Hilter and threatening to take over nuclear command... And it isn't as easy as "just train a model durp". It costs hundreds of millions of dollars just to get a cluster in place that can train chatgpt. There are hundreds of levers to pull to make everything work. Even PhDs are behind by the time they graduate, assuming their thesis took 3-4 years. That is an eternity in AI. * permalink * embed * save * report * give award * reply [-]thrwsitaway4321[S] 38 points39 points40 points 1 day ago (3 children) A colleague of mine had a similar reply. But it feels like a cop out. The product itself is very vulnerable to a startup/competitor who has a ChatGPT like model. It's hard to say the organization made a mistake by underinvesting in that area, but at that same time, what are these highly paid scientists doing? Our budget is very large and some of them make close to 1M a year. * permalink * embed * save * parent * report * give award * reply [-]bubudumbdumb 31 points32 points33 points 1 day ago (1 child) Let me tell you about quants : they do more math, they get more money and what do they do? They clean data to be fed into linear models. * permalink * embed * save * parent * report * give award * reply [-]Deto 4 points5 points6 points 1 day ago (0 children) Yeah but they can easily point to a lot of money that they are making the company so they're probably not as worried. (Though in reality I imagine there's a lot of fudging going on when most funds have a hard time beating the S&P) * permalink * embed * save * parent * report * give award * reply [-]WildlifePhysics 2 points3 points4 points 1 day ago (0 children) what are these highly paid scientists doing? That's an important question to be asking. Depending upon your role in the organization, why not change up the initiatives being worked on and aim bigger? Life's too short to do otherwise. * permalink * embed * save * parent * report * give award * reply [-]GravityWavesRMS 26 points27 points28 points 1 day ago (2 children) OpenAI is a startup that has no reputation to lose if their models start telling people to cut their wrists or leave their wives. Big tech absolutely has customers to answer if that crap happens. It is why both Microsoft and Google abandoned their 1st generation chat Respectfully, this argument seems to unravel itself. OpenAI is a startup with nothing to lose, but why is Microsoft embedding it in its search engine, and why is Google promising to come out with a similar product within the next few months? * permalink * embed * save * parent * report * give award * reply [-]maxToTheJ 11 points12 points13 points 1 day ago (1 child) but why is Microsoft embedding it in its search engine, reputation laundering. It's the genius part of MS play in LLMs. Look at Bard and Llama people largely pooped on those despite metrics not being in the chatGPT ballpark with just uck *insert company name and journalists were extremely skeptical. * permalink * embed * save * parent * report * give award * reply [-]MootVerick 1 point2 points3 points 1 day ago (0 children) Intersting. Can you expand? * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]Insighteous 11 points12 points13 points 1 day ago (0 children) Wasn't it obvious that this would happen? I mean did you have something comparable to gpt3 before? * permalink * embed * save * report * give award * reply [-]user838989237 34 points35 points36 points 1 day ago* (2 children) Amazing double-standard on display. You were being paid to automatize other people's jobs and now you whine when your own job becomes obsolete. * permalink * embed * save * report * give award * reply [-]hattulanHuumeparoni 12 points13 points14 points 1 day ago (0 children) lmao I didn't even think of that, but it's true * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]royal_mcboyle 5 points6 points7 points 1 day ago (0 children) I can tell you our leadership is in full on panic mode because of it lol. We definitely have some product teams that are freaking out as now their value proposition is in question. * permalink * embed * save * report * give award * reply [-]Tiny_Arugula_5648 4 points5 points6 points 1 day ago (0 children) I also work in a major tech company with scientists.. one problem I've seen is the academic SME mindset, people get so obsessed in their own line of research that they don't pay enough attention to upcoming technologies/solutions. Sure sometimes you're on a multi year journey and the destination is worth the effort but other times you're just pushing forward on an approach that isn't going to work or won't be able to release in time to beat a competing approach. The big problem is institutional inertia and politics.. to many teams get caught up chasing a vision and they get to much sunk cost (political capital, labor, resources, and money) and can't be agile and adapt. * permalink * embed * save * report * give award * reply [-]whizzwr 6 points7 points8 points 1 day ago (2 children) for a product you've all probably used. So... a translator? * permalink * embed * save * report * give award * reply [-]wind_dude 1 point2 points3 points 1 day ago (1 child) Maybe not, while chat-gtp performed well on more common languages google translate outperformed it significantly on less common language. Same reports I've heard for gpt-4. But that can probably be solved with more training data for the lower performing languages. It's how readily available that training data is, and if it's a priority, and if increasing that will lead to other side effects in the model. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]amp1212 3 points4 points5 points 1 day ago* (0 children) By definition, any revolutionary technology will leave a lot of incremental projects high and dry. Google, Microsoft, etc -- they were cautious and incremental in their approach. There were solid business reasons for that, much like there were solid business reasons for IBM to be conservative about PCs in the 70s and 80s; disruptive change isn't in your interest, hence "the innovator's dilemma". But, we're now in what Geoffrey Moore once called the "tornado" -- a step change not an iterative change, where people want this new thing "everything, everywhere all at once". And not only do they want it a lot today . . . they'll want it more tomorrow, and more the day after this. Just looking at things like new accounts on Leonardo.ai, the demand is vastly different from two weeks ago, and different again from a month ago. Hard to see the reason for "panic" -- but lots of reason to see that folks who were working on more iterative less disruptive projects . . . most are likely to find their way to new more exciting projects. Look at all the engineering resources that went into Alexa . . . true, it got speech recognition very good, was part of a lot of interesting engineering of microphones and so on . . . but ultimately it was a very stagnant project, with very little to show for it. People got paid, sure -- but did Alexa generate anything much, beyond asking for music? Seems to me that the folks with those skills can be employed much more profitably in related projects which are more disruptive. Whether they remain at Amazon or not . . . hard to see that those folks don't have a very bright future. I'm already seeing lots of folks deploying these models to AWS . . . not hard to see that Amazon would be shifting their AI resources from the loss leading Alexa to enhancing the capabilities of a platform that's been wildly successful and makes them tons of money. * permalink * embed * save * report * give award * reply [-]VinnyVeritas 5 points6 points7 points 22 hours ago (0 children) I saw Google make a panic blogpost announcement for PaLM or something like that. Seems it's mostly vaporware at this point. But this announcement closely following GPT-4 definitely has the smell of despair. * permalink * embed * save * report * give award * reply [-]ChinCoin 12 points13 points14 points 1 day ago (6 children) The panic is justifiable but there is a subtext to it. Unlike past innovations, which were achieved by human ingenuity, i.e., somebody came up with a good idea and disrupted with it, This whole field is fairly black box. Things get better and better but no one truly understands why except at a very broad level, e.g., more data, more parameters. What you don't understand is much scarier than what you do. * permalink * embed * save * report * give award * reply [-]corporate_autist 10 points11 points12 points 1 day ago (2 children) I think this isnt spoken about enough. This innovation is different, theres no real theory to learn. Its a huge black box and it can do almost everything in NLP. * permalink * embed * save * parent * report * give award * reply [-]hexagonshogun 2 points3 points4 points 1 day ago (1 child) GPT 3.5 was terrible at anaphora resolution - a basic task I used to work on. Still haven't checked GPT 4 though. * permalink * embed * save * parent * report * give award * reply [-]xoogl3 1 point2 points3 points 22 hours ago (0 children) "terrible" is not what I would call this response with the chatGPT with GTP3.5 Prompt In the following two sentences: 1) John took two trips around France. 2) They were both wonderful What does the word "both" refer to? Response: The word "both" in the sentence "They were both wonderful" refers to the two trips that John took around France mentioned in the first sentence. And just for good measure here's chatGPT with GPT-4 Prompt: In the following two sentences: 1) John took two trips around France. 2) They were both wonderful What does the word "both" refer to? Response: In these two sentences, the word "both" refers to the two trips John took around France. The pronoun "both" indicates that each of the trips was wonderful. * permalink * embed * save * parent * report * give award * reply [-]yaosio 1 point2 points3 points 23 hours ago (0 children) They'll just have to improve models until the models can explain how they work. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]ktpr 2 points3 points4 points 22 hours ago (0 children) One thing large teams can do is use ChatGPT output to train or induce smaller models that are experimentally shown to have the same accuracy while requiring far less compute and inference resources than OpenAI offerings. Beating the price point for the same service is a no brainer for executives and finance * permalink * embed * save * report * give award * reply [-]Educational-Net303 8 points9 points10 points 1 day ago (8 children) The only future job for humans - training data generator * permalink * embed * save * report * give award * reply [-]emotionalfool123 10 points11 points12 points 1 day ago (5 children) The next big data regime is video understanding. I can't even speculate on what the model will be able to learn if it could grok all the videos available on the internet. Too bad the video data is locked up in YouTube which OpenAI can't use. * permalink * embed * save * parent * report * give award * reply [-]dI-_-I 5 points6 points7 points 1 day ago (0 children) Yep, this is the next big thing, understanding the world through training on videos like your train GPT on text. Then robots. * permalink * embed * save * parent * report * give award * reply [-]corporate_autist 1 point2 points3 points 1 day ago (0 children) I think the technology for this is already here, its just extremely expensive. Someone will do it eventually. * permalink * embed * save * parent * report * give award * reply [-]nirehtylsotstniop 1 point2 points3 points 1 day ago (0 children) It will learn how to be an idiot. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) [-]serge_cell 0 points1 point2 points 1 day ago (1 child) Human were trained for functioning in harsh enviroments with limited access to electric power and with proper training could be much smarter then low-cost DNN. There are many niches where they could be more cost-effective then high-power DNN and robots. * permalink * embed * save * parent * report * give award * reply [-]yaosio 1 point2 points3 points 23 hours ago* (0 children) Not for long. https://mobile.twitter.com/miolini/status/ 1634982361757790209 I've sucefully runned LLaMA 7B model on my 4GB RAM Raspberry Pi 4. It's super slow about 10sec/token. But it looks we can run powerful cognitive pipelines on a cheap hardware. Hardware will get faster, software will get more efficient. It's happening fast now. A Pixel 5 can do 5 tokens/second. Originally it was doing 1 token/ second. https://mobile.twitter.com/simonw/status/1636084164654170112 * permalink * embed * save * parent * report * give award * reply [-]edunuke 2 points3 points4 points 1 day ago (0 children) It is clear by reading the technical report on gpt-4 that they killed it on most benchmark dataset. In that regard, I understand it may have reached a dead end. However, doesn't that mean that benchmarks need to evolve? Achieving GPT-4 performance on those benchmarks may have cost tens of millions. If benchmarks evolve to represent harder concepts and become more difficult, would MSFT dump another 100s of millions to solve those benchmark too? Or would they just stay the way they are and focus on adding more modalities rather than increasing performance to solve more complex benchmark yet to be created? * permalink * embed * save * report * give award * reply [-]melodyze 2 points3 points4 points 1 day ago* (0 children) I pivoted my org off of custom finetuning LLMs and just using openai apis for those tasks once GPT3.5 with few shot was comparable to our finetuned GPT-J. We still have a lot of work to do on top of those models though, including a lot of models that consume the outputs of those models, so it's about adapting, not terminating roles or anything. We get a lot more done now, so it's really a win for our team. * permalink * embed * save * report * give award * reply [-]TitusPullo4 2 points3 points4 points 1 day ago (0 children) I think you've been made more valuable with any experience working on NLP models. * permalink * embed * save * report * give award * reply [-]small_big 4 points5 points6 points 1 day ago (0 children) If you're at a big tech company, then you don't exactly get paid millions every year to sit on your asses. There's no reason why you or other big tech companies can't match GPT-4. If you're at a smaller firm or a startup, then please discard what I just said. * permalink * embed * save * report * give award * reply [-]johnnd 12 points13 points14 points 1 day ago (0 children) No offense to OP, but it was obvious like 4 years ago when GPT-2 came out that this was gonna happen. * permalink * embed * save * report * give award * reply [-]bsenftner 5 points6 points7 points 1 day ago (3 children) I am very curious if the top economists are realizing what these advances in LLM mean for the sustainability of basic Capitalism. I'm a long term developer, been a pro-coder for 44 years, if you ever played PlayStation games you own my software; I also have an MBA with a double major of Finance & International Business. From my perspective, the existence and capabilities of GPT-4 and successors is seriously troubling for the sustainability of many markets, and how those industries operate and are composed. Where "software is eating the world" may have identified the last few decades, this is some new, this is more than "just software". I think human civilization just entered a whole new age, which may not be understood until after all of use are long gone. * permalink * embed * save * report * give award * reply load more comments (3 replies) [-]cajmorgans 2 points3 points4 points 1 day ago (0 children) I mean from a historic point of view, this situation must not be rare * permalink * embed * save * report * give award * reply [-]londons_explorer 4 points5 points6 points 1 day ago* (7 children) If your big org has an NLP department... then they should be capable of replicating OpenAI's work in just a few weeks. It turns out what OpenAI did isn't akin to building Linux or MS Office... The actual codebase is small. It's just a matter of renting in a lot of compute, and paying $$$ for it. They took a lot of risk, because they weren't sure it would work. But to replicate what they have done has far less risk, because now you know it is possible. * permalink * embed * save * report * give award * reply [-]leviaker 7 points8 points9 points 1 day ago (5 children) I think the way they trained with human input along with web crawl is what makes them impressive. Maybe Google missed the bus there? * permalink * embed * save * parent * report * give award * reply load more comments (5 replies) [-]el_chaquiste 6 points7 points8 points 1 day ago (0 children) Yes, very much like rocket science has been well understood for decades, but most rocket launcher companies still die in the crib. The entry level barriers are too high. * permalink * embed * save * parent * report * give award * reply [-]ObiWanCanShowMe 1 point2 points3 points 1 day ago (0 children) The model is a catch all because it can be (in the near future) trained. That is where the future lies. It shouldn't be too had to pivot? * permalink * embed * save * report * give award * reply [-]Borrowedshorts -1 points0 points1 point 1 day ago (0 children) Yes they're in denial. Just look at r/NaturalLanguageProcessing to see how behind the times they are. * permalink * embed * save * report * give award * reply load more comments (38 replies) * about * blog * about * advertising * careers * help * site rules * Reddit help center * reddiquette * mod guidelines * contact us * apps & tools * Reddit for iPhone * Reddit for Android * mobile website * <3 * reddit premium * reddit coins Use of this site constitutes acceptance of our User Agreement and Privacy Policy. (c) 2023 reddit inc. All rights reserved. REDDIT and the ALIEN Logo are registered trademarks of reddit inc. Advertise - technology [pixel] p Rendered by PID 53 on reddit-service-r2-loggedout-5d6dc6c9b5-sjtk6 at 2023-03-16 23:00:56.111963+00:00 running b2f2428 country code: US.