[HN Gopher] Advancements in machine learning for machine learning
___________________________________________________________________
Advancements in machine learning for machine learning
Author : atg_abhishek
Score : 289 points
Date : 2023-12-16 02:50 UTC (20 hours ago)
(HTM) web link (blog.research.google)
(TXT) w3m dump (blog.research.google)
| ikers wrote:
| Feels like they bury the lede with the first paragraph, but
| otherwise cool stuff!
| RyanShook wrote:
| The pace that ML seems to be advancing right now is amazing. I
| don't believe in the singularity but it's changing software and
| then society in ways no one can predict.
| greatpostman wrote:
| I really don't people will be programming like we do today in
| five years
| Gabriel_Martin wrote:
| I don't people will be either brotha, I don't people will be
| <3
| greatpostman wrote:
| Cool
| mortenjorck wrote:
| People don't like it is, but it
| HaZeust wrote:
| People don't think it be like it is, but it do.
| SirMaster wrote:
| I don't see why not.
|
| I like programming how I do now. I don't plan to stop.
|
| People do lots of things manually that machines have been
| able to do for a long time.
| kypro wrote:
| You can do what you want, you just won't be paid to do it
| anymore.
| jocoda wrote:
| I think most are going to wait on the side lines watching
| the bleeding edge. The promise is great but so is the
| risk of disaster. Imo it's going to be a generational
| shift.
| vbezhenar wrote:
| I still get paid for project involving Oracle 9i running
| on Itanium HPUX and Delphi application running on Windows
| XP. This project is not going anywhere in the next 5
| years. And there are numerous other projects which will
| not go anywhere either. I just don't believe that
| programming landscape will change much. May be in
| California startup world. My world moves slower.
|
| I don't think anything significantly changed in my
| approach to the code since 2013. We will see how 2033
| goes, but I don't expect nothing big either. ChatGPT is
| just Google replacement, Copilot is just smart
| autocomplete. I can use Google instead of ChatGPT and I
| could use Google 10 years ago. I can use vim macros
| instead of Copilot. This AI stuff helps me to save few
| hours a months, I guess, so it worth few bucks of
| subscription, but nothing groundbreaking so far.
| smabie wrote:
| And they generally get outcompeted sooner or latter.
|
| All disciplines evolve over time and those who fail or
| refuse to keep up will be left behind.
| fjkdlsjflkds wrote:
| Just because you can have a robot/machine that can
| efficiently churn out 1000 frozen lasagnas a second
| doesn't necessarily mean that italian restaurants have
| been "outcompeted" or "left behind" by not using such a
| machine in their business.
|
| Sometimes quality and responsibility matter. Even if a
| machine is really good at producing bug-free code, often
| _someone_ is going to have to read /understand the code
| that the machine produces and take resposibility for it,
| since machines cannot take responsibility for things.
|
| In the end, you'll always need a human that understands
| the formal algorithmic language (i.e., a programmer),
| capable of parsing the formal program that will be run,
| if you want to be able to trust the program, since any
| way of mapping an _informal_ request (described
| imperfectly in an informal language) to a _formal_
| construct is always going to be "lossy" and prone to
| errors (and you'll know this if you ever used an
| automatic code generator). Just because someone is
| willing to blindly trust automatically-generated code,
| doesn't mean everyone else is: there are contexts in
| which you need a _person_ to blame, when things go wrong.
| rybosome wrote:
| Ok, but to continue this analogy, industrialization and
| the ability to create 1000 frozen lasagnas a second had
| an enormous impact on the world. Not only on the
| economics of production, but ultimately on human society.
|
| Sure, handmade lasagna still exists, but the world looks
| nothing like it did 200 years ago.
| pixl97 wrote:
| Heh, most cooking these days is like using libraries to
| build an application.
|
| I don't slaughter an animal, I buy a cut of meat.
|
| It's very rare I make pasta, I rehydrate dried husks.
|
| The cheese comes in some kind of jar or package. The
| vegetables come from a store.
|
| This has been the general move in applications too. I see
| companies with very large programs that are just huge
| sets of node modules joined together with a small amount
| of code.
| nonrandomstring wrote:
| > and take responsibility for it, since machines cannot
| take responsibility for things.
|
| That's an interesting thought; people "taking
| responsibility" as a form of labour, for machines that
| stole their lunch. Probably be around zero applicants for
| that job.
|
| Responsibility is a complex quality needing capacity and
| competence. Right now even the manufacturers of "AI" are
| unable to assert much about behaviour.
|
| Where "responsibility" exists around robotics and ML it
| will more likely be a blunt legal instrument tied to
| ownership, like owning a dangerous animal.
| samus wrote:
| AI can be used to support that activity too. Models can
| just as well used to explain existing code, possibly
| cranked out by another AI. I bet many companies are
| thinking about fine-tuning or LoRA-ing language models on
| their moldy codebases and outdated pile of documentation
| to make onboarding, refactoring, and routine extensions
| easier.
|
| To interpret what AI models themselves are actually
| doing, researchers employ AI models as well.
| SirMaster wrote:
| People do things because they enjoy doing them. And
| people will continue to do things they enjoy doing such
| as programming.
|
| I don't think it has anything to do with competition or
| being left behind.
| samus wrote:
| The point is that people _won 't get paid anymore_ to do
| it. It has happened before: many activities that have
| been replaced by technology have been almost forgotten
| (for example the newspaper reader in the factory) or are
| practiced as art or niche crafts only. Careers built on
| these are either wildly successful or have highly
| unsteady income since they are literally reliant on the
| whims, not on the needs, of people.
| ben_w wrote:
| We can't all run a YouTube channel for the programming
| equivalent of Primitive Technology, fun though that would
| be. 99.99% of us will have to adapt to AI being a coworker,
| who will probably eventually replace us.
|
| Right now we're still OK because the AI isn't good enough;
| when it gets good enough, doing things manually is as
| economically sensible as making your own iron by gathering
| a few times your mass in wood, burning some of in a sealed
| clay dome to turn it into charcoal, digging up some more
| clay to make a porous pot and a kiln to fire it in, filling
| it with iron rich bacterial scum from a creek, letting the
| water drain, building a furnace, preheating it with the
| rest of the wood, then smelting the bacterial ore with the
| charcoal, to yield about 7 grams or iron.
| ChatGTP wrote:
| who will probably eventually replace us.
|
| no one is going to be using AI and then just have it
| 'replace them', they're going to use it to augment their
| abilities and avoid replacement.
| ben_w wrote:
| The people using AI to write code aren't necessarily
| former professional programmers, for the same reasons
| people using AI to make pictures aren't necessarily
| former professional artists, and those using aim bots
| aren't necessarily former professional snipers or olympic
| shooters.
|
| A manager can prompt a chatbot to write a thing instead
| of prompting me to write the same thing -- for the
| moment, what (I hope) keeps me employable is that the the
| chatbot is "only" at the level of getting good grades
| rather than n-years professional experience.
|
| I have no expectation for any specific timeline for that
| to change. Perhaps there are enough incentives it will
| never get trained to that level, but also perhaps it was
| already trained 4 months back and the improvement to
| capabilities are what caused the OpenAI drama.
| djmips wrote:
| Nice analogy! I saw an estimate recently on the cost of
| programming and they predicted that automated coding will
| cost 10,000 times less than human coders. It was all back
| of the envelope and questionable but still it was food
| for thought. Will we be 10,000 times more productive or
| will we be out of work? I think a lot of people will be
| out of work.
| ben_w wrote:
| Thanks! :)
|
| > Will we be 10,000 times more productive or will we be
| out of work? I think a lot of people will be out of work.
|
| It can be both. Automation of farming means we've gone
| from a constant risk of starvation to an epidemic of
| obesity, while simultaneously reducing the percentage of
| the workforce in agriculture.
| Der_Einzige wrote:
| They're making fun of your typo, but you're right. Pretty
| much every software job in 5 years will be an AI job. This
| rustles a lot of feathers, but ignoring the truth will only
| hurt your career.
|
| I think the era of big tech paying fat stacks to a rather
| larger number of technical staff will start to wane as well.
| Better hope you have top AI paper publications and deep
| experience with all parts of using LLMs/whatever future
| models there are, because if not, you'll be in for a world of
| pain if you got used to cushy tech work and think it's
| inevitable in a world where AI is advancing so fast.
| greatpostman wrote:
| I give it two years. Salaries will drop like a rock.
| flatline wrote:
| LLMs are cool and will continue to change society in ways
| we cannot readily predict, but they are not quite _that_
| cool. GPT3 has been around for a little bit now and the
| world has not ended or encountered a singularity. The
| models are expensive to run both in compute and expertise.
| They produce a lot of garbage.
|
| I see the threat right now to low-paid writing gigs. I'm
| sure there's a whole stratum of those they have wiped out,
| but I also know real live humans still doing that kind of
| work.
|
| What developers may use in five years is a better version
| of Copilot trained on existing code bases. They will let
| developers do more in the time they have, not replace them.
| Open source software has not put us all out of jobs. I
| foresee the waning of Big Tech for other reasons.
| ben_w wrote:
| > GPT3 has been around for a little bit now and the world
| has not ended or encountered a singularity.
|
| And they won't right up until they do. Reason why is
| that...
|
| > The models are expensive to run both in compute and
| expertise.
|
| ...doesn't extend to the one cost that matters: money.
|
| Imagine a future AI that beats graduates and not just
| students. If it costs as much per line of code as 1000
| gpt-4-1106-preview[0] tokens, the cost of rewriting _all_
| of Red Hat Linux 7.1 from scratch[1] is less than 1
| million USD.
|
| [0] $0.03 / 1K tokens
|
| [1] https://dwheeler.com/sloc/
| flatline wrote:
| I like financial breakdowns like this. The thing an LLM
| cannot do is all the decision making that went into that.
| Framing the problem is harder to quantify, and is almost
| certainly an order of magnitude more work than writing
| and debugging the code. But a sufficiently good LLM
| should be able to produce code cheaper than humans. Maybe
| with time and outside sources of truth, better.
| sublinear wrote:
| Have you ever worked in tech and had to deal with the
| typical illiteracy and incompetence of management and
| execs?
|
| If LLMs got this good, the brick wall these orgs will hit
| is what will really ruffle feathers. Leadership will have
| to be replaced by their technical workers in order for the
| company to continue existing. There's simply not enough
| information in the very high level plain english
| requirements they're used to thinking about. From a
| theoretical and practical perspective, you very likely
| cannot feed that half-assed junk to any LLM no matter how
| advanced and expect useful results. This is very much
| already the case human-to-human for all of history.
|
| Either that or nothing happens, which is the current state
| of things. Writing code is not even 10% of the job.
| Fbnkigffb66tfbj wrote:
| > you very likely cannot feed that half-assed junk to any
| LLM no matter how advanced and expect useful results
|
| Why don't you think that a sufficiently advanced AI can
| do the same as what technical humans do today with vague
| directions from managers?
| ben_w wrote:
| Indeed.
|
| I can give a vague, poorly written, poorly spelled
| request to the free version of ChatGPT and it still gives
| me a correct response.
|
| As correct as usual at least (85-95%), but that's a
| different problem.
| vbezhenar wrote:
| I feel that issue with AI is similar to issues with AI
| cars.
|
| AI car won't ever reach its destination in my city.
| Because you need to actively break the rules few times if
| you want to drive to the destination. There's a stream of
| cars and you need to merge into it. You don't have an
| advantage, so you need to wait until this stream of cars
| will end. However you can wait for that for hours. In
| reality you act aggressively and someone will allow you
| to join. AI will not do that. Every driver does that all
| the time.
|
| So when AI will try to integrate into human society,
| it'll hit the same issues. You sent mail to manager and
| this mail got lost because manager does not feel like
| answering it. You need to seek him, you need to face him
| and ask your question, so he has nowhere to run. AI does
| not have physical presence, neither he have aggression
| necessary for this. He'll just helplessly send emails
| around, moving right into spam.
| euos wrote:
| I've been programming since middle school. That would be 30
| years. Nothing really changed much. C++ is incrementally more
| convenient but fundamentally the same. Code editors are same.
| Debugger are same. Shell is same.
|
| I am certain in 30 years everything will still be the same.
| valine wrote:
| The way I write code was fundamentally altered in the last
| year by GPT4 and copilot. Try having GPT4 write your code,
| you won't be so certain about the future of programming
| afterward I guarantee it.
| mianos wrote:
| I am the same, 35 years. I use GPT 4 every day now. It
| sure is handy. It speeds up some things. It is a time
| saver but it does not seem to be better than me. It is
| like an OK assistant.
|
| I would agree, not a fundamental or radical improvement
| yet.
|
| Will it be? I hope so.
| vbezhenar wrote:
| GPT 4 does not produce code that I'm ready to accept. The
| time it takes to convince it to produce code that I'll
| accept significantly larger than the time it takes to
| write that code myself.
|
| GPT 4 is fine for absolutely foreign tasks to me, like
| write a power shell script, because I know almost nothing
| about power shell. However those tasks are rare and I
| generally competent about things I need to do.
| euos wrote:
| I have free Copilot due to my OSS work. This week I
| disabled it for C++ because it is chronically incapable
| to match brackets. I was wasting too much time fixing the
| messes.
|
| I use it for TypeScript/React. But it's just a more
| comprehensive code complete. Incremental.
| valine wrote:
| Uh huh, try GPT4 and report back. It's a generational
| leap above copilot. I use copilot to auto complete one
| liners and GPT4 to generate whole methods.
| pixl97 wrote:
| Other than 30 years ago you were writing a whole shitload
| more buffer/integer overflows. Hell, that's why we've
| written numerous languages since that point to ensure it's
| a hell of a lot harder to footgun yourself.
|
| If coding hasn't change much in 30 years, it may mean _you_
| have not changed much in 30 years.
| xbmcuser wrote:
| I think the biggest blind spot for many programers/coders is
| that yes it might not change much for them but it will allow
| many more people to code and do stuff that they were not able
| to before. As the the models get better and people use them
| more and learn how to use them more efficiently they will
| start changing things.
|
| I am hoping we get to the point where the models are good
| enough that classes in schools are introduced on how to use
| them rather than just build them as the number of people
| wanting to or willing to learn programming is a lot smaller
| than the number of people to looking for ways to do things
| more efficiently.
| moffkalast wrote:
| It's not like schools have any other option on the table,
| students will find a way to use all the help they can get
| like they always have. Embracing it is the only way they
| can stay relevant in the coming age of one-on-one AI
| tutors.
|
| It reminds me of the middle ages where only the priest was
| allowed to read and interpret the bible, mostly through the
| virtue of knowing latin. Then suddenly the printing press
| comes around and everyone can get their own cheap bible in
| their language. You just can't fight and enforce this kind
| of thing in the face of such insane progress. In 100 years
| (if we're not extinct then) people will probably look back
| on mass education where one overworked teacher tries to
| explain something in a standard way to 30 people (over half
| of who are bored or can't keep up) as some kind of old age
| savagery.
| wait_a_minute wrote:
| This + FunSearch make it seem like Singularity is imminent.
|
| https://deepmind.google/discover/blog/funsearch-making-new-d...
| tommychillfiger wrote:
| At great risk of sounding completely ignorant, this approach
| is basically what I thought the point of machine learning was
| - cleverly using feedback loops to improve things
| automatically. The thing that sticks out to me as
| particularly cool about FunSearch is the use of programs as
| inputs/outputs and the fact that they managed to automate
| feedback.
|
| I'm pretty naive in terms of granular understanding here as I
| am barely proficient in Python, to be clear, but when I
| daydream about things you could solve with machine
| learning/AI, this is the approach I always think of and I
| guess is how I thought it already worked. Load it up with the
| best information we have currently, define the desired
| results as clearly as possible, implement some form of
| automatic feedback, and let it run iteratively until it
| produces something better than what you had before.
|
| Is this a case of "well no shit, but actually implementing
| that effectively is the hard part"? Is it being able to
| quickly apply it to a wide variety of problems? I guess I'm
| trying to understand whether this is a novel idea (and if so,
| what parts are novel), or if the idea has been around and
| it's a novel implementation.
| IanCal wrote:
| The important thing is "how do you change X so that it
| heads towards the goal". And "how to do it quickly and
| efficiently".
|
| Otherwise the description is the same as "select randomly,
| keep the best, iterate".
|
| The goal is also complex. You might be thinking of "find
| the most efficient program" but that's not what we're doing
| here iiuc. We're trying to get a program that makes _other
| unseen_ programs more efficient. That 's hard to define as
| a goal.
| Jensson wrote:
| > Otherwise the description is the same as "select
| randomly, keep the best, iterate".
|
| That is what they did though. The LLM didn't know what
| problem it was "solving".
| IanCal wrote:
| That's not really true unless you're ignoring the rest of
| my points. The process did not just uniformly randomly
| create programs.
|
| They also don't just keep the best and search from that
| point but feed the resulting programs and their scores
| into an LLM.
| Jensson wrote:
| They did remove the worst results from the group over
| time, the others was just uses a seed to generate new
| examples from instead of starting each function from
| scratch.
| moffkalast wrote:
| Some speculate that this is what OpenAI's Q* model is about
| and what caused the Altman/Sutskever split.
| m3kw9 wrote:
| I want to see it come out with a cure for a disease that is
| tough to cure first. Singularity itself is pointless unless it
| benefits humans which is mainly in health/lower suffering
| educaysean wrote:
| I'd say advancement in mathematics, computer science, and
| heck, even art is far from "pointless". Why does it feel like
| goalposts get moved everytime there is a significant progress
| in AI?
| impegh wrote:
| I read this same empty "goalposts" lament multiples times a
| day when reading this website, and though I know what the
| words mean I'm confused what you all think they mean.
|
| https://hn.algolia.com/?dateRange=all&page=0&prefix=false&q
| u...
|
| Is this just some performative grousing or do you really
| think what has been developed to date is "artificial
| intelligence"?
|
| These comments all conveniently fail to define their
| author's goalposts that apparently have been reached or
| surpassed. What were yours?
| quickthrower2 wrote:
| Well for it to be Artificial and convey Intellgence. I
| think that goal has been met!
| F-Lexx wrote:
| How do you define intelligence?
| quickthrower2 wrote:
| I don't have a exact definition but I will claim that
| helping me to write code that usually works first time or
| has a minor bug based on natural language is
| intelligence.
| resonious wrote:
| Back in my CS undergrad we learned Dijkstra's Algorithm
| in AI class. Once upon a time, that was AI. I think AI
| just refers to newly discovered useful algorithms.
| cornel_io wrote:
| There was a very recent time when passing some version of
| the Turing Test would have been a fairly commonly
| accepted goalpost. Many experts thought that was 20+
| years away, and were perfectly comfortable saying that it
| was their "bar", primarily because they thought
| generating convincing conversational text was _so_
| difficult that you 'd basically have to solve All The
| Problems(TM) first.
|
| Notice how nobody is talking about the Turing Test
| anymore now that it's either already been passed or is
| very damn close? We can argue back and forth about
| whether the real stupidity was the earlier expectation
| that the Turing Test was a useful threshold for AI, but
| it's impossible to claim that it wasn't a somewhat common
| and well-known one, so that goalpost really has been
| moved in a very dramatic way (or rather, removed
| altogether and replaced with nothing in particular other
| than a vague "I'll know it when I see it", in most
| cases).
| creer wrote:
| Exactly so. The posts have been moved safely waayyyy over
| there at AGI, and at "super-human" or at "critical
| thinking". And several stages or degrees of AGI have been
| hierarchized. There is a serious reluctance at accepting
| how dumb an algorithm can be and still compare with
| humans.
|
| But it is also true that numerous ground techniques are
| issue of the field of AI and generally called AI as they
| come out. It makes for good press. And that too was
| silly.
| ChatGTP wrote:
| On the other hand, we've passed the turing test, what's
| changed?
| creer wrote:
| You mean practically? What did this achieve? What did we
| gain now that the turing test is passed?
|
| For actually already done: Actually believable chat-bots?
| Summarizers and question answerers? Generative text and
| graphics actually usable for generation of text, graphics
| and (mostly) photo-realistic renderings? Architecture
| brainstorming? (And logos, etc.) Kinda working self-
| driving cars? New Go playing strategies? A super-human Go
| champion? AI is on a roll these days.
|
| That's not counting the more proprietary and discreet
| applications being already used all over the place. I
| fully expect there are already several.
| jodrellblank wrote:
| > " _Why does it feel like goalposts get moved everytime
| there is a significant progress in AI?_ "
|
| Why does it feel like people think this is a useful or
| interesting whine? OK you brute-force solved tic-tac-toe,
| you built an AI. Congratulations, everyone recognises the
| dawn of Artificial Intelligence - and truly, enumerating
| all states of tic-tac-toe is all we ever dreamed of, all we
| could want, it's really all there is to intelligence. The
| term "AI" will never ever be used to mean anything else.
|
| The term "AI" means (solving the first problem that was
| ever suggested to be AI decades ago) - how is that a better
| state of the world? What has anyone gained from "not moving
| the goalpoasts"?
| melagonster wrote:
| This is impossible, we can makesure that more possible
| scenario is that most of people lose job and starve. it is
| not sure whether we can reach to a society have UBI.
| sbierwagen wrote:
| In order for an AI to evaluate the effect of a small molecule
| on the brain, it would have to... simulate the operation of a
| human brain in a simulated environment. Similarly, to avoid
| Thalidomide-style disasters, it would have to simulate the
| conception, development and growth to adulthood of a human.
|
| These things are... physically possible, but have WBE and
| uploads as a hard requirement. Those are going to affect a
| hell of a lot of things more than the drug industry!
|
| Amusingly, machine-phase nanotechology and blood nanobots
| would be easier to evaluate, since simple cell-level
| mechanical interventions (reading surface proteins on cancer
| cells and chopping them up, say) will have fewer interactions
| than a small molecule that diffuses into every cell in the
| body.
| jocoda wrote:
| WBE?
| AnthonyMouse wrote:
| > In order for an AI to evaluate the effect of a small
| molecule on the brain, it would have to... simulate the
| operation of a human brain in a simulated environment.
| Similarly, to avoid Thalidomide-style disasters, it would
| have to simulate the conception, development and growth to
| adulthood of a human.
|
| This is how the human doctors who have cured things in the
| past have done it, is it?
|
| The way this is going to work, when it happens, is that
| you'll ask the AI for a cure and it will give you a hundred
| candidates. A human doctor will look at the list and throw
| half of them out because they're toxic, several of the
| remainder will be excluded by animal trials, the few
| remaining will proceed to human clinical trials and one of
| them will actually work.
| zer00eyz wrote:
| >> The way this is going to work...
|
| Google is already doing something like this:
| https://arstechnica.com/ai/2023/11/googles-deepmind-
| finds-2-...
|
| This is great if you want to use well understood pathways
| or make new drugs that you can then patent and mark up.
|
| New pathways are gonna require feeding data into these
| models in the first place. Your not getting ozempic out
| of ML without doing the ground work first:
| https://globalnews.ca/news/9793403/ozempic-canada-
| scientist-...
| AnthonyMouse wrote:
| > New pathways are gonna require feeding data into these
| models in the first place. Your not getting ozempic out
| of ML without doing the ground work first
|
| Sure, but a lot of the ground work has already been done,
| or is susceptible to simulation. They're getting a lot of
| results out of simulating protein folding and things like
| that.
| sbierwagen wrote:
| Then AI will have no effect on the drug industry at all.
|
| The rate limiting step isn't "thinking up molecules." The
| University of Bern enumerated all possible molecules
| composed only of hydrogen, carbon, nitrogen, oxygen,
| sulfur and chlorine, up to 17 atoms. That produced 166
| _billion_ molecules.
| https://pubs.acs.org/doi/10.1021/ci300415d There are
| commercial drugs considerably larger than that. We've got
| molecular structures out the nose. There is no shortage
| of molecules.
|
| The problem _is_ the clinical trial. Putting drugs in
| humans and seeing what they do. That 's the part that
| takes years and tens of millions of dollars. Using AI for
| anything else is like saying Microsoft Powerpoint
| accelerated drug development. Sure, it made presentations
| easier, but did it do anything for the problem of putting
| chemicals in people?
| AnthonyMouse wrote:
| > That produced 166 _billion_ molecules.
|
| Which is useless, because you can't run 166 _billion_
| clinical trials.
|
| But you could run half a dozen if there's a strong chance
| one of them will be a success. Filtering the list down to
| 100 molecules from 166 billion, some of which can be
| further eliminated by human evaluation without the
| expense of clinical trials, is actually useful.
|
| You still ultimately have to do the clinical trial,
| because there is no substitute for empiricism.
|
| > That's the part that takes years and tens of millions
| of dollars.
|
| It doesn't matter if it takes tens of millions of dollars
| if the result is a billion dollar drug.
| blackbear_ wrote:
| > The problem is the clinical trial. [...] That's the
| part that takes years and tens of millions of dollars.
|
| Clinical trials only start after about five years of
| research and development. While they do represent a large
| part of the budget (even in the hundreds of millions of
| dollars), there are countless of other necessary steps
| before, during, and after trials to ensure that drugs are
| both safe and effective. The problem is that we still
| don't understand how most of these molecules behave in
| the body, and how we can produce them reliably and
| efficiently enough, which brings me to the next point:
|
| > [...] but did it do anything for the problem of putting
| chemicals in people?
|
| Yes, there are plenty of problems that AI and
| computational chemistry already help with in the
| pharmaceutical industry, including predicting solubility,
| stability, crystallization, granulation, toxicity,
| pharmacokinetics, developing the formulation, optimizing
| and scaling up both the synthesis and production process,
| developing appropriate techniques for quality control,
| and so on.
|
| In all these cases and more, AI can help reduce the
| amount of experiments that need to be done in the lab,
| which require highly specialized equipment, personnel,
| and a lot of time. Oh and design of experiments is also a
| very important topic, again aiming at reducing the amount
| of lab time needed.
|
| Admittedly, most of these things aim at ensuring that we
| do not put the _wrong_ chemical in people, but they do
| represent most of the R &D effort spent in pharma, and
| reducing everything to clinical trials is not correct.
| There is a very wide gap between "AI will design drugs
| entirely on its own" and "AI is useless".
| nodogoto wrote:
| >In order for an AI to evaluate the effect of a small
| molecule on the brain, it would have to... simulate the
| operation of a human brain in a simulated environment.
|
| Humans aren't capable of doing this, but still make useful
| drug discoveries. AI can be empowered to conduct research
| in the real world, it doesn't need to simulate everything.
| ben_w wrote:
| We start by doing them on mice (well, in vitro first,
| mice as the first in vivo), who have no say in the
| matter; and as mice are only rough analogues of humans,
| the human trials are still cautious once the animal
| trials are over.
| ChatGTP wrote:
| We also develop a lot of drugs with a which have side-
| effects, which will is probably better than no drugs in
| most cases, the side effects are because it's a lot of
| educated guesswork.
| ben_w wrote:
| AI advancements are why we have affordable genome reading.
|
| AlphaFold was a nice surprise when it happened, too.
| bart_spoon wrote:
| "Cure" is a tough bar, but I believe Paxlovid, the anti-viral
| used to reduce Covid severity, was identified using ML.
| There's many companies like Recursion Pharma which are
| entirely focused on using ML for drug discovery, and from
| what I can tell seem to have promising results, but drug
| development is slow enough that nothing will come of it for a
| while.
|
| Also, while not medicine focused, Google's GNOME project
| results announced a few weeks ago was pretty remarkable. They
| discovered more theoretical new materials using their ML
| approach than the rest of human history combined, and they
| are already confirming many of the results in laboratory
| settings. That has the potential to be a revolution in
| limitless scientific and engineering applications.
| DeathArrow wrote:
| For me it's just another gold rush after dotcom, mobile, cloud,
| VR.
| falcor84 wrote:
| I'm not sure what the purpose of the word "just" there is.
| There indeed seems to be quite a lot of gold to be had by
| whoever gets a foothold.
| Tao3300 wrote:
| "just" is there to distinguish a gold rush from a
| singularity
| xbmcuser wrote:
| The first 3 have and did result as of today in trillions in
| dollars of economic activity. And have changed societies,
| politics, political participation, access to knowledge etc
| worldwide for good and bad. So I don't get why you are so
| dismissive of them.
| rvnx wrote:
| AI is definitively here to stay forever. It's not a hype,
| it's 100% here for the long-term.
|
| The hype may be specific to some companies for now, but AI
| is deeply going to change many industries, especially due
| to open-source, specialized chips to allow running in
| local, and new hardware (I strongly hope a clone of H100
| A80G comes quickly).
|
| The next step is to add limbs to the LLMs.
|
| Then we get Tesla bot who is going to help you with daily
| chores, and to execute tasks in a factory.
|
| The bot can ask its own internal knowledge base to know
| what action to execute next, and because the model can
| output JSON, the action commands can be sent to motors for
| in-real-life execution.
| Jensson wrote:
| > It's not a hype, it's 100% here for the long-term.
|
| You mean it isn't overhyped, hype is just what
| expectations people have it is underhyped or overhyped
| that says how those expectations related to reality.
| sfn42 wrote:
| Pretty sure we're quite a ways off that stuff yet,
| especially for consumers. But sure, maybe something like
| that will be reality in the coming decades.
| DeathArrow wrote:
| I am not dismissive. I think, though, that for a hundred
| companies engaged in the race one or less might succeed.
| GreedClarifies wrote:
| How's Gemini looking?
| sbierwagen wrote:
| It is interesting how persistently dominant GPT-4 is:
| https://twitter.com/lmsysorg/status/1735729398672716114
|
| Off the top of my head, I can think for at least five
| foundation models (Llama, Claude, Gemini, Falcon, Mistral) that
| are all trading blows, but GPT is still a head above them and
| has been for a year now. Transformer LLMs are simple enough
| that, demonstrably, anyone with a million bucks of GPU time can
| make one, but they can't quite catch up with OpenAI. What's
| their special sauce?
| kccqzy wrote:
| Their only special sauce is the first-mover advantage. Then
| it attracted users (data), brand recognition, talent and
| became a positive feedback cycle.
| vitorgrs wrote:
| GPT4 was created before most feedback cycle. They had GPT4
| ready before ChatGPT launch.
|
| If I recall right, GPT4 got done in October. After that, it
| was RLHF and safety work (Bing starts using GPT4 publicly
| in February, a month earlier than official launch)
| og_kalu wrote:
| GPT-4 was done training August 2022
| vitorgrs wrote:
| Thanks!
| kccqzy wrote:
| If I recall right, before ChatGPT launched Google already
| had LaMDA which an employee believed to be sentient and
| was subsequently fired. The foundation model was
| definitely done, but to launch Bard, Google needed a kick
| in the ass in additional RLHF, safety and groundedness
| work.
|
| Ultimately though, it's futile to argue which model got
| done first, as long as the models were behind closed
| doors. But ChatGPT launched before Bard did and that's
| the pertinent part that gave OpenAI the first-mover
| advantage.
| rvnx wrote:
| LaMDA is really far from being sentient.
|
| It's outputs non-sensical (aka highly hallucinating) or
| relatively useless but coherent text.
|
| It really needs further refinement.
|
| This is one big reason why GPT-4 is still the most
| popular.
| dindobre wrote:
| The LaMDA is sentient guy gave me the impression of being
| a bit nuts. I'm sure google would show their weight and
| out-compete openai if they could. We all know all this
| "AI safety" is for show, right?
| staunton wrote:
| > We all know all this "AI safety" is for show, right?
|
| No. A lot of people think it really matters
|
| A lot of other people pretend to care about it because it
| also enables stifling the competition and attempting
| regulatory capture. But it's not all of them.
| snewman wrote:
| I'm personally devoting my career to AI safety, on a
| volunteer basis, because I think it's is legitimately of
| high importance. (See my blog, e.g.
| https://amistrongeryet.substack.com/p/implications-of-
| agi, if you want to understand where I'm coming from.)
|
| What makes you think it is for show?
| y04nn wrote:
| No, it's for brand safety and reputation. In 2016
| Microsoft released Tay [1] without or lacking guards and
| it ended up being a failure and hurter the Microsoft
| brand.
|
| [1] https://en.wikipedia.org/wiki/Tay_(chatbot)
| ben_w wrote:
| The RHLF is probably quite important even on top of a
| good base model.
| huytersd wrote:
| That's not it. It's not just hype. The underlying model is
| better.
| code51 wrote:
| Their special sauce is most probably the quality of data and
| the amount of data cleaning effort they put in.
|
| I'm speculating here but I think Google always refrains from
| getting into the manual side of things. With LLMs, it became
| obvious so fast that data is what matters. Seeing Microsoft's
| phi-2 play, I'm convinced more about this.
|
| DeepMind understood the properties, came up with Chinchilla
| but DeepMind couldn't integrate well with Google, in terms of
| understanding what kind of data Google should supply to
| increase model quality.
|
| OpenAI put annotation/cleaning work almost right from the
| start. Not too familiar with this but human labor was heavily
| utilized to increase training data quality after ChatGPT
| started.
| staunton wrote:
| Indeed, making poor people in 3rd world countries rate the
| worst sludge of the internet for 8+h a day might backfire
| on your marketing... OpenAI could risk it, Google maybe
| doesn't want to...
| blowski wrote:
| Given that many western companies hire poor people to do
| all sorts of horrible work I doubt it's that. More likely
| it's to avoid suggestions of bias across their product
| range.
| Palmik wrote:
| This is a naive take. How do you think Google collects or
| collected data for their safe-search classifiers? Now
| that's a sludge.
|
| Or how do you think Google evaluates search-ranking
| changes (or gather data for training various ad-ranking &
| search-ranking models).
| staunton wrote:
| I don't know. How do they?
| pixl97 wrote:
| I was going to make a joke about all those CAPTCHAs we've
| solved, but I don't have an answer here.
| NavinF wrote:
| Their instructions for human raters is public info.
|
| Overview: https://blog.google/products/search/overview-
| our-rater-guide...
|
| Full PDF: https://static.googleusercontent.com/media/guid
| elines.raterh...
| dmarchand90 wrote:
| I kinda wonder if maybe it's at least partially due to openai
| hitting a kind of hyperparameter lottery. When each
| experiment costs millions it might be that (aside from good/
| unique data) they just have a good set of hyperparameters
| used in training and it's too expensive for a competitor to
| find equal or better settings
| porompompero wrote:
| Sorry for my ignorance: why does each experiment cost
| millions?
| Jensson wrote:
| Because training a model costs millions, so each time you
| experiment with trying to create a new kind of model it
| costs millions.
| bart_spoon wrote:
| It's the cost of compute hardware required to train a
| model of that size
| jwuphysics wrote:
| I would be surprised if this is the case. Neural scaling
| laws are well known and are used by all big industry
| players to extrapolate experiments.
| dmarchand90 wrote:
| Are they really "laws" my impression is its all just a
| bunch of empirical trends.
|
| We cannot know truly how these parameters interact at
| large scale and also how they interact with each other.
|
| Is it really the case that openai has data that Google
| doesn't?
| jazarwil wrote:
| You cannot compare GPT 4 to Gemini Pro. They are different
| classes of models.
| dwaltrip wrote:
| Note, Gemini Ultra, which they claim is competitive with or
| possibly even better than GPT-4, isn't out yet. They have
| released a weaker model, Gemini Pro.
|
| It will be interesting to see how capable Gemini Ultra
| actually is. For now we wait.
| summerlight wrote:
| Beside the fact that Gemini pro is more comparable to
| GPT-3.5, one more interesting observation is that even OpenAI
| themselves was not able (or didn't intend) to deliver a
| significantly better model than GPT-4 almost over a year. And
| OpenAI does not seem to hide their own magical "AGI" behind
| the scene as they've been more focused on efficiency and
| engineering works reportedly, primarily driven by Sam, rather
| than developing a new model. I'm reasonably sure that the
| current transformer itself as an architecture is at its peak
| and most improvements will be mostly incremental.
| dalbasal wrote:
| Can anyone bring this down to earth for me?
|
| What's the actual state of these "ML compilers" currently, and
| what is rhe near term promise?
| voz_ wrote:
| Check out torch.compile
| d3m0t3p wrote:
| One of the easiest approache is torch.compile, it's the latest
| iteration of pytorch compiler (previous methods were :
| TorchScript and FX Tracing.)
|
| You simply write model = torch.compile(model)
|
| "Across these 163 open-source models torch.compile works 93% of
| time, and the model runs 43% faster in training on an NVIDIA
| A100 GPU. At Float32 precision, it runs 21% faster on average
| and at AMP Precision it runs 51% faster on average."[1]
|
| What google is trying to do, is to involve more people in the
| R&D of these kind of methods.
|
| [1]https://pytorch.org/get-started/pytorch-2.0/
| larodi wrote:
| Thanks for this summary
| PartiallyTyped wrote:
| The near term promise is that you can use AMD, CUDA, TPUs, CPUs
| etc without explicit vendor support for the framework on which
| the model was developed.
|
| Disclaimer: I will be very handwavey, reality is complex.
|
| This is achieved by compiling the graph into some intermediate
| representation. And then implementing the right backend. For
| projects here, look at stableHLO, IREE, openXLA.
|
| You can argue that Jax's jit compiler is a form of such
| compiler, mapping the traced operations down to XLA, which then
| does its own bit of magic to make it work on your backend.
|
| It's transformations and abstractions all the way down.
| seydor wrote:
| What about transformer itself, any indication that it is optimal
| in some way?
| potac wrote:
| Can anyone explain how conv works in that graph. You have a
| tensor of shape [2,4,16] and you convolve with a kernel of shape
| [4,16,8] and that gives you a [2,8] tensor? How's that possible?
| phillengel wrote:
| Does this help?
|
| *1. Input:*
|
| * Tensor shape: [2,4,16] * `2`: This represents the *batch
| size*, meaning there are two independent data samples being
| processed. * `4`: This is the *input feature dimension*,
| indicating each sample has 4 features. * `16`: This is the
| *input channel dimension*, suggesting each feature has 16
| channels of information.
|
| *2. Kernel:*
|
| * Shape: [4,16,8] * `4`: This is the *kernel size*, meaning the
| filter window used to convolve has a width of 4. * `16`: This
| matches the *input channel dimension*, ensuring the filter
| operates on the same number of channels as the input. * `8`:
| This is the *output channel dimension*, indicating the
| convolution produces 8 new channels of information per sample.
|
| *3. Output:*
|
| * Shape: [2,8] * `2`: This remains the *batch size* as the
| operation is applied to each sample independently. * `8`: This
| matches the *output channel dimension* of the kernel,
| signifying the final tensor has 8 new features extracted from
| the input.
|
| *4. How is it possible?*
|
| Despite the seemingly mismatched dimensions in the input and
| output, convolution on graphs works by leveraging the
| *neighborhood structure* of the graph. Here's a simplified
| explanation:
|
| * The kernel slides across the graph, applying its weights to
| the features of the current node and its neighbors within a
| specific radius. * This weighted sum is then aggregated to form
| a new feature for the current node in each output channel. * As
| the kernel moves across the graph, it extracts information from
| the local neighborhood of each node, creating new features that
| capture relationships and patterns within the graph.
|
| *Additional considerations:*
|
| * The graph structure and edge weights likely play a role in
| how information propagates during the convolution process. *
| Specific details of the convolution implementation, including
| padding and stride, might also influence the output shape.
| owlbite wrote:
| These ML-compilers are being overhyped. It's all the same trade-
| off as a traditional compiler: you get a lot more throughput than
| hiring a specialist performance programmer, but the latter will
| typically outperform, possibly by orders of magnitude.
|
| These things are inferior at many levels: - Algorithmic: These
| things aren't feeding back to their human masters tips and tricks
| on how to modify the network to go faster beyond some very basic
| signals. - Loss of intent: ML network designers are specifying
| architecture in python, and by the time it's gone through many
| layers of lowering, you can get some complete garbage. Highly
| efficient garbage, but still garbage. (recent example, we caught
| one of these compilers doing a slice update operation by first
| forming the range of all possible indices to the array, slicing
| that to get indices to update, and then doing a scatter; we
| replaced it with a single memcpy call). - Inefficient kernels.
| Every time we see the output of these compilers go head-to-head
| with an expert assembly programmer, the compiler loses, often by
| 30%+. This always seems like the sort of thing that should be
| easy to solve, but given no-one seems to have cracked it in the
| past 50 years, it's obviously not as simple as it sounds.
| hotstickyballs wrote:
| Hardware (and performance) can always be improved without
| involvement of users so this is actually pretty useful.
| stabbles wrote:
| Take a look at the chess engine Stockfish: they tossed out
| years and years of human written heuristics in board
| evaluation, to a small neural net that does the same but
| better.
|
| Now consider all the heuristics for inlining, loop unrolling,
| vectorization etc in compilers, certainly a neural net can be
| beneficial and possibly easier to maintain than tons of human
| written heuristics.
| asah wrote:
| big +1 - IMHO the future of optimizers (and probably
| compilers...) are almost certainly ML-based.
| ldjkfkdsjnv wrote:
| Humans designing algorithms by hand will go the way of the
| dodo bird
| YawningAngel wrote:
| My understanding is that stockfish retains and uses its
| classical evaluation model in addition to the NNUE model
| owlbite wrote:
| We'll have to see. I could definitely see someone spending a
| lot of time training for a specific algorithmic kernel and
| microarchitecture and beating the best human results (by a
| few percent).
|
| I'd be very surprised if that can be extended to a large
| complex algorithmic system that is amenable to mathematical
| reformulations (at least within the next 10 years).
| dbecker wrote:
| > These ML-compilers are being overhyped. It's all the same
| trade-off as a traditional compiler
|
| Funny you should say that. Because traditional compilers have
| been incredibly useful.
| owlbite wrote:
| Right, but we still tend to sidestep the compiler and/or
| spend hours of human time tuning the input to get the right
| output for core kernels.
| JyB wrote:
| Comment seem extremely dismissive and close minded.
| jhardy54 wrote:
| Exactly! Why would anyone use gcc/clang when you can just hire
| someone to hand-write assembly instead?
| summerlight wrote:
| > It's all the same trade-off as a traditional compiler: you
| get a lot more throughput than hiring a specialist performance
| programmer, but the latter will typically outperform, possibly
| by orders of magnitude.
|
| That throughput is the point though? You cannot have
| performance specialists on every single ML workload. It's still
| significantly better than not having these kinds of
| optimization.
| aconz2 wrote:
| summary: improve prediction of run-time performance of a
| computation graph using GNN, they use an embedding dictionary for
| each node's opcode along with some other node features (eg shape,
| bits, window size, see [1]), they released a big dataset of these
| graphs in [2] with varying XLA compilation configurations and
| their resulting perf on TPUs, they did some stuff to improve
| prediction on bigger graphs than before in [3] by partitioning
| the graph (METIS graph partition, new to me) and other training
| things
|
| This is only about predicting performance of a given graph and
| not about improving/suggesting/editing a new equivalent graph. As
| in FunSearch, models which have decent predictive power could be
| used with evolutionary search.
|
| [1] https://github.com/google-research-
| datasets/tpu_graphs#featu...
|
| [2] TpuGraphs: A Performance Prediction Dataset on Large Tensor
| Computational Graphs https://arxiv.org/abs/2308.13490
|
| [3] Learning Large Graph Property Prediction via Graph Segment
| Training https://arxiv.org/abs/2305.12322
___________________________________________________________________
(page generated 2023-12-16 23:01 UTC)