[HN Gopher] Andrew Ng says AI has a proof-of-concept-to-producti...
___________________________________________________________________
Andrew Ng says AI has a proof-of-concept-to-production gap
Author : TN567
Score : 29 points
Date : 2021-05-03 21:59 UTC (1 hours ago)
(HTM) web link (spectrum.ieee.org)
(TXT) w3m dump (spectrum.ieee.org)
| fny wrote:
| Clickbait headline. He did not say it's a long way from use, but
| instead that it's challenging to ensure models translate well to
| real world conditions.
|
| Yes, it's a challenge, especially with vision models, but it's
| doable. Health care models I've worked on have been put into
| production, and they just need to be monitored to remain
| effective.
| dang wrote:
| Ok, we've replaced the title with something he actually said.
| Der_Einzige wrote:
| While this is indeed clickbait as mentioned by others - I am
| consistently shocked with how little the most common technique
| for ensuring that a model you trained works on unseen data,
| cross-validation, is used in the real world.
|
| I had it drilled into my brain that I really shouldn't trust
| anything except the average validation score of a (preferably
| high K value) K-fold cross evaluated model when trying to get an
| idea of how well my ML algorithm performs on unseen data.
| Apparently most people in my field (NLP) did not have this
| drilled into their head either. This is partly why NLP is filled
| with unreproducable scores (because the magically score if it was
| ever found was only found on seed #3690398 train/test split)
|
| As far as I'm concerned, if you didn't cross-validate, the test
| set score is basically useless.
| Pokepokalypse wrote:
| "In theory, there is no difference between theory and practice.
| In practice, there is."
|
| - Benjamin Brewster . . . 1882
| sesuximo wrote:
| Title seems somewhat misleading. He said ML often performs poorly
| on out of sample inputs. Seems different from being "a long way
| from real world use." I don't think anyone would argue ML is not
| being used in the real world!
| tharkun__ wrote:
| Yes and no. Let's quote a bit more:
|
| > "All of AI, not just healthcare, has a proof-of-concept-to-
| production gap," he says. "The full cycle of a machine learning
| project is not just modeling. It is finding the right data,
| deploying it, monitoring it, feeding data back [into the
| model], showing safety--doing all the things that need to be
| done [for a model] to be deployed
|
| Healthcare has some special needs in regards to what "real
| world use" means. Especially the "showing safety" part he
| mentions.
|
| That's way different from some recommendation engine
| application, where it doesn't really matter, whether your ML
| approach just creates a bunch of bad feedback loops and people
| get sent into rabbit holes of bad music. No lives are at stake
| in that sense but the recommendation engine still "performs
| poorly on out of sample inputs" and is so to speak, "a long way
| from real world use". It's just that either nobody notices or
| even if they do, again, no lives are at stake and so it's OK
| that we're getting banana software (i.e. software that ripens
| in the hands of customers).
| master_yoda_1 wrote:
| They are trying to generalized "Andrew Ng comment about AI
| application to healthcare" to all the application of AI. When
| these journalist learn to properly report.
| throwaway287391 wrote:
| And besides that the current HN title is further misleading
| IMO: "Andrew Ng says ML may work on test sets, but is a long
| way from real-world use". The actual headline was "... but
| _that 's_ a long way from real-world use".
|
| Those are very different statements even though they're only
| one word off. The HN title implies ML is not in real-world use
| (which is certainly not true); Andrew Ng is saying ML
| performance as measured on IID held-out dataset splits isn't
| (always) a good proxy for performance on the data you'll run
| into when the model is deployed in the real world.
| retendo wrote:
| From the last paragraph: "This gap between research and
| practice is not unique to medicine, Ng pointed out, but exists
| throughout the machine learning world.
|
| "All of AI, not just healthcare, has a proof-of-concept-to-
| production gap," he says."
| nerdponx wrote:
| Only because it's been wildly over-hyped and tech journalists
| + startups have over-promised on it. It's perfectly effective
| in its place.
___________________________________________________________________
(page generated 2021-05-03 23:01 UTC)