[HN Gopher] Can synthetic data help train your AI model?
___________________________________________________________________
Can synthetic data help train your AI model?
Author : belter
Score : 32 points
Date : 2022-04-18 12:21 UTC (10 hours ago)
(HTM) web link (www.theregister.com)
(TXT) w3m dump (www.theregister.com)
| tchalla wrote:
| The end is interesting
|
| > "I think that medical data is a really good example of a use
| case that we don't want to work on," says Elbaz. "In order to
| model medical diseases, you need real doctors to help you.
| "There's a lot of specialized knowledge that you would need in
| order to create this medical synthetic data. Even though medical
| data is extremely valuable. It's something that I think requires
| a separate company. It's just too hard. Anything that requires
| very, very, specialized knowledge is hard," he concluded.
|
| One of the clear uses of synthetic data is areas where data is
| inaccessible and/or expensive to gather like medical imaging.
| Turns out, there may be "no free lunch".
| mring33621 wrote:
| So, if you already understand the patterns in your domain, then
| you can leverage your understanding to generate data, to train
| your models to recognize the patterns in your domain's data?
|
| It sounds silly, but maybe it's easier than hand-encoding the
| rules yourself?
| oneoff786 wrote:
| Can synthetic data help a human learn? Sure. Hopefully it doesn't
| learn you wrong.
| time_to_smile wrote:
| In statistical modeling the equivalent of synthetic data is
| heavily used, but not for training the actual model on.
|
| Building a generative model allows you to sample observations
| from a known distribution and confirm that your approach to learn
| is even able to recover the correct parameters. It's used as a
| basic sanity checking tool.
|
| Synthetic data is great for this purpose. Before setting out on a
| complex and expensive modeling task, at least verify that in the
| best case (i.e. your model is literally the data generating
| process) you could even learn the specific problem. It's
| surprisingly easy to define models that are fundamentally not
| learnable.
|
| However the idea that you could learn from synthetic data doesn't
| make sense, as you could just literally use the model that
| generates the data in the first place. If you're just learning
| known parameters, why waste the time training when you already
| have the answer?
|
| Using real data that has been heavily anonymized and stripped of
| PII (perturbing dates, hashing strings etc) makes some sense,
| assuming that PII contains no information (which is unlikely).
| But the more clever you're trying to be with synthetically
| generating features the more your are essentially just manually
| building a model of how the world works with obviates the need to
| learn a model anyway.
| rozgo wrote:
| When your AI models operate on a vectorized space, then yes.
| Pipelines from real-to-vectorspace and synthetic-to-vectorspace
| can be extremely indistinguishable from each other. Basically a
| video game. This makes training on synthetic data extremely
| effective, if not for real inference, at the very least for
| transfer and curriculum learning.
| photochemsyn wrote:
| > "And bias-wise we can generate whatever distribution of
| ethnicities, ages, genders you want in your data, so we are not
| biased in any way," he says as he shows us a three-dimensional
| fake face.
|
| Am I missing something, or isn't this a tool that can be used to
| generate a highly biased dataset to train an ML system on? This
| is how you'd insert biases into your 'algorithm' to get the
| desired result, isn't it? And then you could use that system to
| say, select from a pool of loan applicants, and any biases would
| just be 'due to the algorithm'?
| Kalanos wrote:
| it's a great way to upsample under-represented classes
| tedivm wrote:
| I worked at a company (Rad AI) and we evaluated another company
| that made synthetic data for medical companies. Our physician
| data scientists reviewed the data- it was bad. Really, really
| bad. We ultimately decided that it would be irresponsible to use
| it (and would make for a pretty crappy model anyways).
|
| I'm really glad to hear that this company acknowledges how hard
| that problem is and avoids it. The risk is so high that there is
| just no value in it.
| jcadam wrote:
| At my last job we were doing ML with full motion video. Our
| data scientists kept looking at synthetic data (being able to
| use it would have solved a lot of practical problems for us)
| and rejecting it :)
| srvmshr wrote:
| In my PhD research (on computer vision at University of Tokyo),
| one of the topics I was exploring was reliability of models to
| synthetic data. Using generative models, we could show that
| models which were built on binary data sources were somewhat more
| reliable than others. Consider different geometries of rashes on
| skin & different skin location - the model could still be closer
| to ground truth (i.e model performance on real skin lesions)
|
| However when other variables came into the mix (different tones,
| ethnicity, participant age etc), the model predictions were up
| for tosses.
|
| Now, medical data is much more harder in ML. The variability is
| quite extreme since all our bodies are little different from one
| another. In my thesis, I very firmly proposed that generative
| models should never ever be used in synthetic medical information
| as part of the thesis proposal - and there were those field
| studies to back it up. (The thesis is under non-disclosure at
| University of Tokyo until September 2023 since an embargo is
| involved from participating hospitals & data sources. I would
| have been eager to share otherwise). I am glad people aren't
| wildly trusting if task A, B & C are doable, task AB, CA, ABC, BC
| etc should be similarly tractable.
| deadmutex wrote:
| > I very firmly proposed that generative models should never
| ever be used
|
| Isn't this a bit strong? E.g. just because it is not a good
| idea for your use case _right now_ , it may not mean that in
| the future something else happens and it could invalidate your
| firm proposal.
| srvmshr wrote:
| Apologies. Yes it seems a bit strong, but going by the
| understanding of field-data variability - I and few others in
| the area, feel that it would take nothing short of an
| absolute miracle to get to human-level error rate in
| diagnostic medicine. Hence, the confidence for the near
| future (5-10 years) and a disclaimer against the false notion
| of improving ML based outcomes. Sometimes negative results
| paint much truer pictures than amazing ones. Maybe if the
| paradigm changes for how we design generator networks,
| perhaps one day this presumption will be invalidated as you
| correctly pointed.
|
| The whole craze about my application i.e. dermatology
| successes in ML spurred from the 2017 Nature paper, where
| skin cancer was detected as good or better than
| dermatologists. But technically, such experiments have design
| problems: We had apriori knowledge of the dataset (White N
| American Melanoma data) & hence we could ascertain the model
| performance. Real world data is much more variable. Further,
| later it was revealed that ML model latched on to the little
| marking physicians made rather than generalizing on lesions.
| 3 years later my experiments could model reliably only on 10
| very common diseases & of a very uniform skin type and
| ethnicity. Those results were nowhere close to perfect.
|
| The proposal to keep Human-in-the-loop is a much fairer
| alternative in ML aided medicine, than end to end machine
| learning. Most direction of research is headed that way.
| Physician assistance is much more reliable than potential
| replacement.
|
| The trouble with synthetic data is that it doesn't address
| the extended variability in real population & generalization
| will always suffer. Also, doctors take multi-path decision,
| choice by elimination, past cases - based on several
| diagnostic inputs & even gut intuition. At that scale of
| input multimodality, Type I & II errors are at a scale higher
| than correct identifications. And we don't know how to teach
| intuition or imagination to machine models well enough. Those
| fall back to rule based methods & edge cases.
| axg11 wrote:
| Generative models cannot do better than a well balanced and
| representative training dataset. I've worked on generative models
| in the medical space in the past. If you are looking to use a
| generative model to fix class imbalance, there is always the
| alternative of weighting your training dataset to account for the
| class imbalance. Both methods are effectively doing the same
| thing: oversampling the underrepresented class.
|
| The issues that come with generative models are not unique to
| machine learning. Here's a human example: imagine you are a
| radiologist with 20 years of experience. Throughout your career
| you have specialized in MRI scans. You are able to identify a
| wide range of pathologies. Now one day a friend walks into your
| office with a brain MRI of a giraffe. They ask you to determine
| if the giraffe is healthy. You have _some_ prior from the tens of
| thousands of human scans you have analyzed. Despite that, you
| can't make a determination with any high degree of confidence.
| You simply don't have a good baseline for giraffes. Any analysis
| you do make is assuming that learnings from human scans extend to
| giraffes.
|
| Generative models are useful if we are _certain_ that training
| data covers essentially all possible variations. Training a model
| on MNIST to produce numbers/digits is trivial, capturing 99.99%
| of the perceptible variation in handwriting is (relatively) easy.
| The issue with doing this for most medical applications is
| twofold: 1) medical training data is difficult to acquire and
| much less abundant than most other data types, 2) we don't have a
| good measure for how much of the possible variation we have
| captured and how much we have missed
| bl0b wrote:
| Synthetic data was used to train the pose estimation model that
| shipped with the microsoft kinect [0].
|
| [0] https://www.microsoft.com/en-us/research/wp-
| content/uploads/...
___________________________________________________________________
(page generated 2022-04-18 23:02 UTC)