[HN Gopher] AI can diagnose childhood autism from retinal photos
___________________________________________________________________
AI can diagnose childhood autism from retinal photos
Author : bookofjoe
Score : 56 points
Date : 2023-12-22 20:13 UTC (2 hours ago)
(HTM) web link (petapixel.com)
(TXT) w3m dump (petapixel.com)
| FirmwareBurner wrote:
| What's the difference between AI and just pattern matching here?
|
| I mean, you take a data set of how people with X health condition
| look like, train your model on that set, then the model can, with
| a certain probability threshold, tell you if other people have X
| condition as well. But to me, a non-AI average joe programmer,
| that's more pattern matching, and less AI.
|
| That kind of application was something I remember some colleagues
| in university were working on over 10 years ago in OpenCV and
| Tensorflow, using labeled sets of X-rays and CT scans to pattern
| match conditions for aiding in diagnostics for radiology lab
| techs, well before the AI explosion.
|
| What am I missing here?
| Racing0461 wrote:
| AI (LLMs etc) exhibits "emergent behavior" properties. things
| like reasoning and other higher order effects. It could be
| mimicry (parrotting or cargo culting) but it seems like there
| is something that that isn't in a bunch of if/else statements
| or pattern matching.
| FirmwareBurner wrote:
| If this is remotely like my experience with ChatGPT, doesn't
| that run the risk of hallucinating? Versus pattern matching
| just telling you "This X-ray shows a 75% chance this person
| has condition X".
|
| Can you at least interact with it and have it explain to you
| its reasoning like "so my diagnostic is based on seeing a
| lump in the lower right hand corner"? That would be an
| innovation indeed.
| therobot24 wrote:
| > AI (LLMs etc) exhibits "emergent behavior" properties.
| things like reasoning and other higher order effects.
|
| I would disagree with this statement. The model is
| _reasoning_ via the method of processing the data since the
| weights are informed from the relationships of prior similar
| information, but at its heart the model is still just doing
| pattern matching. The primary distinction and what I disagree
| with in your statement is that these AI models are closer to
| AGI because there aren't a bunch of if/else statements or
| basic pattern matching.
| lokar wrote:
| What is AI, or what counts as AI, has always had a large
| element of "things seem hard must be AI"
|
| For years most people agreed that a computer beating the best
| human at chess would be "real" AI. Then they managed it, but
| people said "not that does not count, you just brute forced
| it".
|
| What you describe, the automatic construction of classification
| models was textbook AI for over 20 years (using a whole range
| of methods), now it's often just seen as "data science".
| itsoktocry wrote:
| > _What am I missing here?_
|
| Perhaps _how_ the pattern matching is performed?
|
| Whether this is "AI" or not, I don't know. But dismissing it as
| mere "pattern-matching" grossly simplifies the technology and
| it's potential usefulness.
|
| > _I remember colleagues in university were working on over 10
| years ago_
|
| Yes, this is how we get better technology, by continuing to
| work on problems.
| delecti wrote:
| The history of AI research has been one of moving goalposts.
| Technologies only stay "AI" until they're shown to be useful.
| At various points in the past, computing in general, self-
| modifying code, search algorithms, computer vision, chat bots,
| neural networks, stable diffusion, large language models, and
| more, have all been considered "AI" at various points in their
| development. Occasionally, people will demonstrate that a topic
| in "AI" has a broader usefulness, and the specific name will be
| used, and people will argue online whether it's " _really_ AI
| ".
|
| About 15 years ago I learned about several of those topics in
| an AI course in college, and now they're all frequent
| occurrences in debates about whether they're _really_ AI.
| alach11 wrote:
| While there's prior associations between retinal photos and
| autism, I'm by default very skeptical of any AI algorithm
| purporting "100% accuracy". It smells like data leakage.
|
| I would bet that even physicians aren't 100% consistent in their
| diagnosis of autism. If that's the case, then it should be more
| or less impossible for any other diagnostic approach to be 100%
| consistent with the physician diagnoses.
|
| Edit: After reading the study closer, this criticism might be a
| bit harsh. In their autism subjects, they excluded those with
| mild/moderate autism. Limiting to severe cases should mean
| there's a higher degree of confidence/consistency in the
| diagnoses.
| karaterobot wrote:
| I'm not actually sure where the Petapixel article authors are
| getting the phrase "100% accuracy", as it does not show up in
| the article they are writing about, nor does it appear in the
| only other article they link to. Putting it in quotes makes it
| seem like a claim the model creators are making about their
| model, but I don't see them making that claim in general. They
| say the model matched all the sample data in this case, not
| that it's 100% accurate--presumably for the same reasons you
| are hesitant to do so. Unless I'm missing something, Petapixel
| should correct their headline.
| reqo wrote:
| You can see the results if you download the pdf from [0].
|
| [0] https://jamanetwork.com/journals/jamanetworkopen/fullarti
| cle...
| Eisenstein wrote:
| > I would bet that even physicians aren't 100% consistent in
| their diagnosis of autism.
|
| That's because autism is diagnosed by using the DSM. You can
| take an x-ray of an arm and see the fracture, but in order to
| diagnose autism you have to determine 'persistent deficits in
| social communication and social interaction across multiple
| contexts'.
|
| It is all dependent on how society defines things, and is fluid
| (and IMO, somewhat dubious).
| tastyfreeze wrote:
| The whole DSM is dubious. Nothing more than a made up tool to
| pathologize normality and sell more drugs.
|
| https://youtu.be/6JPgpasgueQ?si=dn3muYeOe-cSSKM2
| bitshiftfaced wrote:
| I couldn't find any confirmation that they tested on out of
| sample ASD populations.
| hervature wrote:
| From TFA:
|
| "" The data sets were randomly divided into training (85%) and
| test (15%) sets. We used 10-fold cross-validation to obtain
| generalized results of model performance. Data splitting was
| performed at the participant level and stratified based on the
| outcome variables. Because the data classes were imbalanced for
| symptom severity (ADOS-2 and SRS-2), we performed a random
| undersampling of the data at the participant level before
| conducting data splitting. """
|
| 100% indicates a major case of data leakage. Particularly,
| "When we generated the ASD screening models, we cropped 10% of
| the image top and bottom before resizing because most images
| from participants with TD had noninformative artifacts (eg,
| panels for age, sex, and examination date) in 10% of the top
| and bottom." tells me that there are known issues with the
| photos. Quite possible the photos of the groups were taken on
| distinct days and/or times and the lighting conditions is
| enough to distinguish the groups. Possibly, the background for
| the ASD candidates have a different background or a different
| camera sensor.
| DrStartup wrote:
| FTP: Findings - In this diagnostic study of 1890 eyes of 958
| participants, deep learning models had a mean area under the
| receiver operating characteristic curve of 1.00 for ASD screening
| and 0.74 for symptom severity. The optic disc area was also
| important in screening for ASD.
|
| https://jamanetwork.com/journals/jamanetworkopen/fullarticle...
| n-e-w wrote:
| I try not to immediately call BS on these types of studies...but
| in this case there are some concerns.
|
| "The data sets were randomly divided into training (85%) and test
| (15%) sets. We used 10-fold cross-validation to obtain
| generalized results of model performance. Data splitting was
| performed at the participant level and stratified based on the
| outcome variables. Because the data classes were imbalanced for
| symptom severity (ADOS-2 and SRS-2), we performed a random
| undersampling of the data at the participant level before
| conducting data splitting. Moreover, we examined different split
| ratios (80:20 and 90:10) to assess the robustness and consistency
| of the predictive performances across diverse splitting
| proportions."
|
| * undersampling is problematic here and probably introduced some
| bias. These imbalanced class problems are just plain hard.
| Claiming one hundred percent on an imbalanced class problem
| should probably cause some concern. * data split at the
| participant level has to be done really careful or you'll over
| fit * multiple comparisons bias by testing multiple split ratios
| on the same test data. Same with the 10-fold cross Val. * not
| sure if they validated results on any external test data *
| outcome variable stratification also has to be done really
| carefully or it will introduce bias; seems particularly sensitive
| in this case * using severity of symptoms as class labels is
| problematic. These have to really have been diagnosed the same
| way / consistently to be meaningful.
|
| I also note a long time history in collection of these images (15
| years iirc). Hard to believe such a diverse set of images
| (collection, equipment etc) led to perfect results.
|
| ML issues aside, super interested in the basic medical concept. I
| wasn't aware retinal abnormalities could be indicative of issues
| like ASD.
| Closi wrote:
| Another potential issue:
|
| > The photography sessions for patients with ASD took place in
| a space dedicated to their needs, distinct from a general
| ophthalmology examination room. This space was designed to be
| warm and welcoming, thus creating a familiar environment for
| patients. Retinal photographs of typically developing (TD)
| individuals were obtained in a general ophthalmology
| examination room. Each eye required an average of 10-30 s for
| photography, although some cases involved longer periods to
| help the patient calm down, sometimes exceeding 5-10 min. All
| images were captured in a dark room to optimize their quality.
| Retinal photographs of both patients with ASD and TD were
| obtained using non-mydriatic fundus cameras, including EIDON
| (iCare), Nonmyd 7 (Kowa), TRC-NW8 (Topcon), and Visucam NM/FA
| (Carl Zeiss Meditec).
|
| So two questions:
|
| 1. Are we positive that the difference in rooms does not effect
| these images?
|
| 2. If we are in a dark room, and ASD patients are in it for
| 5-10 minutes longer, are we sure this doesn't effect the
| retina?
|
| 3. Were all cameras used for both ASD and TD images?
|
| Want to make sure the AI is being trained to detect autism, and
| wasn't accidentally trained to identify camera models, length-
| in-dark-room or room-welcomingness.
|
| Hopefully not, but I assume you have to be _so_ careful with
| these sort of things when the model is entirely black-box and
| you can 't actually validate what it's actually doing inside.
| dataangel wrote:
| Came here to say this. 100% is too good to be true and it's
| almost certainly the AI has figured out a signal leak from
| the camera, image format, room, etc.
| Closi wrote:
| Yes! I would also be surprised if the ground truth didn't
| have some errors in it.
|
| If a model was 100% accurate, considering the
| nature/accuracy of manually diagnosing autism you would
| probably expect the AI to either find new cases or identify
| a few incorrect diagnosises.
| shubb wrote:
| It appears they also report good results for predicting
| symptom severity. It's less obvious how the cameras etc would
| leak into severity. Unless it actually works (it does seem a
| bit too good to be true), I'm thinking the test set was in
| the base model or something
| Closi wrote:
| Unsure, but there are lots of variables there and there
| could be even more we don't know about not mentioned in the
| paragraph! Maybe more severe cases involved longer periods
| to help the patient calm down in the dark environment? I
| dunno! Just something smells fishy. You are right, could
| have also been training data leaking, just looks like there
| are multiple leaky elements here potentially!
|
| Also, the study checked ASD participants were autistic by
| using structured interviews with psychologists against the
| DSM-5, but the TD participants were never assessed by
| psychologists, so if autism under-diagnosis is a thing,
| there could theoretically be false-negatives.
| jiggawatts wrote:
| If they consistently captured the images in different
| settings, then I guarantee you that that's what the AI
| learned.
|
| Just being in a dark room longer is sufficient to make
| changes that an AI could pick up on.
| ajb wrote:
| Darn, was excited for a minute. This sort of experiment needs
| double blinding.
|
| Ideally, they should capture the images from children before
| diagnosis, then see if they can predict the diagnosis.
| bookofjoe wrote:
| Also concussions according to the article, which is news to
| this retired former neurosurgical anesthesiologist.(38 years in
| practice; stopped 2015 at age 67 because I believed [still do])
| it's better to retire [from my profession, at least] too early
| than too late.
| ortusdux wrote:
| Similarly - https://ascopost.com/news/october-2023/ai-software-
| demonstra...
|
| In addition to demonstrating 100% (n = 59/59) sensitivity for
| detecting melanoma, the AI software had a detection rate of 99.5%
| (n = 189/190) for all skin cancers and 92.5% (n = 541/585) for
| precancerous lesions.
| eddiewithzato wrote:
| Next we need to train them for the pregnancy stage. We already
| screen for autism and Down syndrome, with AI it will be even
| more accurate.
| syspec wrote:
| There's a screening for autism during pregnancy?
| therobot24 wrote:
| Anything with 100% accuracy is suspect, either via the model, the
| dataset, or the means of measuring
| etothepii wrote:
| There's a famous story (probably apocryphal) about the military
| of a powerful nation training an early AI to find pictures of
| submarines beneath the sea.
|
| There was great excitement as it was near 100%.
|
| It later transpired the pictures with submarines in had a white
| border.
| morkalork wrote:
| I've heard this one before but with soviet tanks back in the
| 2000s!
| erehweb wrote:
| May be an urban legend, according to this site:
| https://gwern.net/tank
| sva_ wrote:
| I think there was something about an ANN detecting skin
| melanoma, assigning a high likelihood if there is a
| clinic/doctors office in the background.
| sterlind wrote:
| haven't modern model architectures gotten better at avoiding
| this kind of overfitting? like obviously data quality is still
| very important, but my understanding is that dropout mitigates
| this by randomly cutting out these unwanted feature channels.
| the models learn to distinguish all differences, rather than
| just one, or fixed combinations of several.
| dragonwriter wrote:
| > haven't modern model architectures gotten better at
| avoiding this kind of overfitting?
|
| Overfitting is, AIUI, a training method and data issue, not a
| model issue alone. I doubt any model is resistant to
| overfitting if you give it data where the answer is reliably
| encoded some aspect it can use but outside of what you want
| it to look at.
|
| Now, you can notice suspicious results and investigate (or
| you can just publish a 100% success rate and call it a day.)
| daveguy wrote:
| It really doesn't have to do with most ML architectures. It
| has to do with experiment design. If some data used in
| testing is part of the training process there will be over
| fitting. That's why a final test set is required for unbiased
| evaluation.
| RecycledEle wrote:
| I do not believe they achieved 100% accuracy because the original
| diagnoses could not have been 100% accurate.
| hereme888 wrote:
| Good point. I wonder how severe the autism was to make sure the
| reference subjects were 'certainly' within the ASD.
| wrsh07 wrote:
| This reminds me of the ml model that realized if an x-ray was
| taken at the center for cancer research it was likely to be an
| image of a cancer patient
| bikerideronmoon wrote:
| 100% accuracy? I strongly doubt it. I would bet anything over 10%
| and they have got a billion dollar corporation.
| kossTKR wrote:
| I wonder if physiognomy will come back as a field, if AI scans
| like these have any validity.
|
| I remember stumbling upon multiple esoteric accounts on both
| Twitter and Tiktok with communities seemingly obsessed with
| characterising various psychological traits from purely looking
| at facial features, importantly without racial undertones.
|
| While this on the surface sounds ridiculous and has various
| horrible historical echoes, i've always had a hunch there was
| actually something to this science from a purely intuitive
| perspective and knowing lots of people - again very importantly
| disregarding anything about race - instead focusing on the myriad
| of hormone linked features, neurotypicalism, alcohol,
| environmental factors, whatever traits that seemingly somehow go
| "across races".
|
| Or maybe there's nothing there.
| hereme888 wrote:
| Does anyone know if the code can be found anywhere? I'd like to
| check it out.
| theferalrobot wrote:
| Am I reading this wrong? They only had validation sets with no
| final test set which makes the results kinda worthless because we
| don't know how overfit they were to these validation sets (which
| can easily happen with any sort of parameter tuning). There is a
| reason why a proper study needs three splits train/validation
| (and possible multiple of these if you use k-fold) and a final to
| be used as sparingly as possible 'test' set.
|
| See the paper, "On estimating model accuracy with repeated cross-
| validation"
|
| Even the sklearn docs for cross validation show this split:
| https://scikit-learn.org/stable/_images/grid_search_cross_va...
| robocat wrote:
| I wonder if autism is a 100% binary diagnosis, versus a partial
| measurement. It will be interesting to see how many adults might
| get diagnosed - especially women since they have a lower rate of
| autism. Autism is 4 times more common in boys
| than girls, and women are diagnosed with autism later in life and
| less frequently than men
| samstave wrote:
| So I had seen this study that showed that autistic kids really
| like to spin objects... I didn't think anything of it aside from
| storing it.
|
| A few years later I met my wife's nephew, who was about 5 or so -
| and he was constantly spinning things... and I asked if he was
| autistic, and they said no.
|
| A few month passed by, and he was officially diagnosed as
| autistic.
|
| I wonder if something in the retina, and the way that it signals
| to the brain is soothing if there is a spinning image signal
| coming through.
|
| I wonder if one were able to apply a HUD either in a contact or
| such, where there might be a slightly spinning halo/ring that one
| might look through, if it is that spinning things sooth an
| autistic signal processor. There are two different spin stimuli
| it appears that autistic people find soothing, visual or physical
| (spinning of themselves).
| javajosh wrote:
| An interesting application for a virtual reality headset. You
| could at least do anecdotal testing with your friends who have
| autistic children.
___________________________________________________________________
(page generated 2023-12-22 23:02 UTC)