[HN Gopher] Machine-learning improves the prediction of stroke r...
       ___________________________________________________________________
        
       Machine-learning improves the prediction of stroke recovery
        
       Author : rustoo
       Score  : 16 points
       Date   : 2021-07-11 15:05 UTC (7 hours ago)
        
 (HTM) web link (actu.epfl.ch)
 (TXT) w3m dump (actu.epfl.ch)
        
       | light_hue_1 wrote:
       | Papers like this need machine learning reviewers and even better,
       | coauthors. I cannot believe that anyone with competence in ML
       | reviewed this paper.
       | 
       | For one thing, the figure that the linked article presents is
       | very misleading. It is not the performance of the model! It's
       | just the raw data. Patients by how well we think they might
       | recover vs. how well they discovered. The goal of the model is to
       | figure out how will recover well. Literally, nothing in that
       | figure is an output of the model.
       | 
       | > In case of stroke lesions affecting the performance of
       | parcellation, the correspondence part of the lesion on the
       | unaffected hemisphere was used and interpolated over the lesioned
       | brain as a brain transplant and fed into the above-described
       | structural imaging processing.
       | 
       | The classifier will trivially discover that you are doing this,
       | and it's almost certainly correlated with terrible outcomes. (PS:
       | should be "corresponding" part) This alone calls into question
       | all of the results and never should have gotten past anyone. Just
       | because you can't tell as a human, doesn't mean the data
       | manipulation isn't immediately obvious to even a simple model
       | like an SVM.
       | 
       | > By means of k-means clustering, patients were identified as
       | fitters, i.e. revealing a proportional recovery, and non-fitters,
       | i.e. patients lacking proportional recovery.
       | 
       | Good god, why? Just set up a simple criteria and use it. This
       | kind of strange arbitrary boundary in your data that comes from
       | k-means can only hurt your model.
       | 
       | Finally: The results are terrible. When cross-validating the
       | model does well, 83% accuracy (87% precision). But on the
       | external data the model is worthless. 60% accuracy, 53%
       | precision.
       | 
       | They provide no trivial baselines. They don't even list of the
       | base rates for their dataset (I'm guessing that out of their main
       | dataset the classifier is just operating at chance at 60%
       | accuracy / 53% precision is just the base rate).
       | 
       | Their data seems to be horrific. Half the patients in the 2nd
       | dataset don't have data for both time points. How can they be
       | assigned to groups that fit or don't fit without this basic data?
       | They don't even describe results on their 3rd dataset.
       | 
       | They don't bother to compute a confidence interval, which will be
       | huge because they have so little data.
       | 
       | Seriously. Authors. If you are reading this, reach out to a
       | machine learning person. If you don't know anyone, reach out to
       | me, I'm a scientist at a major US university. Papers like this
       | just pollute our science and are a total waste of time on your
       | part.
        
         | rscho wrote:
         | You're just not used to clinical research. A large part of the
         | field output is no better than this. And this is only what the
         | authors think will look good on paper, which is often quite far
         | from what really happened during data collection and analysis.
         | I should know, I'm a clinical researcher at a major european
         | uni ;-) And due to peer pressure, my output is also no
         | better... :-(
        
           | AlecSchueler wrote:
           | Would be curious to hear more about the systemic issues that
           | lead to you knowingly outputting poor research. Could you
           | perhaps blog about it anonymously?
        
         | riahi wrote:
         | Agreed. I try to accept all ML focused paper review requests in
         | my disciplines, but modern academics do not reward you for the
         | time you take as a reviewer, which then distracts me from
         | working on my own projects.
         | 
         | It's very much an uncompensated service to the community...
        
       | rob_c wrote:
       | Hope this doesn't mean that we hold back some treatment for those
       | who would have bucked the trend and recovered better than
       | expected... Very interesting if this can be used to tailor
       | recovery and treatments in an adaptive way. :)
        
         | karmicthreat wrote:
         | Yea, this seems more like a tool insurance companies will use
         | to deny coverage.
        
       | deeviant wrote:
       | What is the value of such a prediction?
        
       ___________________________________________________________________
       (page generated 2021-07-11 23:01 UTC)