[HN Gopher] The Neural Net Tank Urban Legend
       ___________________________________________________________________
        
       The Neural Net Tank Urban Legend
        
       Author : belter
       Score  : 20 points
       Date   : 2023-06-21 10:51 UTC (12 hours ago)
        
 (HTM) web link (gwern.net)
 (TXT) w3m dump (gwern.net)
        
       | morkalork wrote:
       | I vividly remember hearing that story from a friend when I was in
       | highschool around the mid 2000s.
        
       | [deleted]
        
       | riskneutral wrote:
       | Oh wow, I read this story as evidence that neural nets are not
       | the future back when I was an undergrad studying machine
       | learning. Maybe I should have followed my instincts back then
       | because even then neural nets seemed intuitively very interesting
       | even if the statistics and math professors hated them because
       | they weren't derived from any first principals.
        
       | teraflop wrote:
       | A fictional variant of this story also shows up in Peter Watts'
       | novel _Starfish_.
       | 
       | In that version, the neural net is trained to manage the
       | ventilation system of an underground train station, based on data
       | about passenger movements. Unfortunately, it ends up simply
       | paying attention to the display of a clock that happens to be
       | visible through one of its cameras. And when the clock breaks, a
       | bunch of people asphyxiate. A bit implausible, but memorable.
        
       | jumploops wrote:
       | My friend once asked me to help him with an autonomous RC car
       | race, where we'd go to a warehouse and train the car around the
       | track.
       | 
       | After a few weekends of iterating and improving the
       | model/training set, we were convinced we'd win the next race...
       | only to lose almost immediately by crossing the lines on the
       | track.
       | 
       | We did some inverse model explanation work which quickly showed
       | that our car was paying attention to the overhead skylights more
       | than the actual tracks. Unlike the other weekends, it was a foggy
       | day!
       | 
       | A quick hack to cut out the top 50% of each training image
       | brought our car back to its prior reliability.
        
       | api wrote:
       | While this may be an urban legend, I have definitely played with
       | ML and seen things overfit in comical ways.
        
         | morkalork wrote:
         | It reminds me of a similar story about using deep learning on
         | medical imagery for diagnosis. In one study, it supposedly
         | learned the to tell which device was used to take the images.
         | Apparently that was somewhat correlated with the results as
         | more serious / heavily suspected cases were sent to a specific
         | hospital in the region.
        
       ___________________________________________________________________
       (page generated 2023-06-21 23:00 UTC)