[HN Gopher] Explaining machine learning pitfalls to managers (2019)
       ___________________________________________________________________
        
       Explaining machine learning pitfalls to managers (2019)
        
       Author : yamrzou
       Score  : 35 points
       Date   : 2022-10-28 22:26 UTC (3 days ago)
        
 (HTM) web link (gudok.xyz)
 (TXT) w3m dump (gudok.xyz)
        
       | mistrial9 wrote:
       | the communication skills here are good IMHO -- essay-style
       | writing with digestable parts, a decent graphic, basic page
       | layout. I don't disagree with the content -- exactly.. from
       | practical experience I believe that this is an
       | oversimplification, but a useful one. The practical tips to Watch
       | Out For are worth the time to read this.
        
       | angarg12 wrote:
       | I have been interviewing ML Engineers for a while now and I've
       | seen companies fall into the same pitfall over and over. I
       | informally call this the "low hanging fruit" or the "illusion of
       | competence" pitfall.
       | 
       | It goes something like this: someone (a PM, manager, or sometimes
       | an engineer) has a brilliant idea to use ML to enhance some part
       | of the business, say automate a manual process. A random folk is
       | asked to do a PoC, and they slap together a model in a couple of
       | days.
       | 
       | This model often shows impressive performance, say 80% accuracy
       | in a problem where 90% is considered acceptable. Leadership gets
       | all excited and they sign off the project. And they get
       | themselves into a world of pain.
       | 
       | The pain takes many forms, but the most common ones are:
       | 
       | a) That extra 10% accuracy is extremely hard to achieve.
       | 
       | b) Running ML in production is really difficult and the company
       | doesn't have the skills/maturity/expertise to run this new,
       | possibly mission critical components.
       | 
       | The pitfall that lead to this situation are:
       | 
       | 1) Assuming ML is easy from a PoC.
       | 
       | If we built a PoC that achieves 80% accuracy in 2 days, how hard
       | can it be to achieve 90%? it turns out it can be really
       | difficult. Performance improvement of ML models is not linear and
       | it can be really difficult to get even a few % points better.
       | 
       | The second part is running ML in production. It might be obvious
       | to an engineer that there is a big difference between slapping
       | together a prototype in a hurry and running a mission critical
       | service in production, but people unfamiliar with ML tend to
       | assume that the process of building the model is all there is to
       | it.
       | 
       | 2) Assuming ML is not all or nothing (for your particular
       | problem).
       | 
       | One might tend to think that a model with 80% accuracy is just a
       | little bit worse than one with 90% accuracy. However, depending
       | on the domain, this isn't true. For some problems models need to
       | perform better than a certain threshold to be of any use. In that
       | case, a model with 80% accuracy is as good as one with 0%.
       | 
       | This is an oversimplified explanation but I've seen it happen
       | often enough that I consider it an (anti) pattern.
        
       ___________________________________________________________________
       (page generated 2022-10-31 23:00 UTC)