https://sloanreview.mit.edu/article/when-algorithms-rule-values-can-wither/ MIT Sloan Management Review Logo Menu Close Menu and Search Search [ ] Topics < Back to Menu * Data, AI, & Machine Learning * Innovation * Leadership * Managing Technology * Marketing * Operations * Social Responsibility * Strategy * Workplace, Teams, & Culture * All Topics * Trending * Culture * Diversity & Inclusion * Corporate Social Responsibility Our Research < Back to Menu * Big ideas Research Projects * Artificial Intelligence and Business Strategy * Future of the Workforce * Responsible AI * Future of Leadership * All Research Projects Spotlight < Back to Menu * Most Popular * Leading With Impact * Measuring Culture * AI in Action * MIT SMR Strategy Forum Magazine < Back to Menu Winter 2023 Issue Winter 2023 Issue This issue offers insight on organizing project teams, fostering collaboration in hybrid work arrangements, and supporting employees through periods of high uncertainty. * Past Issues Webinars & Podcasts < Back to Menu * Upcoming Events * Video Archive * Podcasts * Me, Myself, and AI * Three Big Points Subscribe Now and Save 22% on Unlimited Access. [WI23_Cover] Subscribe Topics * Data, AI, & Machine Learning * Innovation * Leadership * Managing Technology * Marketing * Operations * Social Responsibility * Strategy * Workplace, Teams, & Culture * All Topics * Trending * Culture * Diversity & Inclusion * Corporate Social Responsibility Our Research * Big ideas Research Projects * Artificial Intelligence and Business Strategy * Future of the Workforce * Responsible AI * Future of Leadership * All Research Projects Spotlight * Most Popular * Leading With Impact * Measuring Culture * AI in Action * MIT SMR Strategy Forum Magazine Winter 2023 Issue Winter 2023 Issue This issue offers insight on organizing project teams, fostering collaboration in hybrid work arrangements, and supporting employees through periods of high uncertainty. * Past Issues Webinars & Podcasts * Upcoming Events * Video Archive * Podcasts * Me, Myself, and AI * Three Big Points Search Store Sign In Subscribe MIT Sloan Management Review Logo Magazine Winter 2023 Issue When Algorithms Rule, Values Can Wither Building responsible AI systems starts with recognizing that technology solutions implicitly prioritize efficiency. Dirk Lindebaum, Vern Glaser, Christine Moser, and Mehreen Ashraf December 05, 2022 Reading Time: 12 min Topics * Data, AI, & Machine Learning * Ethics * AI & Machine Learning Subscribe Permissions and PDF Share Twitter Facebook Linkedin [WINTER22-L] Aad Goudappel/theispot.com Interest in the possibilities afforded by algorithms and big data continues to blossom as early adopters gain benefits from AI systems that automate decisions as varied as making customer recommendations, screening job applicants, detecting fraud, and optimizing logistical routes.1 But when AI applications fail, they can do so quite spectacularly.2 Consider the recent example of Australia's "robodebt" scandal.3 In 2015, the Australian government established its Income Compliance Program, with the goal of clawing back unemployment and disability benefits that had been made inappropriately to recipients. It set out to identify overpayments by analyzing discrepancies between the annual income that individuals reported and the income assessed by the Australian Tax Office. Previously, the department had used a data-matching technique to identify discrepancies, which government employees subsequently investigated to determine whether the individuals had in fact received benefits to which they were not entitled. Aiming to scale this process to increase reimbursements and cut costs, the government developed a new, automated system that presumed that every discrepancy reflected an overpayment. A notification letter demanding repayment was issued in every case, and the burden of proof was on any individuals who wished to appeal. If someone did not respond to the letter, their case was automatically forwarded to an external debt collector. By 2019, the program was estimated to have identified over 734,000 overpayments worth a total of 2 billion Australian dollars ($1.3 billion U.S.).4 Get Updates on Leading With AI and Data Get monthly insights on how artificial intelligence impacts your organization and what it means for your company and customers. [ ] sign up Please enter a valid email address Thank you for signing up Privacy Policy The new system was designed to optimize efficiency, but without being attentive to the particulars of individual cases. The idea was that by eliminating human judgment, which is shaped by biases and personal values, the automated program would make better, fairer, and more rational decisions at much lower cost. Unfortunately, choices made by system designers both in how the algorithm was designed and how the process worked resulted in the government demanding repayments from hundreds of thousands of people who had been entitled to the benefits they had received. Some were compelled to prove that they had not illegitimately claimed benefits as long ago as seven years earlier. The consequences for many individuals were dire. Subsequent parliamentary reviews pointed to "a fundamental lack of procedural fairness" and called the program "incredibly disempowering to those people who had been affected, causing significant emotional trauma, stress, and shame. Topics * Data, AI, & Machine Learning * Ethics * AI & Machine Learning About the Authors Dirk Lindebaum is a senior professor in organization and management at Grenoble Ecole de Management. Vern Glaser is associate professor of entrepreneurship and family enterprise and the Eric Geddes Professor of Business in the Strategy, Entrepreneurship, and Management Department at the University of Alberta. Christine Moser is associate professor of organization theory at the Vrije Universiteit Amsterdam. Mehreen Ashraf is a doctoral student at Cardiff Business School. References 1. T.H. Davenport and R. Bean, "Becoming an 'AI Powerhouse' Means Going All In," MIT Sloan Management Review, June 15, 2022, https:// sloanreview.mit.edu. 2. C. O'Neil, "Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy" (New York: Crown Publishers, 2016). 3. "Accountability and Justice: Why We Need a Royal Commission Into Robodebt," PDF file (Canberra, Australia: Senate Community Affairs Reference Committee, May 2022), https://parlinfo.aph.gov.au. 4. "Centrelink's Compliance Program: Second Interim Report," PDF file (Canberra, Australia: Senate Community Affairs Reference Committee, September 2020), chap. 1, https://parlinfo.aph.gov.au. 5. "Centrelink's Compliance Program," chap. 2. 6. Ibid. 7. D. Lindebaum, C. Moser, M. Ashraf, et al., "Reading 'The Technological Society' to Understand the Mechanization of Values and Its Ontological Consequences," Academy of Management Review, July 2022, https://journals.aom.org. 8. O'Neil, "Weapons of Math Destruction." 9. M. Rokeach, "The Role of Values in Public Opinion Research," Public Opinion Quarterly 32, no. 4 (winter 1968-1969): 550. 10. O'Neil, "Weapons of Math Destruction." Tags: Algorithms Artificial Intelligence Ethics Reprint #: 64223 More Like This Add a comment Cancel reply You must sign in to post a comment. First time here? Sign up for a free account: Comment on articles and get access to many more articles. MIT Sloan Management Review Logo Copyright (c) Massachusetts Institute of Technology, 1977-2022. All rights reserved. * Home * Subscriber Services * About Us * Newsletters * Careers * Advertise With Us * Contact Us * Masthead * Help * Author Guidelines * MIT SMR Connections Get free, timely updates from MIT SMR with new ideas, research, frameworks, and more. [ ] sign up Please enter a valid email address Thank you for signing up Privacy Policy Follow Us * Facebook * Twitter * Linkedin * Youtube * Instagram Login Create an Account Business Access