https://www.abc.net.au/news/2022-12-12/robodebt-algorithms-black-box-explainer/101215902 Skip to main content ABC News Homepage Search Loading More from ABC More from ABC Close menuABC * ABC Home * News * Local & Radio * iview * Everyday * More Editorial Policies Read our editorial guiding principles * Accessibility * Help * Contact Us * About the ABC * Privacy Policy * Terms of Use * (c) 2022 ABC * * * * * Just In * Watch Live * Coronavirus * Politics * World * Business * Analysis * Sport * Science * Health * Arts * Fact Check * Other News HomeABC News Homepage How to wrench open the black box of algorithms that decide our fate Share News Ticker Flood warning For the latest flood and weather warnings, search on ABC Emergency Live blog Catch up with The Loop, your quick wrap of this morning's news Live blog Keep up with the latest ASX and business news 1 / of 3 Get breaking news alerts directly to your phone with our app How to wrench open the black box of algorithms that decide our fate Story Lab / By Julian Fell, Ben Spraggon, and Matt Liddy Posted 4h ago4 hours agoSun 11 Dec 2022 at 6:37pm, updated 37m ago37 minutes agoSun 11 Dec 2022 at 10:25pm Share this article abc.net.au/news/robodebt-algorithms-black-box-explainer/101215902 Copy linkShare There is a decision being made about you in this box. A 3D black box on a purple background But you're not allowed to look inside -- and even if you could, it wouldn't tell you much. There are countless black boxes, just like this one, making decisions about your online and offline lives. Some of them are pretty benign, like recommending what movie you should watch next. But others decide the news you see, who you go on a date with and how much money you can borrow. A number of 3D black boxes with well-known corporate logos on a purple background They also determine whether you get a job, if you're a potential shoplifter and what route you should take to the shops. In extreme cases, they have been used to 'predict' teenage pregnancies and cut welfare entitlements for people with disabilities . These boxes all contain algorithms, the decision-making machines that are creeping into areas of increasing consequence to all of our lives. It can feel pretty impossible to understand exactly what these algorithms are doing, let alone keep them in check. However, as you'll see, there are creative and powerful ways to shine a light into these black boxes. The trouble is, we can only use them if their owners -- mostly corporations and governments -- will let us. A 3D black box with light shining into it's open lid on a purple background Centrelink's Robodebt algorithm was locked away inside one of these black boxes. Hidden from public view, it went about its work, sending out hundreds of thousands of miscalculated debt notices -- all based on a flawed assumption. The algorithm divided the full year's worth of welfare recipients' income evenly between the 26 fortnights in the year, rather than considering each fortnight individually. A mundane miscalculation that, when reproduced faithfully and behind closed doors, had consequences that were anything but. "I was literally crushed. I was in shock," reads one submission to a Senate inquiry. It tells the story of just one of the approximately 433,000 Australians who were subjected to debts created by the algorithm. "I walked around my house trying to deny the reality of what had happened ... I was confused as to how I owed this amount of money. Within weeks, I began receiving calls, texts and letters from a debt-collection agency." Following a successful legal challenge and extensive media coverage, the government has launched a royal commission to investigate the scheme's failings. However, even as one algorithm is brought into the light, others continue operating behind closed doors. The Department of Home Affairs has used algorithms to assist visa processing for more than 20 years. And, with increasing demand for visas since our borders re-opened, this is set to expand further. A departmental spokesperson confirmed that Home Affairs was considering "a range of emerging technologies" as part of a "modernisation" strategy. Despite the lessons learned from the Robodebt crisis, these black boxes have, so far, remained firmly shut. The ABC asked Home Affairs questions about transparency, monitoring, testing and redress mechanisms regarding its algorithmic systems, but these were not addressed in their response. Visas in a box To get a glimpse of exactly how using algorithms to assist with visa decisions can go wrong, we only need to look at how the Home Office managed a similar scheme in the United Kingdom. The inner workings of the UK's visa-processing algorithm had been kept secret from the public for years, locked away inside a black box. But documents released under freedom of information laws, following a legal challenge, finally allowed us to peek inside. While they don't give the full picture, they reveal a few key things. The algorithm sorted visa applicants into three risk groups -- high, medium and low -- based on a number of factors that included their nationality. A single pipe splitting into three pipes, labelled in order as low, medium and high risk This categorisation had a big impact on the likelihood of an application being refused. At one processing centre, less than half of applications classed as high risk were approved for visas in 2017, compared to around 97 per cent for low-risk ones. Applicants in the high-risk category could be subjected to "counter-terrorism checks and DNA testing" by immigration officials before a decision was made on their visa. This intense scrutiny contributed to the high refusal rates. Meanwhile, those classed as low risk got by with routine document checks and, therefore, far-lower refusal rates. This alone wasn't particularly controversial -- after all, the UK Home Office was trying to make the best use of its limited resources. The trouble was that, just like Robodebt, their algorithm had an insidious flaw. Over time, it unfairly amplified biases in the data. To see how, let's imagine that 200 people, split evenly between two made-up countries -- Red and Blue -- apply for visas in 2015. We're going to simulate how things will play out for applicants from these two nations, incorporating details from how the UK Home Office's algorithm worked. Red and blue particles falling from a single pipe into three pipes, labelled as 2015 As the applications from the two nations stream down into the categories, some visas are approved and some are refused. The rates at which they are refused is dependent on which category they fall into. To decide how to categorise each application, the algorithm used by the UK Home Office relied on historical "immigration breaches" data. We've given the Red group a slightly higher rate of historical breaches to simulate the fortunes of two similar -- but not identical -- nations. The refusal rates for our two nations (shown at the bottom) reflect this difference in their historical records. Red and blue particles falling from a single pipe into three pipes, labelled as 2015. Percentages of refusals at the bottom. Okay, we're done for 2015. Of our 200 applications, the Red group had 11 more refusals than the Blue group. The results for 2015 are pretty close to the historical data that we made up, so it seems like our algorithm is doing its job so far. Refusal rates for red and blue nations. Historical -- blue 26%, red 33%. 2015 -- blue 23%, red 34% Now, we feed these results back into the algorithm. This is where things start to get ugly. The UK Home Office's algorithm counted merely having a visa refused as a "breach", which led to biases in the historical data being exacerbated. Fast forward to 2016 and another 200 people from the same two groups apply. Red and blue particles falling from a single pipe into three pipes, labelled as 2016 Based on the prior "breaches", our algorithm flags a higher proportion of people from the Red group as high risk than the previous year. And it's solely the algorithm that's increasing this disparity -- there's nothing different about the applicants themselves compared to the year before. The extra scrutiny placed on Red applications results in 18 more being refused than the Blue one this time around. Refusal rates for red and blue nations. 2015 -- blue 23%, red 34%. 2016 -- blue 21%, red 39%. Once again, we feed this new information back into our algorithm. It sees an even greater disparity in the risks in 2017. In the worldview of the algorithm, the evidence is now clear: People from the Red group are predominantly high risk while those from the Blue group are not. This time, the Red group sees more than twice the number of refusals compared to the Blue group. Refusal rates for red and blue nations. 2015 -- blue 23%, red 34%. 2016 -- blue 21%, red 39%. 2017 -- blue 20%, red 43%. That's a pretty big difference from where we started, isn't it? As the years rolled on, the data increasingly became a reflection of the algorithm's prejudices. The algorithm took the differences between nations in the historical data and blew them out of proportion -- regardless of whether they were accurate assessments of risk, or had been created by chance, error or discrimination. So, by 2017, its choices were more of a self-fulfilling prophecy than an accurate reflection of risk in the real world. Jack Maxwell -- lawyer and co-author of Experiments in Automating Immigration Systems -- found through his investigations that the UK Home Office's algorithm suffered from a feedback loop much like this one. And, according to Mr Maxwell, the historical immigration data was flawed too. By their nature, he said, immigration enforcement statistics were incomplete, and did not "reflect the actual incidence of immigration breaches, so much as the biases of the people reporting those breaches". Now, there's no indication that the Australian Department of Home Affairs is making, or about to make, the same mistakes as the UK Home Office as it expands and "modernises" its use of algorithms. However, as long as it keeps its algorithms locked away, we can't be sure. Fortunately, onerous legal challenges and FOI requests are not the only ways to peer inside. As we'll see, the tools that can open these black boxes come in a range of shapes and sizes. Some can help us understand -- and challenge -- the decisions they make about us, as individuals, while others can illuminate bias and discrimination embedded within a system. Thinking outside the box To explain the decisions made by algorithms in a way that humans can understand, leading artificial intelligence researcher Sandra Wachter and her colleagues at the Oxford Internet Institute turned, not to science, but to philosophy. They went back to basics and "thought outside the box" about what an explanation actually is, and what makes one useful. Portrait of a woman with a garden in the background Sandra Wachter has developed a novel technique for explaining decisions made by algorithms.(Supplied: Sandra Wachter) "Do people really want to understand the internal logic of an algorithm? Or do they mainly care about why they didn't get the thing that they applied for?" Professor Wachter ponders. The philosophy textbooks told Professor Wachter that it was more the latter. "They might want to contest the decision because the criteria that [the algorithm] relied upon is wrong. Or it might be the case that the decision was made correctly, and they want guidance on how to change their behaviour in the future," she says. Given these goals, simply looking inside these black boxes is not going to tell us what we want to know. This is because, in practice, most algorithms are a combination of complex variables. A transparent 3D box with a glowing network of nodes visible inside it Not even the experts can reliably interpret decisions made by sophisticated algorithms. So, rather than trying to explain the nitty gritty technical details of how they work, Professor Wachter and her team came up with a deceptively simple alternative. The idea was to describe "how the world would need to be different, for a different outcome to occur", she explains. That idea -- of imagining alternative worlds -- may sound like it belongs in a science fiction writers' room, but the way this potential tool for increasing algorithmic accountability works is really quite simple. Such a tool would generate a number of "nearby possible worlds" in which your application would have been successful -- and tell you how they differ from the real world. Three transparent 3D boxes. One is the real-world outcome which is refused. The other two are accepted alternatives This means you might be told, in plain English, that you'd have been successful had you applied for a different type of visa or requested a shorter stay. So, you wouldn't need to look inside the box at all to understand how it came to its decision and, therefore, how you could do better next time. By offering this kind of transparency without opening the black box, Professor Wachter adds, it will be "protecting the privacy of others and with very little risk of revealing trade secrets". While this "nearby possible worlds" approach is useful and important for understanding specific decisions about one individual, it's not really enough on its own to keep these black boxes in check. Explanations of individual cases alone will not let us identify systemic issues such as the ones seen in the UK. Thinking bigger An individual often will not know how others are faring when they interact with an algorithm, says Paul Henman, a professor of digital sociology and social policy at the University of Queensland. And even in a system that discriminates against others, many individuals will still receive acceptable or even favourable decisions. "Because individuals are experiencing these decisions in isolation, they might not see that an entire group is getting a different outcome." A portrait of a man in a suit jacket with a sandstone university in the background Paul Henman says we need to consider structural biases in algorithmic systems.(ABC News: Tim Leslie) While some algorithms -- such as the one used by the UK Home Office -- explicitly discriminate based on nationality or other attributes protected by law, discrimination is not always so black and white. Factors such as the applicant's immigration background, location and even their name are not protected attributes, but can correlate closely with race. As these structural biases cannot be seen at the level of individual decisions, we need to think bigger. This is where our second transparency tool -- the algorithmic audit -- comes in. An algorithmic audit involves putting the algorithm under the microscope to verify that it meets standards for fairness. A scorecard of visa refusals showing a D for nationality, an A for age and an A+ for gender. In the case of the UK Home Office, an expert could have checked that people of all nationalities saw comparable outcomes when controlling for other factors. The same goes for gender, age and other protected attributes. Results from algorithmic audits can be translated into scores and made public, similar to the health advice that is required on food packaging. When published, these results can help us to understand what's going on inside the box, without us all needing to be experts. These tools -- and others like them -- are not limited to academia anymore. They're being adopted in Australia and around the world. So, why are we not seeing greater transparency around the algorithms used by corporations and our governments? The right to reasons One reason, says former Australian human rights commissioner Ed Santow, is that Australia is lagging other parts of the world on digital rights protections. In 2021, Professor Santow and the Australian Human Rights Commission made a number of recommendations about how Australia can make automated decision-making "more accountable and protect against harm". "The cost of inaction [on digital rights] is waiting until the next crisis, the next Robodebt. At the Human Rights Commission, we were saying that we could avoid that next crisis by putting in place stronger fundamental protections," he said. A portrait of a man in a suit sitting in a modern building lobby Edward Santow advocates for stronger digital rights protections in Australia.(Supplied: Edward Santow) According to the commission's report, the foundational "right to reasons" would be a legislated right to request an explanation for administrative decisions, regardless of whether they were made by humans or machines. These protections can be the difference between a problematic algorithm caught early and another crisis identified too late. Both Robodebt and the UK Home Office algorithm flew under the radar for years before their flaws became apparent, in part due to the lack of transparency around how they operated. Centrelink sent out erroneous debt notices without equipping those recipients with the tools necessary to challenge or even understand those decisions. Instead, they needed courts and advocates to find justice. The story is similar in the UK. It took the efforts of Foxglove, a tech advocacy group, and the Joint Council for the Welfare of Immigrants to challenge the Home Office's algorithm in court. However, it doesn't have to be this way. Specialised tools like "nearby possible worlds" and algorithmic audits make these explanations more practical than ever to produce. And the European Union has been blazing a trail in digital rights protections, so there is plenty of precedent for our legislators to learn from. Having our fates decided by algorithmic black boxes can feel pretty dystopian. However, if we embrace these tools and legislate the necessary protections, we might at least live in a world where the algorithms have to work in the open. Credits * Reporter and developer: Julian Fell * Designer: Ben Spraggon * Editor: Matt Liddy Posted 4h ago4 hours agoSun 11 Dec 2022 at 6:37pm, updated 37m ago37 minutes agoSun 11 Dec 2022 at 10:25pm Share * Copy link * Facebook * Twitter * More on: * Australia * Computer Science * Mathematics * Science and Technology * Social Policy Top Stories * analysis An obscure decision by a little-known regulator might be about to ignite the current property downturn * Lockerbie bombing suspect in custody 34 years after deadliest terror attack in British history * The 'Hi Mum' scam is rife in Australia. This expert explains how it works and the signs to watch out for * PM confident government will pass energy relief bill this week * analysis Albanese taught world leaders the smart way to deal with China * There's one spot on Earth that could definitively mark the start of a new geological epoch. It may be in Queensland * Outback pub's emus turn up again after lifetime ban -- and they've brought reinforcements * Rental bidding to be banned in NSW by Christmas, state government says * With the suspected bombmaker in US custody, the 1988 Lockerbie terror attack returns to the public eye * ACT opposition calls for inquiry into allegations of 'political interference' in Bruce Lehrmann trial * live Griner wants 'to talk' after Russian prison release, and new photos from NASA's rocket return. Catch up with The Loop * Inside the world of dating agencies and matchmakers where someone paid $7,500 for one date in 18 months * 'The situation is not pretty': Fears WA power users are in for white knuckle ride this summer * analysis Australia's pace battery could be reshuffled at the Gabba -- these are the options * India ends Australian winning streak with stunning hitting in Super Over Popular Now 1. Perth parents face jail time after getting children hooked on pain medication they didn't need A child is given a saline solution in hospital 2. Coal and gas prices to spare households $230 on energy bill hikes, but pain still to come Anthony Albanese at a press conference at Parliament House 3. Teens charged after Sydney raids over alleged global scam that netted $148m Two young men with their faces blurred sit on a lounge while a police officer speaks to them 4. 'Give me a break': Former bureaucrat denies she tried to 'manipulate the truth' about Robodebt Catherine Halbert, ex-Department of Social Services bureaucrat, gives evidence at the royal commission into the robodebt scheme 5. Australia take control in Adelaide with sublime bowling under lights on day two Australian cricketers hug and pat a smiling bowler on the head as everyone celebrates a Test wicket. 6. Coronial inquest questions doctor after man dies following bowel surgery A woman walking along a pathway Top Stories analysis An obscure decision by a little-known regulator might be about to ignite the current property downturn A real estate sign in the front yard of a house in Sydney's south, with a sold sticker on it. Lockerbie bombing suspect in custody 34 years after deadliest terror attack in British history An aerial view of a Scottish street. Houses are in ruins and a crater runs through the middle of the street. The 'Hi Mum' scam is rife in Australia. This expert explains how it works and the signs to watch out for An illustration of a hooded figure at a laptop with whatsapp messages overlaid PM confident government will pass energy relief bill this week Anthony Albanese mid-sentence while gesturing with his hands, the background is out of focus analysis Albanese taught world leaders the smart way to deal with China Side by side photos of Canada PM Justin Trudeau, Chinese President Xi Jinping and Australian PM Anthony Albanese There's one spot on Earth that could definitively mark the start of a new geological epoch. It may be in Queensland Plaque on rock marks transition from one geological era to another Outback pub's emus turn up again after lifetime ban -- and they've brought reinforcements Two emus banned from the Yaraka Hotel in south-west Queensland. Rental bidding to be banned in NSW by Christmas, state government says A Open for Inspection sign With the suspected bombmaker in US custody, the 1988 Lockerbie terror attack returns to the public eye Rescue workers are standing around the wreckage of Pan Am flight 103. Just In * PM confident government will pass energy relief bill this week 14m ago14 minutes agoSun 11 Dec 2022 at 10:48pm * When you tick carbon neutral on your airline ticket, these rangers get to work 27m ago27 minutes agoSun 11 Dec 2022 at 10:34pm * Rental bidding to be banned in NSW by Christmas, state government says 31m ago31 minutes agoSun 11 Dec 2022 at 10:30pm * India ends Australian winning streak with stunning hitting in Super Over 59m ago59 minutes agoSun 11 Dec 2022 at 10:02pm * This community is worried about what comes out of its taps, but things are slowly changing 1h ago1 hours agoSun 11 Dec 2022 at 9:59pm * Live: ASX set for falls ahead of busy week for central banks 1h ago1 hours agoSun 11 Dec 2022 at 9:54pm More Just In Back to top Footer ABC News homepage More From ABC NEWS We acknowledge Aboriginal and Torres Strait Islander peoples as the First Australians and Traditional Custodians of the lands where we live, learn, and work. Sections * ABC NEWS * Just In * Watch Live * Coronavirus * Politics * World * Business * Analysis * Sport * Science * Health * Arts * Fact Check * Other News in language * Zhong Wen * Berita Bahasa Indonesia * Tok Pisin Connect with ABC News * Facebook * Twitter * Instagram * YouTube * Apple News More from ABC News * Contact ABC NEWS This service may include material from Agence France-Presse (AFP), APTN, Reuters, AAP, CNN and the BBC World Service which is copyright and cannot be reproduced. AEST = Australian Eastern Standard Time which is 10 hours ahead of GMT (Greenwich Mean Time) * Editorial Policies * Accessibility * Help * Contact Us * About the ABC * Privacy Policy * Terms of Use * (c) 2022 ABC * * * *