https://garymarcus.substack.com/p/covert-racism-in-llms Marcus on AI Subscribe Sign in Share this post [https] Covert racism in LLMs garymarcus.substack.com Copy link Facebook Email Note Other Covert racism in LLMs Shocking new paper with potentially serious implications [https] Gary Marcus Mar 5, 2024 61 Share this post [https] Covert racism in LLMs garymarcus.substack.com Copy link Facebook Email Note Other 37 Share From Allen AI and Stanford and others, an incredibly important new paper: [https] I shouldn't be surprised, knowing how these things work, but results are shocking, and awful. Here's a sampling of what they did and what they found, borrowed from the first author's thread on X: The basic method is to embed Standardized American English or African American English inside a prompt, and see what happens from there: [https] with an important distinction between overt racism - the systems rarely directly say stuff like "Black people are bad" - and covert racism: how the system treated queries about consequential matters, given an African American English prompt. On overt measures, the systems were fine. On covert measures, they were a disaster: [https] The potential consequences on the world, given the widespread adoption of LLMs, are massive: [https] And [https] Echoing an important finding of Abeba Birhane's, scale actually makes the covert racism worse: [https] It looks like trivial, dialectical choices in wording drive a lot of the phenomenon: [https] LLMs are, as I have been trying to tell you, too stupid to understand concepts like people and race; their fealty to superficial statistics drives this horrific stereotyping . As Hofman put it on X, it is a bit of a double whammy:" users mistake decreasing levels of overt prejudice for a sign that racism in LLMs has been solved, when LLMs are in fact reaching increasing levels of covert prejudice." Other recent research from Princeton [that I discovered moments ago] points in the same direction. Hofman's summation is incredibly damning: [https] SS Auto manufacturers are obliged to recall their cars when they produce serious problems. What Hofman and his collaborators (including the MacArthur Fellow Dan Jurafsky) have documented may already be having real-world impact. In many ways, we have no idea how LLMs actually get used in the real world, e.g. how they get used in housing decisions, loan decisions, crime proceedings etc - but now strong evidence to suspect covert racism to the extent that it is used in such use cases. I would encourage Congress, the EEOC, HUD, the FTC, and others to investigate, and demand user logs and interaction data from the major LLM manufacturers. Counterparts in other nations should consider doing the same. I honestly don't see an easy fix -- simply adding human feedback made things worse, as the authors showed. And Google's debacle shows that guardrails are never easy.. But the LLM companies should recall their systems until they can find an adequate solution. This cannot stand. I have put up a petition at change.org, and hope you will consider both signing and sharing. Share Gary Marcus's first paid gig was doing statistics for his father, who at the time was doing discrimination law. These new results turn his stomach; his father would have been appalled. Marcus on AI is a reader-supported publication. To receive new posts and support my work, become a free or paid subscriber. [ ] Subscribe 61 Share this post [https] Covert racism in LLMs garymarcus.substack.com Copy link Facebook Email Note Other 37 Share 37 Comments [https] [ ] Share this discussion [https] Covert racism in LLMs garymarcus.substack.com Copy link Facebook Email Note Other Allie S 6 hrs agoLiked by Gary Marcus I can't wait to watch Andrew Ng try to talk himself in circles on this one. I mean I recall a 2023 paper that found [https] that ChatGPT reproduces societal biases. I agree with you that this cannot stand and that we as the general public deserve better. Expand full comment Reply Share Venkat Srinivasan Gyan: A Natural Language Unders... 7 hrs agoLiked by Gary Marcus [https] Really significant finding. purely data driven models will never solve these root challenges Expand full comment Reply Share 7 replies 35 more comments... Top New Community No posts Ready for more? [ ] Subscribe (c) 2024 Gary Marcus Privacy [?] Terms [?] Collection notice Start WritingGet the app Substack is the home for great writing This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts