https://alecmuffett.com/article/16257 Skip to content dropsafe dropsafe a blog about network security, digital rights, nerdity, food, and bicycles Menu * Home * Blog + Blog (All) * RSS + RSS (All) + RSS (Comments) * Tools + Search + Pre-Flight Travel Checklist + Muffett on Passwords * About + About + Login Posted on 2022/08/072022/10/18 by alecm Abuse Prevention is Tradecraft (Was: "Como is Infosec" by @doctorow) #ContentModeration Is Not #Infosec It's sunday evening; there's a lot to be said for being a stay-at-home-dad, and honestly I actually like having stuff like meal-prep and laundry to do, after 35+ years of "All Unix Network Security, All The Time" living. But I don't like having to write this post. Not least I need to be bringing in the laundry and loading the dishwasher and prepping so my partner can start her 9-to-5 -- more accurately online-7:30-to-6 -- tomorrow without fuss, whilst I prep breakfast for everyone and begin a day full of Hey Duggee, Bluey, Teletubbies, and other make-cultural-benefit-between-nappy-changes for my glorious 13mo daughter. But I also don't want to be writing this because I like Cory, who's written something extraordinarily ill-advised and misconceived; I owe some small debt to Cory, not only for working vaguely simultaneously for the Open Rights Group together, but also for helping sediment my thinking some years ago that if I ever I had the chance -- it seemed unlikely at the time -- that I would quit work in order to raise a family, continuing to make impact on the world in time outside of that, rather than as most of my peers had been: wishing that they'd had more time with their kids. Long story short: Cory wants [S:Facebook:S] tech platform content and community moderation to be more open, to be brought into the daylight, to be where everyone can see how it works, because... well, Cory never exactly gets around to a "x therefore y" reason as far as I can see (?) other than to reasonably point out that having your account shut down without apparent explanation is a pain in the ass, and also to analogise that any amount of secrecy around content moderation is "security through obscurity" -- which as any infosec practitioner knows is a "bad thing". Except that it's not -- not exactly -- a fair comparison; but we'll come to that. Cory's blogpost ends abruptly, as if left hanging by some sudden realisation. I don't know why or what that is, but leaving the discourse hanging does possibly leave the reader with some incorrect impressions, unless they bother to dig deeper. There's a couple in particular that I would like to fix, because: * I used to work at Facebook * I worked as part of the Site Integrity team * I designed and wrote software to provide signals for abuse detection * I worked with and watched the team taking-down a bunch of different kinds of abuse * I understand how this stuff really works, and... * ...having wrong information flying around in the debate at the moment is really politically dangerous for digital rights. [Screenshot-2022-08-07-at-21][Screenshot-2022-08-07-at-21]https:// transparency.fb.com/en-gb/policies/community-standards/hate-speech/ [Screenshot-2022-08-07-at-21][Screenshot-2022-08-07-at-21]https:// transparency.fb.com/en-gb/policies/community-standards/hate-speech/ Santa Clara Principles? Cory writes: That's why companies like Facebook keep both the rules they apply to community moderation and the tools they use to automate discovery of violations of those rules a secret. The first half of this is certainly not correct; Facebook at writes at extraordinary length -- apparently including historical versions -- regarding what activity will result in you being kicked off Facebook. Perhaps he meant to claim that Facebook are lying, instead? The second half of this bleeds into the subsequent paragraph: They insist that revealing the policies and enforcement mechanisms will help bad actors who want to harass, defraud or impersonate their users and upload prohibited materials, from disinformation to misinformation, from Child Sex Abuse Material to copyright infringements to terrorist atrocity videos. Yes, and Facebook's analysis is correct, and Cory's critique ("And yet, the same tech giants... routinely use and defend security through obscurity as the only way to keep their content moderation programs on track") is technically accurate but functionally incorrect, because: "Information Asymmetry" is not the same as "Security Through Obscurity" The problem with Security Through Obscurity is (yes) that it's easily defeated; the usual example is "there's a spare key for the house, kept under the doormat", viz: there exists a simple trick which entirely obviates all security mechanisms for the thing being defended. But Informational Asymmetry (IA) is not the same as STO, and it's a fundamental of Information Security -- or Infosec, since we're in the land of sexy terminology. * IA is knowing the password which hashes to "opbHC1Tv.ZsH.", but a hacker does not * IA is knowing what number is on-screen on your TOTP authenticator , but a hacker does not * IA is knowing that your poker opponent winces when they are dealt a bad hand * STO is every system in the world having the same magic incantation backdoor password * STO is expecting nobody to bother reverse engineering your API But when you're in the land of (for instance) anti-abuse, you're not even in the yes-or-no world of binary truths; instead you're in the world of aggregate signals: * IA is knowing that the Russian spammers use a Curl client with a particular TLS fingerprint * IA is knowing that a troll farm is trying to reduce grammatical language errors by (detectably) posting wholesale chunks of Wikipedia articles as conversational content * IA is knowing what image-library compression quirks hint that a posting was not sent from an "official" platform client, but instead from some custom bot-farm So, yes, "revealing the... enforcement mechanisms [absolutely will] help bad actors who want to harass, defraud or impersonate their users and upload prohibited materials, from disinformation to misinformation, from Child Sex Abuse Material to copyright infringements to terrorist atrocity videos". And you want to stop that happening? Cory again: This is the same failure mode of all security-through-obscurity. Secrecy means that bad guys are privy to defects in systems, while the people who those systems are supposed to defend are in the dark, and can have their defenses weaponized against them. Okay, maybe I get it; perhaps Cory's goal is an "accelerationist evolutionary" one, that if we force the big tech platforms to reveal how they are detecting spam then innocent victims whose accounts get closed will obtain some transparency, and the result of all the "bad actors" suddenly getting a free pass towards raising their game / getting better at not being caught, will require Facebook (et al) to once-and-for-all work out a way to "fix" abuse. Not even regrettably, this is a terrible idea; if anything is going to force Governments around the world to start demanding digital identity cards or some other cure that is worse than the disease, it is this. We should not even go there. We already have (see below) a developing set of principles and an implicit plan to assist people who are victims either of misclassification or of abuse. Being distracted from this towards absolutism, is unwise. So: Facebook (to name but one) does not keep its rules secret, and it (and the rest of the platform community) is correct and probably wise to be reticent about how it attempts day-to-day to react to the ever changing behavioural "tells" of spam and abuse. Yes it's a pain that abusers can infer the (current) rules well enough to know that if they stop just short of referring to a member of an oppressed minority group as [some pre-existing slur] then they can act with some degree of impunity -- this appears to be what Cory is presenting as the "key under the doormat" of abuse prevention. However this is actually an argument for finer-grained, better resourced and (ideally) community-integrated moderation -- so the communities themselves can police their own membership -- noting in passing that such will of course permit (e.g.) white supremacists to protect themselves from harmful, hurtful ideas such as liberalism, equality and equity. But the opposite -- "perfect" top down control --would be worse. So: Cory's blogpost is ill-founded; but perhaps he has a point that platforms at least should be sharing these anti-abuse poker "tells" more transparently amongst themselves and each other, in order to collectively better-prevent abuse? The sooner we start treating como as infosec, the better. A good first step would be to adopt the Santa Clara Principles, a multistakeholder document that sets out a program for accountable and transparent moderation. The big platforms already have conferences about sharing the generic -- and sometimes the detailed -- techniques of abuse-prevention mechanisms with each other; e.g. at the significantly-Facebook-sponsored At Scale conferences with tracks on "Spam Fighting" and "Fighting Abuse" (search for videos, well worth the effort) and of course there is venerable USENIX's "Enigma" conferences which cover some of the best-of-the-best work in this space. So when Cory links to, and calls for adoption of, the Santa Clara Principles, I am like... "what?" because: In 2018, alongside the Content Moderation at Scale conferences ... wait, haven't we seen that name before somewhere? ... In 2018, alongside the Content Moderation at Scale conferences in the United States, a group of human rights organizations, advocates, and academic experts developed and launched a set of three principles for how best to obtain meaningful transparency and accountability around Internet platforms' increasingly aggressive moderation of user-generated content. [...deletia...] Since 2018, twelve major companies--including Apple, Facebook (Meta), Google, Reddit, Twitter, and Github--have endorsed the Santa Clara Principles and the overall number of companies providing transparency and procedural safeguards has increased, as has the level of transparency and procedural safeguards provided by many of the largest companies. [...deletia...] For these reasons, a broad coalition of organizations, advocates and academic experts worked together in 2020 and 2021 to develop this second iteration of the Santa Clara Principles... Oh, and that explains everything. We've gone from a 2018 Version 1 checklist of "what companies need to do in order to be fair to users" to an much-expanded 2021 Version 2 "human rights with design principles" document; and this blogpost is Cory (and perhaps, by extension, the entire EFF) thinking that the best way to get platforms to adopt the new document version is to get people angry about the matter, as-if the document hadn't come out of the platform community in the first place? I'm not saying that the new document version is bad -- there's quite a lot in v2 which is a sensible and proportionate evolution of the v1 document, although some new parts are massively onerous and clearly designed to pander to the interests of civil society data scientists who want material with which they can flog the wicked, capitalist platforms in order to justify their salaries: * The number of times a post was viewed before it was removed. Stakeholders emphasized that the number of views a post received before it was removed, or its virality, is important for understanding the impact a piece of content had before it was removed. * The timeline for content removal. This includes data on: + Time between when a post was published and when it was removed + Time before a post that was erroneously removed was reinstated (either as a result of an appeal or as a result of proactive identification of an error by a company) + Time between user flagging and response from platform + Time for a piece of content to be identified by an automated tool or the company and then removed + Time for a piece of content to be flagged to a company and removed by an automated tool. This is data sought for individual takedowns; there must be millions of these per year, and if nothing else the risks of forensic reidentification of some posts from this data will pose concerns for privacy of the authors; not to mention how to address retrospective actions where a report today causes the takedown of content posted several days, if not weeks previously? With this much data, comes a greater amount of context which will never be available; subsequent punditry and opinion based upon this will be misrepresentative or dubious, at best. But I digress, because this bit of analysis is immaterial for critique of Cory's blogpost; instead I'll just summarise: 1. reticence about how abuse-prevention works is not security through obscurity 2. some reticence about how abuse-prevention operates, helps maintain value of those mechanisms 3. platforms, especially Cory's bete noire, generally already write at great length about how they judge abusive material... 4. ...and, if anything, they risk being criticised for being too verbose about it 5. publishing full details of how platforms detect abusive content will lead to a crisis of abuse occurance, which will further lead to demands for illiberal "crackdowns" and further pursuit of state control of the internet, digital identity cards, etc, to "prevent" the same 6. not to mention: the methods change from week to week, are tweaked from week to week, because people are involved at every level; comparisons with poker are not inaccurate, and yet the v2 document seems not to acknowledge that combatting abuse is a dynamic and diverse pursuit 7. trying to make people angry to get them to beat up the platforms / force the platforms to adopt v2 of something they were involved in writing in the first (and second) place, is not a terribly fair nor charitable tactic I'd like to see better in the public debate. Share this post: * Click to share on Twitter (Opens in new window) * Click to share on Facebook (Opens in new window) * Click to share on LinkedIn (Opens in new window) * Click to share on WhatsApp (Opens in new window) * Click to share on Reddit (Opens in new window) * Click to share on Pocket (Opens in new window) * Click to email a link to a friend (Opens in new window) * More * * Click to share on Telegram (Opens in new window) * Click to share on Pinterest (Opens in new window) * * Click to share on Skype (Opens in new window) * Click to share on Tumblr (Opens in new window) * * Click to print (Opens in new window) * Tagsabuse, infosec, online abuse, scraping, spam 6 Replies to "Abuse Prevention is Tradecraft (Was: "Como is Infosec" by @doctorow) #ContentModeration Is Not #Infosec" 1. [df85ead4fb0f6][df85ead4fb0f6] Geoff Arnold says: 2022/08/07 at 23:25 Useful. I hope your partner and daughter weren't inconvenienced. Reply 1. [e00a726c7070d][e00a726c7070d] alecm says: 2022/08/08 at 07:48 We'll survive, but I am operating today with less sleep so daddy will likewise be having a morning nap. Reply 2. [db81640bba876][db81640bba876] authpor says: 2022/10/18 at 15:30 uff, a lot to unpack. I'm having a complicated thought... the same points about information asymetry in relation to the preservation of value are at play in the political (i.e. public) games. I didn't even know there were santa clara principles, in a rough sense, this is maintining some sort of value from the people who have read those to them who don't even know about such principles. I seem to be thinking that information assymetry is statecraft, a "super-set" of the notion of abuse prevention (IA and security through obscurity) as trade craft (because the state contains the market/trade) Reply 1. [e00a726c7070d][e00a726c7070d] alecm says: 2022/10/18 at 22:53 @authpor: "information asymmetry" is as simple as the password you use to log into your computer. You know it, someone else does not. In a sense it is "obscure" to them, but it is not security-through-obscurity. Reply 3. [14a468eaa938d][14a468eaa938d] Maya says: 2022/10/18 at 16:56 Presenting "civil society data scientists" as motivated by money seems to rhetorically rely on the reader not knowing how much more money you can make with a set of data science skills working on something else. It also seems to border on disingenuity to present: Perhaps he meant to claim that Facebook are lying, instead? when every person who's spent more than a year or so on social media can point to the deltas between policy and reality, the rules on paper and the rules as they are actually enacted - inconsistency. Even non-human systems have huge gaps between intent and observed behavior. Part of the argument here hinges on the idea that it's a huge effort with lots of complicated surface area to try to realize those moderation standards, involving abstruse and specific details. I don't think it's surprising that there's a meaningful gap between the principles of offense Facebook says it will kick you off for and the gritty reality of "what activity will result in you being kicked off Facebook"; that's what you're saying people are taking advantage of with the key-under-the-doormat slur boundaries, for instance. This is an important topic to get informed opinions on, and yours are particularly informed and therefore particularly valuable, so the rhetorical details that make this seem not entirely-in-good-faith are ... well, dispiriting. I'd be very, very, very happy to be wrong, though. Reply 4. [e00a726c7070d][e00a726c7070d] alecm says: 2022/10/18 at 22:47 Hi Maya! I have no axe to grind, and no faith to be without. I literally am beholden to no-one. What I am trying to convey is (a) that abuse prevention stands upon a certain amount of secrecy because it needs to, and (b) there's no way of changing that because the things which would need to be emplaced in order to "do abuse prevention properly" have no place in a liberal society. You might enjoy: https://medium.com/@alecmuffett/ a-billion-grains-of-rice-91202220e10e Reply Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name * [ ] Email * [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Post navigation Previous PostPrevious @naomicfisher on screen time Next PostNext I'm chairing the Encryption session at @UKIGF / UK Internet Governance Forum, on November 1st ... #ukigf2022 #privacy # security #infosec #encryption #endtoendencryption Recent Comments * alecm on Abuse Prevention is Tradecraft (Was: "Como is Infosec" by @doctorow) #ContentModeration Is Not #Infosec * alecm on Abuse Prevention is Tradecraft (Was: "Como is Infosec" by @doctorow) #ContentModeration Is Not #Infosec * Maya on Abuse Prevention is Tradecraft (Was: "Como is Infosec" by @doctorow) #ContentModeration Is Not #Infosec * authpor on Abuse Prevention is Tradecraft (Was: "Como is Infosec" by @doctorow) #ContentModeration Is Not #Infosec * alecm on Okay, yes, some people I know are quite upset that the Apple #AirTag may be used to track people; but the anti-stalking features hamper return of stolen goods, so... About Dropsafe Dropsafe is the personal blog of Alec Muffett (more...) with occasional contributions from friends & guest bloggers; it is a blog populated entirely by personal opinions of the author(s). Hopefully you will find the content to be fun, interesting, or both. All original content - i.e.: not where clearly and transparently linked or copied from elsewhere - is licensed under CC-BY-SA terms. Please log a comment if you find any deviations from this intention. Also: there are no paid adverts, nor affiliate links -- unless some of the latter have leaked in via copypasta, or during an experiment a few years ago. Again, if you find any, please log a comment and I'll remove them. This blog is also available as an Onion Site for users of Tor-capable web browsers. Archives Archives [Select Month ] Proudly powered by WordPress Loading Comments... Write a Comment... [ ] Email (Required) [ ] Name (Required) [ ] Website [ ] [Post Comment]