http://www.antipope.org/charlie/blog-static/2021/03/lying-to-the-ghost-in-the-mach.html Charlie's Diary Being the blog of Charles Stross, author, and occasional guests ... [ Home ] [ FAQ ] [ Contact me ] [ Older stuff ] Back to: Central Banking on Mars! Lying to the ghost in the machine By Charlie Stross (Blogging was on hiatus because I've just checked the copy edits on Invisible Sun, which was rather a large job because it's 50% longer than previous books in the series.) I don't often comment on developments in IT these days because I am old and rusty and haven't worked in the field, even as a pundit, for over 15 years: but something caught my attention this week and I'd like to share it. This decade has seen an explosive series of breakthroughs in the field misleadingly known as Artificial Intelligence. Most of them centre on applications of neural networks, a subfield which stagnated at a theoretical level from roughly the late 1960s to mid 1990s, then regained credibility, and in the 2000s caught fire as cheap high performance GPUs put the processing power of a ten years previous supercomputer in every goddamn smartphone. (I'm not exaggerating there: modern CPU/GPU performance is ridiculous. Every time you add an abstraction layer to a software stack you can expect a roughly one order of magnitude performance reduction, so intuition would suggest that a WebAssembly framework (based on top of JavaScript running inside a web browser hosted on top of a traditional big-ass operating system) wouldn't be terribly fast; but the other day I was reading about one such framework which, on a new Apple M1 Macbook Air (not even the higher performance Macbook Pro) could deliver 900GFlops, which would put it in the top 10 world supercomputers circa 1996-98. In a scripting language inside a web browser on a 2020 laptop.) NNs, and in particular training Generative Adversarial Networks takes a ridiculous amount of computing power, but we've got it these days. And they deliver remarkable results at tasks such as image and speech recognition. So much so that we've come to take for granted the ability to talk to some of our smarter technological artefacts--and the price of gizmos with Siri or Alexa speech recognition/search baked in has dropped into two digits as of last year. Sure they need internet access and a server farm somewhere to do the real donkey work, but the effect is almost magically ... stupid. If you've been keeping an eye on AI you'll know that the real magic is all in how the training data sets are curated, and the 1950s axiom "garbage in, garbage out" is still applicable. One effect: face recognition in cameras is notorious for its racist bias, with some cameras being unable to focus or correctly adjust exposure on darker-skinned people. Similarly, in the 90s, per legend, a DARPA initiative to develop automated image recognition for tanks that could distinguish between NATO and Warsaw Pact machines foundered when it became apparent that the NN was returning hits not on the basis of the vehicle type, but on whether there was snow and pine forests in the background (which were oddly more common in publicity photographs of Soviet tanks than in snaps of American or French or South Korean ones). Trees are an example of a spurious image that deceives an NN into recognizing something inappropriately. And they show the way towards deliberate adversarial attacks on recognizers--if you have access to a trained NN, you can often identify specific inputs that, when merged with the data stream the NN is searching, trigger false positives by adding just the right amount of noise to induce the NN to see whatever it's primed to detect. You can then apply the noise in the form of an adversarial patch, a real-world modification of the image data being scanned: dazzle face-paint to defeat face recognizers, strategically placed bits of tape on road signage, and so on. As AI applications are increasingly deployed in public spaces we're now beginning to see the exciting possibilities inherent in the leakage of human stupidity into the environment we live in. The first one I'd like to note is the attack on Tesla car's "autopilot" feature that was publicized in 2019. It turns out that Tesla's "autopilot" (actually just a really smart adaptive cruise control with lane tracking, obstacle detection, limited overtaking, and some integration with GPS/mapping: it's nowhere close to being a robot chauffeur, despite the marketing hype) relies heavily on multiple video cameras and real time image recognition to monitor its surrounding conditions, and by exploiting flaws in the image recognizer attackers were able to steer a Tesla into the oncoming lane. Or, more prosaically, you could in principle sticker your driveway or the street outside your house so that Tesla autopilots will think they're occupied by a truck, and will refuse to park in your spot. But that's the least of it. It turns out that the new hotness in AI security is exploiting backdoors in neural networks. NNs are famously opaque (you can't just look at one and tell what it's going to do, unlike regular source code) and because training and generating NNs is labour- and compute-intensive it's quite commonplace to build recognizers that 'borrow' pre-trained networks for some purposes, e.g. text recognition, and merge them into new applications. And it turns out that you can purposely create a backdoored NN that, when merged with some unsuspecting customer's network, gives it some ... interesting ... characteristics. CLIP (Contrastive Language-Image Pre-training) is a popular NN research tool, a network trained from images and their captions taken from the internet. [CLIP] learns what's in an image from a description rather than a one-word label such as "cat" or "banana." It is trained by getting it to predict which caption from a random selection of 32,768 is the correct one for a given image. To work this out, CLIP learns to link a wide variety of objects with their names and the words that describe them. CLIP can respond to concepts whether presented literally, symbolically, or visually, because its training set included conceptual metadata (textual labels). So it turns out if you show CLIP an image of a Granny Smith, it returns "apple" ... until you stick a label on the fruit that says "iPod", at which point as far as CLIP is concerned you can plug in your headphones. NN recognizing a deceptively-labelled piece of fruit as an iPod And it doesn't stop there. The finance neuron, for example, responds to images of piggy banks, but also responds to the string "$$$". By forcing the finance neuron to fire, we can fool our model into classifying a dog as a piggy bank. The point I'd like to make is that ready-trained NNs like GPT-3 or CLIP are often tailored as the basis of specific recognizer applications and then may end up deployed in public situations, much as shitty internet-of-things gizmos usually run on an elderly, unpatched ARM linux kernel with an old version of OpenSSH and busybox installed, and hard-wired root login credentials. This is the future of security holes in our internet-connected appliances: metaphorically, cameras that you can fool by slapping a sticker labelled "THIS IS NOT THE DROID YOU ARE LOOKING FOR" on the front of the droid the camera is in fact looking for. And in five years' time they're going to be everywhere. I've been saying for years that most people relate to computers and information technology as if they're magic, and to get the machine to accomplish a task they have to perform the specific ritual they've memorized with no understanding. It's an act of invocation, in other words. UI designers have helpfully added to the magic by, for example, adding stuff like bluetooth proximity pairing, so that two magical amulets may become mystically entangled and thereafter work together via the magical law of contagion. It's all distressingly bronze age, but we haven't come anywhere close to scraping the bottom of the barrel yet. With speech interfaces and internet of things gadgets, we're moving closer to building ourselves a demon-haunted world. Lights switch on and off and adjust their colour spectrum when we walk into a room, where we can adjust the temperature by shouting at the ghost in the thermostat, the smart television (which tracks our eyeballs) learns which channels keep us engaged and so converges on the right stimulus to keep us tuned in through the advertising intervals, the fridge re-orders milk whenever the current carton hits its best-before date, the robot vacuum comes out at night, and as for the self-cleaning litter box ... we don't talk about the self-cleaning litterbox. Well, now we have something to be extra worried about, namely the fact that we can lie to the machines--and so can thieves and sorcerors. Everything has a True Name, and the ghosts know them as such but don't understand the concept of lying (because they are a howling cognitive vacuum rather than actually conscious). Consequently it becomes possible to convince a ghost that the washing machine is not a washing machine but a hippopotamus. Or that the STOP sign at the end of the street is a 50km/h speed limit sign. The end result is people who live in a world full of haunted appliances like the mop and bucket out of the sorcerer's apprentice fairy tale, with the added twist that malefactors can lie to the furniture and cause it to hallucinate violently, or simply break. (Or call the police and tell them that an armed home invasion is in progress because some griefer uploaded a patch to your home security camera that identifies you as a wanted criminal and labels your phone as a gun.) Finally, you might think you can avoid this shit by not allowing any internet-of-things compatible appliances--or the ghosts of Cortana and Siri--into your household. And that's fine, and it's going to stay fine right up until the moment you find yourself in this elevator ... Posted by Charlie Stross at 11:12 on March 6, 2021 | Comments (53) 53 Comments | Leave a comment Therion667 # Therion667 | March 6, 2021 11:29 | Reply 1: I wonder what they'd make of "This is not a pipe"? Charlie Stross # Charlie Stross replied to this comment from Therion667 | March 6, 2021 11:32 | Reply 2: Am now envisaging a hotel evacuation at 3am in midwinter because some prankster scribbled FIRE on a whiteboard in the hotel lobby. Elderly Cynic # Elderly Cynic | March 6, 2021 11:53 | Reply 3: I am a bit less out of touch, and have little to add, except: Q: "OK, gang, it's Stravinsky night - what shall we show?" A: "Firebird! Firebird! Firebird!" Or when a SWAT team is called to shoot up a conference of winemakers for talking about terroire .... LAvery # LAvery | March 6, 2021 11:59 | Reply 4: Similarly, in the 90s, per legend, a DARPA initiative to develop automated image recognition for tanks that could distinguish between NATO and Warsaw Pact machines foundered when it became apparent that the NN was returning hits not on the basis of the vehicle type, but on whether there was snow and pine forests in the background (which were oddly more common in publicity photographs of Soviet tanks than in snaps of American or French or South Korean ones). About that: it probably never happened. Paul # Paul | March 6, 2021 12:04 | Reply 5: I followed the link to the video. I'm having a problem with my PC sound at present, so I turned on subtitles. The subtitles have obviously been generated by a voice recognition algorithm, and were wrong enough that I couldn't quite follow what was going on, but I think this is a video about problems with voice recognition in an elevator that can't understand "eleven" in a Scottish accent. However the captioning algorithm seems to have no problem with that bit. I'm going to fix my audio problems and come back to this. Charlie Stross # Charlie Stross replied to this comment from Paul | March 6, 2021 12:05 | Reply 6: You really need working audio to appreciate the sketch (in the video). It's the accents that carry it. Jamesface # Jamesface | March 6, 2021 12:11 | Reply 7: Looking forward to a computer responding to deepfake requests with a picture of an apple with "Tom Cruise" written on it. Jelliphiish # Jelliphiish | March 6, 2021 12:12 | Reply 8: My black and white Tuxedo cat gained an additioinal name after the 'AI' mode on the smartphone identified him as a Panda. Jack Tingle # Jack Tingle | March 6, 2021 12:30 | Reply 9: To be fair to the machines, apparently American Q-cumbers aren't clear on the concept of being lied to either. Elderly Cynic # Elderly Cynic replied to this comment from LAvery | March 6, 2021 12:37 | Reply 10: Actually, it probably DID happen, though perhaps not in the exact forms quoted. I was close to some of the early research (and dabbled myself), and the effect is real - I have observed it many times since, when crude forms of recognition are used to suggest matches. The main reason that it is hard to track down is that such failures are embarrassing, and hence swept under the carpet. My apologies, but here is a bit of a lecture. All such methods (and they are both much more general than automated image recognition and predate it by decades) are statistical in nature, and it is NOT feasible to eliminate mistakes. The cost (weighting) of type I and II errors (false positives and negatives) is part of the training. But, inherently, when it encounters a condition that is not well-represented in the training set, the reliability goes to pot. If the 'AI' is largely programmed by telling it what characteristics are of consequence, it will not pick up extraneous ones; if they were written to disclose their criteria, then they could be checked; but most modern training does neither. Let's skip the rant about that. I could go on, with a few more subtle aspects, but I am many, many decades rusty and this area has developed unimaginably since. So, no matter HOW universal the traning set is, there will always be some circumstances where it is using an unrelated characteristic to make the judgement. Humans are no different, but we DO have a censoring mechanism as part of our intelligence that says "Based on other experience, something is not right here." And it is the lack of that "common sense" that makes me say that AI is not even approaching human intelligence, and agree that this is a disaster in the making. Actually, I do have one further point that absolutely horrifies me. In the UK, the courts have increasingly accepted such 'evidence' from the spooks and police, and we have trials where the accused is not even told what the evidence is against him. Greg Tingey # Greg Tingey | March 6, 2021 12:38 | Reply 11: Arrgh,Therion 667 beat me to it, never, mind .... Actually, nothing new here, at all: - Apple (fruit) & IPod ... "Ceci ne's pas un pipe" as Rene Magritte showed us, many years ago! It's all distressingly bronze age, but we haven't come anywhere close to scraping the bottom of the barrel yet. THAT modern? Neolithic, or possibly earlier IMHO. A re-read of U K le G on the Rule of Names & the True Names of Things might be in order. As for "Elevator" - all too possible - I have already been caught twice by this. Once, because my ( By modern standards ) clipped RP English has not been recognised by the at least "mid-Atlantic" fucked-over voice recognition & again in an automated basic German class, ("Babbel" ), where, again, my old-fashioned clear speech has screwed the machine software. Can I start screaming now? Jelliphish I once has a small b-&-w rescue cat called "Panda" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OK peoples, what ahem, "pranks" do we want to perpetrate to really screw over the state & company fake "security" systems, just to show them? CharlesW # CharlesW | March 6, 2021 13:13 | Reply 12: "WebAssembly framework (based on top of JavaScript running inside a web browser hosted on top of a traditional big-ass operating system) wouldn't be terribly fast" I thought WebAssembly was supposed to be compiled to its own virtual machine and to be faster than JavaScript, rather than sitting on top of JavaScript. Paul # Paul | March 6, 2021 13:20 | Reply 13: There are a number of SF stories about images or thoughts that crash the brain, one being The Riddle of the Universe and Its Solution, and another of course being the . But if human brains are basically neural networks, and you can fool a specific neural network with the right image, could there be ways of doing that with the specific neural network in the visual cortex? Probably not, at least in general. And the brain being evolved, it's reasonable to suppose that there are multiple neural nets in there with overlapping jobs. So one can imagine an image which makes a particular person say "It looks like a giant spider crawling towards me. Except I can also see its a picture of a dog at the same time. That's wierd!" Of course it might turn out that there are some nets in the visual cortex that are the same for everyone, or for large groups, so there may be illusions that work on some people but not others. Either way, reverse engineering wetware in order to develop such images could be a fruitful area of research. Elderly Cynic # Elderly Cynic replied to this comment from Paul | March 6, 2021 13:27 | Reply 14: The human brain and 'neural networks' in the 'AI' sense have little in common, and the latter term was invented by people with a rudimentary understand of brain function, which turned out to be incorrect. We do not know how the brain works, but it it is very different and much less 'crashable'. However, the visual cortex is much better understood, is largely 'hard-wired' (unlike the auditory one), and there are plenty of known ways to fool it. The simpler ones have been known for ages, and your example isn't just imaginary - there are lots of such examples, such as this classic: http://www.optical-illusionist.com/illusions/ young-lady-or-old-woman-illusion Charlie Stross # Charlie Stross replied to this comment from Paul | March 6, 2021 13:29 | Reply 15: could there be ways of doing that with the specific neural network in the visual cortex? Dave Langford got a Hugo award for that back in 2000: see also. Geoff Hart # Geoff Hart | March 6, 2021 13:40 | Reply 16: I suspect we're going to need a lot more (and a lot more sophisticated) use of "supervised classification". Short version: That's the term used in remote sensing to describe when humans "supervise" the classification performed by a computer algorithm to confirm or reject the computer's decision. Particularly when the human supervisor is really two or more people who discuss the classifications and reach consensus on any questionable pixels, this can greatly improve the classification accuracy. If you do ground-truthing (going to the site of a remote sensing image to see what's actually there), the accuracy can improve further. For example, in Charlie's example with tanks in a forest, the supervisor could have classified the image pixels containing part of a tank as "is tank" (!) and the image pixels containing only forest as "is not tank" (0). This would have minimized (possibly eliminated) the problem of classifying trees as tanks. Relying on the neural network to make this distinction seems... um... lazy? Unsophisticated? Hacking supervised systems should prove harder, since (for example) a training set that includes only words (e.g., "ceci est une pipe") would not appear in the training set for pipes. For that matter, they could be included in the "is not pipe" training set. You could still hack such a system by feeding it a high-resolution image of a pipe or simply tapping into the camera's connection to the computer and directly streaming images of a real pipe. Hackers seem to always find a backdoor, so you'd need to be vigilant and keep monitoring the literature for new exploits. Back-dooring neural nets will also be a thing, but I suspect this will be easier if it involves hacking the host computer (e.g., to substitute your own images for the camera's images) will be more effective. And so long as the computer is controlled by a human with authority to override the computer, social engineering will remain a serious problem. Geoff Hart # Geoff Hart | March 6, 2021 13:43 | Reply 17: Oops... substitute (1) = binary 1 = true for (!) in the previous comment. I didn't mean to imply "not". Geoff Hart # Geoff Hart | March 6, 2021 13:47 | Reply 18: On a related note: https://www.thelocal.de/20210305/ intern-at-german-prison-faces-hefty-bill-after-sending-photo-of-master-key-to-friends / Elderly Cynic # Elderly Cynic replied to this comment from Geoff Hart | March 6, 2021 14:33 | Reply 19: Unfortunately, that's not the way things are going :-( Your example wouldn't help, because it would have correctly classified the trees as non-tank - what made it a Russian tank was the presence of trees. So you can't just supervise in training - you have to do it in real life. And the supervisor needs the authority and ability to override the AI, which is often not the case. I had this problem just recently, when the DVLA automation point-blank refused to recognise my driving licence to renew it, and the bureaucrat who was the fallback merely mimicked the AI. My wife solved it by telephoning, finding an actual human, and being walked through the use of a loophole to renew it. If they had closed that loophole, I would have been stuffed. This sort of thing is getting increasingly common. Niala # Niala replied to this comment from Elderly Cynic | March 6, 2021 14:35 | Reply 20: Elderly Cynic @ 10 : : "And it is the lack of that "common sense" that makes me say that AI is not even approaching human intelligence." Don't despair! The people working on common sense at the Cyc project are still at it! https://en.wikipedia.org/wiki/Cyc Who knows, they might succeed in another 37 years. Greg Tingey # Greg Tingey | March 6, 2021 15:17 | Reply 21: EC Except that, under some circs, the human brain is all-too-crashable. Just not visually. But, look at the multiple cases, over thousands of years of (usually, but not always) insanity/mania of a religious nature. The person behaves utterly irrationally & it is very difficult to crash them out of such a state, once in it. Oh yes: In the UK, the courts have increasingly accepted such 'evidence' from the spooks and police, and we have trials where the accused is not even told what the evidence is against him. Where - examples, please ... because this sort of thing is very worrying. We've had past examples of this in a slightly different context, & it's taken many years to sort out the fuck-ups. Example (?) Here, for a start - "Computer says no" & no-one is interested in actual reality. Elderly Cynic # Elderly Cynic replied to this comment from Greg Tingey | March 6, 2021 15:37 | Reply 22: That's not crashing, any more than the fact that some lighting and sound effects can cause some people to lose balance, be unable to think, have panic attacks, or trigger migraines is. Yes, it happens - some lighting triggers epiletic fits in susceptible people, and those definitely count as 'crashes'. So far, nobody has found anything that does the same in 'neurotypical' people, but there is no reason to say nothing can. The point is that the brain is fairly 'uncrashable', because evolution selected against that failure. gordycoale # gordycoale replied to this comment from LAvery | March 6, 2021 15:53 | Reply 23: Spoilsport ;) Next you'll be telling us that Kangaroo's can't use SAMS. https://www.snopes.com/fact-check/shoot-me-kangaroo-down-sport/ Allen Thomson # Allen Thomson replied to this comment from Elderly Cynic | March 6, 2021 16:16 | Reply 24: because evolution selected against that failure. One problem with current AI might be that, in evolutionary terms, it's just getting into the Ediacaran. But it's evolving fast and I'd not want to make any bets on where it will ten years from now. Also, it isn't obvious to me that several of the problems AI is currently manifesting are all that different from those that afflict the natural sort even after more than half a gigayear of fierce evolution. Unholyguy # Unholyguy | March 6, 2021 16:34 | Reply 25: Yes especially once AR takes off Read a book awhile ago where a guy built a suit covered in led displays running advertising banners Since everyone walked around with augmented reality contact lenses on all the time, the ad blocking software in the AR edited him out and he achieved effective invisibility unrated # unrated | March 6, 2021 16:42 | Reply 26: "shitty internet-of-things gizmos"... that paragraph reads like a current sysadmin wrote it. Don't sell yourself short Charlie... the tech changes every 5 years but for the BOFH, the song remains the same. Also a fun shout-out to Stephen Blackmoore... one of his illusion spells involves writing the desired effect on a post-it note and attaching it to the target (e.g. "I'm a cop" or "I'm not here"). Who knew it came from AI? Michael Hutson # Michael Hutson | March 6, 2021 16:47 | Reply 27: If you've watched the animated series "Futurama" you may have noticed something: all the robots are subtly imbecilic. They're intelligent but can't see outside the bounds of their programming. So for example you can talk one into doing something by using arguments that are nonsensical but which meet the requirements of their parameters. Geoff Hart # Geoff Hart | March 6, 2021 16:59 | Reply 28: Elderly Cynic notes: "Your example wouldn't help, because it would have correctly classified the trees as non-tank - what made it a Russian tank was the presence of trees." No, what made it a Russian tank was the fact that it was large, made of metal, and had a turret with a gun attached. Adding trees doesn't change the definition of "tank". Trees do not make it a tank, let alone a Russian tank. (Unless it was a Potemkin tank, in which case the AI was correct: the wooden boards made it a faux tank.) You didn't understand my explanation of supervision: the point is to eliminate the extraneous so that the AI learns to focus on the things that define the target as a tank, and learn to ignore the trees. EC: "So you can't just supervise in training - you have to do it in real life." "Supervise" is remote-sensing jargon. It means that the AI is not left to go its merry way, but is rather taught by the human to correct its errors so that it learns the correct details to focus on. It's analogous to the way a trainee surgeon is supervised by an experienced surgeon before they're allowed to fly solo. EC: "And the supervisor needs the authority and ability to override the AI, which is often not the case." That's another issue, and it's definitely a problem. To augment your example, I was once denied approval for a credit card and nobody at the bank could explain why... until I asked the manager to intervene. The problem turned out to be that I applied for an adult credit card while in university and should have applied for a student card. Ioan # Ioan | March 6, 2021 17:14 | Reply 29: My 2 cents on how this is going (since I work in the field) At this point, NN that interact with people can input 3 types of data: text, audio, images (video is a subset of images) 1. Text (NLP): Before the BERT transformer models came out in 2018, there was a thought that most practical uses of NLP was maxed out. Most business applications didn't have enough relevant data to avoid overfitting, and those that did were generalists rather than specialist models. Right now on the SuperGLUE tests, transformers exist which beat the human benchmark. https://super.gluebenchmark.com/leaderboard/ https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) 2. Audio. Speech-to-text is surprisingly accurate, to the point that this area is relatively neglected. It doesn't help that audio-based NN are niche applications. For those applications, the thought is that there isn't a lot of data out there to train anything this complex. Perhaps transformers are revolutionizing that field as well? 3. Images/Video. This is a huge area of research, but here there's a huge concentration problem. Other than medical and military uses, there's a sense that most companies don't need this capability. That's why a lot of the research in this field comes from the FAANG + Microsoft and their Chinese equivalents. I've heard the opinion that the only thing that would break this concentration would be VR/AR. This was before Zoom and Clubhouse made it big. Robert Prior # Robert Prior replied to this comment from Geoff Hart | March 6, 2021 17:15 | Reply 30: what made it a Russian tank was the fact that it was large, made of metal, and had a turret with a gun attached Well, the large metal turret thing makes it a tank. It's the trees that make it Russian :-) On the topic of supervision, I'm reminded of Peter Watts' story "Malak". If you haven't read it, I think you'll enjoy it. https://www.rifters.com/real/shorts/PeterWatts_Malak.pdf Barry [typepad_lo] # Barry replied to this comment from Geoff Hart | March 6, 2021 17:38 | Reply 31: "I suspect we're going to need a lot more (and a lot more sophisticated) use of "supervised classification". Short version: That's the term used in remote sensing to describe when humans "supervise" the classification performed by a computer algorithm to confirm or reject the computer's decision." A note on terminology: Everything I've read about 'supervised' learning/classification refers to ML techniques with labels for each case. Human involvement is not necessarily involved, and of course it is very expensive. That case of NATO vs Warsaw Pact vehicles (and dogs vs wolves, another similar situation) was 'supervised classification'. AaronB # AaronB | March 6, 2021 17:54 | Reply 32: Invisible Sun...50% longer than previous books in the series. Ah, good. I was thinking it might take an awful lot of plot to get from the close of Dark State to a satisfying ending, more than would fit in a normal-sized volume. Personally I'd be happier with a tetralogy or even a short wrap-up and another trilogy, but let's not break the Charlie, shall we? Brett Hale # Brett Hale | March 6, 2021 18:07 | Reply 33: I was an undergrad CS student in the early to mid 90s. The joke with AI courses was like the joke with nuclear fusion: it was only 5 years away from completely changing the world - and had been for the last 40 years. The highlight seemed to be the A* algorithm, and the idea that functional languages might someday be 'competitive'. Well, it may still be right about fusion. Wrong about AI with the proliferation of massively parallel compute resources - my Raspberry PI 4 would be considered a super-computer when I started working in the industry. I don't mind admitting I feel a massive gap in my technical corpus right now. Like a missing tooth. I haven't seen anything comparable to the rate of progress in 'AI' (or really, multi-dimensional tensor weight training) since the explosion in graphics hardware capabilities. The difference being, I was all across that - being related to work / interests. I'm literally at the point where I will admit I don't even know about the things *I don't know* in this field. (Unknown unknowns). I'm curious about how the results we see today can be traced back to pure academia - long before any practical hardware was even on the horizon. Purely anecdotal: someone set up a google 'nest' / 'mini' whatever in my home. I asked it to recite PI to 100 places which it did. I asked it if PI was transcendental, which it succinctly answered with a well-regarded reference. You do not get this from `if ... then ...` paradigms. I predict a lot of posts will pontificate from the peak of 'Mt. Stupid', and how it's just neural nets, analogues to meat-brains, Markov chains, etc... we'll see. Geoff Hart # Geoff Hart | March 6, 2021 18:09 | Reply 34: Barry notes: "Everything I've read about 'supervised' learning/ classification refers to ML techniques with labels for each case. Human involvement is not necessarily involved, and of course it is very expensive." Note that I was referring to the terminology used in remote sensing. For example, . My point was that this approach should be applied to AI classification of visual, auditory, or other information if it isn't already being applied. It's not like it's a new technique; I recall reading about it back in the 1980s (when I was studying forest ecosystems), and it's become SOP in many forms of land-use classification these days. Geoff Hart # Geoff Hart | March 6, 2021 18:12 | Reply 35: The link seems to have disappeared. Here it is again, minus angle brackets: https://mapasyst.extension.org/ whats-the-difference-between-a-supervised-and-unsupervised-image-classification / If that disappears, try Googling "What's the difference between a supervised and unsupervised image classification?" and "Cooperative Extension" Allen Thomson # Allen Thomson replied to this comment from Robert Prior | March 6, 2021 18:35 | Reply 36: On the topic of supervision, I'm reminded of Peter Watts' story "Malak". If you haven't read it, I think you'll enjoy it. I did, and thank you for that. mal@ak@ https://biblehub.com/hebrew/4397.htm whitroth # whitroth | March 6, 2021 18:43 | Reply 37: Charlie mentions tens of thousands of random pics from the 'Net. My instant reaction is a) how many of those actually had alt-text, and how many of those that did had *garbage*, put there by advertisers? *sigh* Time to pull up General Semantics again, and use that, with code that recognizes the levels of abstraction, as a filter before it tries to identify something. Charlie Stross # Charlie Stross replied to this comment from AaronB | March 6, 2021 19:34 | Reply 38: Ah, good. I was thinking it might take an awful lot of plot to get from the close of Dark State to a satisfying ending Pretty much. When I checked the copy edits I gave it a thorough read. It's going to be a 450-500 page brick, but it reads fast and twisty. Bill Arnold # Bill Arnold | March 6, 2021 19:55 | Reply 39: Direct link to the openai blog entry, which is a fun (savage) read: Multimodal Neurons in Artificial Neural Networks (March 4, 2021) We've discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually. This may explain CLIP's accuracy in classifying surprising visual renditions of concepts, and is also an important step toward understanding the associations and biases that CLIP and similar models learn. This is the version that includes all the images about "Typographic Attacks" Multimodal Neurons in Artificial Neural Networks (March 4, 2021) One of the funnier tweets about this (forget where I saw it): The year is 2030. Amazon-Bots are hunting me down because I liked a tweet about unions. I quickly spray the word TUNNEL on a wall and hide behind a trashcan as they all charge headfirst into the bricks. https://t.co/tUw3rYBT5G -- Joe Koller (@JoeKllr) March 5, 2021 Re speed, Cerebras Wafer Packs 2.6 Trillion-Transistor CPU With 850,000 Cores (Joel Hruska, August 18, 2020) Won't fit in a phone though (well, maybe a very big phone): the previous version: "The Cerebras Wafer Scale Engine is 46,225 mm2" (Argonne National Labs (US) has one of their CS-1 computers.) Barry [typepad_lo] # Barry replied to this comment from Ioan | March 6, 2021 20:08 | Reply 40: "For those applications, the thought is that there isn't a lot of data out there to train anything this complex. Perhaps transformers are revolutionizing that field as well?" For speech to text, what I have heard is that close caption TV shows were providing massive data sets. Paul # Paul | March 6, 2021 20:43 | Reply 41: There are a bunch of suggestions for ways of jamming facial recognition in surveillance systems. Here is another one. The trouble with all these is that if you are the only one wearing it, then you are really going to stand out. So its going to be like this XKCD cartoon (in any discussion of technology there is *always* a relevant XKCD). This stuff only works if lots of people do it. But if lots of people do it, countermeasures will be deployed. Robert Prior # Robert Prior | March 6, 2021 20:53 | Reply 42: I would be worried about trusting systems to make decisions without humans in the mix. Even fairly simple glitches can get nasty fast. https://www.wired.com/story/ null-license-plate-landed-one-hacker-ticket-hell/ Seems to be a subset of 'use alphanumeric string to indicate no plate that can also be given as a custom plate' fault in programming, which causes problems in many cases. https://www.snopes.com/fact-check/auto-no-plate/ Allen Thomson # Allen Thomson | March 6, 2021 21:31 | Reply 43: From the NYT just today: https://www.nytimes.com/2021/03/06/business/ the-robots-are-coming-for-phil-in-accounting.html AI-powered accountancy programs. What could go wrong? Pigeon # Pigeon replied to this comment from Elderly Cynic | March 6, 2021 21:39 | Reply 44: That tank recognition story always suggests to me that it is an updated version of the story about German and Russian tanks in WW2. One side decided to try blowing up the other side's tanks by training dogs to run underneath them with mines on their backs. But the dogs learned to identify a target as being The Thing That Smells Of Fuel, and since one side used petrol and the other side used diesel, in action the dogs ignored the other side's tanks and blew up their own side's instead. Allen Thomson # Allen Thomson replied to this comment from Robert Prior | March 6, 2021 21:43 | Reply 45: I would be worried about trusting systems to make decisions without humans in the mix. Yeah, but humans in the mix aren't all that much of an improvement. Bureaucracies tend to be an example of that. Also, the recent administrations of various nations discussed here. Heteromeles # Heteromeles replied to this comment from Greg Tingey | March 6, 2021 21:53 | Reply 46: In the cage match of Darwin versus Earthsea, Darwin always wins. Always. Much as I love Earthsea, it's complete bullshit, and you know that if you think about it logically. How can a categorical name have utility, when not just every thing, but every part of every thing, has a unique name. Heck, just think about what happens when someone picks an apple off a tree and eats it, in a system of such names? Yes, some of the names both go to shit (what remains of the apple after it's gone through the human), don't go to shit (the apple tree) and change (the molecules of apple that go into the human). What does which True Name apply to when? I'll get back to a better reference in a second, but the key point is that AI categories are always at least somewhat untrue. That's the point of Darwinism. Species (named groups of things) are groups that have individual variation within them because that's how they worked, not because they're flawed examples of some perfect design some cloud cuckoo came up with. Nothing has a True Name, but some names are better than others. So if you're trying to put names on everything, the process is going to get screwed up somehow, whether you call it a lie, a mistake, a gremlin, sexism, discrimination, or a really bad idea. Now, about typological thinking without True Name: The book you want to read is Walter Ong's Orality and Literacy, which is about the profound differences in how non-literate people see the world, compared with literate people. The tl;dr is that the idea of True Names comes fully from a world of widespread, routine literacy. Nonliterate people and their languages didn't generalize to anywhere near that degree, because it wasn't useful. If you're paying attention, you know that every kind of apple tastes different. Also, some apple trees produce decent wood for carving, while some are better as firewood. In texts, there's some utility of generalizing them all as "apple," but in real life, it's better to have an immense diversity of words for them. Words allow a level of abstraction that's impossible with only oral language, because people tend to start thinking in terms of the permanent words they see, not the ephemeral language they hear. It's the difference between a god making the world out of a lump of clay, or God speaking and the Words becoming Things. The latter is a Religion of the Book. The former is animistic. This doesn't mean that nonliterate people are stupider than we are (I think the evidence points the other way, actually), but it does mean that thinking without written words is something any literate person is going to struggle with, and that makes for all sorts of erroneous back-projections about how "primitive" people thought. You may think they're primitive, but we're the idiots who pay other people to own computers to know things for us. Anyway, the fundamental problem now is that we're trying to turn reality into words and numbers, make the actual virtual. Of course this abstraction, simplification involves lying and omissions. How could it not? I suspect Schneier's going to add standard rants on training set hygiene to his useful rants about security theater and the dangers of IoT. Both are equally dangerous. If you want to hack virtualization, pick up your umbrella, go surreal, and also follow the protesters who hack surveillance tech. The umbrella is useful as a perfectly innocuous tool for shielding people from sun, rain, pepper spray, marking dyes, and surveillance cameras. Allen Thomson # Allen Thomson replied to this comment from Allen Thomson | March 6, 2021 21:58 | Reply 47: > From the NYT just today: From which and talking about, I think, the logistic curve model, "The rate of progression of this technology is faster than any previous automation," said Mr. Le Clair, the Forrester analyst, who thinks we are closer to the beginning than the end of the corporate A.I. boom. "We haven't hit the exponential point of this stuff yet," he added. "And when we do, it's going to be dramatic." Pigeon # Pigeon replied to this comment from Paul | March 6, 2021 22:01 | Reply 48: Trouble with all those kinds of suggestions is that there is no way to get any reliable data on whether they actually work or not, and it is an area which is an appallingly prolific bullshit generator. Just as we currently have people convinced that spraying your number plate with a relabelled can of hair spray will prevent automatic systems reading it, but nobody knows what the actual subtle alterations that really do have such an effect really are, so we will see a vast array of silly hats and so forth which are all claimed to break the system, almost certainly don't, but do make it vastly more difficult to determine what things really do. Pigeon # Pigeon replied to this comment from Heteromeles | March 6, 2021 22:10 | Reply 49: "The umbrella is useful as a perfectly innocuous tool..." An umbrella is a dastardly and evil invention for poking random people's eyes out, masquerading as a rain deflector. ("Masquerade" is, of course, just a more old fashioned term referring to adversarial inputs.) allynh # allynh | March 6, 2021 22:16 | Reply 50: Speak like Arnold Schwarzenegger and the voice recognition software will understand you. David L # David L | March 6, 2021 22:46 | Reply 51: And that's fine, and it's going to stay fine right up until the moment you find yourself in this elevator ... This is the favorite Youtube video of my neighbor from Scotland. Pigeon # Pigeon replied to this comment from Allen Thomson | March 6, 2021 22:48 | Reply 52: It's as if things have to balance out, so teaching machines to think goes hand in hand with teaching humans not to. And you get the same problem with the supposedly intelligent processing engine becoming trained to operate on cues that have very little to do with the ostensible intended purpose. So you get the customer service centre zombie learning that by copying and pasting some chunk of company boilerplate that contains two or three keywords in common with a customer's complaint, they can tick off the greatest number of complaints with the smallest chance of being shouted at for displaying initiative, while the customer gets closer and closer to detonation point due to repeatedly getting the same stupid answers that don't actually address the matter of the complaint. Not only do the purely machine parts of the system have no common sense, but the elements which could potentially introduce some common sense are in fact being conditioned into keeping the system as a whole devoid of it. To sort this problem out requires both the application of genuine effort, and the existence of a genuine commitment to actually sorting it. Neither is likely to be forthcoming, and I can only be glad not to have been born at any later date than I actually was. Elderly Cynic # Elderly Cynic replied to this comment from Pigeon | March 6, 2021 22:53 | Reply 53: Yes, indeed. I have been abused for holding my arm up, causing an umbrella to deflect and spin instead of sticking into my eye. Leave a comment Here's the moderation policy. If this is your first time, please read it before you post. If you need to sign in and want to create a local account on this blog, select "Movable Type" from the "Sign in ..." menu. You will need a working email address. Name [ ] Email Address [ ] URL [ ] [ ] Remember personal info? [ ] Comments (This form requires JavaScript. You may use HTML entities and formatting tags in comments.) [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [Preview] [Submit] Buy my Books * US English editions * UK English editions * Deutsche Ausgaben * Ri Ben Yu noBen Quick Stuff * FAQ: Who am I? * FAQ: Moderation Policy * FAQ: Why is there no tipjar? * FAQ: Copyright Notice * FAQ: Inviting Charles Stross to speak * FAQ: Fan Fiction * Bibliography and online fiction * FAQ: Laundry Files reading order * Talk to me * Non-blog writing (old) Specials * Common Misconceptions About Publishing--a series of essays about the industry I work in. * How I Got Here In The End --my non-writing autobiography, or what I did before becoming a full-time writer. * Unwirer--an experiment in weblog mediated collaborative fiction. * Shaping the Future--a talk I gave on the social implications of Moore's Law. * Japan: first impressions -- or, what I did on my holidays * Inside the MIT Media Lab--what it's like to spend a day wandering around the Media Lab. * The High Frontier, Redux -- space colonization: feasible or futile? * "Nothing like this will be built again"--inside a nuclear reactor complex. * Old blog--2003-2006 (RIP) Merchandise About this Entry This page contains a single entry by Charlie Stross published on March 6, 2021 11:12 AM. Central Banking on Mars! was the previous entry in this blog. Find recent content on the main index or look in the archives to find all content. Search this blog [ ] [Search] Propaganda [223224] Categories * Administrative (5) * Computers (10) * Gadget Patrol (7) * Humour (12) * News (6) * Politics (15) * Publicity stuff (11) * Publishing (19) * Travel (7) * Writing (37) Monthly Archives * March 2021 (1) * February 2021 (1) * January 2021 (3) * December 2020 (2) * November 2020 (1) * October 2020 (6) * September 2020 (2) * August 2020 (1) * July 2020 (2) * June 2020 (1) * May 2020 (2) * April 2020 (2) Earlier, and other types of archive.