https://www.science.org/content/blog-post/deliberately-optimizing-harm Advertisement * * news * careers * commentary * Journals * Covid-19 Science Science * * * Log in * Become A Member [science] science [sciadv] science advances [sciimmunol] science immunology [scirobotic] science robotics [signaling] science signaling [stm] science translational medicine [spj-cover] science partner journals Quick Search anywhere Enter Search Term[ ] Quick Search in Journals Enter Search Term[ ] Quick Search in Journals Enter Search Term[ ] Quick Search in Journals Enter Search Term[ ] Quick Search in Journals Enter Search Term[ ] Quick Search in Journals Enter Search Term[ ] Quick Search in Journals Enter Search Term[ ] Searching: Anywhere AnywhereScienceScience AdvancesScience ImmunologyScience Robotics Science SignalingScience Translational Medicine Advanced Search Search Trending Terms: * covid-19 * climate * science policy * genome editing * batteries Log In Become A Member Quick Search anywhere Enter Search Term[ ] science.org * Custom publishing * collections * videos * podcasts * blogs * visualizations * prizes and awards * authors & reviewers * librarians * advertisers * about * help * * * * * * AAAS Logo * Terms of Service * Privacy Policy * Accessibility * Commentary Home * Opinion * Analysis * Blogs GET OUR E-ALERTS HomeCommentaryBlogsIn the PipelineDeliberately Optimizing for Harm Back To In the Pipeline * In the Pipeline Deliberately Optimizing for Harm * 15 Mar 2022 * By Derek Lowe * 2 min read * Comments Share: * Twitter * Linked In * Facebook * Reddit * Email This new paper is worrying but not surprising. It describes a recent effort to (mis)use computational drug design software (especially as applied to avoid compounds with predicted toxicity), flipping the signs to have the model generate the most toxic molecules it could come up with. This was part of a biannual conference in Switzerland that looks at the implication of new technologies on chemical and biological weapons threats, and the authors (from Collaboration Pharmaceuticals) were invited to present on how AI/ML approaches could be weaponized in these areas. They were clearly shaken by the experience: The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting. Our work is rooted in building machine learning models for therapeutic and toxic targets to better assist in the design of new molecules for drug discovery. We have spent decades using computers and AI to improve human health--not to degrade it. We were naive in thinking about the potential misuse of our trade, as our aim had always been to avoid molecular features that could interfere with the many different classes of proteins essential to human life. Even our projects on Ebola and neurotoxins, which could have sparked thoughts about the potential negative implications of our machine learning models, had not set our alarm bells ringing. The effort used known LD[50] data to drive the modeling in the known "nerve gas" space, and it sppears to have worked very well. Relatively speaking, that is: In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD [50] values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD[50 ]model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules. It would not surprise me if this modeling work uncovered a number of nondisclosed structures that have been quietly looked at in the past. It seems clear that there's a lot more structure-activity space in this area than has ever been declassified (the appearance of the "Novichok" compounds is one example of this). Now, keep in mind that we can't deliberately design our way to drugs so easily, so we won't be able to design horrible compounds in one shot, either. Just as there are considerations in drug discovery that narrow down these sorts of lead-generation efforts, there are such factors for chemical weapons: stability on storage, volatility (or lack of it), persistance in the environment, manufacturing concerns, etc. So it's not like all of these predicted highly toxic compounds would be militarily useful - but note that we're not just talking about state actors here. A big worry has always been homebrew types (as witness Aum Shinrikyo and their 1995 nerve-gas release in the Tokyo subway system). As it happened, Aum was not particularly competent, but that shouldn't make any of us feel better, because someone else might be. That goes for the various garage-ricin incidents over the years as well - ricin is (fortunately) not all that easy to turn into a weapon, and the people who try to do it are generally somewhat disconnected from reality (and also from technical proficiency). But there's always that worrisome part of the Venn diagram where competence and evil intent overlap. It hit me in the aftermath of 9/ 11 (during that anthrax period) that I could personally think of a whole list of horrible possibilities for chemical and biological terrorism threats. I'd never thought of such things, but a bit of sustained thought brought all sorts of stuff to mind. That must have been a similar feeling to what the authors of this paper experienced. And while I'm fairly imaginative, I don't think I'm particularly diabolical. What, I wondered, about people who are a bit of both? As mentioned, I'm not surprised at all that computational methods can point the way to such things, although I can see how the authors of this work found demonstrating this to be an unsettling experience. To be honest, I'm not all that worried about new nerve agents, at least not compared to my level of worry about the existing ones for which there is already a wide base of knowledge. That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us. One factor (mentioned in the paper) is that new compounds and new routes might be able to circumvent chemical controls and watch lists, though. The paper discusses ethics training, international agreements and guidelines, pledges of responsibility and so on. That's all fine, but history demonstrates that anyone truly interested in using such things will care nothing for these constraints. I feel about these the way that I felt about the "No First Use" pledge on nuclear weapons - that it guaranteed that the world could only be blown up by a liar. Thin comfort. So we should be vigilant about potential misuse of these technologies, but at the same time we shouldn't imagine that this will be enough. About the author Derek Lowe Derek Lowe emailTwitter Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases. --------------------------------------------------------------------- Comments Please enable JavaScript to view the comments powered by Disqus. IN THE PIPELINE Derek Lowe's commentary on drug discovery and the pharma industry. An editorially independent blog, all content is Derek's own, and he does not in any way speak for his employer. Advertisement YOU MAY ALSO LIKE 7 Feb 2022By * Derek Lowe Omicron Boosters and Original Antigenic Sin 4 Mar 2022By * Derek Lowe An ALS Protein, Revealed 19 Nov 2021By * Derek Lowe Coronavirus Vaccines and Cancer 25 Feb 2022By * Derek Lowe How Much Published Crap Will We Put Up With? View More Advertisement ARCHIVES Select Year: [2020] [January ] View Posts Related Jobs Advertisement RecommendedClose September 1989 A Near-Optimum Parallel Planarization Algorithm 19 Aug 2014By * Derek Lowe Don't Optimize Your Plasma Protein Binding August 2018 Optimizing drug combinations against multiple myeloma using a quadratic phenotypic optimization platform (QPOP) 25 Aug 2020By * Derek Lowe Preparing For the Vaccine Results Skip slideshow Follow Us * * * * * * * NEWS * All News * ScienceInsider * News Features * Subscribe to News from Science * News from Science FAQ * About News from Science * CAREERS * Careers Articles * Find Jobs * Employer Profiles * COMMENTARY * Opinion * Analysis * Blogs * JOURNALS * Science * Science Advances * Science Immunology * Science Robotics * Science Signaling * Science Translational Medicine * Science Partner Journals * AUTHORS & REVIEWERS * Information for Authors * Information for Reviewers * LIBRARIANS * Manage Your Institutional Subscription * Library Admin Portal * Request a Quote * Librarian FAQs * ADVERTISERS * Advertising Kits * Custom Publishing Info * Post a Job * RELATED SITES * AAAS.org * AAAS Communities * EurekAlert! * Science in the Classroom * ABOUT US * Leadership * Work at AAAS * Prizes and Awards * HELP * FAQs * Access and Subscriptions * Order a Single Issue * Reprints and Permissions * Contact Us Follow Us * * * * * * AAAS logo (c) 2022 American Association for the Advancement of Science. All rights reserved. AAAS is a partner of HINARI, AGORA, OARE, CHORUS, CLOCKSS, CrossRef and COUNTER. back to top * Terms of Service * Privacy Policy * Accessibility