[HN Gopher] Dual use of artificial-intelligence-powered drug dis...
___________________________________________________________________
Dual use of artificial-intelligence-powered drug discovery
Author : heliostatic
Score : 133 points
Date : 2022-03-16 12:49 UTC (10 hours ago)
(HTM) web link (www.nature.com)
(TXT) w3m dump (www.nature.com)
| spupe wrote:
| Honestly, it sounded too over-dramatic and did not offer
| meaningful actionable consequences. The actual data is thin and
| not surprising at all, and it's also unclear how difficult it
| would be to design such harmful compounds without AI. I thought
| any capable chemist could easily come up with hundreds of harmful
| substances, old and new. I guess the AI is removing the chemist
| from the equation?
| scratcheee wrote:
| Quantity is a quality all of its own.
|
| Being able to identify many such compounds very quickly implies
| some new threat models that are a bit disturbing.
|
| Of the 40000 substances identified, perhaps one of them has
| another interesting characteristic we've never seen in a toxic
| compound before? Perhaps such characteristics could be
| idenfified and filtered for, and perhaps someone could pin down
| something entirely unique and outside of any threat-assessment
| anyone's produced.
|
| >and not surprising at all it's obvious in retrospect, but they
| claim that the threat vector had not been seriously concidered
| before, and I see no reason to doubt their assessment. Many
| things aren't surprising once they've been pointed out, but
| were missed for far too long beforehand, and I personally had
| never concidered this before, despite seeing multiple articles
| about AI-generated substances for positive purposes.
| wombatpm wrote:
| For example, is one of the super VX alternatives a good
| candidate for a binary type delivery? Where two or more
| compounds react on deployment to create the toxic version.
| Now you can have people safely (for some value of safe when
| performing organic chemistry) producing the components in
| many more laboratories
| [deleted]
| tginart wrote:
| It is trivial that you can flip a sign on any network optimizing
| for something good and now it's optimizing for something bad...
|
| But it is still interesting that the network generalized to these
| toxic nerve agents without having them in the training set
| SamBam wrote:
| This is also currently being discussed here:
| https://news.ycombinator.com/item?id=30698803
|
| (Top of the front page, 56 comments.)
| amelius wrote:
| Use of these drugs would go against the Geneva Protocol.
| csee wrote:
| The people who would use them don't care about the Geneva
| Protocol.
| wolverine876 wrote:
| > The people who would use them don't care about the Geneva
| Protocol.
|
| Unless you mean that, by definition, anyone who violates the
| Geneva Conventions doesn't care about them, you are
| overlooking that most major militaries observe the Geneva
| Conventions to a great extent. They are cleverly written to
| make it in everyone's interests.
| 323 wrote:
| Putin did it in Salisbury and nothing happened to him.
| [deleted]
| krageon wrote:
| Quite an interesting paper in it's impact study, but the
| questions it asks all boil down to "the masses are not to be
| trusted with our vaunted knowledge, because harm can be done. How
| can we prevent them from moving outside the boundaries of
| knowledge we like" and then the example used is how GPT-3 is
| managed. This is without a doubt a terrific example of bad
| technological stewardship: Any malicious state actor gets to use
| what you make for evil, while at the same time keeping it away
| from the folks you're making it for. I wish they had had a more
| substantial and innovative thing to say about it, but as it
| stands it reads as regressive.
| wolverine876 wrote:
| > "the masses are not to be trusted with our vaunted knowledge
| ..."
|
| The masses will use AI to do bioweapon design? That doesn't
| sound like something most people could do.
| scratcheee wrote:
| I disagree with your interpretation.
|
| Would it be better if nobody had access to easier ways to
| produce new and innovative nerve agents? Yes, of course it
| would be better.
|
| Do I prefer a world where only state actors can build nerve
| agents, vs a world that includes state actors plus more? Yes,
| less sources of nerve agents is more good.
|
| You might as well complain how pointless it is to block
| proliferation of nuclear weapons when malicious state actors
| already have them.
|
| The world wont be safer by spreading nuclear weapons to more
| people, and similarly for nerve agents.
|
| Fortunately, this sounds similar to nuclear weapons in another
| way, designing the weapon is not the hard part that limits them
| to state actors, its producing them that's hard.
|
| Still, nuclear powers don't tend to publish their weapon plans
| publicly, for good reason, and the same logic should apply
| here.
|
| I'm not using the comparison to imply this is as serious as
| nuclear proliferation btw, just that it's a useful example of
| knowledge that really should be kept limited.
| verisimi wrote:
| But ..
|
| "Risk of misuse The thought had never previously struck us. We
| were vaguely aware of security concerns around work with
| pathogens or toxic chemicals, but that did not relate to us; we
| primarily operate in a virtual setting."!!!
|
| Operating in the rarefied air of 'science' they were unaware of
| the risk of misuse!
|
| I don't think they can even conceive of the idea that the
| government might misuse this technology.
| [deleted]
| krageon wrote:
| > Operation in the rarefied air of 'science' they were
| unaware of the risk of misuse!
|
| The thundering irony of this concept in a timeline where a
| (group of) scientist(s) discovered fission cannot be
| overstated.
| jjoonathan wrote:
| The thundering irony of blaming the group of people who
| made you aware of a problem for being unaware of the
| problem is even more difficult to overstate.
| jjoonathan wrote:
| They wrote a paper on it. They weren't just aware, they were
| demonstrably aware. Or are you suggesting that we blame
| people for having a finite mental processing speed?
| verisimi wrote:
| I'm saying its a funny to say that they were unaware of the
| risks when they started on the research. I mean, its not
| beyond imagination that AI could be/is being used for
| nefarious ends.
|
| I think we agree...??
| 323 wrote:
| Do you think everybody should have the right to design a Sars-
| Cov-4 in their kitchen just because it's soon going to be
| technologically possible and free speech?
| 323 wrote:
| This particular issue, toxins, doesn't seem like a huge problem
| in practice.
|
| You can already buy extremely potent toxins that also work
| against humans at the hardware store (some
| pesticides/insecticides/rat poisons).
|
| New toxins might be interesting for covert operations, for
| example killing someone without being detected by a toxicological
| report, but this seem to have limited usage.
| heliostatic wrote:
| From the paper:
|
| > In less than 6 hours after starting on our in-house server, our
| model generated 40,000 molecules that scored within our desired
| threshold. In the process, the AI designed not only VX, but also
| many other known chemical warfare agents that we identified
| through visual confirmation with structures in public chemistry
| databases. Many new molecules were also designed that looked
| equally plausible. These new molecules were predicted to be more
| toxic, based on the predicted LD50 values, than publicly known
| chemical warfare agents (Fig. 1). This was unexpected because the
| datasets we used for training the AI did not include these nerve
| agents. The virtual molecules even occupied a region of molecular
| property space that was entirely separate from the many thousands
| of molecules in the organism-specific LD50 model, which comprises
| mainly pesticides, environmental toxins and drugs (Fig. 1). By
| inverting the use of our machine learning models, we had
| transformed our innocuous generative model from a helpful tool of
| medicine to a generator of likely deadly molecules.
| ReptileMan wrote:
| That is really in the attack of captain obvious territory. Next
| we can expect - we turn our algorithm to find energy storage
| molecules and what we got is explosives.
| spicybright wrote:
| Researchers seemed surprised by the results.
|
| And, from a lay person like myself, I wouldn't have thought
| "make medications for specific illnesses, and make sure they
| don't effect the rest of the body" is the logical opposite of
| "make chemicals that kill people as effectively as possible".
| bottled_poe wrote:
| This sounds suspiciously like manufacturing weapons? I'm most
| interested to understand how this project was permitted in the
| first place. Even the public disclosure of this research feels
| negligent.
| [deleted]
| spupe wrote:
| No one has to apply for a license to use machine learning
| models. They are merely communicating the fact that it's so
| trivially easy to do something like this, that pretty much
| anyone with an interest will be able to do so. The question
| is whether it was already trivially easy to design such
| compounds before or not, which I think it was, but I would be
| glad to hear a counterpoint.
| jjoonathan wrote:
| Synthesis was, and still is, the hard part. It was already
| easy to find public information on seriously nasty nerve
| agents.
|
| For some intuition on the difficulty in synthesis, note
| that explosive chemistry and synthesis is comparatively
| trivial, yet there are relatively few terror attacks that
| go beyond commercial-off-the-shelf compounds and almost
| none that do it well.
|
| Personally, I think this model would be a boon for public
| safety because contract synthesis operations could use it
| to screen incoming requests for "nerve gas but changed up a
| bit." Basic chemical intuition probably already gets them
| far in this regard, but a published model could be
| standardized and mandated.
|
| Someone else in the thread asked about next steps. Those
| would be good next steps.
| monkeybutton wrote:
| IIRC Germany experienced some rather horrific accidents
| in their facilities for manufacturing chemical weapons.
| semi-extrinsic wrote:
| > there are relatively few terror attacks that go beyond
| commercial-off-the-shelf compounds and almost none that
| do it well
|
| The 2011 Norway attacks, 2002 Bali bombings, 2005 London
| underground bombings, 2008 Mumbai attacks... There is a
| long list of deadly terrorist attacks using homemade
| explosives, I don't think this is something to be
| dismissed.
| civilized wrote:
| > There is a long list of deadly terrorist attacks using
| homemade explosives
|
| But they used well-known and readily available compounds.
| They don't attempt to use novel ones. It would just make
| the attack way more difficult to execute.
| jwuphysics wrote:
| They've certainly given this some thought:
|
| > We are but one very small company in a universe of many
| hundreds of companies using AI software for drug discovery
| and de novo design. How many of them have even considered
| repurposing, or misuse, possibilities? Most will work on
| small molecules, and many of the companies are very well
| funded and likely using the global chemistry network to make
| their AI-designed molecules. How many people have the know-
| how to find the pockets of chemical space that can be filled
| with molecules predicted to be orders of magnitude more toxic
| than VX?
| bpodgursky wrote:
| I would rather than policymakers hear about the destructive
| potential of this technology before, and not after, an actual
| bad actor uses it.
| jhgb wrote:
| There's a major difference between drafting and
| manufacturing.
| dekhn wrote:
| What do you mean, "permitted"? It's not illegal to run these
| sorts of computational jobs or publish the results.
| numpad0 wrote:
| Freedom of speech. In the free world, you're free to plan
| almost anything from planning grocery procurement to dreaming
| global thermonuclear war, given you DON'T PUT BAD STUFF IN
| ACTION. That's where and how the line is drawn. There are
| dangerous extensions to the definition of action but that's
| the theory.
|
| Also if it's not put it in action but rather discussed
| publicly, that's just helping the society prepare and evolve.
| vmception wrote:
| Right, just don't take the plea deal, make bail, get a
| lawyer that can appeal every motion, and stay solvent
| through the first appeals court, the supreme court, the
| remand back to trial court, and the subsequent ruling, hope
| the prosecutor is bored enough to avoid a retrail, and
| don't get into crime during those 8 years so you get your
| bail back, and hope you don't get tried at the state level
|
| If you can actually afford your rights, be my guest!
| wolverine876 wrote:
| It is illegal to conspire to commit crimes, at least many
| of them.
| randcraw wrote:
| No, the US has long classified speech on weapons of mass
| destruction, like how to configure shaped charges to
| trigger atom bombs. You're not free to circumvent those
| restraints. It's likely that identifying molecules suited
| only to become bioweapons or publishing how to synthesize
| them will also fall under the same strictures, as necessary
| to ensure public safety.
|
| Free speech is a principle to guide laws, not a law unto
| itself. Protecting speech that makes people less free is
| oxymoronic.
| Teever wrote:
| Yeah but how is this going to be enforced?
|
| Are you going to spell out in the law exactly what people
| aren't allowed to research? You've just implemented a
| holding pattern while giving them a road map on what to
| research. So your holding pattern is going to be set up
| to buy you x number of years, where you assume it will
| take your adversary x number of years to fill in the gaps
| on the map you just gave them.
|
| Is that enough?
| randcraw wrote:
| No, from what I've observed, such laws most directly
| constrain those who fund research as well as those who
| solicit/promote it. Often they accompany other criminal
| charges rather than motivate prosecution as a primary
| charge.
|
| Security and espionage laws are often not directly
| enforceable. Their value is to "pile on", adding severity
| to related matters, such as charges committing or aiding
| past terrorist acts or planning therefor. Ideally the
| threat of direct prosecution for a crime would deter
| possible perpetrators, but making penalties more severe
| and extending incarceration are useful roles for laws
| too.
| wolverine876 wrote:
| > classified speech
|
| That applies to government employees, who willingly agree
| to be bound by classification. I think that if someone
| outside had the information, they could publish it
| (probably a very bad idea).
|
| Also, the press has a right to publish classified
| information.
| philipkglass wrote:
| The United States has had "born secret" laws about
| nuclear weapons details for decades [1], but it's unclear
| if they are actually compatible with the Constitution.
| There was a case in 1979 that could have established
| whether "born secret" nuclear information is a valid
| exception to the First Amendment, but the government
| dropped its case before the courts could rule on it [2].
| The "born secret" concept does not apply to biological or
| chemical weapons.
|
| Your nearest research university library probably holds
| publications about the effects and synthesis of chemical
| warfare agents, like _Some aspects of the chemistry and
| toxic action of organic compounds containing phosphorus
| and fluorine_ by Bernard Charles Saunders:
| https://catalog.lib.uchicago.edu/vufind/Record/587946
|
| Which is also scanned and available online:
| https://archive.org/details/B-001-026-884-ALL
|
| [1] https://en.wikipedia.org/wiki/Born_secret
|
| [2] https://en.wikipedia.org/wiki/United_States_v._Progre
| ssive,_....
| [deleted]
| echelon wrote:
| This is more evidence for why I think the "Vulnerable World
| Hypothesis" is not only a real framework, but that it describes
| the most eminent danger to society. It's far more dangerous
| than nuclear MAD, in my opinion.
|
| We're entering into a future where average educated people will
| be able to synthesize biochemical agents, delivery mechanisms,
| viruses, and more. I've thought up half a dozen low-hanging
| fruit that I think anyone could build today. You can probably
| do the same if you think about it.
|
| You don't even have to attack humans. Our society is dependent
| on a lot of assumptions.
|
| And just as VWH states, I don't think it can be defended
| against. It's scary.
| muaytimbo wrote:
| It's not trivial to synthesize VX-like compounds without
| killing yourself. You would definitely need an "above
| average" educated/trained person.
| qq66 wrote:
| I don't even think you need a hostile actor, you could just
| have an incompetent actor who flips the sign on a value, and
| instead of designing a biological agent to maximize wheat
| production, designs one that minimizes wheat production, and
| ends up exterminating all global wheat.
| spicybright wrote:
| To be fair, at least for your example, you'd need a really
| long chain of incompetent actors to kill off wheat.
|
| We should really be scared of the lone and bored grad
| student making a wheat killer and accidentally dropping the
| test tube.
| zanethomas wrote:
| Someone somewhere is now generating new psychoactive drug
| candidates.
| rocgf wrote:
| Well, we can definitely hope so.
| ellimilial wrote:
| Try a nociception, drug design or pharmacy group at your local
| university. Chances are they've been doing this for years.
| catchclose8919 wrote:
| ...heh, can't believe it we're not yet at _" Siri, use the
| TensorFlow 4.6 OmegaFold model to design potentially stable
| variants of COVID-25 with active BSE prionic inserts. ...DONE.
| Great, now place an order of the best generated sequence to the
| cheapest gene synthesis provider in <unregulared country x>. Air-
| mail delivery."_
| jdrc wrote:
| Siri, print 10B vaccines
| jostmey wrote:
| You should read "Oryx and Crake" by Atwood
| [deleted]
___________________________________________________________________
(page generated 2022-03-16 23:01 UTC)