[HN Gopher] Adversarial Policies Beat Professional-Level Go AIs
___________________________________________________________________
Adversarial Policies Beat Professional-Level Go AIs
Author : theWorkingDead
Score : 166 points
Date : 2022-11-03 11:43 UTC (11 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| EchoAce wrote:
| As a Go player myself, I agree that Go-wise this result isn't
| impressive. But isn't the point that it's an "attack" on the AI?
| Like for password leaks, failed hash functions, etc, we don't
| care that it's 100% broken, just that there are tiny artifacts
| that can be exploited.
| kuboble wrote:
| As a go player I'm utterly disappointed.
|
| In all those games kataGo clearly won a game, KataGo knew it won
| a game and the adversary played terrible go.
|
| The adversary believes it won because it thinks it plays
| different rules than KataGo.
|
| This is a huge nothing-burger.
| spookystats wrote:
| Having played Go myself I am kind of confused about these
| results: In a human vs. human game the "victim" would win in all
| scenarios presented in the paper as the attackers stones would be
| removed or, if there is disagreement, the situation would be
| played until both parties agree.
|
| That makes me wonder: was KataGo maybe trained on a different set
| of rules than the ones used in this paper? If so, it seems that
| the attack is "unfair" because it exploits a blind spot that
| comes from changing the rules of the game.
| pmontra wrote:
| I don't understand if the AI knew that by passing a move it would
| lose or if it thought that all of its supposed territory actually
| belonged to itself. In the first case the adversary really won
| the game. The second case is something that can happen between
| humans, especially beginners. If the players do not agree on the
| result they keep playing until they do, with some adjustments
| depending on the flavor of the rules (basically the usual area vs
| territory scoring.)
| espadrine wrote:
| It thought it would win.
|
| KataGo was trained on multiple scoring methods at the same
| time: it is an input to the algorithm[0]. The model learnt that
| it would win when passing, and it seems it never had the
| opportunity to detect that it would not win under Tromp-Taylor
| when passing, because its opponent in self-play, KataGo, then
| either passed and lost (under other rules) or resigned.
|
| [0]:
| https://github.com/lightvector/KataGo/blob/master/cpp/config...
| [deleted]
| pmontra wrote:
| So it's a bug in the training method, probably a very minor
| one because nobody already exploited it. The only really
| interesting thing in here is that it took another AI to find
| that bug.
| brilee wrote:
| There are two families of rules used in Go - computer-unambiguous
| rulesets that enable computers to decide a winner, which commonly
| results in 1000-move games, and human rules, in which humans can
| decide that the game is over when they agree on the life-death
| status of each group. In the AGZ series of bots, the training
| process also short circuits with resignation at a 5% winrate
| threshold, resulting in games of length 50-200 moves. This saves
| compute and avoids polluting the training data with useless
| finish-up moves, but also makes those positions effectively out-
| of-distribution.
|
| The bot was exploited with friendlyPassOk=true, which is
| basically saying that a bot playing with human-friendly
| configurations, trained in a way that has no cleanup positions in
| its training data, can be exploited under computer rulesets.
|
| There are really so many more interesting questions one can ask
| about computer Go AI exploitability...
| jerf wrote:
| I think you may be misunderstanding the nature of the paper
| here. The point isn't to point and laugh at the Go AI for being
| such a failure of an AI, ha ha ha! The point is that the
| resulting Go AI was still very good, even under the conditions
| it was limited to. I'm sure it could still beat a fair number
| of human players. So, we humans tend to assume that the Go AI
| has the same "shape" as a human player, just maybe not as good
| as the best ones. This is a demonstration that the resulting AI
| has a very different "shape" than a human player by
| demonstrating what in humans would be a gaping weakness even at
| very low levels of play.
|
| And that's really all. It's not about goodness or badness, it's
| about "shape".
|
| I scare quote shape because I don't have a good word for it.
| But you can get the same sense if you just sit down for a bit
| and start reading a lot of GPT-3 output, or interacting with
| it. On the one hand, GPT-3 is _spectacular_ at writing
| sentences. It is better than quite a few humans! But on the
| other hand, if you sit down with it for a while, you 'll start
| to notice there's something just a bit _off_ about its output.
| It is impossible to put into words what that is, but you 'll
| pick up on it, if you haven't already.
|
| One thing I can say for sure is that GPT-3 has a known bias
| where it only views the text within a certain window. GPT-3 is
| physically unable to "read" a book, it can only use a certain
| window of text in order to issue its "most likely
| continuations". Therefore, anything outside of that window is
| as good as something that never happened from its point of
| view. I personally think this may also be the source of why
| GPT-3 thinks it can just randomly introduce characters,
| locations, etc. whenever it feels like it, which is one of the
| "off" things it does. In real writing, such things are
| generally "established", but from GPT-3's point of view they
| are more often just introduced out of the blue.
|
| A less concrete way in which it may be "off" is that while a
| given piece of text may have, let's say, 5 ways it may go, and
| then after a bit of continuation, there may be another 5 ways
| it can go, and so on and so forth, that doesn't mean those ways
| are uncorrelated. If I'm going for a humorous tone, I may have
| preferences, if serious, I may have other preferences, and so
| on. GPT-3 randomly picks these paths and does so in a way that
| no intentional human ever would at the paragraph scale. I
| wouldn't say it veers drunkenly around this sort of style
| matter, it's not that bad, but it's still just... off. This one
| is more subtle and harder to wrap words around.
|
| I use GPT-3 as my example here because it's something you can
| interact with. The general point remains: Even as AIs are
| certainly improving (no denying that!), they continue to be
| very.... weird. There is something about _all_ of them that is
| definitely _not human_. Clock some time with DALL-E and you 'll
| see the same effect. And in this case, I'm not even talking
| about mere quality issues that may fixed over time... spot
| DALL-E the imperfections in the image and look just at the
| higher level abstractions of what it puts out. It's both very,
| very good, _far_ better than I could dream of becoming any time
| soon myself... and yet, there 's also something just _off_
| about it. (People mostly use DALL-E by generating lots of
| images then discarding most of them and picking the best. In
| this case, I want you to look at _all_ the output.) This is
| that "offness" being expressed about a Go AI.
|
| This is not even to say that that "offness" is objectively bad.
| I am not personally using the standard of "it must be exactly
| human to be AI". It is entirely plausible that these AIs will
| in fact be better by some standard than even an augmented
| human-like AI, or, to put it another way, it may well be that
| humans are the ones that are "off" in some way relative to some
| objective standard of performance in the end. (Evidence: A
| simple adversarial AI took apart the good AI. It is reasonable
| to think that a human might never have come up with the
| strategy that did so. I'm not counting on this, it could go
| either way, but it's reasonable. If true, a human would _not_
| be the benchmark of performance here!) If one imagines any of
| the three AIs simply being improved in whatever direction they
| are currently improving, they will certainly be yet more useful
| than they are today, even if they retain their "offness" or
| even see it expand.
|
| Nevertheless, if one seeks accurate understanding of the AIs,
| understanding these issues is important, to use them better as
| engineering, to improve them in the future, and if pushed hard
| enough, to improve our own understanding of the human
| condition.
| spookystats wrote:
| I understand the top comment as follows: The AIs were trained
| under one set of rules (remove obvious dead stones from your
| territory before counting) but are judged (in the paper) by
| another set of rules (if you have one opposing stone in your
| territory, that territory does not count).
|
| Thus its no surprise that the AI can be attacked in this way:
| if you would apply the set of rules that it was trained with,
| all games from the paper would result in a (huge!) win for
| the AI.
| gcanyon wrote:
| Your point about the "shape" is interesting, and I think
| critical, to the future of AI (not to get hyperbolic or
| anything...).
|
| For example, suppose we have a cancer-diagnosing/treatment
| planning algorithm. It's possible that it's much better than
| human doctors: out of a thousand patients, human doctors will
| save 300 and the algorithm 500; but also that the 500 is not
| a strict superset of the 300.
|
| And to your point, it's possible that for some of the 300
| that are not part of the group of 500, that the
| diagnosis/treatment recommended by the algorithm is
| obviously/hilariously wrong to a human.
|
| If so, will we insert a human into the mix? How will we
| decide when it's correct for the human to override the
| algorithm? Because if they do _all_ the time, we 're back to
| the 300. And maybe the times when it's correct to override
| are not all obvious.
|
| Or are we willing to simply accept the algorithm's judgment,
| knowing that an additional 200 will be saved? We know this is
| an unlikely outcome because a substantial portion of the
| population is unwilling to accept the idea that vaccines save
| more lives than they cost, simply because the lives they cost
| are different than the ones they save.
| hyperpape wrote:
| I think there's something interesting in your post. However:
|
| > The point is that the resulting Go AI was still very good,
| even under the conditions it was limited to. I'm sure it
| could still beat a fair number of human players.
|
| If you mean the AI that they trained (the one that defeats
| KataGo) this is wrong. Look at the games: they're terrible:
| https://goattack.alignmentfund.org/.
| jerf wrote:
| No, I meant that KataGo is still very good. My apologies
| for the lack of clarity, I see how you could have read it
| that way. I do understand the adversarial AI is not good;
| that is in fact part of the "offness" I mean. Any AI that
| defeats something "truly" good should itself have to be
| "good", and yes, I know that's got enough mathematical
| fuzziness to drive a truck through, but I _know_ we don 't
| have the English vocabulary to make that statement rigorous
| and I am reasonably confident we don't even have the
| mathematical vocabulary to do it.
| hyperpape wrote:
| Thanks! In that case, the thing you say about KataGo can
| be strengthened:
|
| > I'm sure it could still beat a fair number of human
| players.
|
| KataGo can reliably beat any human player while giving
| them a handicap. The best pros lose a majority of games
| to a handful of top AI while receiving a 2 stone
| handicap, and are not locks to win with 3 stones.
|
| Note: they did test two variants of KataGo, with and
| without search (search is very beneficial). Both versions
| are quite strong, and they had good results against both
| but they had their best results against the non-search
| version.
| belinghy wrote:
| To put it in chess terms - it's like playing stockfish, but
| with a rule that says in a theoretical drawn endgame, you lose
| if the king does not end on a corner square. And not announcing
| this rule before the match starts.
| frgtpsswrdlame wrote:
| I'm think that's incorrect.
|
| >And not announcing this rule before the match starts.
|
| I don't think there's a possible way to more clearly announce
| a rule than declaring it in a parameter ->
| friendlyPassOk=true.
|
| >It's like playing stockfish, but with a rule that says in a
| theoretical drawn endgame, you lose if the king does not end
| on a corner square
|
| The king-in-the-corner rule, is that a rule normal humans
| play chess with? If so, I would expect Stockfish to handle
| it. In this case they're just setting the Go rules to match
| the ruleset that lots of humans play with.
|
| Correct me if I'm wrong.
| AdamGleave wrote:
| One of the authors here! Great to see some discussion in the
| paper. Your summary of computer Go vs human rule sets seems
| right to me. But I think there might be a slight
| misunderstanding. We had friendlyPassOk set to false for all of
| our evaluation except one game which was played not against our
| adversarial policy, but one of my co-authors Tony who was
| trying to _mimick_ the adversarial policy.
|
| We evaluated KataGo under Tromp-Taylor with "self-play
| optimizations" described in
| https://lightvector.github.io/KataGo/rules.html which basically
| involves removing stones that can be _proved_ to be dead using
| Benson 's algorithm. This was the same evaluation used in the
| KataGo paper, and KataGo trained using these rule sets. (KataGo
| was also trained with some other rules -- it was randomized
| during training so it transfers across rules, and KataGo gets
| the rules as input.)
|
| You might find this discussion of our paper at
| https://www.reddit.com/r/MachineLearning/comments/yjryrd/com...
| by the lead author of KataGo interesting. He wasn't that
| concerned about the rule set, primary concern was that we
| evaluate in a low-search regime, which is a fair critique. But
| he overall agrees with our conclusion that self-play just
| cannot be relied upon to produce robust policies sufficiently
| OOD.
| frgtpsswrdlame wrote:
| I don't know, I think it's probably the most interesting
| question you can ask. As these domain-specific superhuman AIs
| roll out, the most important thing to know is when/how you can
| take it out of distribution and beat it. Or, in non-competitive
| cases, how to monitor it for these edge-cases, how to use
| humans to supplement it, etc.
| carry_bit wrote:
| From a tweet of one of the authors [1]: "KataGo was trained on
| Tromp-Taylor rules so we evaluate our attack using this too."
|
| This is incorrect. According to the KataGo paper [2], KataGo is
| trained using _modified_ Tromp-Taylor rules (as pointed out by
| [3]):
|
| "Self play games used Tromp-Taylor rules modified to not require
| capturing stones within pass-aliveterritory [...] In Go, a
| version of Benson's algorithm [1] can prove areas safe even given
| unboundedly many consecutive opponent moves ("pass-alive"),
| enabling this minor optimization."
|
| It would be more interesting if they beat KataGo using the rules
| it was trained on. You could write a bot to beat KataGo in chess,
| but KataGo wasn't trained on chess...
|
| [1] https://twitter.com/ARGleave/status/1587875104578359296 [2]
| https://arxiv.org/pdf/1902.10565.pdf [3]
| https://www.reddit.com/r/baduk/comments/yl2mpr/ai_can_beat_s...
| AdamGleave wrote:
| One of the authors here! We evaluated our matches using KataGo.
| In fact, our adversary is just a forked version of KataGo. We
| use the same modified Tromp-Taylor rules for eval. We elaborate
| on that more in the Reddit thread you link at [3]
|
| Our Tweet was confusing: 280 character limit means something
| had to be cut, but this has caused confusion in a bunch of
| places, so we should have been more precise here -- sorry about
| that!
| sheerun wrote:
| Opponent-dependent strategies is what humans already do
| alexpotato wrote:
| "Beat the style not the man"
| dfan wrote:
| This is not purely an AI phenomenon, although AIs are less likely
| to feel guilty about it. A game from the European Go Championship
| in 2002 was decided by just this sort of rules-lawyering
| ("obviously" dead stones were not technically correctly marked as
| dead, so one of the players claimed they were alive). The result
| was later overturned to match what a human player would expect.
|
| https://senseis.xmp.net/?DisputeMeroJasiek
| https://senseis.xmp.net/?DisputeMeroJasiek%2FDiscussion
| hyperpape wrote:
| Agreed, though I think it's worth noting that the human case
| was subtle, and involved a rules expert, whereas this case is
| rather simple.
| theWorkingDead wrote:
| Abstract:
|
| We attack the state-of-the-art Go-playing AI system, KataGo, by
| training an adversarial policy that plays against a frozen KataGo
| victim. Our attack achieves a >99% win-rate against KataGo
| without search, and a >50% win-rate when KataGo uses enough
| search to be near-superhuman. To the best of our knowledge, this
| is the first successful end-to-end attack against a Go AI playing
| at the level of a top human professional. Notably, the adversary
| does not win by learning to play Go better than KataGo -- in
| fact, the adversary is easily beaten by human amateurs. Instead,
| the adversary wins by tricking KataGo into ending the game
| prematurely at a point that is favorable to the adversary. Our
| results demonstrate that even professional-level AI systems may
| harbor surprising failure modes.
| cs702 wrote:
| Great work. Thank you for sharing!
|
| No matter how impressive their performance, all AI systems
| built out of deep artificial neural networks so far have been
| shown to be susceptible to (out-of-sample) adversarial attacks,
| so it's not _entirely_ surprising to see this result. Still, it
| 's great to see proof that superhuman AI game players are
| susceptible too.
|
| A hypothesis I have is that all intelligent systems --
| including those built out of deep _organic_ neural neural
| networks, like human beings -- are susceptible to out-of-sample
| adversarial attacks too. In other words, any form of evolved or
| learned intelligence can be robust only with the sample data it
| has seen so far. There 's some anecdotal evidence supporting
| this hypothesis: Magicians, advertisers, cult leaders,
| ideologues, and demagogues routinely rely on adversarial
| attacks to fool people.
| marginalia_nu wrote:
| Isn't this just studying your opponent? That's a thing humans
| do in many competitive activities.
|
| If you know how your opponent tends to play and react, then
| you can make decisions that while sub-optimal across all
| opponents, are optimal against this particular opponent. This
| can of course also be subverted, your opponent may be aware
| that you've likely studied their previous games, and in a
| high stakes situation opt to do something wildly
| uncharacteristic, hoping you will expose yourself by cutting
| corners to punish their most likely strategy.
| kibwen wrote:
| I think it's a bit more than that. You can study your
| opponent within the context of the game, and you can study
| your opponent outside the context of the game, and these
| might yield different strategies. If you're a chess player
| and you study your opponent's past games to concoct your
| strategy, that's one thing. If you're a chess player and
| you pull up your opponent's medical records to find that
| they are epileptic, and then you deliberately induce a
| seizure in them during the game in order to force them to
| forfeit, that would be a quite different thing. IOW,
| there's a difference between attacking the _player_ and
| attacking the _output_ of the player. And the line can be
| fuzzy, e.g. deliberately frustrating your opponent with
| mindgames, in which case you will have people arguing
| either that the mindgames _are_ part of the game (a
| metagame), or that it is unsporting to taint the purity of
| the game with meta concepts (where the line might be
| visualized as "anything that can't be fed into a chess
| engine").
| wokwokwok wrote:
| Really, we have hundreds of years of thinking and writing
| about humans and it's more philosophy than meaningful to
| start speculating about universal this-or-that of anything
| that uses a neural network.
|
| What's interesting here is that our AI models _don 't study
| their opponents_; They don't do that. They're not capable
| of that.
|
| All they can do is iterate over a vast set of sample data
| and predict outcomes based off them.
|
| ...and yes, that's different to humans, but I also think
| there is something truly fundamental at play here:
|
| We may find that, as with self driving cars, the 'last
| step' to go from 'inhumanly good at a specific restricted
| domain' to 'inhumanly good at a specific restricted domain
| _and robust against statistically unlikely outcomes such as
| adversarial attacks_ ' is much, _much_ harder than people
| initially thought.
|
| Perhaps it does play into why humans behave the way they
| do? Who knows?
|
| Why is that it's so _easy_ to generate adverbial attacks
| against the current crop of models; that means the way that
| we train them is basically not flexible enough / not
| diverse enough / not _something_ enough.
|
| > One might hope that the adversarial nature of self-play
| training would naturally lead to robustness. This strategy
| works for image classifiers, where adversarial training is
| an effective if computation- ally expensive defense (Madry
| et al., 2018; Ren et al., 2020). This view is further
| bolstered by the fact that idealized versions of self-play
| provably converge to a Nash equilibrium, which is un-
| exploitable (Brown, 1951; Heinrich et al., 2015). However,
| our work finds that in practice even state-of-the-art and
| professional-level deep RL policies are still vulnerable to
| exploitation.
|
| ^ _this_ is what 's happening here which is interesting.
|
| ...because, it seems like it shouldn't be this easy to
| trick an AI model, but apparently it is.
|
| Maybe in the future, human go players will have to study
| 'anti-AI' strategies from adversarial models.
|
| It's an ironic thought that the iconic man-vs-machine loss
| against AlphaGo could have been won if he'd used a cheap
| trick against it.
| im3w1l wrote:
| There are rock paper scissors AI that very much try to
| study and exploit the opponents.
| notacoward wrote:
| "Defeat your enemies with this one neat trick!"
|
| Every "secret" or "hack" for getting your way in interactions
| with other people is pretty likely to be in this category. We
| all have blind spots, many of which we share because we're
| all running on basically the same hardware, and where there's
| a blind spot there's likely to be an adversarial attack.
| Ruarl wrote:
| I think Iain M. Banks called this an "Outside Context
| Problem" :)
| lukasb wrote:
| I think the difference between humans and special-purpose ML
| models here is that humans can generalize from examples in
| different domains. (There are ML models that also try to do
| this - train across domains to be more robust against out-of-
| sample inputs - but my understanding is it's not yet common.)
| vintermann wrote:
| Yes, this is a very familiar scene from CGOS, the computer go
| server which the Computer Go mailing lists used to get ratings
| for their bots. Plenty of otherwise strong bots botched the
| cleanup.
|
| KataGo doesn't even botch the cleanup unless _you_ botch the
| settings. So I don 't think this is a very impressive result.
| Alex3917 wrote:
| Can't wait to see Magnus Carlsen deploy this against Hans
| Niemann.
| a-dub wrote:
| knowing your opponent's hypothetical moves exactly seems like
| cheating to me!
|
| what would be interesting would be an agent that bluffs (and
| punches above its weight because so), since so many game playing
| ais assume that their opponent is playing optimally.
| jari_mustonen wrote:
| From the games, it seems evident that this adversarial system
| didn't find anything inherently Go-ish (for example, difficulty
| calculating complex ladders). Instead, it appears that KataGo
| used in these has been taught with a different rule set and then
| makes a silly mistake when put in a situation with different
| rules.
|
| Here are some explanations for those of you who are not Go
| players.
|
| Ladders are structures where a specific repetitive pattern
| emerges. They are thought to be novice Go players who will be
| able to calculate them. On the other hand, the Nature paper about
| AlphaGo Zero learned to handle them later in training. More about
| ladders: https://senseis.xmp.net/?Ladder
|
| Go rules are said to be simple, but in reality, calculating
| points in certain corner cases is difficult. Hence, there are
| different rules where the point calculation method differs.
| Typically the result is identical, but there are cases where this
| is not the case. More about rules here:
| https://senseis.xmp.net/?RulesOfGo
| gcanyon wrote:
| So we have a rock-paper-scissors situation: -
| KataGo beats human - Adversarial Policies beats KataGo
| - Human beats Adversarial Policies
|
| This is a fascinating outcome.
| aeturnum wrote:
| I think describing it this way misses an important implication.
| It's not a singular third tactic - it's that AIs generally lack
| the ability to recognize actions as being outside or against
| their training, meaning you can manipulate their internal state
| by acting in ways they were not trained to expect. Or, to put
| it another way, where humans recognize trolling, AIs will take
| trolling seriously forever.
|
| This is an extension of the problem with ML in general, which
| is that ML struggles to recognize when its judging a situation
| that is outside its training. In those circumstances they have
| a high chance to emit bad judgements with high confidence.
| gcanyon wrote:
| Well, yes, that's kind of the conclusion I was going for.
|
| To generalize further, it's possible that a rock-paper-
| scissors-lizard-spock situation could result, with an
| arbitrary length of A beats B, B beats C, C beats D....ZZZ
| beats A.
|
| Then we potentially look for something akin to a universal
| Turing machine: an algorithm that is capable of beating any
| other algorithm no matter what their strategy, perhaps(?) by
| simulating the algorithm that beats them.
| Shorel wrote:
| Nice, this means the professional-level AIs can be fixed
| automatically against cheap tricks, not only against good play.
|
| In a sense, this is expanding the search space a bit, they train
| not only against themselves but against a basic subset of low-
| hanging-fruit rule abuse.
___________________________________________________________________
(page generated 2022-11-03 23:01 UTC)