6

Even with all the compute in the world, chess engines cannot compute very deeply. So, they have to make use of pruning heuristics and discard moves from the analysis.

Seems possible that even the best heuristics will mistakenly discard good moves, and conversely go down bad paths that initially look good from a heuristic standpoint.

Have researchers tried to identify when this tends to happen, and if this can be taken advantage of, similar to adversarial examples in Deep Learning? E.g. an expert with deep understanding of a chess engine internals could concoct a series of moves to force the chess engine to make bad moves.

I play chess.com occasionally, and I've noticed something like this happening, where the chess engine will evaluate the same move as exceedingly good, and then as exceedingly bad.

Improve this question
7
  • 4
    There are certain engines out there which are supposedly designed to play against Stockfish in particular. Though I haven't heard much about them in the past couple years. It's probably harder now that Stockfish uses a neural net for its evaluation, as opposed to hand-crafted heuristics.
    – Inertial Ignorance
    Commented Nov 2, 2022 at 21:37
  • I did notice the use of neural networks, which also led to the question about adversarial examples. The imagery deep learning networks are easily fooled by minor image tweaks. I was wondering if the chess engine neural networks are similarly brittle.
    – yters
    Commented Nov 2, 2022 at 21:42
  • 2
    @yters In general, neural nets that are trained though self-play reinforcement are much more robust against adversarial inputs than supervised models constrained by the size and quality of the annotated datasets on which their learning was based. That said, potential weaknesses can still remain due to the relation between feature encoding and the horizon effect on learning rate. For example, early Alphago-inspired go engines would tend to misjudge ladders, whereas recent stronger engines avoid the problem by encoding the presence/absence of...
    – Will
    Commented Nov 3, 2022 at 11:33
  • 1
    ...ladders as additional input features (providing the model with new "vocabulary" so to speak). So if I were to look at finding such "exploits", I'd look at (lack of) feature encoding first.
    – Will
    Commented Nov 3, 2022 at 11:35
  • 1
    chess.stackexchange.com/questions/41867/… This was actually asked and answered recently, you might be interested.
    – Allure
    Commented Apr 20, 2023 at 5:16

2 Answers 2

Reset to default
5

Yes. Antifish was an attempt in 2018 to train a LC0 style network specifically to beat stockfish. It turns out that even when searching specifically for weaknesses, chess engines are really strong.

Improve this answer
1

The problem with this question is that it's not clear how an adversarial engine would work. In a comment you write:

making moves where the payoff is beyond the chess engine's calculable horizon

But what is "calculable horizon" to you? Let's say you define it as "the engine does not find the refutation after thinking for 10 minutes on a $1000 2023 desktop". Now what if:

  1. Someone lets the engine think for 20 minutes
  2. Someone uses a $2000 desktop

And the engine finds the refutation - is the adversarial engine still adversarial? If yes, then the definition of "adversarial" is very arbitrary. If no, then it's not well-defined in the first place.

Accordingly, the question is too vague to answer.

PS: There are positions where the engine does not find the key move after thinking for a "very long time" (inverted commas because this term is also not well-defined, c.f. the above), but they're rare, so an engine that makes a good move in rare positions at the cost of making not-so-good moves in more common positions will lose elo in self-play.

Improve this answer
6
  • Horizon is a parameter for the adversarial engine. It is adversarial against a specific version of a specific chess engine, such as stockfish. Someone did this sort of thing recently for Go. I've also seen specific instances for chess where chess engines fail, due to the horizon problem.
    – yters
    Commented May 24, 2023 at 15:01
  • I'm not familiar with computer Go, but I'm pretty familiar with computer Chess, and I don't understand how horizon can be a parameter for the adversarial engine. Can you explain?
    – Allure
    Commented May 25, 2023 at 2:12
  • Tell the adversarial engine the target has enough computational resources to calculate N-ply deep. Or, find situations where pruning is not effective, so N is always small regardless of the chess engine.
    – yters
    Commented May 25, 2023 at 13:28
  • @yters but N depends on position. And pruning is extremely effective in chess engines, they're virtually unbeatable from the starting position. The positions they misevaluate are few and far between. Maybe people have tried to create adversaries for chess engines (and some of them are linked in a comment), but there is no known way to reliably beat SF/Lc0, not from the starting position.
    – Allure
    Commented May 25, 2023 at 14:10
  • this answer of yours is the sort of thing I'm asking about. chess.stackexchange.com/questions/33098/… Have people researched ways to find booms for a specific chess engine? Then they could use the boom to beat chess engines that don't calculate far enough.
    – yters
    Commented May 26, 2023 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.