Approximating a rating for too strong engines

Question

Suppose we have an extremely strong engine that happens to win every game it participates in even against the most powerful engines and players. If it continues to win every game, its rating (for example ELO) will grow indefinitely. But we do not have an infinite amount of time, so we will have to stop evaluating sometimes with some rating R. On the other hand, it feels wrong to stop at rating R because it is likely our engine will win its next game and get a rating R' > R. Moreover, we cannot give this engine a rating of infinity, because in theory it can draw or lose, it just hasn't yet faced such an opponent.

How would you find a rating approximation for such an engine?

Answer 1

I'd treat this as a Bayesian inverse probability problem. Per Laplace's rule-of-succession solution of the sunrise problem (see https://en.wikipedia.org/wiki/Rule_of_succession and https://en.wikipedia.org/wiki/Sunrise_problem), it follows that if an engine wins n out of n games, its rating should be estimated as if it played n+3 games, winning n+1, drawing one, and losing one - playing, of course, against the same pool of opponents. (A point of note: a chess game differs from a sunrise in that there are three possible outcomes, rather than just two. For the sake of simplicity, we'll assume the number of games with black and white pieces is equal.)

Using the example provided above, the rule-of-succession method returns an estimate of 2230 (1000 games), 3330 (1M games), and 3450 (2M games) - not that far off from Bayeselo. I'm not familiar with the method employed by Bayeselo and cannot say how mathematically sound it is, but in the end this is a kind of problem that doesn't have a "correct" solution because it hinges on one's choice of prior probability. In Laplace's approach, it is posited that, if we knew nothing about the history of sunrises, we should assume that the probability of Sun rising tomorrow is uniformly distributed between 0 and 1. Whether that assumption is applicable to a chess game between two players about whom we know nothing is anyone's guess.

(Additional note: I'm assuming here that the question is how to estimate Elo once n games have already been played, rather than when to stop playing more games. The answer to the latter question is significantly more complex as it obviously depends on additional factors.)

Answer 2

There're a few ways to do it:

• Add 400 rating to the top opponent and take that as an approximation (used by USCF)
• Add a draw for the engine against itself
• Assign an infinite rating
• Use linear approximation

By the way "win every game against the most powerful engines and players" is realistically not happening, because chess is a draw** and top engines on strong hardware guided by a human are already so close to perfect play that the great majority of all correspondence chess games are drawn.

**It's not proven that chess is a draw, but you won't find any top player seriously disagreeing with this assessment. See also this answer illustrating what things are like at the top level of play (world correspondence chess championship, which is stronger than both engines and humans alone).

Answer 3

While there will be no concrete answer to this, there are few methods of approximation. We can play it against a 7 man tablebase, and for the sake of ELO rating we can assume that its strength in the middlegame or the opening is the same as the endgame. The number of losses that an endgame tablebase delivers to it (giving the borderline winning side to the tablebase and a blessed loss to the engine) determines its strength.

Another method we can use is the number of moves it takes to defeat the opposing engine. While not always correct, the general rule is that the more moves it takes to defeat the opponent, the weaker the player.

Answer 4

According to bayeselo program if the best wins 1k games without loses and draws and setting the opponent rating to 1000, it will get 2092.

Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
   1 best:  2092  170   83  1000  100%  1000    0%  1000     0     0
   2 p1  :  1000   83  170  1000    0%  2092    0%     0  1000     0

1M games:

Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
   1 best:  3203  167   82 1e+06  100%  1000    0% 1e+06     0     0
   2 p1  :  1000   82  167 1e+06    0%  3203    0%     0 1e+06     0

2M games:

Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
   1 best:  3314  161   83 2e+06  100%  1000    0% 2e+06     0     0
   2 p1  :  1000   83  161 2e+06    0%  3314    0%     0 2e+06     0

Answer 5

You could let the engine play with a handicap (e.g. rook down).

Then, you need to calibrate how much rating delta a missing rook means. I saw some numbers that estimated one point of material to be a delta of 100 points.

This calibration could be done through self-play. Let the engine play against itself and see how much a rook is worth.

A downside of this handicap method is that it changes the game dynamics. Engines are not optimized for starting the game in an unusual position with a massive disadvantage. This can throw off some heuristics.