Why not use computers to evaluate strength of players?

Question

A computer like Stockfish can evaluate the rating performance of individual players based on a game. Why don't we take the average rating evaluation of a player's games to measure the player's strength instead of having elo systems etc?

Answer 1

Elo is a system to summarize past results into a meaningful number that informs about the (relative) strength of a player. Since the only objectives a player has during a game of chess are getting a win or a draw, only success and failure in those two specific objectives should be taken into account. It's true that how accurately someone played in a game can be used to predict how many wins he'll get in the future, but at least if we want our rankings to have an "official" value (granting titles, qualifying for tournaments...), we should base those rankings on past results alone rather than trying to make predictions about potential future outcomes.

But even if we want to use an engine-similarity ranking as just a way to determine who is the best player of all time (parallel to the official Elo rankings), it still comes with its share of problems:

• First we'd need to choose a specific engine to use. If we just take the one with the highest Elo at the moment, we're going back to taking past results as the ultimate metric of strength. We'd also need to decide of how much time per move and on what hardware the engine should run. These are again arbitrary choices that could leave to dramatically different outcomes.

• Since new and stronger engines are coming up all the time, we'd need to be constantly recalculating past ranking lists. It'd seem unfair to dismiss "Player A" as the strongest player of 2024 just because the engines of the time weren't good enough to understand the mistakes of their opponents.

• Chess has many different time controls yet FIDE does fine with just 3 different ratings (Blitz, Rapid and Classical). An accuracy-based system would require to introduce some correction for specific time controls (or to generate separate lists for 60+30, 90+30 and so on).

• Chess is a two-player game. Forcing your opponent into making a mistake is just as important as avoiding mistakes yourself. Engines have no concept of the "difficulty" of a move. For instance in a theoretically drawn endgame like rook+bishop vs rook, giving the bishop away for free is just as "accurate" as making a challenging move that forces your opponent into a difficult defense.

• Quiet games usually lead to easier positions where fewer mistakes are played. An accuracy-based ranking would be heavily biased against aggressive players. Many of Tal's sacrifices are "blunders" even though they actually led to wins.

• In an Elo system, you win the biggest amount of points by playing higher-rated opposition. If you just "farm" lower-rated players, your rating will increase very slowly. But it's much easier to get high accuracies against bad players, so you can massively inflate your score by brilliantly defeating newcomers.

In short, a system that rewards a player for a Berlin draw against a beginner but punishes him for a win in a chaotic double-edge position against a Grandmaster can never be a valid way to measure chess strength.

Answer 2

Why don't we take the average rating evaluation of a player's game to measure the player's strength instead of having elo systems etc?

The answer is simple and obvious. The elo system requires only the results of games and the statistical calculations are straightforward and quick. What you suggest would require all the moves of all the over-the-board FIDE rated games be available before it could even start. Then enormous computer power would be required to analyse the hundreds of thousands of games played every month.

That just applies to standard time control games. It would be completely impossible to do for the vast majority of blitz and rapid games which are not played on electronic DGT boards which record the moves. There is no record of the moves played hence rapid and blitz ratings using this system would be impossible.

Answer 3

Stockfish does not evaluate the rating performance of a player in a game, it can only give an opinion on the accuracy of a player's moves. I say "opinion" because stockfish does (rarely) lose games to other computers, so it is not 100% accurate.
At master level, certain openings can be played with high accuracy very easily. Other openings are too complicated for a human to navigate with computer precision. This does not matter to the player, because they are playing other humans. Should a player receive a higher rating because they are consistently drawing with 98% accuracy in simple openings rather than seizing advantages in messy, imperfect middlegames? I don't think so.
Some sites like chess.com offer an estimate of a player's Elo following every game. I think this is mostly a novelty and another way to "gamify" chess improvement. The numbers it gives should be taken with a grain of salt.

EDIT: Brian Towers answer is maybe the "real" answer here. Even if a perfect engine was created that could also judge how difficult moves are to find from a human perspective, tournament directors don't want to have to input all moves from every game. It may seem like that's something they already do since you can find massive databases of master games. However, it's easy to forget that FIDE and USCF sanction tournaments for all skill levels. The Elo system is fast and practical, it allows a player to receive an official rating using only their tournament results.

Answer 4

Complementary to the other more pragmatic answers let's take a step back and approach this from a more theoretical angle: The essence of a rating system is the prediction of the result of a game between two players. And since human players typically do not play deterministically, they essentially constitute or employ a mixed strategy from a game theoretical perspective. Therefore, the rating is a predictor for the statistically expected score between two such mixed strategies with only limited information about these strategies/players.

Therefore you need two things to define a rating:

1. Knowledge about the reference population (rating pool) of strategies.
2. Knowledge about the strategy/player to be rated.

What is the optimal way to use this knowledge to construct a predictor for the expected score? To me, it is not a given that only using game results is the optimal way. If you could build an approximate model of the mixed strategy and the reference strategies (e.g., using machine learning) from game records, you could do a monte-carlo simulation to give an expected score and hence a rating.

All I want to say with this is that while it might practically currently not be very accurate or feasible to predict playing strength from just moves without game results, it is far from an absurd question. It is just that purely result based rating estimation has been the most transparent and robust so far, especially given the incomplete knowledge and constant changes of both the player pool and individual players.

Answer 5

If the aim is to find the best player of all time using a computer systems that looks at the moves a player makes, it is clearly not possible as explained by other answers.

If we believe

The essence of a rating system is the prediction of the result of a game between two players

We can't hope to use a computer based measurement of the accuracy of a player's moves to replace rating systems, I will not repeat other answers that expain why.

What what if we believe?

The primary aim of a rating system to to divide players in a chess festival between separate Swiss Tournaments so it is likely that most games will be fun, eg the outcome is not predictable.

And we also believe?

Players who have not played in formal tournaments against anyone outside of their school chess club should be able to be allocated to the correct Switt Tournament so they get many fun games

This includes when none of the club members have regularly played people with a reliable ratings

Then we have an interesting and solvable problem with two likely solutions:

• Use a computer to get accuracy of a player's moves, then use the result as the 1st approximation of their rating.
• Get them to play agaist a system like Maia, finding the level that they tend to win as many games as lose, then use the result as the 1st approximation of their rating.

Once we have that 1st approximation of their rating and the 1st approximation of the rating ratings of other members of the schools chess club we can combine them with known recent game results from the club and feed the information into a linear programming solver to give an improved approximation of their ratings.

In other words we could consider each school chess club to be a disconnected rating pool, and use the computer based methods to work out an approximation of the mapping function between these rating pools.

---

But we could likely practically do as well by using the opponents Chess.com rating to work out the "tournament rating" of each person in a rating pools who played recent OTB games, then use these "tournament ratings" to work out the mapping between each chess clubs internal rating system. But that is no what the question asked!

Answer 6

The ELO system is bias towards a game's result (winning, drawing, losing). By this, I mean it doesn't care about what happens in a game, only the result of a game.

From my understanding (these algorithms are always in flux) and simplifying, a computer rates a game based on the amount of centipawns advantage given up throughout the game (not a linear sum relationship). This is biased towards what happens in a game.

Let's think about a player like Richard Rapport. He is famous for his crazy openings. He does something crazy, throwing their opponent out of prep. He then plays a killer mid and endgame.

In your suggested system, Richard Rapport compared to other (current) top players would see his rating fall badly because he consistency gives up advantage in the early game. Whereas some players who play pretty bland games and tie mostly would see very large (relative) gains.

Does it make sense for Rapport to be lowered rated than a player (a player he could beat more often than he loses to), just because that player plays straight, serious chess? That is not a rhetorical question.

I am not saying one system is better than another but pointing out that they value two different things.

Answer 7

It can be done, and in fact it has already been done. Several times, even.

A great starting point is the Wikipedia page "Comparison of top chess players throughout history", in particular the paragraph Moves played compared with computer choices.

Let me quote a sentence from that Wiki article, about the Markovian model proposed by Jean-Marc Alliot of the Toulouse Computer Science Research Institute in 2017:

These predictions have proven not only to be extremely close to the actual results when players have played concrete games against one another, but to also fare better than those based on Elo scores.

Granted, this is applied only to the best players ever, to reduce the number of games that must be analysed. But I don't think applying this method to "normal" players is that impractical. Moves are already recorded, they just have to be saved to a PGN file and then the analysis is easy.

My personal experience with Chess.com is that the Elo estimator (which, if I understand correctly, is based on a player's accuracy) works very poorly when evaluating bots that are rated 1000-1500 (e.g. a bot rated 1300 can be evaluated around 400). And I use bots exactly because in theory their gameplay should be consistent. Maybe the problem is that at lower levels there is too much variability for the accuracy to be a good indicator of a player's strength? This is something that some research could assess.

So the approach does have some problems (like every other method, I'd say), but it also has some merits. Notably, it makes it possible to compare players who have lived at different times and who have never played against each other.