9

Suppose I have a completed game. I don't have Elo ratings of the players. My purpose is to evaluate a player's performance in the game based solely on his moves. Can this can be done automatically using a chess program?

The result can be his approximate Elo rating, or just some value indicating his strength or error rate.

If it helps, a database of the player's games can be given. Again, with no Elo ratings.

My motivation is simple. I play chess over the internet and would like to automatically track my progress, based on the games themselves, not on the rating on the sites. I'm a (upper)-beginner level.

A simple solution is to annotate the game using any computer engine and track number of ?!, ? and ?? marks. However, it's not very accurate, and I'd like to get more ideas :)

1
  • Any evaluation based on a single game will come with a huge error margin. Your "progress" will be going up and down constantly and I doubt that you will be able to track anything besides very long term "progress" by this method (which basically comes down to an average over games). Chess ratings (or ratings in just about any other sport) avoid this problem and I don't see anything wrong with using the online rating as indicator for your strength. Sep 28, 2018 at 16:07

5 Answers 5

17

The Site ratings at slow time controls can be quite reliable for servers where strong players congregate (ICC, FICS to name a few) as the ratings VERY closely reflect your true playing strength if you've played enough games. For very standardized rating systems such as USCF and FIDE/ELO, you will notice that the different rating classes tend to point to the types of mistakes those players are still making. NM Dan Heisman's Improving Chess Thinker does an excellent job discussing the types of errors players make across the rating classes.

Have you tried the many self-test books out there? Igor Khmelnitsky's Chess Rating Exam and Danny Kopec's Test, Evaluate and Improve your chess are excellent books that allow you to track your progress by seeing how you perform against graded test positions.

Your compare-my-moves-with-an-engine approach is another way to do this but once again, the ??/? moves are really only indicating tactical errors, not strategic or positional or even behavioral or time-management mistakes you might be making.

That's why playing slow time-control OTB/online games against equal-to-stronger opposition and getting them reviewed + critiqued by stronger players is an efficient way to improve. Your mistakes in every category (tactics, knowledge, thought process, time management etc.) get highlighted and you can simply measure progress in terms of the mistakes you've stopped making.

Though one fun variant you can try with an engine at home: Why not extend your engine-evaluation method to visually observe a player's quality/performance via evaluation graphs? In other words, take engine evaluation scores per move and plot them (some free software like SCID does this for you) over the moves.

For example: Two rank beginners would have a game that looks like:

enter image description here

Notice how jagged these are. Both sides make many terrible mistakes (slopes of the spikes!) and also how often they fail to exploit the other person's terrible mistakes.

The spikes are always fun to look at : enter image description here

Two intermediate (USCF 1400-1600) players might have games that look like: enter image description here

It does look jagged, but notice how the y-axis (engine evaluation) is way smaller ... indicating that these players are more seasoned and play higher quality chess than the novices.

For a final comparison, a 1911 Grandmaster game would look like this:

enter image description here

No comments necessary here :) These guys really don't make many mistakes, do they?

If you could devise your own heuristic for mapping the slopes + scale of an evaluation graph to player skill/performance, perhaps this is one way to go? :)

2
  • 2
    Note that I'd like to throw in a caveat that Engine evaluation scores are not so reliable in some nuanced positions and some material-hungry flavors will consider a Gambit type-opening very differently than a human would. How long you set your engine on a half-move position while going over the game will also influence things a bit. Be warned! :)
    – shivsky
    Nov 24, 2013 at 12:06
  • What program do you use to generate these graph? I've scid + stockfish, do I have this option?
    – Uri London
    Nov 28, 2018 at 10:59
5

For a very accurate rank of a player's quality, you can use the excellent tool provided by www.chess-db.com. It lets you upload your games and after some minutes it outputs the quality of both players in percentage compared with the best moves of a strong engine.

This is the page to upload a PGN file: http://chess-db.com/public/game_upload.jsp

And this is an example of the results: http://chess-db.com/public/game.jsp?id=Pablo%20Bento.Shredder%20Android.107755008

2

chess.com CAPS. Compare CAPS score from chess.com (requires subscription), to table found in a graphic on this: https://www.chess.com/article/view/better-than-ratings-chess-com-s-new-caps-system . Also of interest: https://www.chess.com/article/view/who-was-the-best-world-chess-champion-in-history

Note of caution: Caps scores for anyone particular game are volatile; best to make some average of CAPS scores over a batch of games. Right now, to my knowledge, CAPS can, by extension, predict an Elo rating from a set of moves. Another note: CAPS scores exist in a vacuum without respect to the time controls. I play better chess at slower time controls than I do in a 1-minute bullet game. CAPS will see this difference in play strength and accordingly assign a lower rating to the bullet games. This does not mean that I am not the same person who played the slower time control games!

1

What you're asking does not exist by my knowledge. However, this is my idea:

You'll need samples with the following features, (1) Chess position, (2) Move made in the position, (3) Rating of the player that made the move.

Let's say you have 1 billion samples. You can train a computer algorithm on these samples that can predict for each move in a position the quality of the move with respect to a rating. All ratings for all moves can be averaged out to get the approximate rating of the player and in effect the quality of his game.

This is a rough idea that can be further polished.

2
  • This doesn't really work. In most games, you will relatively quickly be in positions not in the database - yes, even if you somehow get access to a billion game database. And you get punished if your opponent plays weakly - after 1.e4 e5 2.Qh5, no matter what Black plays it's going to have a low average rating, because that opening is played way more at lower levels than higher ones. (Heck, even 1...e5 itself probably has a rather low average rating.) What happens after the known openings is way more indicative of a player's strength than how long they follow theory.
    – D M
    Sep 29, 2018 at 4:42
  • I gave this answer 5 years before AlphaZero. Now we can all just ask AlphaZero to annotate our games for us and tell us how well we are doing. We are probably almost at the point that we can even ask WHY a move is good or bad. That's what you really need to improve. Current human ratings are overestimated.
    – Rafiek
    Oct 1, 2018 at 5:08
0

Here are a few ideas on what parameters to measure. Number of blunders per game. How often you have an equal or better position after 10 moves, 15 moves, 20 moves. How often you succeed on using your opponent's blunders. How often you draw or win objectively drawn endgames. How often you succeed to find forced mating combinations. How often you successfully defend objectively lost endgames. How often you lose on time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.