"How are the scores decided in Chess?" - Correct?

Question

Recent question:

How are the scores decided in Chess?

It has been argued that PV scores for Stockfish (the engine being used by chess.com) can be interpreted as something like:

+1.0 is a pawn advantage for White. -0.5 is a half-pawn advantage for Black.

Is that correct? The weights and parameters in Stockfish are either manually calibrated or tuned by something like the SPSA optimiser. We only care if the optimization can give the engine gradients to a local optimum solution.

The objective function is to maximise the playing strength. Maximising playing strength has nothing to do with making sure "+1.0 is a pawn advantage for White". In fact, we should be able to scale all all the PV scores by a constant. As long as the engine makes the same move, the magnitude of the scores shouldn't matter.

This is consistent to the minimax algorithm, where both players only care how to minimize/maximise the evaluation. Again, there are no efforts to make the PV scores human-interpretable.

To me, the PV scores are only important relatively, not absolutely. For instance, we should be able to say "+2.0 is about two-times advantage for White than a position with +1.0 for White.", but that shouldn't have anything to do with pawns. Similarly, if I modify the source code and scale the outputs by a factor of 10. I can still say: "+20.0 is about two-times advantage for White than a position with +10.0 for White". We shouldn't be able to connect the scores to pawns, knights, bishops etc.

I haven't seen anybody in the Stockfish development team spent any efforts to make the outputs interpretable. To me, the scores have no direct interpretation to chess.

Should I call all answers in the question technically incorrect?

EDIT

This is in response to @D_M. How would you scale the outputs to make it human interpretable? What's the algorithm? How would you define the loss function? How to test for the changes? I don't see it's a trivial problem.

Answer 1

How does the algorithm work?

Starting here, the rest become trivial extensions.

The algorithm runs under a tree structure. For example, it looks at Move A, find every corresponding move from the opponent. Then it finds all moves from yourself, over and over. We list all of these plausible positions a score. The process Balances itself out so in each case the algorithm spends most of it's time following branches of the tree that seem most likely to win, rather than obviously losing branches. The score is the key here - I have run under the assumption that it is not directly decided by position (although I could be wrong now), but based upon the material value of future positions.

As an example, if you make move A now, in a few more moves, you might have 200 plausible positions where you're up a rook, 250 where you're down a rook, 300 plausible positions where you're up a knight, 250 where you're down, 1000 where you're up a pawn, and 500 where you're down a pawn, as determined by the algorithm playing a branch. Then it just averages the material, divided by the total number of positions):

(200*(5) + 250*(-5) + 300*(3) + 250*(-3) + 1000*(1) + 500*(-1))/2500 = 400/2500 = 0.16.

The branch is assigned that value. But in the same branch of that tree, a slightly different course might get a score of 0.56. We accumulate a bunch of scores by looking in depth 5,6,7 (in the endgame way more) moves ahead this way for all the branches with this many moves, then average all of them and get a score for move A. We use this logic of assigning scores based upon future material gains for each of the possible moves available, until a set amount of time is up, or we've thought through a set number of moves ahead.

Finally, if it finds a branch that would make all branches lead to a capture of the king (which is in itself mate), then it would follow this as such with a mate in X moves. From playing with the machine, it seems the computer has arbitrarily set the king as worth 100 points, rather than the infinite game-ending value it really has. This simply means it's worth more than the cumulative sum of the other pieces though, so has the desired effect.

With this knowledge, we can then answer:

Is this correct:

Yes. Going a single move ahead that has a 100% chance for being up a single pawn, a +1.0 would be the algorithm's response. A single move for black which has a 50% chance for being up a single pawn would result in an outcome from the algorithm of -0.5

How would you scale the outputs to be human interpret-able:

To me, a better output would be % chance of win. However, while computers are beating humans, they haven't beaten the game. Just to much math. Knowing that move A gives me 4% more chance to win matters more to me then pawns I might win in the future. The computer could only spit out based upon plausible positions where wins and losses were detected, but long term strategy usually isn't to aim for repeated Scholar's Mates.

How would you define the loss function?

It's the probability of having more material in the future. A +0.01 means you have 1% chance of being up one pawn in the future if you made the intended move.

How to test for the changes?

This needs clarification.

Answer 2

Well, I found a place where a member of the Stockfish support staff says:

Your intuition is correct. Positive values mean white is winning. Negative values mean black is winning. The units are in pawn units, so +1.00 means white is winning by one pawn.

So, it seems that they at least claim it corresponds. I don't know how exact it is; there really isn't an "exact" value for any piece since it depends on the position anyway. But I think that's your answer.

You say:

We only care if the optimization can give the engine gradients to a local optimum solution.

I disagree. Your premise that Stockfish is only used to find the best move is incorrect - some of us actually look at the evaluation score. If I'm using Stockfish to look at a game I played, I absolutely care whether the move I played lost me 5 pawns or 0.05 pawns compared to the best move (one is a blunder, the other is a perfectly reasonable move and might not even be objectively worse.) And it makes a difference whether a move took the evaluation from -10 pawns to -20 (it was completely hopeless anyway and I shouldn't worry too much), or -1 pawns to -2 (going from disadvantaged to probably lost, and maybe I should study that a bit more to see what went wrong.)

Yes, you can scale it however you like - and since you can, why not just scale it so a pawn is 1?

Answer 3

In fact, we should be able to scale all all the PV scores by a constant. As long as the engine makes the same move, the magnitude of the scores shouldn't matter.

True. We can even add a constant to all scores (e.g. an equal position would have a score of +1) without any effect on the engine's decisions.

However, it's useful to keep the well-known score definitions:

+1.0 is a pawn advantage for White

for 'backwards compatibility', just like we still have a 'meter bar' in Paris while the meter is currently defined with Krypton 86, so the 'meter bar' is technically slightly longer or shorter than a meter.