# How to compute variance of a set of Chess move evaluations

## Introduction

You are given a set of Chess move evaluations. They contain centipawn values and also mate values.

For example, an example set: `[#1, +234, -17, #-4]`

The above example contains the evaluations for 4 random moves. The board position and moves themselves are not important.

The problem I am trying to solve is to generate a variance for this set. I want to know how "close" together the evaluations are. I'd like to then normalize this final value into a reasonable range such as `0-100` where `0` is not close at all and `100` is extremely close (all moves have same strength).

Using the statistics Variance formula, I am able to do this easily and well for just centipawn values. It works flawlessly.

Some examples for centipawn values to illustrate the idea:

Evaluation Set Variance
{300, 300, 300, 300} 0.0
{0, 0, 0, 0} 0.0
{300, 299, 298, 50} 11625.688
{300, 200, 100, 0} 12500.0
{500, 400, 300, 200} 12500.0
{600, 500, 400, 300} 12500.0
{300, 299, 100, 50} 12912.688
{300, 299, 298, 0} 16763.188
{500, 400, 300, 0} 35000.0
{600, 500, 400, 0} 51875.0
{1000, 50, -200, -750} 400625.0

In the above examples, you can see sets of 4 moves in order from very close to not close. The first set `{300, 300, 300, 300}` has variance of 0 because all items are the same. This should map to `100` for my normalized output.

`{500, 400, 300, 200}` and `{600, 500, 400, 300}` are equivalent because they all spread out the same (even though one set has stronger moves in each position).

`{1000, 50, -200, -750}` has a very large variance and thus is very spread out. This should map to near 0. Obviously variance can be unbound but I have an upper bound I enforce (currently using 4002).

## Problem

How can I introduce check mates into my variance equation? I am struggling with this.

Obviously a `#1` by itself should be very 'not close' in an example like `[#1, +300, +100, -100]`.

However, what if there are two mates in the set like `[#1, #2, +300, +100]`? A `#1` and `#2` are (very?) close to each other but there are also centipawn values to compare.

What about this scenario: `[#1, #2, +300, #-1]` which contains all the different possibilities mates for white/black + centipawns.

Hopefully someone has ideas or can lead me in the right direction.

## Context

For additional context, I am making a fun Chess game and need a way to compare the difficulty of sets of 4 moves. Basically answer the question "How strong are these 4 moves?".

• You should simply assign a large arbitrary value to checkmates. Anyway I see a major problem with the concept of variance applied to move evaluations. A set like {+9, +15} may seem to have more variance than the set {-1, +1} but in practical terms it doesn't. Specially if you want to relate it to the difficulty of a position Commented Jul 30 at 15:34
• I tried assigning a large value to checkmates (like `32000-N`) but it dominates the variance equation which isn't correct. Thanks for clarification on my assumption on variance. You are right, evals closer to 0 are "more varied" in the context of difficulty of position. Commented Jul 30 at 15:55
• I spend a lot of time on this and this is the best I could come up with: gist.github.com/mobeigi/2abe90d5f6d6cb2eb3ecfb67b29fb326 Its not perfect by any means but it somewhat works (and my brain hurts thinking about this so much!). Commented Jul 31 at 17:06
• Please keep in mind that comments are a sign of bad code, not good one. Let the code speak for itself! Commented Jul 31 at 18:14
• I tend to over comment when I don't understand the domain :D Commented Aug 1 at 13:11

Instead of var(x_i), maybe try var(y_i) where y_i =sign(x_i)*log(abs(1+x_i)). Using logarithmic scale means the answer is more sensitive to the smaller values than the larger ones.

Using sign & absolute value ensures that positive and negative numbers are handled equally. Adding 1 before taking the log means 0 maps to 0.

This doesn’t directly answer the question of how to map checkmates, but it does mean that the way you do so matters less!

• I headed in a different direction by using `exponential decays` for mate values and passing it into a `sigmoid` function. Then finally taking a `harmonic mean`. Its not perfect by any means but its something (link to code in question comment if interested). I hope I can improve it shortly to be more accurate / realistic. Commented Jul 31 at 17:08

Before even calculating the second order moment of this, are you sure you are having the same ordered domain to do the statistics on?

What is the ordering relation between #n and #m? I guess one could impose some exaggeration of the 50 move rules that shorter is always better (without any other information, about players and board).

But then what kind of measure to use in a mixture of such outcome quantities, one of categorical ordering, and the other from forecasting learned or programmed leaf evaluation function of outcome odds from best play continuation (if having such conversion from elsewhere on some statistical basis, if the evaluation non-terminal value is not itself derived from a probabilistic mathematical model, say LC0, or A0, using RL training).

When integrating over those leaf evaluation, there might be a need for some floating parameter, and you might have to do, like fishtest, do a global optimization search on that moment definition. Average or variance, they all need such definition. Between the terminal outcome scale. And the forecasting one. By forecasting, I mean the non-terminal leaf evaluations. e.g. 1,3,3,4,9 might be a possible forecasting system. But we found that it would not always be enough. Etc.

• I don't have a good answer to what the ordering relation between `#n` and `#m`. We are making a broad and general assumption that lower ones #1 is easier than #30. I'm also introducing a strong exponential decay that drop off rapidly after #7 (on anecdotal evidence that strong human GM players can find a #7 but may struggle finding higher mates). Its not ideal for any really valuable work but for my use case, any rough approximation that isn't totally random / nonsense will do. Commented Aug 1 at 13:13

Here is a working solution to the problem.

The first step is to compute evaluation strength as normal for mates & centipawns.

1. Take `sigmoid(centipawn / downscale_factor)` for every centipawn value. The downscale factor allows you to vary the spread. This puts all centipawn values into a [1, 0] normalised range.
2. Take `sigmoid(-1 * (mate_value / downscale_factor)` for every white mate. The downscale factor allows you to vary the spread. Mate value is the mate number like mate in 3. This give you the white mates in [0.5, 0] range.
3. Repeat step 2 for black mates.
4. Now create a list of normalised values that range from [1.5, -0.5] for all of your values. We get this by simply adding 1 to all the white mate values from before, and taking away from 0 the values from the black mate.

At this stage, we now have a normalised [1.5, -0.5] range of values including mates & centipawns. Note, that the range includes exponential effects in multiple places (i.e. there is a large exponential decay as we go from 1.5 to 1. There is a decay from -0.5 to 0. And a decay outwards from 0 (neutral). This is exactly what we want.

1. Convert this [1.5, -0.5] range into a [1, 0] range using linear scalar.

Calculating difficulty / variance. This section is a little domain specific to Chess game I made where you sort 4 chess moves.

1. Take the strengths from before and your list of move evaluations
2. Calculate the adjacent diffs between each value
3. Remove all 0 value adjacent diff values (because if its 0 it means two adjacent moves have same evaluation and that means its easy to sort because they're correct in either position).
4. Take `harmonic_mean` on value so there is strong bias towards smaller gaps.

Done!

For anyone interested in a code example of this, you can see: https://github.com/mobeigi/chessort/tree/main/server/chessortserver/utils/engine

New contributor
Mo Beigi is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.