At zero depth/search, i.e. only using its neural net, Stockfish's tactics would, of course, be abysmal, but how strong is its positional evaluation? Can it be compared to grandmasters or to club players?
Note: Stockfish NNUE uses a mixture of AB search and neural network. This question is asking how strong the neural network positional evaluation part of Stockfish's evaluation is.
Without searching we can get the nnue score for each legal move from a given position.
Stockfish has an eval command to return the static evaluation of the position. It can be classical, nnue and hybrid.
Run stockfish and enter the following as an example.
position fen 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 w - - 0 1
eval
Output
Classical evaluation +0.22 (white side)
NNUE evaluation +0.45 (white side)
Final evaluation +0.50 (white side) [with scaled NNUE, hybrid, ...]
We are interested on
NNUE evaluation +0.45 (white side)
To get the nnue score of the legal moves without searching, we need the position and the legal moves.
position fen 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 w - - 0 1 moves d4d8
eval
With that we will get the nnue score of the move d4d8.
Output
NNUE evaluation +0.12 (white side)
eval = 0.12
evalcp = 0.12 * 100 = 12
The side to move after the move is black.
evalcp = -1 * evalcp = -12
The move is pushed so we negate the eval again.
evalcp = -1 * evalcp = -1 * (-12) = 12
So the nnue eval of the move d4d8 from the position 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 w - - 0 1 is 12 cp (spov).
Do the same for other legal moves to get the eval.
Sort the move eval in descending order to get the top move and eval from the given position. Remember the top move, as we will get the score of this move from the stockfish full strength multipv analysis.
Error Calculation
To get the error, we run stockfish say 10 sec at full strength at multipv value equal to the legal moves of the positions.
The top 1 score less the score of the top move from nnue eval is the error. Take the absolute difference to get a positive error value.
Where to get the test positions?
You can take any game in pgn format. I use tata steel 2023 games. These games are played by some top human players.
Parse each position in the game, record the game move, get the score of the move and calculate the error based from top 1 move score from stockfish. The score of game move can be found from the stockfish multipv analysis.
code
The full code below will save its analysis in csv file for later processing.
main.py
"""Estimate the strength of nnue eval.
requirements:
chess
pandas
installation:
pip install chess
pip install pandas
"""
import subprocess
import chess
import chess.pgn
import chess.engine
import pandas as pd
pd.set_option('display.width', 1000)
pd.set_option('display.expand_frame_repr', False)
pd.set_option('max_colwidth', 800)
def nnue_eval(enginefn, epd):
"""Find the best move with highest eval.
Run engine, setup position and send eval command.
Save eval and legal moves from the given epd.
"""
board = chess.Board(epd)
epds, moves, scores = [], [], []
engine = subprocess.Popen(enginefn, stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True, bufsize=1,
creationflags=subprocess.CREATE_NO_WINDOW)
engine.stdin.write('uci\n')
for eline in iter(engine.stdout.readline, ''):
line = eline.strip()
if 'uciok' in line:
break
engine.stdin.write('isready\n')
for eline in iter(engine.stdout.readline, ''):
line = eline.strip()
if 'readyok' in line:
break
# Visit all moves and get the eval.
for move in board.legal_moves:
b = board.copy()
san_move = b.san(move)
b.push(move)
# If side to move is in check, don't evaluate.
if b.is_check():
continue
fen = b.fen()
engine.stdin.write(f'position fen {fen}\n')
engine.stdin.write('eval\n')
pawn_value_wpov = None
for eline in iter(engine.stdout.readline, ''):
line = eline.strip()
# NNUE evaluation +0.19 (white side)
if 'NNUE evaluation' in line and not 'info string' in line:
value = line.split('NNUE evaluation')[1]
value = value.split('(white side)')[0].strip()
pawn_value_wpov = float(value)
epds.append(epd)
moves.append(san_move)
score = round(100*pawn_value_wpov)
# Convert to spov.
if b.turn == chess.BLACK:
score = -score
score = -score # move was pushed
scores.append(score)
break
engine.stdin.write('quit\n')
df = pd.DataFrame({'epds': epds, 'moves': moves, 'scores': scores})
df = df.sort_values(by=['scores'], ascending=[False])
return df
def main():
max_games = 5
movetimesec = 10
pgnfn = r'F:\project\pgn\tatamast23.pgn'
enginefn = r'F:\Chess\Engines\stockfish\stockfish_15.1_win_x64_popcnt\stockfish-windows-2022-x86-64-modern.exe'
gcnt = 0
engine = chess.engine.SimpleEngine.popen_uci(enginefn)
alldf = []
with open(pgnfn) as pgn:
while True:
game = chess.pgn.read_game(pgn)
if game is None:
break
gcnt += 1
print(f'game {gcnt}')
wp = game.headers['White']
bp = game.headers['Black']
data = []
for node in game.mainline():
board = node.parent.board()
gmove = node.move
gmove_san = board.san(gmove)
gply = board.ply()
epd = board.epd()
fmvn = board.fullmove_number
gmove_is_check = True if '+' in gmove_san else False
if board.is_check():
continue
# Don't analyze moves below 20.
if gply < 40:
continue
legal_moves = board.legal_moves.count()
# Analyze this board with the engine. Get all the legal moves evaluation.
info = engine.analyse(board, chess.engine.Limit(time=movetimesec), multipv=legal_moves)
epds, scores, moves = [], [], []
top_move_is_tactical = False
top1_move = None
for i in range(min(legal_moves, board.legal_moves.count())):
score = info[i]['score'].relative.score(mate_score=32000)
pv = info[i]['pv']
move = pv[0]
san_move = board.san(move)
if i == 0:
top1_move = san_move
# Exclude tactical moves.
if i <= 5 and ('+' in san_move or 'x' in san_move or '=' in san_move):
top_move_is_tactical = True
break
epds.append(epd)
scores.append(score)
moves.append(san_move)
if not top_move_is_tactical and not gmove_is_check:
top_df = pd.DataFrame({'epds': epds, 'moves': moves, 'scores': scores})
print(f'top_df:\n{top_df}\n')
nnue_df = nnue_eval(enginefn, epd)
nnue_df = nnue_df.reset_index(drop=True)
print(f'nnue_df:\n{nnue_df}\n')
print(f'game move:\n{gmove_san}\n')
# Get top nnue score based from top_df data.
top1_score = top_df['scores'].iloc[0]
top_nnue_move = nnue_df['moves'].iloc[0]
top_nnue_move_score_from_top_df = top_df.loc[top_df.moves == top_nnue_move]['scores'].iloc[0]
# Get actual score of game move based from top_df data.
gmove_score_from_top_df = top_df.loc[top_df.moves == gmove_san]['scores'].iloc[0]
print(board)
print()
print(epd)
print(f'top_1_score: {top1_score}, top_nnue_score: {top_nnue_move_score_from_top_df}, gmove_score: {gmove_score_from_top_df}')
print()
if board.turn == chess.WHITE:
data.append([gcnt, epd, fmvn, wp, gmove_san, gmove_score_from_top_df, top1_move, top1_score, top_nnue_move, top_nnue_move_score_from_top_df])
else:
data.append([gcnt, epd, fmvn, bp, gmove_san, gmove_score_from_top_df, top1_move, top1_score, top_nnue_move, top_nnue_move_score_from_top_df])
if abs(top1_score) > 500 and abs(gmove_score_from_top_df) > 500:
break
# Save to dataframe.
df = pd.DataFrame(data, columns=['GameNum', 'Epd', 'MoveNum', 'Name', 'GameMove', 'GameMoveScore', 'EngineMove', 'EngineScore', 'NNUEMove', 'NNUEScore'])
alldf.append(df)
if gcnt >= max_games:
break
engine.quit()
# Save data to csv file. Print average error in cp.
df = pd.concat(alldf, ignore_index=True)
df.to_csv('data.csv', index=False)
print(df)
df_error = df.copy()
df_error['HumanError'] = abs(df_error['EngineScore'] - df_error['GameMoveScore'])
df_error['NNUEError'] = abs(df_error['EngineScore'] - df_error['NNUEScore'])
print(df_error)
average_human_error = df_error['HumanError'].mean().round()
average_nnue_error = df_error['NNUEError'].mean().round()
print(f'AveHumanError: {average_human_error}')
print(f'AveNNUEError: {average_nnue_error}')
if __name__ == '__main__':
main()
Modify the stockfish and pgn paths to suit your needs. See,
pgnfn = r'F:\project\pgn\tatamast23.pgn'
enginefn = r'F:\Chess\Engines\stockfish\stockfish_15.1_win_x64_popcnt\stockfish-windows-2022-x86-64-modern.exe'
command line
python main.py >log.txt
output
GameNum Epd MoveNum Name GameMove GameMoveScore EngineMove EngineScore NNUEMove NNUEScore HumanError NNUEError
0 1 5rk1/1p1q1ppb/2p1p2p/4P3/R7/2Q3P1/1P2PPBP/6K1 ... 23 Carlsen, Magnus Rd4 52 Rd4 52 Rd4 52 0 0
1 1 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 ... 24 Carlsen, Magnus Rd6 62 Rd6 62 Rd7 -548 0 610
2 1 5rk1/1pq2ppb/2pRp2p/4P3/8/2Q3P1/1P2PPBP/6K1 b - - 24 Aronian, Levon Ra8 -73 Qb6 -51 Qxd6 -520 22 469
3 1 r5k1/1pq2ppb/2pRp2p/4P3/8/2Q3P1/1P2PPBP/6K1 w - - 25 Carlsen, Magnus Bf3 59 h4 69 Rxc6 -466 10 535
4 1 r5k1/1pq2ppb/2pRp2p/4P3/8/2Q2BP1/1P2PP1P/6K1 b... 25 Aronian, Levon Bg6 -38 Bg6 -38 Qxd6 -504 0 466
.. ... ... ... ... ... ... ... ... ... ... ... ...
71 5 8/p7/1p5R/3P1kp1/8/1K5P/r7/8 b - - 38 Abdusattorov, Nodirbek Ra1 67 Rd2 67 Rd2 67 0 0
72 5 8/p7/1p5R/3P1kp1/8/1K5P/8/r7 w - - 39 Rapport, Richard Kc4 -124 Kc2 -59 Kc4 -124 65 65
73 5 8/p7/7R/1pKP1kp1/8/7P/8/3r4 b - - 42 Abdusattorov, Nodirbek b4 205 b4 205 a5 35 0 170
74 5 7R/p7/8/2KP1kp1/1p6/7P/8/3r4 b - - 43 Abdusattorov, Nodirbek b3 205 b3 205 a5 103 0 102
75 5 4R3/p7/8/2KP2p1/8/1p1k3P/8/3r4 w - - 46 Rapport, Richard d6 -686 Rb8 -641 d6 -686 45 45
[76 rows x 12 columns]
AveHumanError: 8.0
AveNNUEError: 251.0
The function nnue_eval() will calculate the nnue eval of the legal moves from the given position or epd.
I place the code in github repository to easier maintain it.
I don't think it's possible to isolate positional understanding from tactics completely when discussing human players.
We can make some assessments when comparing engines among themselves. For instance, with the introduction of deep-learning-based engines, positional understanding has improved. But we can't really find an objective measurement of "positional strength" that would allow us to compare engines and humans.
The main things which stockfish considers when choosing a move are:
Nearly all of these things could be considered parts of "positional understanding" but to really understand how "strong" positional understanding is, looking for ideas and plans is necessary, stockfish does this using a optimized alphabeta pruning search, but this doesn't demonstrate any understanding of the board in front of it at all, so while stockfish with the ability to search has great ability to find and execute ideas, without that it has far less of an understanding than humans.
To put it another way, humans try to come up with general plans and find good squares, while stockfish can place value on squares, it can't picture the pieces going to them without the ability to search, so its positional understanding in that case is really quite bad. I haven't tested it or seen data for it, but i would wager it is far below club players without search depth.
The idea that positional understanding and tactics are different is wrong. Even a beginner like myself needs both to play reasonably well. Without tactics I will never beat any decent player. Of course, being that this is Stockfish, it does a bit better than I would. Also, here are some in important things to consider in chess. They could be said to be either:
So, to say how stong positional understanding is without tactics (or vice versa) is moot. Also consider the following opposite question. How strong is tactics without positional understanding? The answer. Nonexistent. How can one have tactics if you have no understanding of your position? You can't. How can one have positional understanding without any tactics? You can't.
One other thing to note is that when the NNUE (Efficiently Updatable Neural Network) was added to Stockfish 12 it is increased the rating only by 100. While a bunch, it is not that large. This does indicate that Alpha Beta search was probably able to have positional understanding (because they are the same thing).
Based on this blog:
Stockfish uses a static evaluation function, or a function which calculates an estimate evaluation of the position using a heuristic (a manually constructed algorithm designed to fit any arbitrary chess position) which has been hand-crafted by computer scientists and chess grandmasters for over a decade. Since the evaluation function is very powerful we can assume it will do a good job most of the time to evaluate a position. However, this is not enough for a game as complex as chess – it is a Dynamic Game as opposed to a simultaneous-move game and involves niche game tree paths which can never be fully encapsulated in the problem space of an evaluation heuristic. There is a high probability that the function will evaluate a position as winning and miss a line or path of moves by the opponent which causes us to actually lose. It can also evaluate a position as losing and miss a line which is actually winning. The static evaluation function capitalizes on the deterministic nature of the game, however we need to do more to deal with the element of randomness. Hence, it is necessary for the engine to be able to calculate into the future.
It states that depending on the position of the game, Stockfish when only using its static evaluation function can miss a far better outcome than what it plays and misevaluate the position. Therefore without the minimal and Alpha beta search methods, stockfish isn't considered very good when it comes to positional understanding, since it can misevaulate.
