8

At zero depth/search, i.e. only using its neural net, Stockfish's tactics would, of course, be abysmal, but how strong is its positional evaluation? Can it be compared to grandmasters or to club players?

Note: Stockfish NNUE uses a mixture of AB search and neural network. This question is asking how strong the neural network positional evaluation part of Stockfish's evaluation is.

13
  • 1
    At zero depth/search, stockfish will not be able to produce a move. What it can do is output the static evaluation. Can you measure the strength of an engine without knowing its recommended bestmove from a given position?
    – ferdy
    Commented Apr 17, 2023 at 10:17
  • @ferdy Thanks for pointing out my mistake. I must have meant depth 1, right? Where Stockfish uses its neural net to evaluate every possible move, but does not consider any responses to them.
    – Anna
    Commented Apr 17, 2023 at 12:37
  • Like many engines, stockfish has two main searches the main search and the quiescence search. Search depth 1 refers to the main search. After main search depth 1, the program will use quiescence search where it will evaluate tactical moves such as capture, checking, check evasion and promote moves. So even at search depth 1, the engine is capable of resolving tactical moves behind the scene. We can revise stockfish and disable its quiescence search if that make sense of what your are trying to achieve.
    – ferdy
    Commented Apr 18, 2023 at 1:24
  • Thank you @ferdy, I had no idea about quiescence search. I thought this could be answered by playing a couple games vs Stockfish at depth 1, but apparently it's more complicated!
    – Anna
    Commented Apr 18, 2023 at 7:55
  • This is the reason why even at depth 1 the engine can see tactics because of the presence of this secondary search. The evaluation function (NNUE, hybrid, classical) is mostly applied on quiet positions at quiescence search. It is not very strong as it is limited to depth 1, but can still challenge some titled players. Also since you are interested on NNUE only, stockfish has to be revised to only use NNUE. I think the default is hybrid.
    – ferdy
    Commented Apr 18, 2023 at 11:37

5 Answers 5

5
+500

Without searching we can get the nnue score for each legal move from a given position.

Stockfish has an eval command to return the static evaluation of the position. It can be classical, nnue and hybrid.

Run stockfish and enter the following as an example.

position fen 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 w - - 0 1
eval

Output

Classical evaluation   +0.22 (white side)
NNUE evaluation        +0.45 (white side)
Final evaluation       +0.50 (white side) [with scaled NNUE, hybrid, ...]

We are interested on
NNUE evaluation +0.45 (white side)

To get the nnue score of the legal moves without searching, we need the position and the legal moves.

  1. Setup the position + legal move to stockfish
  2. Send eval the command to get nnue score.
  3. Negate the score as the move is pushed on the board.
  4. Convert wpov (White Point of View) score to spov (Side Point of View) score.
position fen 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 w - - 0 1 moves d4d8
eval

With that we will get the nnue score of the move d4d8.

Output

NNUE evaluation        +0.12 (white side)
eval = 0.12
evalcp = 0.12 * 100 = 12

The side to move after the move is black.

evalcp = -1 * evalcp = -12

The move is pushed so we negate the eval again.

evalcp = -1 * evalcp = -1 * (-12) = 12

So the nnue eval of the move d4d8 from the position 5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 w - - 0 1 is 12 cp (spov).

Do the same for other legal moves to get the eval.

Sort the move eval in descending order to get the top move and eval from the given position. Remember the top move, as we will get the score of this move from the stockfish full strength multipv analysis.

Error Calculation

To get the error, we run stockfish say 10 sec at full strength at multipv value equal to the legal moves of the positions.

The top 1 score less the score of the top move from nnue eval is the error. Take the absolute difference to get a positive error value.

Where to get the test positions?

You can take any game in pgn format. I use tata steel 2023 games. These games are played by some top human players.

Parse each position in the game, record the game move, get the score of the move and calculate the error based from top 1 move score from stockfish. The score of game move can be found from the stockfish multipv analysis.

code

The full code below will save its analysis in csv file for later processing.

main.py

"""Estimate the strength of nnue eval.

requirements:
  chess
  pandas

installation:
  pip install chess
  pip install pandas
"""


import subprocess

import chess
import chess.pgn
import chess.engine
import pandas as pd

pd.set_option('display.width', 1000)
pd.set_option('display.expand_frame_repr', False)
pd.set_option('max_colwidth', 800)


def nnue_eval(enginefn, epd):
    """Find the best move with highest eval.

    Run engine, setup position and send eval command.   
    Save eval and legal moves from the given epd. 
    """
    board = chess.Board(epd)
    epds, moves, scores = [], [], []

    engine = subprocess.Popen(enginefn, stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.STDOUT,
                              universal_newlines=True, bufsize=1,
                              creationflags=subprocess.CREATE_NO_WINDOW)
    
    engine.stdin.write('uci\n')
    for eline in iter(engine.stdout.readline, ''):
        line = eline.strip()
        if 'uciok' in line:
            break

    engine.stdin.write('isready\n')
    for eline in iter(engine.stdout.readline, ''):
        line = eline.strip()
        if 'readyok' in line:
            break

    # Visit all moves and get the eval.    
    for move in board.legal_moves:
        b = board.copy()
        san_move = b.san(move)
        b.push(move)

        # If side to move is in check, don't evaluate.
        if b.is_check():
            continue

        fen = b.fen()
        engine.stdin.write(f'position fen {fen}\n')
        engine.stdin.write('eval\n')

        pawn_value_wpov = None
        for eline in iter(engine.stdout.readline, ''):
            line = eline.strip()

            # NNUE evaluation        +0.19 (white side)
            if 'NNUE evaluation' in line and not 'info string' in line:
                value = line.split('NNUE evaluation')[1]
                value = value.split('(white side)')[0].strip()
                pawn_value_wpov = float(value)

                epds.append(epd)
                moves.append(san_move)

                score = round(100*pawn_value_wpov)

                # Convert to spov.
                if b.turn == chess.BLACK:
                    score = -score
                score = -score  # move was pushed
                     
                scores.append(score)
                break
            
    engine.stdin.write('quit\n')
            
    df = pd.DataFrame({'epds': epds, 'moves': moves, 'scores': scores})

    df = df.sort_values(by=['scores'], ascending=[False])

    return  df


def main():
    max_games = 5
    movetimesec = 10
    pgnfn = r'F:\project\pgn\tatamast23.pgn'
    enginefn = r'F:\Chess\Engines\stockfish\stockfish_15.1_win_x64_popcnt\stockfish-windows-2022-x86-64-modern.exe'
    gcnt = 0

    engine = chess.engine.SimpleEngine.popen_uci(enginefn)

    alldf = []

    with open(pgnfn) as pgn:
        while True:
            game = chess.pgn.read_game(pgn)
            if game is None:
                break

            gcnt += 1
            print(f'game {gcnt}')

            wp = game.headers['White']
            bp = game.headers['Black']

            data = []

            for node in game.mainline():
                board = node.parent.board()
                gmove = node.move
                gmove_san = board.san(gmove)
                gply = board.ply()
                epd = board.epd()
                fmvn = board.fullmove_number

                gmove_is_check = True if '+' in gmove_san else False

                if board.is_check():
                    continue

                # Don't analyze moves below 20.
                if gply < 40:
                    continue

                legal_moves = board.legal_moves.count()

                # Analyze this board with the engine. Get all the legal moves evaluation.
                info = engine.analyse(board, chess.engine.Limit(time=movetimesec), multipv=legal_moves)
                epds, scores, moves = [], [], []
                top_move_is_tactical = False
                top1_move = None
                for i in range(min(legal_moves, board.legal_moves.count())):
                    score = info[i]['score'].relative.score(mate_score=32000)
                    pv = info[i]['pv']
                    move = pv[0]
                    san_move = board.san(move)

                    if i == 0:
                        top1_move = san_move

                    # Exclude tactical moves.
                    if i <= 5 and ('+' in san_move or 'x' in san_move or '=' in san_move):
                        top_move_is_tactical = True
                        break

                    epds.append(epd)
                    scores.append(score)
                    moves.append(san_move)

                if not top_move_is_tactical and not gmove_is_check:
                    top_df = pd.DataFrame({'epds': epds, 'moves': moves, 'scores': scores})
                    print(f'top_df:\n{top_df}\n')

                    nnue_df = nnue_eval(enginefn, epd)                
                    nnue_df = nnue_df.reset_index(drop=True)
                    print(f'nnue_df:\n{nnue_df}\n')

                    print(f'game move:\n{gmove_san}\n')

                    # Get top nnue score based from top_df data.
                    top1_score = top_df['scores'].iloc[0]
                    top_nnue_move = nnue_df['moves'].iloc[0]
                    top_nnue_move_score_from_top_df = top_df.loc[top_df.moves == top_nnue_move]['scores'].iloc[0]

                    # Get actual score of game move based from top_df data.
                    gmove_score_from_top_df = top_df.loc[top_df.moves == gmove_san]['scores'].iloc[0]

                    print(board)
                    print()
                    print(epd)
                    print(f'top_1_score: {top1_score}, top_nnue_score: {top_nnue_move_score_from_top_df}, gmove_score: {gmove_score_from_top_df}')
                    print()

                    if board.turn == chess.WHITE:
                        data.append([gcnt, epd, fmvn, wp, gmove_san, gmove_score_from_top_df, top1_move, top1_score, top_nnue_move, top_nnue_move_score_from_top_df])
                    else:
                        data.append([gcnt, epd, fmvn, bp, gmove_san, gmove_score_from_top_df, top1_move, top1_score, top_nnue_move, top_nnue_move_score_from_top_df])

                    if abs(top1_score) > 500 and abs(gmove_score_from_top_df) > 500:
                        break

            # Save to dataframe.
            df = pd.DataFrame(data, columns=['GameNum', 'Epd', 'MoveNum', 'Name', 'GameMove', 'GameMoveScore', 'EngineMove', 'EngineScore', 'NNUEMove', 'NNUEScore'])
            alldf.append(df)

            if gcnt >= max_games:
                break

    engine.quit()

    # Save data to csv file. Print average error in cp.
    df = pd.concat(alldf, ignore_index=True)
    df.to_csv('data.csv', index=False)
    print(df)

    df_error = df.copy()

    df_error['HumanError'] = abs(df_error['EngineScore'] - df_error['GameMoveScore'])
    df_error['NNUEError'] = abs(df_error['EngineScore'] - df_error['NNUEScore'])
    print(df_error)

    average_human_error = df_error['HumanError'].mean().round()
    average_nnue_error = df_error['NNUEError'].mean().round()

    print(f'AveHumanError: {average_human_error}')
    print(f'AveNNUEError: {average_nnue_error}')


if __name__ == '__main__':
    main()

Modify the stockfish and pgn paths to suit your needs. See,

pgnfn = r'F:\project\pgn\tatamast23.pgn'
enginefn = r'F:\Chess\Engines\stockfish\stockfish_15.1_win_x64_popcnt\stockfish-windows-2022-x86-64-modern.exe'

command line

python main.py >log.txt

output

    GameNum                                                Epd  MoveNum                    Name GameMove  GameMoveScore EngineMove  EngineScore NNUEMove  NNUEScore  HumanError  NNUEError
0         1  5rk1/1p1q1ppb/2p1p2p/4P3/R7/2Q3P1/1P2PPBP/6K1 ...       23         Carlsen, Magnus      Rd4             52        Rd4           52      Rd4         52           0          0
1         1  5rk1/1pq2ppb/2p1p2p/4P3/3R4/2Q3P1/1P2PPBP/6K1 ...       24         Carlsen, Magnus      Rd6             62        Rd6           62      Rd7       -548           0        610
2         1  5rk1/1pq2ppb/2pRp2p/4P3/8/2Q3P1/1P2PPBP/6K1 b - -       24          Aronian, Levon      Ra8            -73        Qb6          -51     Qxd6       -520          22        469
3         1  r5k1/1pq2ppb/2pRp2p/4P3/8/2Q3P1/1P2PPBP/6K1 w - -       25         Carlsen, Magnus      Bf3             59         h4           69     Rxc6       -466          10        535
4         1  r5k1/1pq2ppb/2pRp2p/4P3/8/2Q2BP1/1P2PP1P/6K1 b...       25          Aronian, Levon      Bg6            -38        Bg6          -38     Qxd6       -504           0        466
..      ...                                                ...      ...                     ...      ...            ...        ...          ...      ...        ...         ...        ...
71        5                 8/p7/1p5R/3P1kp1/8/1K5P/r7/8 b - -       38  Abdusattorov, Nodirbek      Ra1             67        Rd2           67      Rd2         67           0          0
72        5                 8/p7/1p5R/3P1kp1/8/1K5P/8/r7 w - -       39        Rapport, Richard      Kc4           -124        Kc2          -59      Kc4       -124          65         65
73        5                  8/p7/7R/1pKP1kp1/8/7P/8/3r4 b - -       42  Abdusattorov, Nodirbek       b4            205         b4          205       a5         35           0        170
74        5                 7R/p7/8/2KP1kp1/1p6/7P/8/3r4 b - -       43  Abdusattorov, Nodirbek       b3            205         b3          205       a5        103           0        102
75        5               4R3/p7/8/2KP2p1/8/1p1k3P/8/3r4 w - -       46        Rapport, Richard       d6           -686        Rb8         -641       d6       -686          45         45

[76 rows x 12 columns]
AveHumanError: 8.0
AveNNUEError: 251.0

The function nnue_eval() will calculate the nnue eval of the legal moves from the given position or epd.

I place the code in github repository to easier maintain it.

3

I don't think it's possible to isolate positional understanding from tactics completely when discussing human players.

We can make some assessments when comparing engines among themselves. For instance, with the introduction of deep-learning-based engines, positional understanding has improved. But we can't really find an objective measurement of "positional strength" that would allow us to compare engines and humans.

2
  • Do you mean that discussing positional strength is meaningless since it can't be objectively measured? I'm not asking for numbers, but understanding.
    – Anna
    Commented Apr 14, 2023 at 6:33
  • @Anna not ncessarely. There is no objective measurement but we can still make assessments like "Magnus Carlsen has a better positional understanding than you and I". The problem here is that the way engines think about chess is very different from that of humans, so in this case the comparison is harder to make.
    – David
    Commented Apr 14, 2023 at 7:30
2

The main things which stockfish considers when choosing a move are:

  • Material Imbalance
  • Positional Advantage
  • Material Advantage
  • Strategic Advantage for Pawns
  • Strategic Advantage for other Pieces
  • Incoming threats
  • Passed Pawns
  • Space
  • King Safety

Nearly all of these things could be considered parts of "positional understanding" but to really understand how "strong" positional understanding is, looking for ideas and plans is necessary, stockfish does this using a optimized alphabeta pruning search, but this doesn't demonstrate any understanding of the board in front of it at all, so while stockfish with the ability to search has great ability to find and execute ideas, without that it has far less of an understanding than humans.

To put it another way, humans try to come up with general plans and find good squares, while stockfish can place value on squares, it can't picture the pieces going to them without the ability to search, so its positional understanding in that case is really quite bad. I haven't tested it or seen data for it, but i would wager it is far below club players without search depth.

3
  • Could you please explain what you mean by modern stockfish not using a neural net? I'm referring to the NNUE.
    – Anna
    Commented Apr 13, 2023 at 18:53
  • my apologies, i misread an article, it does use a neural net, but not ordinary machine learning techniques. i'll correct it now
    – brekker
    Commented Apr 13, 2023 at 18:56
  • Thanks for spending time on the answer. It seems though that "its positional understanding in that case is really quite bad" is mostly speculation? I'm also confused by your list, that seems to relate to pre-NNUE stockfish?
    – Anna
    Commented Apr 14, 2023 at 6:38
0

The idea that positional understanding and tactics are different is wrong. Even a beginner like myself needs both to play reasonably well. Without tactics I will never beat any decent player. Of course, being that this is Stockfish, it does a bit better than I would. Also, here are some in important things to consider in chess. They could be said to be either:

  1. Threats:You can make the arguement for both. Threats could be positional understanding. After all, they are something in your position that you need to understand (to avoid a blunder).
  2. King safety:Again, you can make the arguement for both. Understanding my position, I can realize that I must move my bishop to avoid mate in 1. Or I could say that I use tactics and realize that in order to win, I can't be checkmated, and reach the same conclusion
  3. A strategic advantage of any kind:The same as above. I can promote a pawn soon. I can checkmate soon. I can win a rook for a knight. Etc. I understand my position that I can promote a pawn. Or promoting that pawn is part of my tactics.
  4. Etc:Literally anything can be considered both. After all, everything a chess engine does is trying to win something. That is tactics. Also, there is no move that ignores the position, as that would be idiotic.

So, to say how stong positional understanding is without tactics (or vice versa) is moot. Also consider the following opposite question. How strong is tactics without positional understanding? The answer. Nonexistent. How can one have tactics if you have no understanding of your position? You can't. How can one have positional understanding without any tactics? You can't.

One other thing to note is that when the NNUE (Efficiently Updatable Neural Network) was added to Stockfish 12 it is increased the rating only by 100. While a bunch, it is not that large. This does indicate that Alpha Beta search was probably able to have positional understanding (because they are the same thing).

2
  • There's plenty of chess literature that distinguishes positional understanding from tactics, so unless I'm misunderstanding you this argument seem a bit reductive?
    – Anna
    Commented Apr 16, 2023 at 10:06
  • Really then name one concrete thing that is one and not the other? @Anna Commented Apr 16, 2023 at 10:07
0

It depends

Based on this blog:

Stockfish uses a static evaluation function, or a function which calculates an estimate evaluation of the position using a heuristic (a manually constructed algorithm designed to fit any arbitrary chess position) which has been hand-crafted by computer scientists and chess grandmasters for over a decade. Since the evaluation function is very powerful we can assume it will do a good job most of the time to evaluate a position. However, this is not enough for a game as complex as chess – it is a Dynamic Game as opposed to a simultaneous-move game and involves niche game tree paths which can never be fully encapsulated in the problem space of an evaluation heuristic. There is a high probability that the function will evaluate a position as winning and miss a line or path of moves by the opponent which causes us to actually lose. It can also evaluate a position as losing and miss a line which is actually winning. The static evaluation function capitalizes on the deterministic nature of the game, however we need to do more to deal with the element of randomness. Hence, it is necessary for the engine to be able to calculate into the future.

It states that depending on the position of the game, Stockfish when only using its static evaluation function can miss a far better outcome than what it plays and misevaluate the position. Therefore without the minimal and Alpha beta search methods, stockfish isn't considered very good when it comes to positional understanding, since it can misevaulate.

2
  • 1
    Thanks for your answer. If I'm not misunderstanding, Stockfish does not use the static evaluation function anymore, but instead a neural network (the NNUE)? It is the positional understanding of this network that I'm wondering about.
    – Anna
    Commented Apr 17, 2023 at 14:34
  • @Anna is right, Stockfish uses a NNUE since stockfish 12 Commented Apr 17, 2023 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.