Optimising Stockfish parameters when batch-analyzing with python-chess

Question

I am running a python script that analyzes a large batch of PGN games (each of ~5,000 games) across and exports the games evaluations. It calls Stockfish 14 to analyze each game at depth 20. **[EDIT: I've since read that newer versions of Stockfish can actually take LONGER to reach the same depth as older versions, because they have a wider search and thus more nodes per second. So I am considering setting a position time limit rather than depth limit.] **I am using a virtual machine with 32 vCPUs and 64GB RAM but processing is slow (though htop claims it is using 99% of CPU%). I haven't modified any Stockfish settings since installing.

1. What parameters should I set to minimize the processing time?
2. How can I change these settings so that they will be recognized by the call from python-chess? The relevant lines of the python code are:

engine = chess.engine.SimpleEngine.popen_uci(stockfish)
engine_depth = 20
def evaluate_game(game, engine, limit=chess.engine.Limit(depth=engine_depth)):
   board = game.board()
   info = engine.analyse(board, limit=limit)

Answer 1

The link refers to stockfish 11 and you are now using stockfish 14, so that article is irrelevant.

1. To minimize processing time you can adjust the depth or movetime or both. The question is what analysis quality are you up to?

2. To control analysis time use the following example.

engine = chess.engine.SimpleEngine.popen_uci(stockfish)
movetimesec = 5
def evaluate_game(game, engine, limit=chess.engine.Limit(time=movetimesec)):
   board = game.board()
   info = engine.analyse(board, limit=limit)

To use both depth and time you can use the following.

engine = chess.engine.SimpleEngine.popen_uci(stockfish)
movetimesec = 5
depth = 20
def evaluate_game(game, engine, limit=chess.engine.Limit(time=movetimesec, depth=depth)):
   board = game.board()
   info = engine.analyse(board, limit=limit)

In this case, when the engine reaches depth 20 before the time limit of 5 sec, then the search is aborted. If analysis time of 5 sec is reached first before depth 20 then the search is also aborted.

In ending phase reaching depth 20 is very fast as there are less pieces to evaluate while in the opening phase reaching depth 20 could be a challenge and movetimesec of 5 might be consumed first, depends on your hardware. It is a matter of analysis quality you want and the total time to finish the analysis.

Measuring quality and analysis time

1. Analyze game1 (more games and get the average) with depth 20, remember the total time.
2. Analyze game1 (more games and get the average) at movetime of 5 sec, remember the total time.
3. Run an engine vs engine match, sf14_depth20 vs sf14_5sec.

step 1 and 2 compare the total analysis time while step 3 compares the analysis quality.

I guess sf14_5sec will win over sf14_depth20, but you have to test this on that hardware.

If sf14_5sec will beat sf14_depth20 in a match of 100 or so games and step 2 is lower than step 1 then sf14_5sec would be a candidate for a good setting.

Now do some comparisons with sf14_5sec vs sf14_5sec_depth20 and sf14_depth20 vs sf14_5sec_depth20. Keep a note on quality and total analysis time.