Does the Stockfish engine use past matches, or does it not need such a "database"?

Question

I have a simple question. Does the Stockfish engine use, amongst the steps of its complicated algorithm, a library/database of past matches? For such an algorithm to work, it would to need store many matches that have had a particular scenario. Each scenario has a player who went on to win. Thus, it is provided some kind of rating point to judge each particular scenario with, allowing it to choose the best move. It's either that or the Stockfish engine works just by working through the possible moves to figure out how to win.

In other words, does the Stockfish engine need to be fed data from past matches or can it work without any such data ?

Answer 1

Technically, Stockfish does make use of past matches, just not when it's playing. The way it uses its past matches is during training, when a new neural network (NNUE in Stockfish's case) is being trained. Once the NNUE is trained, though, it is "standalone" and does not change anymore.

The Stockfish you can download and play has an already-trained NNUE, so it does not use past matches.

Answer 2

does the Stockfish engine need to be fed data from past matches

No.

You are confusing Stockfish with AlphaZero. The two engines work in entirely different ways.

Stockfish works by using brute force calculation plus clever evaluation functions to determine what is the best move in any position. During a game it maintains hash tables of positions and evaluations that have occurred during that game so that when it calculates a transposition of an earlier calculated position it doesn't redo the calculations but just uses the previous one.

If an opening database is available it can use that but it also works perfectly well albeit slower without. Similarly in the endgame it can use tablebases if available which can greatly increase speed and accuracy.

AlphaZero, however, does rely on millions of training games played against itself in order to update its policy (learn to play chess well). It does not need data from past matches after it has been trained in order to play chess - the experience obtained from the past games is implicitly contained in the weights of the neural network.

Answer 3

To answer your question concisely, SF does not need a 'database' as in raw data pulled from a bag, but its NNUE function concatenates the moves, move order, and outcomes of many games played against itself to make more informed move decisions based on learned strategies and deep thinking.

A database is like a grocery store. An NNUE is like a cookbook from a Top Chef (it's gonna contain precise measurements, ingredients, cooking times, and detailed preparation as well as pictures showing garnished finished products--maybe even some backstory)

Answer 4

If you see "NNUE" on the end of Stockfish's name, it means it's using something called a neural network (i.e. "Stockfish 15.1 NNUE"). A neural network works in a similar fashion to how your brain does. For example, you can look at an easy chess puzzle (say, a simple knight fork), and immediately know the right move. You don't have to check that the knight is attacking your opponent's king and rook, or that it is moving to a safe square. You know that it wins the exchange. Or, a better explanation might be the common description of Jose Raul Capablanca's abilities---many said that he could take one glance at a chessboard and know what the position needed.

That's a pretty simple explanation of how a neural network functions. Stockfish trains its neural network by playing millions of games with itself, and with each game the network gets a tiny bit better at evaluating positions. Then in an actual game or evaluation, it uses the network alongside traditional engine calculation to provide more accurate positional play (pawn structures and piece placement are what are really affected by neural network evaluation).

So the answer is yes and no. Stockfish doesn't reference a game database when it plays, but it does use a neural network that essentially makes it learn from its mistakes during self-play.

Answer 5

NNue is not trained through self play. It is using SF evaluations as target output data vector during some form of supervised learning. It does not use game but many positions with SF single root searches of moderate depth, as target output data to fit with generalization power (requires validation data, contrary to fishtest optimization, which tests and "train" on games and their outcome, but not as RL self-play scheduling).

I would love to be contradicted with specific link to support that.

Otherwise, for my claim, I point of SF12 Blog, where it is stipulated as above, in very few economic lines. Later, more recently, the blog mentions using also Leela's data, which I understand still as positions not games.

I currently understand that the master NN (if still called that) is still trained using a single SF search on the input position of the input data vector. It is an approximator of SF with either a simple non-NN evaluation (also looking for pointers), or some iteration of the process, where SF includes previous iteration of NNue approximating SF. A sort of feedback loop of self-approximation. But no self-play, other than in fishtest but self is about the name SF, as none of the versions in SF ever play against own clone (unless pointer). SFx vs SFy. Global optimization scheme based pools of dev. instances (with different parameter values).

There are rumors around that describe this (iteration over NNue SL training) as some sort of reinforcement learning (not at all what I think it means in ML). This might explain many propagated rumors using self-play and games, as if it was using Deep RL as in A0 or LC0. I don't mind being contradicted with new information.

Some old text from nodchip repository is still using such wording. Ai chat bot might still regurgitate the confusion. I think SF should get their explanation reviewed for that important chess interpretation question.

Answer 6

While I understand that Stockfish is working with NNUE instead of having a separate database that contains critical positions with their evaluation, this is not very useful for game analysis, and I would not currently use Stockfish for that purpose. I would rather use the derivative engines SugaR NN 1.1, or SugaR AI 2.5, which are based on the Stockfish code but which CONTAIN data of previous matches, so-called 'experience' files and even tools to view and edit them. Basically, I'm not interested in using the 'strongest' chess engine but the one which allows me to intelligently carry out my analysis, and these experience files are a great part of it, a 'must-have' to make the engine's calculation go in a certain direction.

So my answer is: NO, curently the main Stockfish engine doesn't have a separate 'learning' database, but some important derivative engines do have it. And YES, the main Stockfish engine should have such an accessible learning database, not for defeating other engines (NNUE can do that), but for offering to common chess players a useful tool for training or the analyses of their own games.

For example, I would highly like what every engine could have provided in the last 40 years: A simple possibility to enter manually an evaluation of a certain position that is then considered in future tree calculations. To my knowledge only the older Shredder engine has this feature, and only in its own GUI. With SugaR 2.5 you have a similar feature by editing the experience files, but it's a cumbersome process that could be simplified by providing it in the UCI options.