How to create your own opening book for your own chess engine?

Question

I am currently working on a chess engine in Java. I am using the minimax algorithm with alpha-beta pruning to select the best move. However, it is inefficient to use the algorithm for selecting opening moves.

So here is my question for people who have experience with making their own engines:

How do you make your own opening book? Do you create your own database to store different openings and their successive variations? Or is their a general database that one can connect to and extract moves?

I would gladly appreciate any help from anyone who has experience or knowledge on this.

Answer 1

It may depend a little on what your code can access, but either way you should read Ed Schröder's pages on this topic. Schröder was the author of the Rebel Chess software until his retirement about a decade ago. In the '90s Rebel was the strongest consumer/commercial chess software and the first such to beat a GM (Anand). Since retiring he turned Rebel into ProDeo and released it for free on his website.

Start with this introduction, though you should already know all that, then have a look at this more in depth explanation. Then check this page for details on adding Rebel/ProDeo books to any engine (will require compiling C code for each platform you want your engine to be able to work). You should be able to add to the books from PGN records before compiling them.

EDIT: Not quite sure why this was voted down, but probably the mention of C when the OP states Java. I should mention that most chess processing software is in C or C++ because it provides better processing capabilities; it compiles to assembly language or machine code, unlike Java, which requires the JVM to run and that makes Java less efficient. Yes, it is possible to compile Java into machine code, but hardly anyone ever does it because then it is no longer platform independent (such as it is). So the OP will be running into the C hurdle no matter what.

Beyond that, these sites on the chess programming wiki on opening books and opening suites may be of use. You'll find a bunch of Ed Scröder's stuff there because he is one of the pioneers who made the shift from hardware electronic chess systems (e.g. Mephisto) to personal computing software. With the added bonus that now he's giving his software and, more importantly, his knowledge away to whoever wants it. If that's really not enough to immerse one's self in the subject then nothing is.

Answer 2

Back in 2013 Steve Maughan posted an excellent YouTube video Create a Polglot Chess Opening Book in which he takes you through the steps to extract high quality games from a large pgn database, clean the files and create a Polyglot database bin file. Although almost 8 years old it is still very useful.

He uses the following free software:

Scid, which can be downloaded from Sourceforge. Alternatively Scid vs PC also works.
Polyglot, which can be downloaded from Ed Schröder's Rebel13 site.
pgn-extract, written by our very own David J Barnes (@kentdjb), which can be downloaded from his site.

Steve Maughan starts by downloading a large games database. He uses Ed Schröder's 2.2 million database from 2013. You can source your own such. Or if you want to follow in Maughan's footsteps then a 2.5 million database from 2015 is available for download on Ed Schröder's site. Download file here.

Next he extracts "high quality" white wins and black wins into two separate pgn files. He defines "high quality" to mean 2400+ winner and reasonable strength opponent. He does this using Scid's filter option to select the files and then saves the two resulting search sets. If you use the more modern Scid vs PC (as I do) then you want the "General" option in the Search menu instead of the "Header" option but it works in a similar way. Save the file under the Tools menu "Save all filter games" option.

Then he cleans his two high quality extract files using pgn-extract using the command line:

pgn-extract -s -C -N -V -tstartpos.txt -oclean_white2400.pgn white2400.pgn   

and same again for black2400.pgn

Note that startpos.txt just has the starting fen position, i.e. fen and then "rnbq...NR w KQkq - 0 1". Polyglot, which is coming up next only handles games which start from the start position so this filter is also required.

Next use polyglot to "clean" the files. We want it to run silently and remove comments and variations and annotations.

First we want a file of short variations (max 16 ply) where there are at least 50 matching games in the database. So, mainline, more or less.

polyglot make-book -only-white -pgn clean_white2400.pgn -bin w1.bin -max-ply 16 -min-game 50

Second we want a file of more obscure (minimum of 5 matching games) opening variations in greater depth (up to 60 ply)

polyglot make-book -only-white -pgn clean_white2400.pgn -bin w2.bin -max-ply 60 -min-game 5

Then we merge the two white books:

polyglot merge-book -in1 w1.bin -in2 w2.bin -out w12.bin

Do the same for black and then merge the white and black books into one high quality book so:

polyglot merge-book -in1 w12.bin -in2 b12.bin -out HighQuality.bin

Obviously you can tune the criteria according to your views and requirements, but overall a very powerful set of tools.