Back in 2013 Steve Maughan posted an excellent YouTube video Create a Polglot Chess Opening Book in which he takes you through the steps to extract high quality games from a large pgn database, clean the files and create a Polyglot database bin file. Although almost 8 years old it is still very useful.
He uses the following free software:
Scid, which can be downloaded from Sourceforge. Alternatively Scid vs PC also works.
Polyglot, which can be downloaded from Ed Schröder's Rebel13 site.
pgn-extract, written by our very own David J Barnes (@kentdjb), which can be downloaded from his site.
Steve Maughan starts by downloading a large games database. He uses Ed Schröder's 2.2 million database from 2013. You can source your own such. Or if you want to follow in Maughan's footsteps then a 2.5 million database from 2015 is available for download on Ed Schröder's site. Download file here.
Next he extracts "high quality" white wins and black wins into two separate pgn files. He defines "high quality" to mean 2400+ winner and reasonable strength opponent. He does this using Scid's filter option to select the files and then saves the two resulting search sets. If you use the more modern Scid vs PC (as I do) then you want the "General" option in the Search menu instead of the "Header" option but it works in a similar way. Save the file under the Tools menu "Save all filter games" option.
Then he cleans his two high quality extract files using pgn-extract using the command line:
pgn-extract -s -C -N -V -tstartpos.txt -oclean_white2400.pgn white2400.pgn
and same again for black2400.pgn
Note that startpos.txt just has the starting fen position, i.e. fen and then "rnbq...NR w KQkq - 0 1". Polyglot, which is coming up next only handles games which start from the start position so this filter is also required.
Next use polyglot to "clean" the files. We want it to run silently and remove comments and variations and annotations.
First we want a file of short variations (max 16 ply) where there are at least 50 matching games in the database. So, mainline, more or less.
polyglot make-book -only-white -pgn clean_white2400.pgn -bin w1.bin -max-ply 16 -min-game 50
Second we want a file of more obscure (minimum of 5 matching games) opening variations in greater depth (up to 60 ply)
polyglot make-book -only-white -pgn clean_white2400.pgn -bin w2.bin -max-ply 60 -min-game 5
Then we merge the two white books:
polyglot merge-book -in1 w1.bin -in2 w2.bin -out w12.bin
Do the same for black and then merge the white and black books into one high quality book so:
polyglot merge-book -in1 w12.bin -in2 b12.bin -out HighQuality.bin
Obviously you can tune the criteria according to your views and requirements, but overall a very powerful set of tools.