Program to merge chess opening polyglot database files with recalculation of weights

Question

I have been trying to merge polyglot chess opening databases. When merging databases with existing programs, if any move in a position already exist in the first file, new moves from same position will not be added from the next file.

Here is the problem: When two books (in bin-format) are merged with polyglot, it's expected that lines from both of them can be found in the new combined/merged one. This is not the case: the contents of the second entry are omitted:

pgn1:

[Event "?"]
[Site "?"]
[Date "2021.01.01"]
[Number "1"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
[ECO "B74"]
[PlyCount "12"]

1. e4 c5 1/2-1/2

pgn2:

[Event "?"]
[Site "?"]
[Date "2021.01.01"]
[Number "1"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
[ECO "A00"]
[PlyCount "12"]

1. g3 e5  1/2-1/2

After doing:

polyglot make-book -pgn 1.pgn -min-game 1 -max-ply 12 -bin 1.bin

polyglot make-book -pgn 2.pgn -min-game 1 -max-ply 12 -bin 2.bin

polyglot merge-book -in1 1.bin -in2 2.bin -out combined.bin

the g3 e5 entry gets scraped.

For reference, the above definition of the problem was described here: https://www.mail-archive.com/[email protected]/msg1798406.html

This is something I have found really annoying for a very long time. After years of frustration that there was no program that could properly merge chess polyglot database files with recalculation of weights, I eventually decided to learn how to make a program that could do that.

So, now I have written a program to merge chess opening polyglot database files with recalculation of weights and it solves this problem. I call it merge_polyglot.

But I don´t really have any clue what to do with it. Would others find this useful? Where would I post this program to make it available for other chess enthusiasts?

Answer 1

You may use the more versatile subcommand jja merge of the tool jja:

⇒  jja merge -h
Merge two opening books

Usage: jja merge [OPTIONS] --input1 <input-file-1> --input2 <input-file-2> --output <output-file>

Options:
  -1, --input1 <input-file-1>
          First opening book
      --porcelain[=<FORMAT>]
          Give the output in an easy-to-parse format for scripts [default: auto] [possible values: auto, csv]
  -2, --input2 <input-file-2>
          Second opening book
  -v, --verbose...
          Run verbosely, may be specified multiple times [env: JJA_VERBOSITY=]
  -o, --output <output-file>
          Path to the output chess file
  -T, --threads <threads>
          Specify maximum number of threads [env: JJA_NPROC=] [default: 4]
  -s, --strategy <merge-strategy>
          Merge strategy [default: sum] [possible values: avg, max, min, ours, sum, wavg]
  -w, --weight1 <merge-book1-weight>
          Book 1 weight for merge strategy weighted average, aka `wavg'
  -W, --weight2 <merge-book2-weight>
          Book 2 weight for merge strategy weighted average, aka `wavg'
  -c, --weight-cutoff <merge-weight-cutoff>
          Moves less than this weight will not be included in the book
  -h, --help
          Print help
  -V, --version
          Print version

To solve your problem you should use a merge strategy which suits your needs. If you want to keep both entries you would go for either avg, sum, or wavg, max and min which states how the weight field should be updated for entries with same moves with averaging them, summing them and weighted averaging, using the maximum weight, and using the minimum weight being five options.

If you want to keep a single field, then you should use ours strategy which keeps entries in the first book intact when both book have the same position. I use this very often when I merge a small book with my own openings over a huge book generated from a database of lichess games so I get to use the information from lichess database whilst still having my own preferred moves selected for the positions I want. Another common option I use is --weight-cutoff with which I can get rid off very weak moves in the opening book.

Finally I should admit I am the author of jja so this is in-part self-promotion for which I apologize.