Extract anonymized games out of a pgn

Question

I'm a big fan of pgn-extract, but I am open to other linux-friendly tools. I have a big pgn file (over 5 million games) and a set of ECO codes. My goal is to fetch all the games matching those ECO codes, but with removing or fuzzying the players names and ratings, the year played, and the game result.

I then plan to open a random game with Scid and analyze it, since removing the players' data and game result will help me avoid any bias.

So far Im using pgn-extract like this: pgn-extract -t filter_for_naked --noresults --notags --nocomments --noduplicates -bl20 --output naked.pgn my.pgn

But Scid is having problems opening the file. As expected each game is just the set of moves (no headers at all) and I suspect Scid requires at least some headers.

Any advise on how to use pgn-extract or Scid to achieve this?

Answer 1

Thanks for your compliment about pgn-extract (I am its author). You might be able to make the task a little easier by adding --linelength 1000 to the argument list in order to get each game on a single line, then use sed to add an unknown result to the end of the game and re-process the game with pgn-extract to add an empty 7-tag roster. Finally, use sed again to edit the White and Black tags.

The resulting command would look something like the following, although sed experts might be able to clean it up a bit:

pgn-extract -t filter_for_naked --noresults --notags --nocomments --noduplicates -bl20 --linelength 1000 my.pgn | sed -e 's/..*/& */' | pgn-extract -7 | sed -e 's/White "?"/White "AnonW"/;s/Black "?"/Black "AnonB"/' > naked.pgn

You will get some error reports for games that end in mate having inconsistent results but you can hide those in a logfile.

Answer 2

Scid vs PC requires [Black "AN Other1"] [White "AN Other2"] [Result ""].
Since this was cloned from SCID I'm pretty sure SCID will be the same. I suggest you write a simple program to add these 3 header fields with the [Result] field matching the result at the end of the game.

Answer 3

You can use sed '/Result/c[Result ""]' my.pgn > no_results.pgn to replace lines containing the string Result with whatever you want the line to be. You can adopt this for whatever else you want to replace. You want to keep the c.

Answer 4

You can always copy the pure text from a PGN manually, but that only works for one game.

For more games I am not sure, and unfortunately most chess software is for windows.