5

Is there a fast way to generate FEN strings for every move in a PGN?

I'm generating an opening book using the Lichess elite database and Python Chess, but Python Chess is taking a surprisingly long time to analyze games and generate FEN strings for each move. Like, 1/1000th the speed I expected.

This is how I've done it - it works, but again, slow. board = game.board() is the primary slowdown - but I don't see a way to get FENs directly from the game object.

# PARSE PGNS and create an opening book using a giant dictionary

import chess
import chess.pgn
import io
import datetime

#START WITH SOME DUMMY DATA FOR DEBUGGING

games = '''[Event "Rated Rapid game"]
[LichessURL "https://lichess.org/YcFqJqyM"]
[Date "2020.06.01"]
[Round "-"]
[White "Poecraft94"]
[Black "Germanvince"]
[Result "1-0"]
[WhiteElo "2323"]
[BlackElo "2415"]
[ECO "B50"]
[Opening "Sicilian Defense"]
[TimeControl "600+0"]
[UTCDate "2020.06.01"]
[UTCTime "00:00:01"]
[Termination "Normal"]
[WhiteRatingDiff "+9"]
[BlackRatingDiff "-8"]

1. e4 c5 2. Nf3 d6 3. Nc3 Nf6 4. g3 g6 5. Bg2 Bg7 6. O-O O-O 7. d4 cxd4 8.
Nxd4 a6 9. Re1 Nbd7 10. Bg5 Rb8 11. Nd5 e6 12. Nxf6+ Bxf6 13. Bxf6 Nxf6 14.
Qd2 Qc7 15. Rad1 Kg7 16. Nf3 Rd8 17. e5 Ne8 18. Re3 b5 19. Rd3 d5 20. Rc3
Qe7 21. Nd4 Bb7 22. Rf3 Rbc8 23. g4 h6 24. Rh3 g5 25. f4 gxf4 26. Rh5 Nc7
27. Qxf4 Rh8 28. Rf1 Rcg8 29. Rf5 Rf8 30. g5 h5 31. Bh3 b4 32. b3 Nb5 33.
Rf6 Nxd4 34. Qxd4 Bc6 35. R1f2 Bd7 36. Qa7 Rh7 37. g6 1-0

[Event "Rated Blitz game"]
[LichessURL "https://lichess.org/AY9kSWFt"]
[Date "2020.06.01"]
[Round "-"]
[White "sandstorm00"]
[Black "yendorzerep"]
[Result "1/2-1/2"]
[WhiteTitle "NM"]
[BlackTitle "IM"]
[WhiteElo "2497"]
[BlackElo "2426"]
[ECO "B06"]
[Opening "Modern Defense: Standard Line"]
[TimeControl "180+0"]
[UTCDate "2020.06.01"]
[UTCTime "00:00:19"]
[Termination "Normal"]
[WhiteRatingDiff "-1"]
[BlackRatingDiff "+1"]

1. e4 g6 2. d4 Bg7 3. Nc3 c6 4. Be3 d5 5. e5 f6 6. exf6 Nxf6 7. Bd3 O-O 8.
h3 Nbd7 9. Nf3 Qc7 10. Qd2 e5 11. dxe5 Nxe5 12. Nxe5 Qxe5 13. O-O-O Bf5 14.
Bxf5 Qxf5 15. g4 Qf3 16. Rhg1 Ne4 17. Nxe4 Qxe4 18. h4 a5 19. h5 a4 20. a3
Rae8 21. Rde1 Qe5 22. c3 Qd6 23. f4 Re4 24. hxg6 hxg6 25. f5 Rfe8 26. Bf2
Rxe1+ 27. Bxe1 d4 28. Bf2 Re4 29. fxg6 Qxg6 30. Bxd4 Bxd4 31. cxd4 Rxg4 32.
Rxg4 Qxg4 33. Qe3 Kf7 34. Kd2 Qe6 35. Qf4+ Ke7 36. Qc7+ Qd7 37. Qxd7+ Kxd7
38. Kc3 Kd6 39. Kb4 b5 40. b3 axb3 41. Kxb3 c5 42. dxc5+ Kxc5 43. Kc3 Kc6
44. Kb4 Kb6 45. a4 bxa4 46. Kxa4 1/2-1/2'''

# If you have the Lichess elite database, yoou can parse the entire thing by uncommenting here

#with open('/lichess_elite_2020-06.pgn') as f:
#    games = f.read()

games = games.replace("\n\n[", "||||[")
pgns = games.split("||||")

import time

fens = {}
start = datetime.datetime.utcnow()

for i, pgn in enumerate(pgns):
    game = chess.pgn.read_game(io.StringIO(pgn)) #parse each pgn string
    if game.errors:
        continue

    parent_pos = None
    move_num = 0
    
    while game:
        move_num += 1
        if move_num > 40: #practically, we only need to first 20 moves for our book
            break

        game=game.next()
        
        if game:
            board = game.board()
            pos = board.fen(en_passant='fen') #grab fen string of current position
            move = game.san() #what was the last move made?

            if pos not in fens:
                fens[pos]={}

            if parent_pos:
                #in the previous position, the current move was made X times
                if pos in fens[parent_pos]:
                    fens[parent_pos][move] += 1
                else:
                    fens[parent_pos][move] = 1

        parent_pos=pos

print('Run time: ', (datetime.datetime.utcnow()-start).seconds)

This yields a data structure with FEN strings as keys, and the most common replies as values. E.g. after parsing the 2 games, e4 is followed by g6 once and c5 once.

{
    'rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1': {
        'c5': 1,
        'g6': 1}
...
}```
Improve this question
5
  • 1
    "Fast" is relative: Object-oriented+Python = you see the taillights of me old FORTRAN guy :-) Still, I program in Python a lot and a simple string parsing app should work fast enough (maybe unless you have millions of games), so mind to give some concrete timing values? And structure and example data for your variables? It might work faster the obvious way: starting from RNBQ.../ and parse the moves of the PGN directly, updating the FEN.
    – Hauke Reddmann
    Commented Jul 18, 2021 at 15:28
  • @HaukeReddmann One caveat is that I need the strings to comply with the FEN standard for my downstream use case. That means encoding legal en-passant moves, and so on. So, for that reason, I'd love to NOT generate the FEN strings myself!
    – MattY
    Commented Jul 18, 2021 at 18:38
  • @HaukeReddmann I updated the post with complete code. Note that the script's able to process 100 games every 2 seconds, which means my little database of 400k games will take over 2 hours, and larger databases will take forever.
    – MattY
    Commented Jul 18, 2021 at 22:58
  • I took a look at the documentation of PythonChess. First Impression: Very meticulous, very by-the-book-of-good-programming. But your use case needs speed (100 games/2 seconds sounds slow to me either) and it might be more effective to grin and bear it, throw the nice features of PythonChess out of the window, program from scratch and optimize for speed.
    – Hauke Reddmann
    Commented Jul 19, 2021 at 6:37
  • 1
    FYI, game.board() is expensive because it is re-evaluating the entire game from move one every time you call that method! Since the logic in that method does not look terribly complicated, you could probably rewrite it yourself to iterate over each board in a single pass.
    – Kevin
    Commented Jul 19, 2021 at 22:09

2 Answers 2

Reset to default
5

If you are not tied to using PythonChess and simply want FEN encodings of each position, you might like to consider using my free, open-source pgn-extract program with its -Wepd option to output in EPD format. It is a compiled C program so likely to be significantly faster for this use case. For instance:

pgn-extract --quiet -Wepd games.pgn

Here are the first few lines of its output for one of the games quoted in the OP:

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
rnbqkbnr/pppppp1p/6p1/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;

If you wish to suppress irrelevant en passant squares then there is also the --nofauxep option to do that.

Improve this answer
2
  • 1
    I compiled and tested pgn-extract by piping the first 1000 games in my database. The FENs were generated in about 1 second. So, yes, I think it's worth making the switch! Even considered writing a python-wrapper for pgn-extract, but unsure if there is enough demand to justify that.
    – MattY
    Commented Jul 21, 2021 at 20:48
  • I am glad that worked for you.
    – kentdjb
    Commented Jul 23, 2021 at 6:57
0

As Kevin indicated, you're reevaluating your entire position after every move. A better way to code would be to use the next method. Below is a code snippet that should give you an idea:

import chess.pgn
pgn = open("anderssen_kieseritzky_1851.pgn")
mygame=chess.pgn.read_game(pgn)
while mygame.next():
    mygame=mygame.next()
    print(mygame.board().fen())
Improve this answer
5

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.