Tools for automated notation: webcam movie to PGN

Question

Sometimes during blitz games over the board, or when in zeitnot, it gets very difficult to annotate them.

I've seen several software / hardware to help in writing the notation, even one that's FIDE approved; these require that you repeat the moves on the device. Hardly faster than pen and paper, just as prone to errors, plus you risk typing the wrong move and wasting even more time.

DGT makes a board that will do this automatically, but it will only work with its pieces, and starts at 400€, double that for tournament boards. Some independent projects are available such as usb chessboard 101, or voice controlled chess robots, Chess board step sequencer. Additionally some open projects are available that even move pieces (or gigantic pieces).

I have a playoff tonight, and the tournament rules dictate shorter and shorter games until there is a winner. Hence I thought about shooting a movie of the board should we lose track. But then my programming background kicked in and I thought "there must be some software that can process a movie and write the PGN".

I searched everywhere, and found nothing.

1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?
2. Has anyone attempted such a feat and shared their research?
3. Are there obvious limitations that make this project unlikely to succeed?
4. Would you buy it if it existed?

Answer 1

It is clearly interesting and relevant to automatically record chess games (live boards are more or less tackling the same problem). As far as I know, there is no software for this purpose that exists today. To solve this problem, I think that the "simple" case should be as follows.

1. The chess board should be filmed from top-down to get a two-dimensional view. (Motivation: filming in a three-dimensional view would cause problems because of a skewed view and pieces that will block the view for other pieces behind them. Also, different 3D-views are possible which would counteract a clean procedure.)
2. The film should be manually edited to capture "clean board" moments and remove moments when the two players' hands or heads are blocking the view.
3. The image processing algorithm on the software side should detect changes on the board and record them as moves. It should also auto-rotate and transform the image to detect the board orientation and thereby the coordinates.
4. The software should use the starting position to "memorize" every piece type and then use these for the image recognition during the game. Probably, some clever tricks can be written to solve the problem of piece recognition in a smart way.

A quick Google search with the search string

chess game image processing

returned this interesting article

All in all, it should be realistic to implement this type of software. Yet it might be cheaper for a human to do the same work (transfer moves from a video clip to a computer).

Answer 2

It's 2019-12 now and 5 years after the original question.

1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?

   see

   • https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/19
   • https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/22 which are both closed by now. See http://lichess.bitplan.com for a game successfully recorded. Join https://gitter.im/play-chess-with-a-webcam/community to discuss

2. Has anyone attempted such a feat and shared their research?

   Yes - so far I found some 35 research papers and some 10 open source solutions and one commercial offering

   2.1 http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers

   2.2. http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Open_Source_Projects

   2.3. http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Commercial

   http://wiki.bitplan.com/index.php/PlayChessWithAWebCam is another project aiming at the goals stated by you. The project was started 2019-10-15 and in a few hours I already found a lot of resources e.g. this Question Some two months later I have updated my answer (see links above)

3. Are there obvious limitations that make this project unlikely to succeed? I can only answer from my own experience with a similar project I did 31 years ago see http://wiki.bitplan.com/index.php/MagneticSensorChessBoard1987. At that time i tried to create a chess board that would detect moves. I used magnets in the wooden pieces and reed contacts. The key to success was using a chess engine that new "the rules" to be able to concentrate on movements alone and not having to detect individual pieces. As outlined in an answer above promotions to non queen pieces might be one of the edge usecases that are difficult. Detecting the motion of pieces via picture differences is a well understood problem.

   Chess engines and UCI compatible GUIs are also available these days. So the problem is more an integration problem these days then solving the individual detail issues.

   A major obstacle is the difference between a "prototype" and a product. See e.g. http://www.nandanbanerjee.com/files/ICCVR-08AUG12-011%20paper.pdf

   a paper that is even older than this question but far from a product that you can buy at a reasonable price.

4. Would you buy it if it existed?

   Yes. When I bought a new chess board a few days ago I was looking for one that could record moves and it was still around 500 EUR. As it looks the hardware cost will be well below 150 EUR these days including a decent chess board and pieces.

   The software might well be opensource and free (as in free beer) soon. I'll update this answer accordingly.

Answer 3

In order,

1) Not to my knowledge

2) Not to my knowledge

3) There are several things that make this sort of project non-trivial, even difficult. Not the least of which is following the piece, differentiating it from a hand gesture, for example. Also determining when the move stopped would be an interesting question for computer vision to solve. These kinds of problems have been solved in other situations, but not with 100% accuracy, and certainly not at prices that are even close to being competitive with the DGT board.

Then there's the question of where the camera could be positioned. Directly overhead would offer the best view of the board, making it easier to determine the square with precision. But wherever the camera gets placed, there's a distinct probability its view will be obstructed by one of the players, either leaning over the board, reaching for the clock, or simply gesticulating.

4) Not unless it was really cheap. And I don't expect it would be. My reasoning is simple. As time controls shorten, the quality of the game lessens, to the point where it's no longer interesting to me

Answer 4

That would be a rather complex task. Maybe a reasonable first step is to look into recognizing FEN position from a single image - for which more resources exists, for example:

• Create a FEN from a chess diagram picture
• Recommended pattern recognition technique for chess board

Answer 5

I wrote this software. It contains webcam AI for move recognition, connected to Arena GUI.

https://www.youtube.com/watch?v=APkoE1UDKj0

Answer 6

I worked on the chess tracking project 8x8cam within image processing course in 2013 at FING UdelaR.

Then in 2014 other students created a better version for Android named chesstrack. Under it's web you can find videos and link to the code they used.

I would love to see a finalized app of this project idea I had for years. So if there is some news about this please leave a comment.

Answering your pointed questions:

1. Does any such thing exist (webcam/video to pgn converter? A prototype maybe?

   
   Apart from my project 8x8cam and chesstrack, I have recently found a project named ChessCam. All of these projects are Open Source and only need to be named as resource if you create a new project from them.

2. Has anyone attempted such a feat and shared their research?

   
   Yes. You can see such on previous question's answer.

3. Are there obvious limitations that make this project unlikely to succeed?

   
   The hardest part I believe is to make the software work no matter how the camera is placed and independently of the illumination. And also to work on fast paced games and work with the hand occlusions when the game is being played, in order to detect correctly the move done.

4. Would you buy it if it existed?

   
   Definitely yes.

Answer 7

Edit: promoting to anything different than queen will make everything explode. It will also be very hard to support this (my program for Part 2 supports underpromotion, but storing the info of Part 1 purely by reading pixels is a nightmare).

Edit2: I just noticed you mean real board chess to PGN, so this is all useless and I will most likely delete my answer.

Part 1: storing the squares using pixel readers

It is totally possible, the steps needed would be:

1. change the set up of the board so it highlights the last move (prefereably not just the border)
2. write a program that reads pixels on given (x,y)
3. make it read about 5+ pixels on each square (so it is impossible for the cursor/piece to cover all the pixels to be read)
4. convert the HEX to RBG, then give a range of accepted values of what we deem to be considered close to the "last move" color
5. save the moves on an array with a format ["h1-h8", "e4-e5", ...] just the initial and ending squares (order unimportant)

Note: from here we can't really know from which square was the initial/ending position, but there are only 2 posibilities and only 1 of them is legal.

I have written simple pixel-based macros/bots for RPGs games in C#, I can guarantee this is not that difficult, given that:

1. the cam is static (so the x,y values don't get messed up)
2. you previously know the background color or you can set it up before playing

Basically, it is best if the game is played online and you are recording directly from the screen.

Sorry, no program avaiable for part #1

---

Part 2: exporting that list into PGN

To turn this list into a real PGN, you can use a program I wrote in JavaScript that records moves, and one of the functions moveCaller() you pass two squares positions and it searches the final_pos in the legal moves of initial_pos and if there is a match then it calls makeMove(). So we can call it twice e5 to e4 and e4 to e5 and I guarantee it will ignore the illegal one.

1. download the project (direct link .zip)
2. uncompress the .zip
3. change the JavaScript of the index.html file with this:

'

$(function(){
    var i, len, arr, temp, m_board;

    m_board=IsepicChess.initBoard({name : "main"});

    arr=["e4-e2", "e7-e5", "f3-g1"]; //is ok for the initial and final squares to be swapped

    for(i=0, len=arr.length; i<len; i++){
        temp=arr[i].split("-");

        //one of the two following lines will silently fail
        m_board.moveCaller(temp[0], temp[1]);
        m_board.moveCaller(temp[1], temp[0]);
    }

    m_board.refreshBoard();
});

.4. copy the move list and add some PGN headers above it like [WhiteElo "2400"] etc.

Answer 8

Joey Meyer wrote and built "Raspberry Turk":

The Raspberry Turk is a robot that can play chess—it's entirely open source, based on Raspberry Pi, and inspired by the 18th century chess playing machine, the Mechanical Turk.

This goes beyond the OP's question because it also incorporates (a) the engine to calculate responses and (b) a robotic arm to make the engine's moves on the board.

But it answers the OP's question because one component of Raspberry Turk includes an overhead camera and a machine-learning component to learn how to convert what it sees on the board into a PGN representation of the board.

• You can see it in action at this YouTube video "Chess Playing Robot Powered by Raspberry Pi - Raspberry Turk". (Note that the video is sped up by about a factor of 12, per personal communication with the creator.)
• For an accessible explanation, see Chaim Gartenberg, "The Raspberry Turk is a DIY chess robot with instructions to build your own*," The Verge, March 24, 2017.
• The code is available at GitHub » joeymeyer/raspberryturk
• The project website is raspberryturk.com/

Answer 9

Firstly, great post. I have been surveying this issue recently and the answers here are a great source of information. I have found an excellent free app for Android and iOS called KnightVision. It is relatively new. It was trained on various chess sets. It even managed to properly recognize my hand-crafted pieces which are not identical to the Staunton standard. It works well, although to be on the safe side, it requires a separation of about 3 seconds between each move on the board. So be gentle when swapping pieces, which is something that is usually done quite quickly. It also has a web page, where after logging in you can see the game PGN that you can copy and export to any chess engine for studying. It also has some analysis on the website itself. https://www.knightvision.io/

Answer 10

I won't be answering the question, because I don't know if there's a software out there. I'm just saying how I think that's how to do it, I don't know for sure, so...

---

Okay first you need to make sure that the video is recorded from a chess software, not a live tournament, just to make your job easier.

You need to convert the movie to images

ffmpeg -i foo.avi -vsync 0 -vf select="eq(pict_type\,PICT_TYPE_I)" -s WxH -f image2 foo-%03d.jpeg

• Source

That was easy, now the hard part, you need to start doing image processioning, for that, the most powerful is OpenCV.

For that, there are several techniques, I'll give you the links so you can read them.

• Chessboard picture recognition project
• Recommended pattern recognition technique for chess board
• How does OpenCV find chessboard corners?
• Using Image Processing Techniques to Automate Chess Game Recording
• Automatic Chessboard Detection for Intrinsic and Extrinsic Camera Parameter Calibration
• Camera Calibration and 3D Reconstruction

It is complicated but simpler than you think, because you only need to know what is the starting position, loop through all the images and see the difference. Take a look at OpenCV docs you'll find tons of materials on that subject.

---

Oh by the way, what you're asking for is done already, but not using software as far as I know, there's an old project that was coded using Pascal, cameras placed over the board, robots do image processing and make a move. Nowadays many robots play chess.

You could buy an Arduino uno, and its light censor, it will detect the white pieces and the black pieces, some coding and it will play chess. :)

Answer 11

Squareoff is making another board at 200$. Expensive but not extortionist. They say it will be available March 2021. Automatically moves pieces too. Details. Has Blitz mode but may not be so good for it. May not allow more than one user per board but plays through chess.com and maybe lichess.

Answer 12

I use the Monroi Chess Notation Device for all my chess games and it is basically an electronic board.