19

Sometimes during blitz games over the board, or when in zeitnot, it gets very difficult to annotate them.

I've seen several software / hardware to help in writing the notation, even one that's FIDE approved; these require that you repeat the moves on the device. Hardly faster than pen and paper, just as prone to errors, plus you risk typing the wrong move and wasting even more time.

DGT makes a board that will do this automatically, but it will only work with its pieces, and starts at 400€, double that for tournament boards. Some independent projects are available such as usb chessboard 101, or voice controlled chess robots, Chess board step sequencer. Additionally some open projects are available that even move pieces (or gigantic pieces).

I have a playoff tonight, and the tournament rules dictate shorter and shorter games until there is a winner. Hence I thought about shooting a movie of the board should we lose track. But then my programming background kicked in and I thought "there must be some software that can process a movie and write the PGN".

I searched everywhere, and found nothing.

  1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?
  2. Has anyone attempted such a feat and shared their research?
  3. Are there obvious limitations that make this project unlikely to succeed?
  4. Would you buy it if it existed?
6
  • 3
    it would be an interesting computer vision / machine learning project!
    – Memming
    Feb 11, 2014 at 12:10
  • I think the board has to be filmed from top-down to produce a "2D"-image. Otherwise, there is an additional problem of transposing a 3D-view to a 2D-view and different 3D-angles have to be treated separately.
    – user2001
    Feb 13, 2014 at 13:15
  • actually it's simply a linear transformation to "unstretch" the board and make all its squares square. the main problem is to recognize the square the piece is moved from if the angle is such that pieces can line up behind each other.
    – peter
    Feb 26, 2015 at 11:13
  • Maybe we could do some sort of crowdsourcing for this project. I think it's very interesting. I bet that some college student(s) could do this for senior projects. We need to find a way to communicate the idea to professors. Maybe with two cameras placed in two different locations this can be done reliably. I would be interested in the 3D approach, not 2D from overhead. Mounting an overhead camera is too impractical. I would also be interested in software that would take inputs from a few different cameras shooting several games simultaneously played on a long table and produce notation - for
    – user8450
    Oct 8, 2015 at 12:33
  • 1
    Given that kickstarter.com/projects/infivention/… was already successful a crowdfunding project seems feasible indeed. Especially since the cost for end users would be very low. Oct 21, 2019 at 5:03

12 Answers 12

5
+50

It is clearly interesting and relevant to automatically record chess games (live boards are more or less tackling the same problem). As far as I know, there is no software for this purpose that exists today. To solve this problem, I think that the "simple" case should be as follows.

  1. The chess board should be filmed from top-down to get a two-dimensional view. (Motivation: filming in a three-dimensional view would cause problems because of a skewed view and pieces that will block the view for other pieces behind them. Also, different 3D-views are possible which would counteract a clean procedure.)
  2. The film should be manually edited to capture "clean board" moments and remove moments when the two players' hands or heads are blocking the view.
  3. The image processing algorithm on the software side should detect changes on the board and record them as moves. It should also auto-rotate and transform the image to detect the board orientation and thereby the coordinates.
  4. The software should use the starting position to "memorize" every piece type and then use these for the image recognition during the game. Probably, some clever tricks can be written to solve the problem of piece recognition in a smart way.

A quick Google search with the search string

chess game image processing

returned this interesting article

All in all, it should be realistic to implement this type of software. Yet it might be cheaper for a human to do the same work (transfer moves from a video clip to a computer).

6
  • 1
    It took me a while to go through the study by G.D.Illeperuma you linked, it is most interesting. But it does exclude a priori the possibility of elaborating 3D skewed images. Thank you, a very interesting article. Feb 15, 2014 at 21:17
  • @RiccardoZorn you are welcome! I must admit that I don't see the point of handling 3D skewed images. The situation is that if a human has trouble documenting the moves from a video recording, then an image processing software will have even more difficulty.
    – user2001
    Feb 16, 2014 at 1:02
  • Setting up a vertical camera requires a lot of space, is invasive, and you need a custom support/stand. This makes it impractical :-( Feb 16, 2014 at 8:22
  • A more simple and cheap "live board" technology might be a better way to go in that case. Optionally, film from a reasonable 3D angle and let a human extract the chess notation from the video recording.
    – user2001
    Feb 16, 2014 at 11:58
  • the file researchgate.net/publication/… doesnt seem to exist anymore. does anyone have a copy?
    – peter
    Oct 24, 2016 at 10:58
7

It's 2019-12 now and 5 years after the original question.

  1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?

    see

  2. Has anyone attempted such a feat and shared their research?

    Yes - so far I found some 35 research papers and some 10 open source solutions and one commercial offering

    2.1 http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers

    2.2. http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Open_Source_Projects

    2.3. http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Commercial

    http://wiki.bitplan.com/index.php/PlayChessWithAWebCam is another project aiming at the goals stated by you. The project was started 2019-10-15 and in a few hours I already found a lot of resources e.g. this Question Some two months later I have updated my answer (see links above)

  3. Are there obvious limitations that make this project unlikely to succeed? I can only answer from my own experience with a similar project I did 31 years ago see http://wiki.bitplan.com/index.php/MagneticSensorChessBoard1987. At that time i tried to create a chess board that would detect moves. I used magnets in the wooden pieces and reed contacts. The key to success was using a chess engine that new "the rules" to be able to concentrate on movements alone and not having to detect individual pieces. As outlined in an answer above promotions to non queen pieces might be one of the edge usecases that are difficult. Detecting the motion of pieces via picture differences is a well understood problem.

    Chess engines and UCI compatible GUIs are also available these days. So the problem is more an integration problem these days then solving the individual detail issues.

    A major obstacle is the difference between a "prototype" and a product. See e.g. http://www.nandanbanerjee.com/files/ICCVR-08AUG12-011%20paper.pdf

    a paper that is even older than this question but far from a product that you can buy at a reasonable price.

  4. Would you buy it if it existed?

    Yes. When I bought a new chess board a few days ago I was looking for one that could record moves and it was still around 500 EUR. As it looks the hardware cost will be well below 150 EUR these days including a decent chess board and pieces.

    The software might well be opensource and free (as in free beer) soon. I'll update this answer accordingly.

3
5

In order,

1) Not to my knowledge

2) Not to my knowledge

3) There are several things that make this sort of project non-trivial, even difficult. Not the least of which is following the piece, differentiating it from a hand gesture, for example. Also determining when the move stopped would be an interesting question for computer vision to solve. These kinds of problems have been solved in other situations, but not with 100% accuracy, and certainly not at prices that are even close to being competitive with the DGT board.

Then there's the question of where the camera could be positioned. Directly overhead would offer the best view of the board, making it easier to determine the square with precision. But wherever the camera gets placed, there's a distinct probability its view will be obstructed by one of the players, either leaning over the board, reaching for the clock, or simply gesticulating.

4) Not unless it was really cheap. And I don't expect it would be. My reasoning is simple. As time controls shorten, the quality of the game lessens, to the point where it's no longer interesting to me

1
  • You make a great point at 4: the quality is indeed poor. Not worth the hassle of setting up a camera, much less spending a month to write the software. I just won the playoff, but I am embarassed of how poorly I played. Feb 10, 2014 at 1:23
4

That would be a rather complex task. Maybe a reasonable first step is to look into recognizing FEN position from a single image - for which more resources exists, for example:

4
  • Excellent thank you. I read both and they are really interesting. Chessgrabber is a recent project and is definitely more suited, I contacted the developer. Feb 10, 2014 at 2:21
  • I'd think that from this first step it wouldn't be too difficult to convert a video to PGN - take images every few milliseconds, analyse them, and if the position has changed in a legal way add the new position to the file.
    – rlms
    Feb 11, 2014 at 17:32
  • Update: the developer responded, it will never work for 3-d skewed images with overlapping pieces. Too bad, without that as a base, the project will never start. Feb 12, 2014 at 9:55
  • recognizing a still image is actually harder than having the video leading up to it, since the latter eliminates the problem of having to be able to identify the pieces.
    – peter
    Oct 24, 2016 at 11:01
4

I wrote this software. It contains webcam AI for move recognition, connected to Arena GUI.

https://www.youtube.com/watch?v=APkoE1UDKj0

1
  • Very impressive Marius, congratulations! Is the alpha available for testing already? Can you expose the moves through a service, for use by other software? Jun 15, 2015 at 12:36
3

I worked on the chess tracking project 8x8cam within image processing course in 2013 at FING UdelaR.

Then in 2014 other students created a better version for Android named chesstrack. Under it's web you can find videos and link to the code they used.

I would love to see a finalized app of this project idea I had for years. So if there is some news about this please leave a comment.

Answering your pointed questions:

  1. Does any such thing exist (webcam/video to pgn converter? A prototype maybe?


    Apart from my project 8x8cam and chesstrack, I have recently found a project named ChessCam. All of these projects are Open Source and only need to be named as resource if you create a new project from them.

  2. Has anyone attempted such a feat and shared their research?


    Yes. You can see such on previous question's answer.

  3. Are there obvious limitations that make this project unlikely to succeed?


    The hardest part I believe is to make the software work no matter how the camera is placed and independently of the illumination. And also to work on fast paced games and work with the hand occlusions when the game is being played, in order to detect correctly the move done.

  4. Would you buy it if it existed?


    Definitely yes.

1

Edit: promoting to anything different than queen will make everything explode. It will also be very hard to support this (my program for Part 2 supports underpromotion, but storing the info of Part 1 purely by reading pixels is a nightmare).

Edit2: I just noticed you mean real board chess to PGN, so this is all useless and I will most likely delete my answer.

Part 1: storing the squares using pixel readers

It is totally possible, the steps needed would be:

  1. change the set up of the board so it highlights the last move (prefereably not just the border)
  2. write a program that reads pixels on given (x,y)
  3. make it read about 5+ pixels on each square (so it is impossible for the cursor/piece to cover all the pixels to be read)
  4. convert the HEX to RBG, then give a range of accepted values of what we deem to be considered close to the "last move" color
  5. save the moves on an array with a format ["h1-h8", "e4-e5", ...] just the initial and ending squares (order unimportant)

Note: from here we can't really know from which square was the initial/ending position, but there are only 2 posibilities and only 1 of them is legal.

I have written simple pixel-based macros/bots for RPGs games in C#, I can guarantee this is not that difficult, given that:

  1. the cam is static (so the x,y values don't get messed up)
  2. you previously know the background color or you can set it up before playing

Basically, it is best if the game is played online and you are recording directly from the screen.

Sorry, no program avaiable for part #1


Part 2: exporting that list into PGN

To turn this list into a real PGN, you can use a program I wrote in JavaScript that records moves, and one of the functions moveCaller() you pass two squares positions and it searches the final_pos in the legal moves of initial_pos and if there is a match then it calls makeMove(). So we can call it twice e5 to e4 and e4 to e5 and I guarantee it will ignore the illegal one.

  1. download the project (direct link .zip)
  2. uncompress the .zip
  3. change the JavaScript of the index.html file with this:

'

$(function(){
    var i, len, arr, temp, m_board;

    m_board=IsepicChess.initBoard({name : "main"});

    arr=["e4-e2", "e7-e5", "f3-g1"]; //is ok for the initial and final squares to be swapped

    for(i=0, len=arr.length; i<len; i++){
        temp=arr[i].split("-");

        //one of the two following lines will silently fail
        m_board.moveCaller(temp[0], temp[1]);
        m_board.moveCaller(temp[1], temp[0]);
    }

    m_board.refreshBoard();
});

.4. copy the move list and add some PGN headers above it like [WhiteElo "2400"] etc.

1

Joey Meyer wrote and built "Raspberry Turk":

The Raspberry Turk is a robot that can play chess—it's entirely open source, based on Raspberry Pi, and inspired by the 18th century chess playing machine, the Mechanical Turk.

This goes beyond the OP's question because it also incorporates (a) the engine to calculate responses and (b) a robotic arm to make the engine's moves on the board.

But it answers the OP's question because one component of Raspberry Turk includes an overhead camera and a machine-learning component to learn how to convert what it sees on the board into a PGN representation of the board.

1

Firstly, great post. I have been surveying this issue recently and the answers here are a great source of information. I have found an excellent free app for Android and iOS called KnightVision. It is relatively new. It was trained on various chess sets. It even managed to properly recognize my hand-crafted pieces which are not identical to the Staunton standard. It works well, although to be on the safe side, it requires a separation of about 3 seconds between each move on the board. So be gentle when swapping pieces, which is something that is usually done quite quickly. It also has a web page, where after logging in you can see the game PGN that you can copy and export to any chess engine for studying. It also has some analysis on the website itself. https://www.knightvision.io/

0
0

I won't be answering the question, because I don't know if there's a software out there. I'm just saying how I think that's how to do it, I don't know for sure, so...


Okay first you need to make sure that the video is recorded from a chess software, not a live tournament, just to make your job easier.

You need to convert the movie to images

ffmpeg -i foo.avi -vsync 0 -vf select="eq(pict_type\,PICT_TYPE_I)" -s WxH -f image2 foo-%03d.jpeg

That was easy, now the hard part, you need to start doing image processioning, for that, the most powerful is OpenCV.

For that, there are several techniques, I'll give you the links so you can read them.

It is complicated but simpler than you think, because you only need to know what is the starting position, loop through all the images and see the difference. Take a look at OpenCV docs you'll find tons of materials on that subject.


Oh by the way, what you're asking for is done already, but not using software as far as I know, there's an old project that was coded using Pascal, cameras placed over the board, robots do image processing and make a move. Nowadays many robots play chess.

You could buy an Arduino uno, and its light censor, it will detect the white pieces and the black pieces, some coding and it will play chess. :)

0

Squareoff is making another board at 200$. Expensive but not extortionist. They say it will be available March 2021. Automatically moves pieces too. Details. Has Blitz mode but may not be so good for it. May not allow more than one user per board but plays through chess.com and maybe lichess.

-1

I use the Monroi Chess Notation Device for all my chess games and it is basically an electronic board.

2
  • 5
    That doesn't answer the OP's question, though. Feb 22, 2014 at 11:18
  • you could use a digital presenter to record a game its a camera just above the table. then you could just write an application tracking the objects in a grid.
    – Cferrel
    Feb 23, 2014 at 2:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.