15

Sometimes during blitz games over the board, or when in zeitnot, it gets very difficult to annotate them.

I've seen several software / hardware to help in writing the notation, even one that's FIDE approved; these require that you repeat the moves on the device. Hardly faster than pen and paper, just as prone to errors, plus you risk typing the wrong move and wasting even more time.

DGT makes a board that will do this automatically, but it will only work with its pieces, and starts at 400€, double that for tournament boards. Some independent projects are available such as usb chessboard 101, or voice controlled chess robots, Chess board step sequencer. Additionally some open projects are available that even move pieces (or gigantic pieces).

I have a playoff tonight, and the tournament rules dictate shorter and shorter games until there is a winner. Hence I thought about shooting a movie of the board should we lose track. But then my programming background kicked in and I thought "there must be some software that can process a movie and write the PGN".

I searched everywhere, and found nothing.

  1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?
  2. Has anyone attempted such a feat and shared their research?
  3. Are there obvious limitations that make this project unlikely to succeed?
  4. Would you buy it if it existed?
  • it would be an interesting computer vision / machine learning project! – Memming Feb 11 '14 at 12:10
  • I think the board has to be filmed from top-down to produce a "2D"-image. Otherwise, there is an additional problem of transposing a 3D-view to a 2D-view and different 3D-angles have to be treated separately. – Rauan Sagit Feb 13 '14 at 13:15
  • actually it's simply a linear transformation to "unstretch" the board and make all its squares square. the main problem is to recognize the square the piece is moved from if the angle is such that pieces can line up behind each other. – peter Feb 26 '15 at 11:13
  • Maybe we could do some sort of crowdsourcing for this project. I think it's very interesting. I bet that some college student(s) could do this for senior projects. We need to find a way to communicate the idea to professors. Maybe with two cameras placed in two different locations this can be done reliably. I would be interested in the 3D approach, not 2D from overhead. Mounting an overhead camera is too impractical. I would also be interested in software that would take inputs from a few different cameras shooting several games simultaneously played on a long table and produce notation - for – user8450 Oct 8 '15 at 12:33
4
+50

It is clearly interesting and relevant to automatically record chess games (live boards are more or less tackling the same problem). As far as I know, there is no software for this purpose that exists today. To solve this problem, I think that the "simple" case should be as follows.

  1. The chess board should be filmed from top-down to get a two-dimensional view. (Motivation: filming in a three-dimensional view would cause problems because of a skewed view and pieces that will block the view for other pieces behind them. Also, different 3D-views are possible which would counteract a clean procedure.)
  2. The film should be manually edited to capture "clean board" moments and remove moments when the two players' hands or heads are blocking the view.
  3. The image processing algorithm on the software side should detect changes on the board and record them as moves. It should also auto-rotate and transform the image to detect the board orientation and thereby the coordinates.
  4. The software should use the starting position to "memorize" every piece type and then use these for the image recognition during the game. Probably, some clever tricks can be written to solve the problem of piece recognition in a smart way.

A quick Google search with the search string

chess game image processing

returned this interesting article

All in all, it should be realistic to implement this type of software. Yet it might be cheaper for a human to do the same work (transfer moves from a video clip to a computer).

  • It took me a while to go through the study by G.D.Illeperuma you linked, it is most interesting. But it does exclude a priori the possibility of elaborating 3D skewed images. Thank you, a very interesting article. – Riccardo Zorn Feb 15 '14 at 21:17
  • @RiccardoZorn you are welcome! I must admit that I don't see the point of handling 3D skewed images. The situation is that if a human has trouble documenting the moves from a video recording, then an image processing software will have even more difficulty. – Rauan Sagit Feb 16 '14 at 1:02
  • Setting up a vertical camera requires a lot of space, is invasive, and you need a custom support/stand. This makes it impractical :-( – Riccardo Zorn Feb 16 '14 at 8:22
  • A more simple and cheap "live board" technology might be a better way to go in that case. Optionally, film from a reasonable 3D angle and let a human extract the chess notation from the video recording. – Rauan Sagit Feb 16 '14 at 11:58
  • the file researchgate.net/publication/… doesnt seem to exist anymore. does anyone have a copy? – peter Oct 24 '16 at 10:58
5

That would be a rather complex task. Maybe a reasonable first step is to look into recognizing FEN position from a single image - for which more resources exists, for example:

  • Excellent thank you. I read both and they are really interesting. Chessgrabber is a recent project and is definitely more suited, I contacted the developer. – Riccardo Zorn Feb 10 '14 at 2:21
  • I'd think that from this first step it wouldn't be too difficult to convert a video to PGN - take images every few milliseconds, analyse them, and if the position has changed in a legal way add the new position to the file. – rlms Feb 11 '14 at 17:32
  • Update: the developer responded, it will never work for 3-d skewed images with overlapping pieces. Too bad, without that as a base, the project will never start. – Riccardo Zorn Feb 12 '14 at 9:55
  • recognizing a still image is actually harder than having the video leading up to it, since the latter eliminates the problem of having to be able to identify the pieces. – peter Oct 24 '16 at 11:01
6

In order,

1) Not to my knowledge

2) Not to my knowledge

3) There are several things that make this sort of project non-trivial, even difficult. Not the least of which is following the piece, differentiating it from a hand gesture, for example. Also determining when the move stopped would be an interesting question for computer vision to solve. These kinds of problems have been solved in other situations, but not with 100% accuracy, and certainly not at prices that are even close to being competitive with the DGT board.

Then there's the question of where the camera could be positioned. Directly overhead would offer the best view of the board, making it easier to determine the square with precision. But wherever the camera gets placed, there's a distinct probability its view will be obstructed by one of the players, either leaning over the board, reaching for the clock, or simply gesticulating.

4) Not unless it was really cheap. And I don't expect it would be. My reasoning is simple. As time controls shorten, the quality of the game lessens, to the point where it's no longer interesting to me

  • You make a great point at 4: the quality is indeed poor. Not worth the hassle of setting up a camera, much less spending a month to write the software. I just won the playoff, but I am embarassed of how poorly I played. – Riccardo Zorn Feb 10 '14 at 1:23
1

I won't be answering the question, because I don't know if there's a software out there. I'm just saying how I think that's how to do it, I don't know for sure, so...


Okay first you need to make sure that the video is recorded from a chess software, not a live tournament, just to make your job easier.

You need to convert the movie to images

ffmpeg -i foo.avi -vsync 0 -vf select="eq(pict_type\,PICT_TYPE_I)" -s WxH -f image2 foo-%03d.jpeg

That was easy, now the hard part, you need to start doing image processioning, for that, the most powerful is OpenCV.

For that, there are several techniques, I'll give you the links so you can read them.

It is complicated but simpler than you think, because you only need to know what is the starting position, loop through all the images and see the difference. Take a look at OpenCV docs you'll find tons of materials on that subject.


Oh by the way, what you're asking for is done already, but not using software as far as I know, there's an old project that was coded using Pascal, cameras placed over the board, robots do image processing and make a move. Nowadays many robots play chess.

You could buy an Arduino uno, and its light censor, it will detect the white pieces and the black pieces, some coding and it will play chess. :)

-1

I use the Monroi Chess Notation Device for all my chess games and it is basically an electronic board.

  • 4
    That doesn't answer the OP's question, though. – Dag Oskar Madsen Feb 22 '14 at 11:18
  • you could use a digital presenter to record a game its a camera just above the table. then you could just write an application tracking the objects in a grid. – Cferrel Feb 23 '14 at 2:30
0

Edit: promoting to anything different than queen will make everything explode. It will also be very hard to support this (my program for Part 2 supports underpromotion, but storing the info of Part 1 purely by reading pixels is a nightmare).

Edit2: I just noticed you mean real board chess to PGN, so this is all useless and I will most likely delete my answer.

Part 1: storing the squares using pixel readers

It is totally possible, the steps needed would be:

  1. change the set up of the board so it highlights the last move (prefereably not just the border)
  2. write a program that reads pixels on given (x,y)
  3. make it read about 5+ pixels on each square (so it is impossible for the cursor/piece to cover all the pixels to be read)
  4. convert the HEX to RBG, then give a range of accepted values of what we deem to be considered close to the "last move" color
  5. save the moves on an array with a format ["h1-h8", "e4-e5", ...] just the initial and ending squares (order unimportant)

Note: from here we can't really know from which square was the initial/ending position, but there are only 2 posibilities and only 1 of them is legal.

I have written simple pixel-based macros/bots for RPGs games in C#, I can guarantee this is not that difficult, given that:

  1. the cam is static (so the x,y values don't get messed up)
  2. you previously know the background color or you can set it up before playing

Basically, it is best if the game is played online and you are recording directly from the screen.

Sorry, no program avaiable for part #1


Part 2: exporting that list into PGN

To turn this list into a real PGN, you can use a program I wrote in JavaScript that records moves, and one of the functions moveCaller() you pass two squares positions and it searches the final_pos in the legal moves of initial_pos and if there is a match then it calls makeMove(). So we can call it twice e5 to e4 and e4 to e5 and I guarantee it will ignore the illegal one.

  1. download the project (direct link .zip)
  2. uncompress the .zip
  3. change the JavaScript of the index.html file with this:

'

$(function(){
    initBoard("xboard",null,false);

    var arr=["e4-e2","e7-e5","f3-g1"];//notice the initial-final might be switched

    for(var i=0;i<arr.length;i++){
        var temp=arr[i].split("-");

        //one of the two following lines will silently fail
        moveCaller(bosToPos(temp[0]),bosToPos(temp[1]),window["xboard"]);
        moveCaller(bosToPos(temp[1]),bosToPos(temp[0]),window["xboard"]);
    }
});

.4. copy the move list and add some PGN headers above it like [WhiteElo "2400"] etc.

3

I wrote this software. It contains webcam AI for move recognition, connected to Arena GUI.

https://www.youtube.com/watch?v=APkoE1UDKj0

  • Very impressive Marius, congratulations! Is the alpha available for testing already? Can you expose the moves through a service, for use by other software? – Riccardo Zorn Jun 15 '15 at 12:36
1

I worked on the chess tracking project 8x8cam within image processing course in 2013 at FING UdelaR.

Then in 2014 other students created a better version for Android named chesstrack. Under it's web you can find videos and link to the code they used.

I would love to see a finalized app of this project idea I had for years. So if there is some news about this please post and answer or comment.

1

It's 2019-10 now and 5 years after the original question.

  1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?

    Sort of Yes - definitely yes for a prototype.

    Webcam to UCI has been done as a student project in 2018 see

    https://classes.engineering.wustl.edu/ese205/core/index.php?title=CV_Chess

    my own hope is also that from such a prototype it is not far to a viable "product".

  2. Has anyone attempted such a feat and shared their research?

    Yes. See http://wiki.bitplan.com/index.php/Raspberry_PI_Chessboard and the Links on it. The project was started today 2019-10-15 and in a few hours I already found a lot of resources e.g. this Question

  3. Are there obvious limitations that make this project unlikely to succeed? I can only answer from my own experience with a similar project I did 31 years ago. At that time i tried to create a chess board that would detect moves. I used magnets in the wooden pieces and reed contacts. The key to success was using a chess engine that new "the rules" to be able to concentrate on movements alone and not having to detect individual pieces. As outlined in an answer above promotions to non queen pieces might be one of the edge usecases that are difficult. Detecting the motion of pieces via picture differences is a well understood problem.

    Chess engines and UCI compatible GUIs are also available these days. So the problem is more an integration problem these days then solving the individual detail issues.

    A major obstacle is the difference between a "prototype" and a product. See e.g. http://www.nandanbanerjee.com/files/ICCVR-08AUG12-011%20paper.pdf

    a paper that is even older than this question but far from a product that you can buy at a reasonable price.

  4. Would you buy it if it existed?

    Yes. When I bought a new chess board a few days ago I was looking for one that could record moves and it was still around 500 EUR. As it looks the hardware cost will be well below 150 EUR these days including a decent chess board and pieces.

    The software might well be opensource and free (as in free beer) soon. I'll update this answer accordingly.

New contributor
Wolfgang Fahl is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.