How did the engines improve since Deep Blue?

Question

Computer chess engines have gotten better since Deep Blue beat Kasparov in 1997.

Did the algorithms get better, or were the improvements mostly due to the same algorithms running faster thanks to faster hardware, etc.?

If the former, are these algorithmic improvements public?

And if so, what were the improvements? Where can I read about them?

Answer 1

Maybe you can take a look at TalkChess, a forum dedicated to computer chess. I found a recent thread that might be interesting for you: Progress in 30 years by four intervals of 7-8 years

A couple of matches between (former) top engines are played on the same hardware. The test suggests that in the recent years (2002-2017), the gain is mainly made by software improvements. In the test, Stockfish (2017) scored an impressive 94/100 against RobboLito (2009), while RobboLito, on its turn, crushed Shredder (2002) with 92/100.

An important remark: as parallel computing is not implemented in the older engines, the test was performed on a single core. As a result, the hardware gain by parallel machines is not measured. On the other hand, you could argue that parallel computing is also a software gain: it is not easy to design and implement an efficient and well-scaling parallelization for the search algorithm.

The Stockfish engine is open source, so the algorithmic improvements are public. A lot of documentation can be found on https://chessprogramming.wikispaces.com

Answer 2

I can't speak for the algorithm used for Deep Blue, but I'm going to try and explain the improvements in chess programming. Speed is the greatest improvement. Deep Blue used multi-processor dedicated computers, so a comparison isn't really possible.

https://chessprogramming.wikispaces.com/ is a great source, but it's hard to navigate.

There are 3 main functions that are tweaked to improve a chess engine are the evaluation, move generation, and search functions.

Evaluation is the hardest to program, as there are many exceptions to the rules. With hard drive space getting cheaper, the eval function allows for more exceptions to be evaluated.

Move generation, along with making and unmaking a move, consumes a lot of memory because it has to be preformed so many times. The most common generation functions are mailbox, bitboard, 0x88, 8x8, extended boards (10x10, 10x12), and a predetermined move array/table (*I use an indexed move table). Current opinion is that bitboards are the faster, and using magic bitboards speed this by up to 30%. Dr. Robert Hyatt, professor and creator of cratfy chess engine, claims no significant speed increase.

The early search function was the primitive min-max functions. Basically were you try to maximize the score of the side to move and minimize the opponent's score. Alpha-Beta was the first improvement. They reduced the number of moves being searched by transposition table, cut-off values, aspiration windows, and history heuristics. These are depth-first searches. There is also the internal iterative deepening search which tries to search the "best" move(s) the deepest hoping that searching other moves will prove to be fruitless.

NOTE: My index table. GNUChess and Jester both use an index array to generate their moves. They initialize the engine by filling array with possible moves. The take the six pieces and compute the legal moves that are available from each square. So each piece had a [64][8] array. I took this idea and compressed it to two indexes and a table. The table holds a value which tells if the 16 moves are possible, one index holds the offset of the move, and the other holds the mask.

offset[] = {-8, -1, 1, 8, -9, -7, 7, 9, -17, -15, -10, -6, 6, 10, 15, 17};

mask[] = {1, 2, 4, 8, 16, 32, 64, 128, 256, ...};

Then the generation of a sliding move is as easy as looking up the validity of it's mask in it's allowable offsets against the move table.

Answer 3

Did the algorithms get better?

Obviously, yes a little bit.

or were the improvements mostly due to the same algorithms running faster thanks to faster hardware and software?

Minor nit: If the algorithms got better then that is the software getting better so there is no "or".

Moore's Law tells us that processor speed will double roughly every 18 months. That means it has doubled roughly 13 times in 20 years. That makes modern processors somewhere in the region of 8,000 times faster. So, far and away the biggest improvement in engine performance is due to faster hardware.

If the former, are these algorithmic improvements public?

And if so, what were the improvements? Where can I read about them?

Well, it wasn't the former, it was the latter. Nevertheless the improvements are mostly open source and freely visible by downloading the sources for engines like Stockfish. Perhaps also worth giving the general Stockfish download link since the specific source code link will likely expire when version 9 comes out.

Answer 4

It's all about algorithms.

Taking on a human chess player took one of the most powerful computers in the world at the time. This brute force computing approach allowed Deep Blue to look around six to eight moves ahead. In a closely-fought contest, the machine eventually defeated Kasparov by 3 1/2 games to 2 1/2.

Six years later, Kasparov was involved in another contest of man versus machine. This time he played against Deep Blue’s successor, Deep Junior. The result was a drawn series at three games all. The biggest difference was that Deep Junior ran on a machine with about one per cent of the computing power of Deep Blue. Chess-playing algorithms had improved to the point of achieving virtually the same result with a hundred times less computing power.

Answer 5

Disclaimer: not an expert.

The algorithms got better, and today's best engines running on 1995 (remember Deep Blue was 1999) hardware will handily beat Kasparov. As I understand it, there are two aspects of algorithms:

Search. If for example I take your queen with my queen, a human opponent will automatically look first at recapturing. For a computer however, it will evaluate every possible response to QxQ. Almost all of the time, this is wasted processing power. A good search algorithm cuts down all these "branches" since they're irrelevant anyway.

The standard search algorithm is alpha-beta pruning, and it was used in the earliest chess computers. I don't know if Deep Blue used alpha-beta pruning, but modern engines don't. As a result their searches are "unsafe" - they can miss, for example, that some move other than recapturing the queen would've won the game. However, it's rare that this happens, and in return they push their depths very high. ("Depth" is a technical term for how deep the engine searches, so e.g. an engine that searches to depth 30 is likely to beat one that only searches to depth 20, all other things being equal).

Evaluation. This is the other prong of engine code. Given a particular position, is it better for white, black, or equal? This can involve all sorts of functions, e.g.

• If one side has extra material/space, give it a bonus to eval.
• If white has an advanced knight supported by a pawn, give white a bonus to eval.
• If black's king is stalemated, give white a bonus to eval.
• If white has a rook on the 7th rank, give white a bonus to eval.
• If it's an endgame (and there are algorithms to decide whether the position is an endgame) and both sides have opposite color bishops, impose a penalty to eval (i.e. push it towards 0.00).

Today's engines evaluate positions much better than Deep Blue.

As for whether the algorithms are public, Stockfish is currently the world's strongest engine and it's open source. You can download the code yourself from Github.