It is well known that both hardware and software contributed to the total dominance of computers. Has there been an experiment to run a very modern program (say Alpha Zero) on a museum computer (of course, a memory issue can happen, so don't abuse the Zuse for your amuse) or vice versa, a museum program on a very fast computer? If yes, with what results (hard- or software more important)?
asked Jan 3 at 18:33
-
2I see compatibility problems in running such experiments– Andrea MoriJan 3 at 19:44
-
2It's the software. In computer science we generally say that we would rather run today's algorithms on yesterday's computers than vice versa. Chess shouldn't be an exception.– HauptidealJan 4 at 9:00
-
1@Hauptideal Hmm, hardware acceleration is easily an order of magnitude (in terms of execution time) for modern DL models at least. I'd say software is always on the forefront of computing progress, and you can poorly run something advanced today knowing that tomorrow it will run well. But hardware still imposes limits on this progress, you can feasibly be two steps ahead, but not two thousand. We would not have Deep Blue if all we had was PDP-11.– LodinnJan 4 at 16:51
-
1@Lodinn that's all just constant factors. I'm taking about real Big-O improvements! And yet while something like this didn't happen in chess, we have much much better evaluations today than we had before.– HauptidealJan 4 at 19:13
-
Modern neural nets benefit greatly from GPGPU but only for training. You can then use the generated weights on a smartphone.– qwrJan 4 at 21:24
|
Show 1 more comment
5 Answers
The answer is software is more important. You can test this yourself actually. Get your favorite top engine today and your favorite top engine from 10 years ago, and run both of them against each other at time odds. This simulates superior hardware (since it yields a higher number of positions searched).
This was done at 50-1 NPS (nodes per second, a measure of how fast the hardware is) odds here. Komodo 8 (a top 2014 engine), played on a smartphone against Shredder 10 (top 2006 engine) on an i7 desktop. Komodo won 5-1. Not a large sample size by any means, but clear enough given auxiliary data.
See also How does engine strength scale with hardware?, which in conjunction with rating lists played on equal hardware gives an idea of where hardware effects might dominate over software.
-
2Interesting. You'd run into problems running modern engines on true museum-piece hardware, though, like when memory limitations prevent modern engines from keeping their working set in RAM (especially deep learning models?). Or on truly ancient machines, when the depth they can see (in the time limit you give them) is so much lower than their algorithms / heuristics were designed and tuned for, they might make huge blunders. Maybe even worse than simple chess-playing programs written for such machines that use simpler heuristics. Jan 4 at 8:11
-
Yes software by far. More so once realizing said software isn't, but could be, (re)written to run on toasters. Jan 4 at 12:46
-
1I wonder how well a chess engine could be programmed to perform if constrained to various historical classes of machinery, including e.g. the Fidelity Chess Challenger (a Fairchild F8 I think), a Atari 2600 with first-generation cart (6502 with 4K of code and 128 bytes of RAM) or later-generation cart (32K of code and 256 bytes of RAM), an Apple II+ (6502 with 48K RAM and 140K floppy), a 640K IBM PC using RAM only, a 640K IBM PC with up to 5MB of hard drive space, etc. An Apple II wouldn't be competitive with Stockfish, but modern knowledge might allow vastly better play than any programs...– supercatJan 5 at 16:09
-
-
The test you describe is clearly faulty: if you want to test hardware and software influences seriously, you don't emulate. (That's like whoever thought that the Digital VAX computer was a 1MIPS system because 'it feels like 1MIPS'.) So thumbs down for that. As it is the test also is open to the problem that it may test only improvements of a particular hardware architecture. No, the question deserves a serious test. If no-one has performed that, it is better to say so. Jan 6 at 8:09
I develop mobile chess apps to the App Store. The very first version of the apps were built on the iPhone 3GS. That was many years ago, but even then the apps could play at the GM chess level.
There is a rating list for Android chess engines:
You can then compare the list with more powerful hardware configurations:
No rating list exists for iOS due to technical limitations.
According to the JCER rating list, there is a big gap. Stockfish on Android is 3155 while the PC version is 3734.
Note that there is a massive jump for the software side with the introduction of neural networks like Alpha Zero. Software before that required a large amount of clever humans programming it but these humans could do that with low end hardware. Neural networks need in comparison much less human programming time but require lots of computation power once to train them. After the training is done they can be run on worse hardware than pre neural network software and still play much better.
This makes numerically comparing pre neural network to post neural network software difficult because it is not clear what exactly you want to compare. One could run a modern chess software on say year 2000 hardware and it would trash any software available back then but I don't know how feasible it would have been to train a modern chess software with year 2000 hardware. A chess software probably can't afford to rent a super computer for a month of time for training.
-
The obvious way to do it to me would be to run the new software at the same nps. For example, if a top engine in 2000 evaluates 2 million positions per second on the hardware of the day, we could also run a top engine of today at 2 millions positions per second. If that's not feasible, we could run a top engine today at time odds so that it searches the same number of positions as the old engine. It's not ideal - the complexity of the evaluation functions affect nps - but it should be close enough.– AllureJan 5 at 13:12
-
1What would be the minimum amount of memory and CPU speed to make a neutral-network-designed chess bot usable? Would a desktop computer from 1992 (e.g. 80486 with 4MB of RAM and 120MB of hard drive storage available) be able to benefit? What about one from 1980 (e.g. 6502 with 48KB of RAM and 140K of floppy-drive storage)?– supercatJan 5 at 20:57
-
@supercat This is a good question but I unfortunately don't know enough about hardware requirements of neural-network software to provide a meaningful answer.– quaragueJan 6 at 7:20
-
@supercat: I already did a quick google to find the space specs of Alpha Zero, but with no avail. (They surely are in the original SCIENCE paper.) Some data for Leela: lczero.org/play/troubleshoot Jan 6 at 9:14
"Turochamp", the original chess-playing programme from 1948 by Alan Turing and David Champernowne, was executed manually for a game in 1952, since hardware at the time was deemed incapable of running the software proper. It took about 30 minutes per move, and lost to Alick Glennie after 29 moves.
In 2012, for the Turing anniversary celebrations, the programme was reimplemented on contemporary hardware, losing to Garry Kasparov after 16 moves. Kasparov explained that the basic problem was that Turochamp is "only two ply", that is, it cannot look ahead very far and in particular could be deceived by a sufficiently skilled player who was aware of this limitation, regardless of the speed at which it would produce its moves.
As a caveat, the 2012 programme was not the same as the original, as the authors didn't preserve all details of the algorithm, and may not have implemented it thoroughly for the 1952 game in any case. Nonetheless, this demonstrates a long time gap, and that even with much, much better hardware, the algorithm is still not that capable.
I don't see how any experiment would decide the question of "importance".
If you tried to run modern software on ancient hardware, it would fail, because modern software is designed to take advantage of modern hardware.
If you ran ancient software on modern hardware, it would do whatever it did before, only faster. Would it beat modern software running on modern hardware? Almost certainly not, because it wasn't written to take advantage of the hardware power available.
But consider: a lot of the improvement in computer chess is due to the existence of databases of openings and endgames. There's no clever software behind that, just availability of cheap storage. So do you attribute that to software or hardware advances?
