I just read this question and it made me wonder: when setting up such a game/comparison - shouldn't you try to level the computing capacity field? Or at least limit how much energy each "player" is allowed to use (and then let each team optimize within that limit). Otherwise, isn't it just a matter of building the biggest computer?
You absolutely should, which is why the two most important computer chess tournaments (CCC and TCEC) both make all competitors play on the "same" hardware. "Same" in inverted commas because the recent advent of neural-network engines like AlphaZero, which run best on GPUs, meant that it's no longer fair to play on exactly the same hardware for everyone.
You can see the result of not using equal hardware from the 2019 WCCC results, which Komodo won. Komodo is almost a hundred elo behind Stockfish when run on equal hardware (and Stockfish doesn't play in the WCCC, because it's not a commercial engine and doesn't have full-time staff to attend the event).
The situation right now is traditional engines run on the same CPUs, NN engines run on the same GPUs, and whenever traditional engines play against NN engines, they run on consensus fair hardware. "Fair" here is a very controversial word, because there are several different measures of what is fair that I'm glossing over. Suffice to say unless you are a hardware expert yourself, I'd just trust the admins of CCC and TCEC to use something that's fair.
Edit: to illustrate the impact of hardware, here's the list of hardware the competitors played at in the 2019 WCCC. Note Jonny ran on 1200 (!) cores, and it just barely edged out Ginkgo, which was running on 1/20 of the cores. In TCEC Season 14, the two engines played with the same hardware in Division 1, and Ginkgo comfortably finished ahead of Jonny in spite of multiple crashes (which happen because Ginkgo doesn't run well at 43 cores; they result in auto-losses).
You can also see that these games were played is Division 1, not even Division Premier. Many of the strongest engines in the world simply don't participate in WCCC. I don't know the exact reasons why, but I'm confident hardware is a major issue for participants. In turn, this lack of participation is partly why the community don't take the WCCC very seriously.
Yes, you are right. So computer competitions will have rules about compute to keep things level. For example the chess.com computer championship uses this specification:
- CPUs: 2 x Intel Xeon Platinum 8168 @ 2.70 GHz 33 MB L3
- Cores: 48 physical (96 logical)
- RAM: 256GB DDR4-2666 ECC Registered RDIMM
- SSD: 2x Crucial MX300 (1TB) in RAID1
- OS: Windows Server 2016
- GPU: 4x Tesla V100 (64 GB GPU memory)
- CPU: Intel(R) Xeon(R) CPU
- E5-2686 v4 @ 2.30 GHz Cores: 16 physical (32 virtual)
- RAM: 256 GB
There are a number of reasonable things you could set as constant, which is probably why they have experts determine the rules in a fundamentally arbitrary but hopefully fair fashion. Energy is one, but some other options would be:
FLOPS (floating point operations per second).
I suspect (although my info is limited) hardware cost and energy would help neural network neural based devices, while a flops based metric would benefit classical engines, because GPUs tend to provide great FLOPS/watt and FLOPS/$ -- metrics that CPUs sacrifice for the ability to go (relatively) quickly through branchy logic.
An argument against using a simple metric is that it might induce behavior that doesn't advance the state of the field in an interesting direction. If you ask for performance/watt or performance/$, people start asking questions like "what about an FPGA or ASIC," and eventually it becomes a competition in brute force and low level tuning, rather than interesting new ideas. Plus people wouldn't be able tinker with the engines at home if they needed specific devices to do so.
It is better to have experts in the field push toward interesting results, IMO. Although "energy limited on commodity hardware" might be fun.
It really comes down to the purpose of the event.
If it is a championship that is trying to determine who wrote the best software, and not just the strongest hardware/software combination, then you want to even up the hardware as best as possible so the software is the determining factor. The winner determines the best software.
If it is a demonstration or exhibition match, like the AlphaZero-Stockfish match, or the Kasparov-Deep Blue match years ago, and it is all about just crushing the opponent and winning at all costs, all bets are off, and you can expect to see massive, and outrageous, hardware builds.
Should we limit human players who play by how much they weigh or their IQ?
There is no point in so called 'levelling the field' when computers play each other.
There is much more than building the biggest computer that determines who wins.
Software makes the hardware happen. And how well they came up with the software and how well they determined the algorithms to guide the software as well as practical tricks they use like having table bases or pitting two machines against each other to help them learn faster are more important than the hardware.
Limiting energy would only mean that they would use different compuer chip technology that has nothing to do with chess.
Your suggestion would change what is being optimized and what most people care about are how well the computers play not how much electricity they use.
However since the real concern is the software the latest championship had these rules for software:
The rules for the World Chess SOFTWARE Championship state that competing programs must run on machines with identical hardware specifications.