9

I just read this question and it made me wonder: when setting up such a game/comparison - shouldn't you try to level the computing capacity field? Or at least limit how much energy each "player" is allowed to use (and then let each team optimize within that limit). Otherwise, isn't it just a matter of building the biggest computer?

5

You absolutely should, which is why the two most important computer chess tournaments (CCC and TCEC) both make all competitors play on the "same" hardware. "Same" in inverted commas because the recent advent of neural-network engines like AlphaZero, which run best on GPUs, meant that it's no longer fair to play on exactly the same hardware for everyone.

You can see the result of not using equal hardware from the 2019 WCCC results, which Komodo won. Komodo is almost a hundred elo behind Stockfish when run on equal hardware (and Stockfish doesn't play in the WCCC, because it's not a commercial engine and doesn't have full-time staff to attend the event).

The situation right now is traditional engines run on the same CPUs, NN engines run on the same GPUs, and whenever traditional engines play against NN engines, they run on consensus fair hardware. "Fair" here is a very controversial word, because there are several different measures of what is fair that I'm glossing over. Suffice to say unless you are a hardware expert yourself, I'd just trust the admins of CCC and TCEC to use something that's fair.

Edit: to illustrate the impact of hardware, here's the list of hardware the competitors played at in the 2019 WCCC. Note Jonny ran on 1200 (!) cores, and it just barely edged out Ginkgo, which was running on 1/20 of the cores. In TCEC Season 14, the two engines played with the same hardware in Division 1, and Ginkgo comfortably finished ahead of Jonny in spite of multiple crashes (which happen because Ginkgo doesn't run well at 43 cores; they result in auto-losses).

You can also see that these games were played is Division 1, not even Division Premier. Many of the strongest engines in the world simply don't participate in WCCC. I don't know the exact reasons why, but I'm confident hardware is a major issue for participants. In turn, this lack of participation is partly why the community don't take the WCCC very seriously.

| improve this answer | |
  • 1
    Thank you. Well, since, as you say, hardware requirements are very different between different types of engines, using energy consumption (and, obviously time) seems to be a reasonable way of levelling the field, doesn't it? Of course, it all depends on what you want to measure, but at the end it is energy that propels the world. – d-b Feb 23 at 12:42
  • Using max energy consumption is an interesting way to aim for equivalency between GPU and CPU systems. I don’t know how it is done now, maybe that is what is done. With the specs I provided in my answer the energy could be calculated. – Michael West Feb 23 at 18:36
  • @d-b that's one way to do it. However, GPUs are energy-intensive so equal power consumption advantages CPU engines. Another way to do it is to have equal cost (the best GPU/CPU $5000 can buy, for example) which advantages GPU engines because at this level, CPUs are more expensive than GPUs. There's also the "Leela ratio" (github.com/dkappe/leela-ratio), effectively basing off the AlphaZero paper. I'm sure there are more ways to say what's "fair". – Allure Feb 23 at 21:41
  • 1
    @Mast yeah (TCEC is currently playing a CPU-only league), but people want to know what is the "strongest" engine. Right now the strongest CPU engine is undoubtedly Stockfish, while the strongest GPU engine is most likely Lc0. Which is stronger though, Stockfish or Lc0? That's only answerable by getting them to play each other, which means we need to use "fair" hardware. – Allure Feb 24 at 5:43
  • 1
    @Allure Are you sure about that? I thought GPUs are more efficient than CPUs, simply because it is easier to make HW efficient if it is more specialized. C.f., hardware dedicated to BTC-mining. A typical GPUs might use more energy than a typical CPU in a desktop system but that might be because we in most cases have reached a level of CPU performance that in most applications no longer makes it the bottleneck, hence we don't push the CPU performance that much anymore while GPUs still might be bottlenecks in certain applications. How do you compare CPU and GPU energy consumption in a fair way? – d-b Feb 24 at 7:11
3

Yes, you are right. So computer competitions will have rules about compute to keep things level. For example the chess.com computer championship uses this specification:

CPU Engines:

  • CPUs: 2 x Intel Xeon Platinum 8168 @ 2.70 GHz 33 MB L3
  • Cores: 48 physical (96 logical)
  • RAM: 256GB DDR4-2666 ECC Registered RDIMM
  • SSD: 2x Crucial MX300 (1TB) in RAID1
  • OS: Windows Server 2016

GPU Engines:

  • GPU: 4x Tesla V100 (64 GB GPU memory)
  • CPU: Intel(R) Xeon(R) CPU
  • E5-2686 v4 @ 2.30 GHz Cores: 16 physical (32 virtual)
  • RAM: 256 GB
| improve this answer | |
3

There are a number of reasonable things you could set as constant, which is probably why they have experts determine the rules in a fundamentally arbitrary but hopefully fair fashion. Energy is one, but some other options would be:

  • Hardware cost.

  • FLOPS (floating point operations per second).

I suspect (although my info is limited) hardware cost and energy would help neural network neural based devices, while a flops based metric would benefit classical engines, because GPUs tend to provide great FLOPS/watt and FLOPS/$ -- metrics that CPUs sacrifice for the ability to go (relatively) quickly through branchy logic.

An argument against using a simple metric is that it might induce behavior that doesn't advance the state of the field in an interesting direction. If you ask for performance/watt or performance/$, people start asking questions like "what about an FPGA or ASIC," and eventually it becomes a competition in brute force and low level tuning, rather than interesting new ideas. Plus people wouldn't be able tinker with the engines at home if they needed specific devices to do so.

It is better to have experts in the field push toward interesting results, IMO. Although "energy limited on commodity hardware" might be fun.

| improve this answer | |
  • Far better to let any computer and any software make the best TOTAL SYSTEM to play chess and see how good that is. – edwina oliver Feb 23 at 21:50
  • 4
    I'm not sure about that -- are some of the chess algorithms embarrassingly parallel? In that case it is just a race to spend more money, which seems a bit silly. – Zwuwdz Feb 23 at 22:58
  • It only a race for those who chose to enter. The whole thing is silly if you think about it. Some people choose to spend their money time effort on it but that is their choice and they are free to do it unless the govt becomes totally dictatorial. OTOH it is a useful adjunct to research in AI and computer design and maybe even designing software. – edwina oliver Feb 24 at 0:48
  • The usefulness to AI and computer design is what makes it not silly. The concern I was trying to bring up was: the best way to make a computer more efficient is often to strip out some generality. Thus, if we want to have a competition that has a positive impact on general AI research and computer design, imposing harsh efficiency standards is counterproductive. – Zwuwdz Feb 24 at 21:01
  • you keep saying what i actually said initially. let each person be free to build the best chess playing SYSTEM that they can without any restrictions. – edwina oliver Feb 24 at 21:04
1

It really comes down to the purpose of the event.

If it is a championship that is trying to determine who wrote the best software, and not just the strongest hardware/software combination, then you want to even up the hardware as best as possible so the software is the determining factor. The winner determines the best software.

If it is a demonstration or exhibition match, like the AlphaZero-Stockfish match, or the Kasparov-Deep Blue match years ago, and it is all about just crushing the opponent and winning at all costs, all bets are off, and you can expect to see massive, and outrageous, hardware builds.

| improve this answer | |
  • The AlphaZero-Stockfish preprint was heavily criticized because the hardware was so busted. They corrected the issues in the final paper. Kasparov-Deep Blue I wouldn't say is a good example, since at the time it wasn't apparent how strong computers can become. It also wasn't a computer vs. computer match. – Allure Feb 23 at 21:54
  • @PhishMaster d-b and zwuwdz want to find the best sofware that runs on a specific PC windoze configuration which narrows the solution space way too far and ensures that inferior answers will be created. – edwina oliver Feb 24 at 21:27
-1

Should we limit human players who play by how much they weigh or their IQ?

There is no point in so called 'levelling the field' when computers play each other.

There is much more than building the biggest computer that determines who wins.

Software makes the hardware happen. And how well they came up with the software and how well they determined the algorithms to guide the software as well as practical tricks they use like having table bases or pitting two machines against each other to help them learn faster are more important than the hardware.

Limiting energy would only mean that they would use different compuer chip technology that has nothing to do with chess.

Your suggestion would change what is being optimized and what most people care about are how well the computers play not how much electricity they use.

However since the real concern is the software the latest championship had these rules for software:

The rules for the World Chess SOFTWARE Championship state that competing programs must run on machines with identical hardware specifications.

| improve this answer | |
  • 2
    I think you've misunderstood how relevant the hardware is, your analogy is very wrong. The IQ of human players is equivalent to the quality of the software/algorithms. The hardware/power provided is equivalent to the time on the clock. Saying you shouldn't 'level the field' when computers play each other is the same as saying you should let human players use unlimited time. That then simply becomes a competition of who has the most free time, rather than who is better at chess – Darren H Feb 23 at 0:51
  • Sorry but the hardware makes a big difference too. But software makes the hardware happen. Hardware is like what neurons you work with. Software is how they get used to do something useful. Your analogy with time is totally wrong. Both people and computers are time limited for fairness. The OP was about levelling the hardware which is already done since 2010. No amount of time would make any human better than their ability. Indeed too much time is counterproductive as one gets bored lazy sleepy and makes more errors. – edwina oliver Feb 23 at 1:14
  • 1
    This is completely wrong. There is a reason why the World Computer Chess Championship doesn't command the respect of the community, if you allow competitors to use their own hardware (+ opening book) then you aren't directly measuring chess strength. CCC and TCEC, the two most important computer chess championships, both use equal hardware for all programs (although the GPU engines recently threw a spanner into the works). – Allure Feb 23 at 3:58
  • 6
    @edwinaoliver no, I strongly disagree. Nobody is "making" hardware for these purposes, well almost nobody, you'd need the budget of Google to even think about it. Hardware is bought off the shelf and is dependant only on what funds you have available. The really clever bit is the software and that is what is being compared. I reiterate my analogy, hardware is the time, software is the brain. Allowing people to use their own hardware is equivalent to allowing a human to choose their own time controls. – Darren H Feb 23 at 4:15
  • 1
    @edwinaoliver here is not really the place for going into detail about the intricacies of operating systems, but suffice to say I still strongly disagree with your statement and your response leaves me with the impression that you don't really have a strong understanding of how these things actually work – Darren H Feb 24 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.