Is testing an opening by having a computer play against itself viable?

Question

Recently, I got into a brief argument with a colleague when I mentioned that I wanted to use an engine playing against itself to generate a larger sample size for a rare sideline.

His argument was that human chess and computer chess are two different things, which made sense. Humans don't scan for every possible legal move, and don't use brute calculation like computers do.

However, there may not be a time when masters decide to play a given line, meaning that rare lines would have fewer games played. There would be no other way to build a larger dataset other than playing games yourself or waiting for other people to play it.

Another factor to consider is how seriously people take TCEC openings.

Is the difference between humans and computers substantial enough to dismiss opening testing with an engine playing against itself? Would the use of computer games be a flawed sample size for determining the validity of a move?

Answer 1

If you're concerned about whether an opening is objectively good, then using computer games is a good approach. Computers don't play perfectly, but they are by far our best option. You could also check out correspondence games from ICCF, which involves people sometimes having days to figure out each move, and they can use an engine. These games, as well as TCEC games, are some of the strongest games available.

However, if you're interested in how good an opening is practically (i.e., for human OTB play), then it depends on what you mean. Personally, I'd think an opening is practically good if it is fairly sound objectively, but also something I'd be comfortable playing in an OTB game.

Both of these requirements are somewhat subjective - e.g., what is "fairly sound objectively"? Some amateurs might say anything between 0.00 and -1.00 is fine for their Black opening, while super GMs may not be satisfied with anything worse than -0.20. Even more subjective is whether an opening is comfortable for you, since this comes down to playing styles.

To that end, it could be worthwhile to have an engine play against itself to figure out how good an opening is objectively. Although, there will likely already be enough games out there (TCEC, correspondence, high level GM games). You can also see how well an opening scores amongst all players in big databases, such as TWIC or the Mega Database. This could give an indication of how easy it is to play an opening, for people who aren't professionals or anything.

Answer 2

What you are after is called a novelty, or better: novelties.

That's formally (a) new move(s) in a certain position, but let's instead focus on an alternative, albeit slightly off, definition:

A (prepared) novelty is a move that's in your book, and hopefully missing in your opponents.

What's your book, alternatively your repertoire then?

Your book is, again roughly, the sum of what you know by heart and are willing to play.

How comes you are willing to play a certain line? - Well, you studied the line: you know the ins and outs of every reply your opponents could play; you understand why some replies are losing and how to punish those; why other replies are fine and how to proceed; which procedings can lead to what imbalances, pawn structures, endgames.

And that's exactly where engines come in handy. Nota bene that before our age, when this was being done with brainpower alone, the concept was still the one mentioned above! You investigate, summarize, memorize.

The investigation part is what you'd like to outsource now, and please feel free to do so! However, the follow-ups are still heavy workloads in summarizing (sort of translating into terms understandable to humans, the whys from above) and memorizing (sort of the hows). Thus, great idea, but the flip side is just that oh so many lines will take massive workloads to fully comprehend.

In essence: The difference between humans and computers is substantial enough to exactly build a repertoire upon it - if you're willing to do the comprehensional workloads.

Answer 3

I will not be able to answer your question fully but can guide you here. There is a field of Machine Learning called Reinforcement Learning which deals with this exactly. There has been an example case of a computer playing checkers against itself for a long time and ultimately learning to play better than the programmer itself. I suppose that similar RL Models would be existing for Chess too, you could look for it and it might help you.