Early on, chess is clearly just a game, but as you advance competitively, its character changes. For me, that change happened when I was about 1800-2000.
I was a ranked (state, not nationally) junior tennis player in the early 1970's, so I know what a sport feels like, and master-level chess takes just as much out of me as tennis did when I was a kid. Part of that is because I cannot sit still, and thus, I walk around a lot between moves, but most of it is just the tension.
In addition, the competitive day in chess is longer than most other sports. It is not uncommon for round one to start at 9 a.m., and sometimes I finish the second game at 10 p.m. at night.
Just because you do not run a 100-yard dash during the game, that does not mean that your body is not working harder physically, even though you cannot see it overtly. During the first Karpov vs Kasparov match during 1984-1985, Karpov, not a big man to begin with, lost 22 pounds during the course of the 5-month match due to the stress and other factors. They said he looked like "death" at the end of the match.
In a recent ESPN article on Fabiano Caruana, they noted "In October 2018, Polar, a U.S.-based company that tracks heart rates, monitored chess players during a tournament and found that 21-year-old Russian grandmaster Mikhail Antipov had burned 560 calories in two hours of sitting and playing chess -- or roughly what Roger Federer would burn in an hour of singles tennis."
They also said this, which is phenomenal: "a chess player can burn up to 6,000 calories a day while playing in a tournament."
So, while the tennis burned more calories, the chess certainly was no slouch in that regard, and we play a lot longer. If chess is not a physical sport, even if it is not obvious, why is fitness so important? It would not be if it were not a sport. Mental exertion, clearly, also manifests itself physically.