I think you are on to something, and statistically, there is a very good reason to play the Scotch over the Ruy Lopez, but that might not be all there is to it.
I think that the main reason is that we are taught that the Ruy Lopez is THE most classical opening, and is the best. I have seen that mindset in SO many books written by the top players over the years, in particular, Karpov. Also, I was watching "Method in Chess" videos recently, and GM Dorfman said that white is statically better because white has more space, and black is confined to three ranks, and white can play on the kingside, center, or queenside, and thus, black must always be on the lookout. I think that is just what classical Soviet players were taught, and that mindset has not been broken among the elite, especially since so many of them became trainers in other countries after the breakup of the Soviet Union.
Statistically, you are right on the money.
All the top players I looked at had similar results to Kasparov, meaning an overwhelming plus score with it. I left out Anand, Aronian, So, MLV, and Grischuk from the following list since they played it so few times (less than 6 each), it was clear that their reason was simply surprise.
- Carlsen 11-1-6 (wins-losses-draws) 78%
- Caruana 5-0-6 72%
- Nepomniachtchi 34-6-29 70%
- Nakamura 13-6-7 63%
These are against all players, and the problem is that in Carlsen's case, in particular, is most of the games were from when he was not the Carlsen we know today, and I could have left him out. Caruana's games were virtually all against elite competition!
Then I decided to search only for all games in the Mega 2020 database with players both above 2700, and then the stats fell dramatically down to a typical winning percentage for white, 58% even (53-25-97, 175 games total).
When I then did the same thing for 2700 players playing the Ruy Lopez, there were 2366 games with a breakdown of 580-373-1412, or only 54%.
That is still a healthy plus for the Scotch, but maybe the Soviets still were right, and that the only reason the stats for the Scotch are so good is that the opponent was not expecting it as much. Also, I did not attempt to look at hundreds of games to identify where the games where lost. It might have just been due to other circumstances.