No, there shouldn't and if you live to be 80 you will probably understand why.
A few years ago in a tournament where I was one of the arbiters a 16 year old boy was playing an old guy in his 80's in round 2, both of them were rated about 1950. There had recently been the case of the Bulgarian phone cheat who had consulted a phone hidden behind one of the toilets.
The 16 year old came up to me and said "Can I have a word with you outside?" It sounded very serious so I agreed. He then explained that his opponent was getting up regularly to go to the toilet even once doing it when it was his turn to move. The boy was worried his opponent was a phone cheat.
Had I been the boy's coach and not the arbiter I would have been able to explain the reasons. The old guy had just recently come out of hospital where he had had prostate surgery. The truth was he should really have been at home recovering with his feet up not playing chess but who is going to tell an 80 year-old how to live his life? In the previous round he had blundered his queen against a player rated about 1750. I would have told the boy to just keep playing good moves and the old guy was sure to blunder sooner or later. That is what happened in the game, by the way.
Of course as the arbiter I couldn't give him helpful information like that. Instead I told him I would check the toilets for mobile phones and keep an eye on the old guy for the rest of the round, which I did.
Note that the toilets are generally in the bounds of the playing venue but not in the playing area so going to the toilet when it is your turn is not allowed unless you get the permission of the arbiter. If you have a medical condition which makes this likely then you can speak to the arbiter before play to explain to the arbiter and get this permission.
If you are a woman then you can't have prostate problems but if you have children you may experience similar problems with needing to make regular visits to the toilet for some time after giving birth.
In short imposing limits on toilet breaks would be inhumane.