Is it legal to modify and redistribute a GPL engine, under a more restrictive license (for example closing the source code)?
GPL-licensed software cannot become closed software. If you 'fork' it, then the rights on this fork remain the same (see GNU General Public License on Wikipedia).
Short answer: No. But for chess, the answer: Yes. Read on.
GPL is quite complicated. Essentially, it requires the application be open-source.
Can I sell a GPL project?
You are allowed to sell copies of the modified program commercially, but only under the terms of the GNU GPL. Thus, for instance, you must make the source code available to the users of the program as described in the GPL, and they must be allowed to redistribute and modify it as described in the GPL.
Thus, it's ok to sell the Stockfish chess engine which is licensed as GPL.
Why open my source?
A crucial aspect of free software is that users are free to cooperate. It is absolutely essential to permit users who wish to help each other to share their bug fixes and improvements with other users.
In theory, whoever gets the source code could improve the engine and therefore benefit the whole community. However, in practice this has never worked well.
Can I change the license to something more restrictive?
No. The GPL says that your modified versions must carry all the freedoms stated in the GPL. Thus, anyone who receives a copy of your version from you has the right to redistribute copies (modified or not) of that version. You may not distribute any version of the work on a more restrictive basis.
But... ICC Chess Club bundles their software with the Stockfish engine and I don't have their source code. Is that illegal? No.
CHESS IS DIFFERENT
Chess engine almost always runs as a separate process from the application itself, connected to the application through pipes. This is very different to traditional programming where you bundles the GPL source code within the application.
Technically, there is no need to link (statically or dynamically) the GPL engine while compiling the GUI application. The engine itself is an executable that can run independently. Everything in the FAQ is only valid if the application links with the engine, if that isn't the case, there is no such thing as GPL license restriction.
Try to download Stockfish, do you see something like Stockfish_64bits.exe? This is a working executable that is independent to whatever chess GUI you'd like to use. You can even run it directly in the terminal. The engine is not linked to any chess GUI, thus it's always legal to bundle it with a commercial non-GPL chess GUI.
You can read more here
What does that mean?
It's perfectly legal to bundle a chess engine with a closed-source application if the engine is an independent executable. The GPL license simply don't apply here. But if you compile the engine with the application, you must provide the source code for your entire application. For example, ICC Chess Club is allowed to distribute the Stockfish engine without providing you their source code.