I've some experience with CUDA and chess programming (not engine programming). This question also came to my mind many times. Here is my opinion, I never studied this.
Well, first, it is virtually certain that with current design of chess engines it's impossible. Otherwise someone would simply do it.
Whether you could use it in completely new design, that's a bigger question. There's no problem with integer operations, but clearly there is huge thing that you must run primitive code with no serious branching on many of entities.
In my opinion all the tree calculations should run on CPU.
But I can imagine (I don't say it's possible) some partial evaluation function(s) that just make as much evaluations on positions as possible, to give CPU more time, to help him build the tree in a better way, lets say one such function could calculate how serious calculation should be made on each position.
You can calculate number of squares attacked, pieces attacked, firepower around kings and in some subregions on board, pawn structure related things, you could probably make GPU optimised code for these. You can probably get a lot of useful informations for free (no CPU time, no CPU waiting), but you would have to be real genius to use full horsepower of modern GPUs for such small things.
It is probably not very interesting thing to do to create completely new design of engines to get let's say 30% extra speed. And to get let's say 20x speedup on state of the art GPUs, you would have to make evaluation functions really alien.
And last thing on this what I see to be a problem. You can run one code on CPU with not many thinking, but in CPU+GPU scenario you would have to deal with really different ratios of CPU and GPU. speeds and core counts. You would need different code for 16 cores cluster with weak GPU than on slow CPU with super fast GPU to use full capabilities of hardware. Benchmarking CPU vs. GPU and then running more code on the faster machine would be desirable. But still, I think you will get nowhere close to feeding 3000 cores beast just with evaluation function(s).
Edit: Just to make things clear, CPU would regularly send GPU arrays of positions to make partial evaluations. Maybe that wasn't clear from what I wrote.
Many problems to make this work.