This is a great (and slightly non-trivial) question. In principle, PennyLane manages computations implemented by other libraries and frameworks, and whether GPU computations are supported depends on which frameworks are combined.
For example, if you use
diff_method="backprop" , and
interface="tf", then the entire workflow is executed in TensorFlow, and if you set it up to run on your GPU, the PennyLane computations should be run on your GPU. (It would be interesting to know if you can confirm this?).
If you want to use the parameter-shift rule (which is always much slower than backpropagation, and there to support hardware where backprop is not possible), maybe you could try the Qulacs device. If installed correctly, Qulacs supports GPU computations for the simulations of the circuit - but not for the entire optimisation pipeline. And Qulacs is a very fast device for CPU simulations as well!
Hope this helps?