# Lightning.gpu with multiple gpus **URL:** \ **Category:** PennyLane Help\ **Created:** [November 4, 2022, 8:44am UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267 "2022-11-04T08:44:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ToMago](https://yyz2.discourse-cdn.com/flex012/user_avatar/discuss.pennylane.ai/tomago/32/1367_2.png) [@ToMago](https://discuss.pennylane.ai/u/ToMago)\ **Post date:** [November 4, 2022, 8:44am UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/1 "2022-11-04T08:44:46Z")
Hey pennylane community, does lightning.gpu support scaling to multiple gpus? I noticed that cuQuantum supports multi-node and multi-GPU execution, is this also implemented in lightning.gpu ? If so could someone point to a simple example of setting this up? Thanks! Tom ---
**Author:** ![mlxd](https://yyz2.discourse-cdn.com/flex012/user_avatar/discuss.pennylane.ai/mlxd/32/526_2.png) [@mlxd](https://discuss.pennylane.ai/u/mlxd)\ **Post date:** [November 4, 2022, 10:52pm UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/2 "2022-11-04T22:52:13Z")
Hi @ToMago Do you have a specific use-case in mind? `lightning.gpu` natively supports multi-GPU execution across observables in the adjoint differentiation method. If you are creating circuits with multiple expectation values, the gradients will be carried out by splitting the work-load over multiple GPUs. Feel free to see [here](https://docs.pennylane.ai/projects/lightning-gpu/en/latest/devices.html) on how to enable this. For an example of the improvements we get when using this functionality, you can see [the release notes of `lightning.gpu` 0.25](https://github.com/PennyLaneAI/pennylane-lightning-gpu/releases/tag/v0.25.0). If you are looking to split a single state-vector over multiple GPUs, we do not currently support these offloads to cuQuantum. Let me know if you have any follow-on questions. ---
**Author:** ![ToMago](https://yyz2.discourse-cdn.com/flex012/user_avatar/discuss.pennylane.ai/tomago/32/1367_2.png) [@ToMago](https://discuss.pennylane.ai/u/ToMago)\ **Post date:** [November 8, 2022, 5:15pm UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/3 "2022-11-08T17:15:31Z")
Hey @mlxd and thanks for you answer! I’m looking to train e.g. a PQC with a single expectation value measurement with with adjoint differentiation. So lets say I have a Qnode `circuit` with a single measurement and train it with Adam on some cost function. Does `batch_obs=True` batch the different circuit evaluation even when I only have a single measurement? So I assume I use `batch_input` and the different circuit evaluation are distributed automatically ? Or does `batch_obs` only work for multi measurements ? Thanks! Tom ---
**Author:** ![CatalinaAlbornoz](https://yyz2.discourse-cdn.com/flex012/user_avatar/discuss.pennylane.ai/catalinaalbornoz/32/1196_2.png) [@CatalinaAlbornoz](https://discuss.pennylane.ai/u/CatalinaAlbornoz)\ **Post date:** [November 9, 2022, 11:40pm UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/4 "2022-11-09T23:40:49Z")
Hi @ToMago, Unfortunately `batch_obs` only works for multiple measurements. If you have a single measurement you cannot split it into more than one GPU. Something else you could try is using circuit cutting. I’m not sure whether it will work with GPUs but it could potentially help you reduce the memory needs (although with a time overhead). You can learn more about it in this [demo](https://pennylane.ai/qml/demos/tutorial_quantum_circuit_cutting.html). Please let me know if this helps! ---
**Author:** ![Jim](https://avatars.discourse-cdn.com/v4/letter/j/ac91a4/32.png) [@Jim](https://discuss.pennylane.ai/u/Jim)\ **Post date:** [November 10, 2022, 12:07pm UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/5 "2022-11-10T12:07:21Z")
Hi @mlxd and @CatalinaAlbornoz Is that any plans that pennylane will support splitting a single state-vector or splitting a single measurement over multiple GPUs? For this case it sounds straightforward to do the splitting: for qml.expval(qml.PauliZ(0)), the measurement expectation is the dot product of two vectors: sum over i of (a\_i + b\_i). This can be easily splitted into multiple GPUs: sum over k of (a\_k + b\_k) + sum over j of (a\_j + b\_j) + … Here are two links about matrix multiplication for multiple GPUs: > **[Tutorial 5 - Matrix Multiplication on multiple NVIDIA GPUs](https://pages.nist.gov/hedgehog-Tutorials/tutorials/tutorial5.html)** > > Tutorials for the Hedgehog library, designed to aid in creating task graphs for algorithms to obtain performance across CPUs and multiple co-processors. [https://hal.inria.fr/hal-02282529/document](https://hal.inria.fr/hal-02282529/document) ---
**Author:** ![mlxd](https://yyz2.discourse-cdn.com/flex012/user_avatar/discuss.pennylane.ai/mlxd/32/526_2.png) [@mlxd](https://discuss.pennylane.ai/u/mlxd)\ **Post date:** [November 10, 2022, 3:21pm UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/6 "2022-11-10T15:21:22Z")
Hi @ToMago Just to add onto what @CatalinaAlbornoz has suggested. If your return value is a Hamiltonian composed on sums of terms, this will allow multi-GPU batching as of v0.26.2. However, for the upcoming release, we also add support for `qml.SparseHamiltonian` as part of the single-GPU adjoint pipeline. We have found for many Hamiltonians this often beats the multi-GPU distributed approach in runtime, at the expense of generating the sparse-Hamiltonian value up-front. For the circuit cutting data, we have successfully cut a 62-qubit QAOA problem into many 20-30 qubit circuits allowing us to distribute the work over 100s of GPUs. Some additional details can be found in [this repo](https://github.com/xanaduai/randomized-measurements-circuit-cutting) and [this paper](https://arxiv.org/abs/2207.14734). I recently gave a [talk](https://www.youtube.com/watch?v=MnPAFbueMKQ) on this too here if you are interested. Feel free to let us know if we can be of further assistance. ---
**Author:** ![ToMago](https://yyz2.discourse-cdn.com/flex012/user_avatar/discuss.pennylane.ai/tomago/32/1367_2.png) [@ToMago](https://discuss.pennylane.ai/u/ToMago)\ **Post date:** [November 11, 2022, 8:48am UTC](https://discuss.pennylane.ai/t/lightning-gpu-with-multiple-gpus/2267/7 "2022-11-11T08:48:34Z")
Hi everyone, thanks for all your input, this is very helpful! I will also look further into circuit cutting. Tom