I installed pennylane-lightning gpu with mpi support on Nersc Perlmutter with modules. using instructions - quantum-mini-apps/src/mini_apps/quantum_simulation/distributed_state_vector at main · radical-cybertools/quantum-mini-apps · GitHub. (kind of formulated from previous posts)
It used to work, but some upgrade on the HPC causing the error. I don’t see the installation failing for any reason and could successfully install pennylane-lgpu successfully with mpi support.
Hello! If applicable, put your complete code example down below. Make sure that your code:
- is 100% self-contained — someone can copy-paste exactly what is here and run it to
reproduce the behaviour you are observing - includes comments
from mpi4py import MPI
import pennylane as qml
from pennylane import numpy as np
from timeit import default_timer as timer
import sys
print("starting")
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
print("Initialized mpi")
print(size)
# Set number of runs for timing averaging
num_runs = 3
# Choose number of qubits (wires) and circuit layers
n_wires =int(sys.argv[1])
print(n_wires)
n_layers = 2
# Instantiate CPU (lightning.qubit) or GPU (lightning.gpu) device
# mpi=True to switch on distributed simulation
# batch_obs=True to reduce the device memory demand for adjoint backpropagation
dev = qml.device('lightning.gpu', wires=n_wires, mpi=True, batch_obs=True)
# Create QNode of device and circuit
@qml.qnode(dev, diff_method="adjoint")
def circuit_adj(weights):
qml.StronglyEntanglingLayers(weights, wires=list(range(n_wires)))
return qml.math.hstack([qml.expval(qml.PauliZ(i)) for i in range(n_wires)])
# Set trainable parameters for calculating circuit Jacobian at the rank=0 process
if rank == 0:
params = np.random.random(qml.StronglyEntanglingLayers.shape(n_layers=n_layers, n_wires=n_wires))
else:
params = None
# Broadcast the trainable parameters across MPI processes from rank=0 process
params = comm.bcast(params, root=0)
# Run, calculate the quantum circuit Jacobian and average the timing results
timing = []
for t in range(num_runs):
start = timer()
jac = qml.jacobian(circuit_adj)(params)
end = timer()
timing.append(end - start)
# MPI barrier to ensure all calculations are done
comm.Barrier()
if rank == 0:
print("num_gpus: ", size, " wires: ", n_wires, " layers ", n_layers, " time: ", qml.numpy.mean(timing))
If you want help with diagnosing an error, please put the full error message below:
Traceback (most recent call last):
File "/global/u1/p/prmantha/dist_mem_jacobian.py", line 47, in <module>
jac = qml.jacobian(circuit_adj)(params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/_grad.py", line 517, in _jacobian_function
jac = tuple(_jacobian(func, arg)(*args, **kwargs) for arg in _argnum)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/_grad.py", line 517, in <genexpr>
jac = tuple(_jacobian(func, arg)(*args, **kwargs) for arg in _argnum)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/wrap_util.py", line 20, in nary_f
return unary_operator(unary_f, x, *nary_op_args, **nary_op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/differential_operators.py", line 60, in jacobian
vjp, ans = _make_vjp(fun, x)
^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/core.py", line 10, in make_vjp
end_value, end_node = trace(start_node, fun, x)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/tracer.py", line 10, in trace
end_box = fun(start_box)
^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/wrap_util.py", line 15, in unary_f
return fun(*subargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/qnode.py", line 987, in __call__
return self._impl_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/qnode.py", line 977, in _impl_call
res = self._execution_component(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/qnode.py", line 935, in _execution_component
res = qml.execute(
^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/execution.py", line 624, in execute
results = ml_boundary_execute(tapes, execute_fn, jpc, device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/interfaces/autograd.py", line 147, in autograd_execute
return _execute(parameters, tuple(tapes), execute_fn, jpc)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/tracer.py", line 44, in f_wrapped
ans = f_wrapped(*argvals, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/autograd/tracer.py", line 48, in f_wrapped
return f_raw(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/interfaces/autograd.py", line 183, in _execute
return _to_autograd(execute_fn(tapes))
^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/jacobian_products.py", line 464, in execute_and_cache_jacobian
results, jac = self._dev_execute_and_compute_derivatives(tapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/workflow/jacobian_products.py", line 429, in _dev_execute_and_compute_derivatives
return self._device.execute_and_compute_derivatives(numpy_tapes, self._execution_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/devices/modifiers/simulator_tracking.py", line 97, in execute_and_compute_derivatives
return untracked_execute_and_compute_derivatives(self, circuits, execution_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/devices/modifiers/single_tape_support.py", line 62, in execute_and_compute_derivatives
results, jacs = batch_execute_and_compute_derivatives(self, circuits, execution_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/devices/modifiers/simulator_tracking.py", line 97, in execute_and_compute_derivatives
return untracked_execute_and_compute_derivatives(self, circuits, execution_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane/devices/modifiers/single_tape_support.py", line 62, in execute_and_compute_derivatives
results, jacs = batch_execute_and_compute_derivatives(self, circuits, execution_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane_lightning/core/lightning_newAPI_base.py", line 334, in execute_and_compute_derivatives
results = tuple(
^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane_lightning/core/lightning_newAPI_base.py", line 335, in <genexpr>
self.simulate_and_jacobian(
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane_lightning/core/lightning_newAPI_base.py", line 228, in simulate_and_jacobian
res = self.simulate(circuit, state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/u1/p/prmantha/pennylane-lightning/pennylane_lightning/lightning_gpu/lightning_gpu.py", line 542, in simulate
return self.LightningMeasurements(final_state).measure_final_state(circuit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane_lightning/core/_measurements_base.py", line 261, in measure_final_state
return tuple(self.measurement(mp) for mp in circuit.measurements)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane_lightning/core/_measurements_base.py", line 261, in <genexpr>
return tuple(self.measurement(mp) for mp in circuit.measurements)
^^^^^^^^^^^^^^^^^^^^
File "/pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages/pennylane_lightning/core/_measurements_base.py", line 240, in measurement
return self.get_measurement_function(measurementprocess)(measurementprocess)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/u1/p/prmantha/pennylane-lightning/pennylane_lightning/lightning_gpu/_measurements.py", line 200, in expval
return self._measurement_lightning.expval(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pennylane_lightning.lightning_gpu_ops.LightningException: [/global/u1/p/prmantha/pennylane-lightning/pennylane_lightning/core/src/simulators/lightning_gpu/StateVectorCudaMPI.hpp][Line:2208][Method:applyMPI_Dispatcher]: Error in PennyLane Lightning: an illegal memory access was encountered
terminate called after throwing an instance of 'Pennylane::Util::LightningException'
what(): [/global/u1/p/prmantha/pennylane-lightning/pennylane_lightning/core/src/utils/cuda_utils/DataBuffer.hpp][Line:133][Method:~DataBuffer]: Error in PennyLane Lightning: an illegal memory access was encountered
srun: error: nid200321: task 0: Aborted
srun: Terminating StepId=36872546.1
srun: error: nid200324: task 2: Aborted
slurmstepd: error: *** STEP 36872546.1 ON nid200321 CANCELLED AT 2025-03-15T05:25:01 ***
srun: error: nid200321: task 1: Terminated
srun: error: nid200324: task 3: Terminated
srun: Force Terminated StepId=36872546.1
And, finally, make sure to include the versions of your packages. Specifically, show us the output of qml.about().
Version: 0.39.0
Summary: PennyLane is a cross-platform Python library for quantum computing, quantum machine learning, and quantum chemistry. Train a quantum computer the same way as a neural network.
Home-page: https://github.com/PennyLaneAI/pennylane
Author:
Author-email:
License: Apache License 2.0
Location: /pscratch/sd/p/prmantha/lgpu_env/lib/python3.11/site-packages
Requires: appdirs, autograd, autoray, cachetools, networkx, numpy, packaging, pennylane-lightning, requests, rustworkx, scipy, toml, typing-extensions
Required-by: PennyLane_Lightning, PennyLane_Lightning_GPU
Platform info: Linux-5.14.21-150500.55.65_13.0.73-cray_shasta_c-x86_64-with-glibc2.31
Python version: 3.11.7
Numpy version: 2.0.2
Scipy version: 1.15.2
Installed devices:
- lightning.gpu (PennyLane_Lightning_GPU-0.39.0)
- lightning.qubit (PennyLane_Lightning-0.39.0)
- default.clifford (PennyLane-0.39.0)
- default.gaussian (PennyLane-0.39.0)
- default.mixed (PennyLane-0.39.0)
- default.qubit (PennyLane-0.39.0)
- default.qutrit (PennyLane-0.39.0)
- default.qutrit.mixed (PennyLane-0.39.0)
- default.tensor (PennyLane-0.39.0)
- null.qubit (PennyLane-0.39.0)
- reference.qubit (PennyLane-0.39.0)
>>>