Error in PennyLane Lightning: CUDA driver version is insufficient for CUDA runtime version

PennyLane Plugins PennyLane Lightning
1

Hi guys,
I’m trying to use the pennylane lightning gpu but I get the following error when I create my device object:

/user/.conda/envs/cuda_11.5_env_v2/lib/python3.10/site-packages/pennylane_lightning_gpu/lightning_gpu.py:110: UserWarning: [/project/pennylane_lightning_gpu/src/util/cuda_helpers.hpp][Line:603][Method:getGPUCount]: Error in PennyLane Lightning: CUDA driver version is insufficient for CUDA runtime version
  warn(str(e), UserWarning)
/user/.conda/envs/cuda_11.5_env_v2/lib/python3.10/site-packages/pennylane_lightning_gpu/lightning_gpu.py:977: RuntimeWarning: 
            !!!#####################################################################################
            !!!
            !!! WARNING: INSUFFICIENT SUPPORT DETECTED FOR GPU DEVICE WITH `lightning.gpu`
            !!!          DEFAULTING TO CPU DEVICE `lightning.qubit`
            !!!
            !!!#####################################################################################

I work on an HPC with V100 gpus and with CUDA 11.5 installed inside my Conda environment.
Here are the Conda installation commands that I used:

- CONDA_OVERRIDE_CUDA="11.5" conda install -c "nvidia/label/cuda-11.5.0" cuda-toolkit
- pip3 install pennylane-lightning[gpu]
- pip3 install custatevec-cu11

I also defined the global variable LD_LIBRARY_PATH=/user/.conda/envs/cuda_11.5_env/lib before executing my python code.
Finally here’s a the code I execute to get this error:

import pennylane as qml
dev = qml.device("lightning.gpu", wires=1)

I hope I gave enough informations to help me fit this issue, if not don’t hesitate to ask for more.

Thanks for your help in advance!

2

Hi @Lalou
The issue looks like the CUDA driver is too old on your HPC system to run CUDA 11.5+ code.
Can you post the output of nvidia-smi. Depending on how your system is managed, you may need to request an update of the CUDA driver by your system administrator to ensure to is capable of running the given CUDA version code.

Here is a similar output on the NVIDIA forum (CUDA driver version is insufficient for CUDA runtime version - #4 by Robert_Crovella - CUDA Setup and Installation - NVIDIA Developer Forums), where the recommended course of action is to ensure the CUDA driver matches or exceeds the CUDA runtime version. Since conda only allows you to update the CUDA runtime libraries, this may not be sufficient to ensure a working environment.

I suggest updating everything to support CUDA 11.8 for both drivers and runtime if possible, as this is the most recent (and likely best supported) version of the CUDA 11 releases.

3

Hi @mlxd
Thanks for your answer.
Yes, you’re correct. On my HPC system, the oldest available CUDA version is 11.4. That’s why I wanted to install a newer version in my Conda environment. Is there any way to update the CUDA driver without requesting an update from my sysadmin? (There’s a very slim chance that he’ll accept my request.)
nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000000:5E:00.0 Off |                    1 |
| N/A   31C    P0    25W / 250W |      1MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  On   | 00000000:D8:00.0 Off |                    0 |
| N/A   31C    P0    25W / 250W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
4

Hi @Lalou
Unfortunately drivers need to be installed system-wide to work, and so cannot normally be added by updating user-space libraries.

In this case, I think the only option is to request a more recent CUDA driver to be installed on the system. The 418.87.00 driver version is quite outdated by a number of years (Tesla Driver for Linux x64 | 418.87.00 | Linux 64-bit | NVIDIA), and so will need upgrading to ensure correct operation for both lightning.gpu and cuQuantum.

5

Alright I’ll try to do that then.
Thanks for all :pray:

1 Like
6

Happy to help. Feel free to let us know if you need any further assistance.