Trouble installing lightning-gpu with mpi support under Python3.10

justin6626 · December 12, 2024, 2:44am

Hi all, I am trying to install lightning-gpu with mpi support under python3.10, and I have been having some trouble with it. I am following the instructions here:

https://docs.pennylane.ai/projects/lightning/en/stable/lightning_gpu/installation.html

When I run this command:

CMAKE_ARGS="-DENABLE_MPI=ON" python3.10 -m pip install -e . --config-settings editable_mode=compat -vv

I get the following error:

Usage:   
  /usr/bin/python3.10 -m pip install [options] <requirement specifier> [package-index-options] ...
  /usr/bin/python3.10 -m pip install [options] -r <requirements file> [package-index-options] ...
  /usr/bin/python3.10 -m pip install [options] [-e] <vcs project url> ...
  /usr/bin/python3.10 -m pip install [options] [-e] <local project path> ...
  /usr/bin/python3.10 -m pip install [options] <archive url/path> ...

no such option: --config-settings

I understand now that the --config-settings option was introduced with python 3.11, so I am wondering if there is a way to do this installation under python 3.10

Here is the package version information:
Name: PennyLane
Version: 0.39.0
Summary: PennyLane is a cross-platform Python library for quantum computing, quantum machine learning, and quantum chemistry. Train a quantum computer the same way as a neural network.
Home-page: GitHub - PennyLaneAI/pennylane: PennyLane is a cross-platform Python library for quantum computing, quantum machine learning, and quantum chemistry. Train a quantum computer the same way as a neural network.
Author:
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: appdirs, autograd, autoray, cachetools, networkx, numpy, packaging, pennylane-lightning, requests, rustworkx, scipy, toml, typing-extensions
Required-by: PennyLane_Lightning, PennyLane_Lightning_GPU, PennyLane_Lightning_Kokkos

Platform info: Linux-6.8.0-49-generic-x86_64-with-glibc2.35
Python version: 3.10.12
Numpy version: 1.23.0
Scipy version: 1.8.0
Installed devices:

lightning.qubit (PennyLane_Lightning-0.39.0)
lightning.gpu (PennyLane_Lightning_GPU-0.39.0)
default.clifford (PennyLane-0.39.0)
default.gaussian (PennyLane-0.39.0)
default.mixed (PennyLane-0.39.0)
default.qubit (PennyLane-0.39.0)
default.qutrit (PennyLane-0.39.0)
default.qutrit.mixed (PennyLane-0.39.0)
default.tensor (PennyLane-0.39.0)
null.qubit (PennyLane-0.39.0)
reference.qubit (PennyLane-0.39.0)
lightning.kokkos (PennyLane_Lightning_Kokkos-0.39.0)

CatalinaAlbornoz · December 13, 2024, 8:32pm

Hi @justin6626,

I’m not sure, I’m checking with the team so I’ll let you know what I hear from them.

Alternatively you can create a Docker image with the following steps:

FROM nvcr.io/nvidia/cuquantum-appliance:24.08-x86_64 as base

ENV DEBIAN_FRONTEND=noninteractive
ENV PATH=/usr/local/cuda/bin:$PATH
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

RUN apt-get update && apt-get install -y build-essential openmpi-bin libopenmpi-dev

RUN git clone https://github.com/PennyLaneAI/pennylane-lightning.git && \
cd pennylane-lightning && \
pip install -r requirements.txt --upgrade && \
python -m pip install . && \
PL_BACKEND="lightning_gpu" python scripts/configure_pyproject_toml.py && \
CMAKE_ARGS="-DENABLE_MPI=ON" python -m pip install . -vv

Note that at the moment this doesn’t work with editable mode installation of Lightning_GPU-MPI.

Let me know if this works for you!

justin6626 · December 13, 2024, 9:18pm

Thank you very much! I will try that out and see if it works. I’m not very familiar with installing in editable mode. What would it do in this particular case?

CatalinaAlbornoz · December 13, 2024, 9:57pm

Hi @justin6626 ,

With my instructions above you would not be installing with editable mode. Note that it doesn’t have the “-e” that you had in your original post.

justin6626 · December 13, 2024, 10:14pm

Sorry about that! I was actually wondering if not installing in editable mode would lead to any problems.

CatalinaAlbornoz · December 13, 2024, 11:34pm

No, it shouldn’t lead to any problems.

Regarding your original post I heard back from the team.
The issue you were having might be rather about the pip version than Python version. What version are you using at the moment?

--config-settings is an option of pip, introduced with Changelog - pip documentation v24.3.1

You can run python -m pip install --upgrade pip to upgrade pip to v24.3.1 with python3.10 installation.

Once you upgrade pip you can try building lightning_qubit as follows:

PL_BACKEND="lightning_qubit" python scripts/configure_pyproject_toml.py
SKIP_COMPILATION=True pip install -e . --config-settings editable_mode=compat -vv

If this works then we can confirm that the issue was indeed with pip instead of Python.

Let me know if this works!

justin6626 · December 14, 2024, 3:26am

It looks like the pip version was the problem. Thank you very much for your help!

justin6626 · December 14, 2024, 4:05am

It looks like I spoke to soon. When I run

sudo pip install -e . --config-settings editable_mode=compat -vv

I get the following error:

-- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 139, in run
      self._create_wheel_file(bdist_wheel)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 340, in _create_wheel_file
      files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 263, in _run_build_commands
      self._run_build_subcommands()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 290, in _run_build_subcommands
      self.run_command(name)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 995, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/build_ext.py", line 99, in run
      _build_ext.run(self)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
      self.build_extensions()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
      self._build_extensions_serial()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
      self.build_extension(ext)
    File "<string>", line 151, in build_extension
    File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/home/user/pennylane-lightning', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmpmigozo5f.build-lib/pennylane_lightning', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DENABLE_WARNINGS=OFF', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DPYBIND11_FINDPYTHON=ON', '-GNinja', '-DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-jazjr65m/overlay/local/bin/ninja', '-DPL_BACKEND=lightning_gpu']' returned non-zero exit status 1.
  /tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py:973: _DebuggingTips: Problem in editable installation.
  !!

          ********************************************************************************
          An error happened while installing `PennyLane_Lightning_GPU` in editable mode.

          The following steps are recommended to help debug this problem:

          - Try to install the project normally, without using the editable mode.
            Does the error still persist?
            (If it does, try fixing the problem before attempting the editable mode).
          - If you are using binary extensions, make sure you have all OS-level
            dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
          - Try the latest version of setuptools (maybe the error was already fixed).
          - If you (or your project dependencies) are using any setuptools extension
            or customization, make sure they support the editable mode.

          After following the steps above, if the problem still persists and
          you think this is related to how setuptools handles editable installations,
          please submit a reproducible example
          (see https://stackoverflow.com/help/minimal-reproducible-example) to:

              https://github.com/pypa/setuptools/issues

          See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
          ********************************************************************************

  !!
    cmd_obj.run()
  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
      return hook(wheel_directory, config_settings, metadata_directory)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 476, in build_editable
      return self._build_with_temp_dir(
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 320, in run_setup
      exec(code, locals())
    File "<string>", line 206, in <module>
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 183, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 199, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 954, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 995, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 139, in run
      self._create_wheel_file(bdist_wheel)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 340, in _create_wheel_file
      files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 263, in _run_build_commands
      self._run_build_subcommands()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/editable_wheel.py", line 290, in _run_build_subcommands
      self.run_command(name)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 995, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/command/build_ext.py", line 99, in run
      _build_ext.run(self)
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
      self.build_extensions()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
      self._build_extensions_serial()
    File "/tmp/pip-build-env-jazjr65m/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
      self.build_extension(ext)
    File "<string>", line 151, in build_extension
    File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/home/user/pennylane-lightning', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmpmigozo5f.build-lib/pennylane_lightning', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DENABLE_WARNINGS=OFF', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DPYBIND11_FINDPYTHON=ON', '-GNinja', '-DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-jazjr65m/overlay/local/bin/ninja', '-DPL_BACKEND=lightning_gpu']' returned non-zero exit status 1.
  error: subprocess-exited-with-error
  
  × Building editable for PennyLane_Lightning_GPU (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_editable /tmp/tmp8ro9hm55
  cwd: /home/user/pennylane-lightning
  Building editable for PennyLane_Lightning_GPU (pyproject.toml) ... error
  ERROR: Failed building editable for PennyLane_Lightning_GPU
Failed to build PennyLane_Lightning_GPU
Remote version of pip: 24.3.1
Local version of pip:  24.3.1
Was pip installed by pip? True
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (PennyLane_Lightning_GPU)
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/commands/install.py", line 434, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: ERROR: Failed to build installable wheels for some pyproject.toml based projects (PennyLane_Lightning_GPU)
Removed build tracker: '/tmp/pip-build-tracker-dekwwoqi'

I have tried running the command outside of editable mode, but that does not seem to help. I am now using python3.11.9 instead of 3.10.

CatalinaAlbornoz · December 16, 2024, 4:09pm

Hi @justin6626 ,

So when you install lightning.qubit it works but not lightning.gpu?

I’d say this confirms that your original error with --config-settings is due to the pip version.

Did you try creating a Docker image with the instructions I shared a few days ago?

Topic		Replies	Views
MPI Enabled lightning@v0.32 PennyLane Help	5	413	October 11, 2023
Issues Installing and Running PennyLane Lightning GPU PennyLane Help	15	2671	November 20, 2024
Lightning-GPU for POWER9 PennyLane Help	13	766	November 30, 2022
Issues running LightningGPU PennyLane Help	3	341	January 22, 2024
Issues when using PennyLane Lightning GPU PennyLane Help	7	311	May 30, 2024

Trouble installing lightning-gpu with mpi support under Python3.10

Related topics