How to augment my quantum layer in my classical NN?

Hi! I want to make the hybrid model as one without saving the classical NN first and then loading it again and add it in my quantum layer.
This is the code for my classical model:

    def make_residual_lstm_layers(input, rnn_width, rnn_depth, rnn_dropout):
    """
    The intermediate LSTM layers return sequences, while the last returns a single element.
    The input is also a sequence. In order to match the shape of input and output of the LSTM
    to sum them we can do it only for all layers but the last.
    """
    x = input
    for i in range(rnn_depth):
        return_sequences = i < rnn_depth - 1
        x_rnn = LSTM(rnn_width, recurrent_dropout=rnn_dropout, dropout=rnn_dropout, return_sequences=return_sequences)(x)
        if return_sequences:
            # Intermediate layers return sequences, input is also a sequence.
            if i > 0 or input.shape[-1] == rnn_width:
                x = add([x, x_rnn])
            else:
                # Note that the input size and RNN output has to match, due to the sum operation.
                # If we want different rnn_width, we'd have to perform the sum from layer 2 on.
                x = x_rnn
        else:
            # Last layer does not return sequences, just the last element
            # so we select only the last element of the previous output.
            def slice_last(x):
                return x[..., -1, :]
            x = add([Lambda(slice_last)(x), x_rnn])
    return x
if __name__ == '__main__':
    # Example usage
    from keras.layers import Input
    from keras.models import Model
    from keras.layers.merge import add
    from keras.layers import Lambda
    input = Input(shape=(X_train.shape[1], X_train.shape[2]))
    output = make_residual_lstm_layers(input, rnn_width=24, rnn_depth=3, rnn_dropout=0)
    model = Model(inputs=input, outputs=output)
    model.summary()
And this is my QNN:
n_qubits =16
dev = qml.device("lightning.qubit", wires=n_qubits)
tf.keras.backend.set_floatx('float64')
@qml.qnode(dev, diff_method="adjoint")
def qnode(inputs, weights):
    QAOAEmbedding(features=inputs, weights=weights, local_field='Z', wires=range(n_qubits))
    return [qml.expval(qml.PauliZ(wires=i)) for i in range(n_qubits)]
n_layers = 1
weight_shapes = {"weights": (n_layers, 20)}
print(weight_shapes)
weights = np.random.random(QAOAEmbedding.shape(n_layers, n_wires=10))
features = np.array([1., 2.])
qlayer = qml.qnn.KerasLayer(qnode, weight_shapes, output_dim=n_qubits)
tf.keras.backend.set_floatx('float64')
from keras.models import load_model
prev_model = load_model(r"C:\Users\user\residual.h5") # loading the previously saved model.

model_gru = Sequential()
model_gru.add(prev_model)
model_gru.add(Dense(units=10, activation= 'relu'))
model_gru.add(qlayer)
model_gru.add(Dense(units=24, activation= 'linear'))
model_gru.add(Dense(units=24, activation= 'linear'))
model_gru(X_test[:2])
model_gru.summary()

Thank you

Hey @aouie! Welcome back :slight_smile:

To make a hybrid model with Keras, you don’t have to save the model, load it back, and then create your hybrid model. Our demo on turning quantum circuits into Keras layers contains an example where everything is defined on the fly (i.e., no classical elements of the network are loaded in).

Thank you for this! I tried to add it but I got this error:

TypeError Traceback (most recent call last)
in
11 model.add=Dense(units=10, activation= ‘relu’)
12 model(X_test[:2])
—> 13 model.add(qlayer)
14 model.add(Dense(units=24, activation= ‘linear’))
15 model.add(Dense(units=24, activation= ‘linear’))

~\anaconda3\lib\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.traceback)
—> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb

~\anaconda3\lib\site-packages\keras\engine\input_spec.py in assert_input_compatibility(input_spec, inputs, layer_name)
194 # have a shape attribute.
195 if not hasattr(x, ‘shape’):
–> 196 raise TypeError(f’Inputs to a layer should be tensors. Got: {x}’)
197
198 if len(inputs) != len(input_spec):

TypeError: Inputs to a layer should be tensors. Got:

This is the updated code:

if name == ‘main’:
from keras.layers import Input
from keras.models import Model
from keras.layers.merge import add
from keras.layers import Lambda
input = Input(shape=(X_train.shape[1], X_train.shape[2]))
output = make_residual_lstm_layers(input, rnn_width=24, rnn_depth=3, rnn_dropout=0)
model = Sequential(Model(inputs=input, outputs=output))
model.add=Dense(units=10, activation= ‘relu’)
model.add(qlayer)
model.add(Dense(units=24, activation= ‘linear’))
model.add(Dense(units=24, activation= ‘linear’))
model(X_test[:2])
model.summary()
opt = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(opt, loss=“mse”, metrics=[“accuracy”])
history = model.fit(X_train, y_train, epochs=100,validation_data=(X_test, y_test), batch_size=5000, shuffle= True)

It looks like part of your error message got cut off when you copy pasted! Can you resend that? Also, it would be awesome if you could gather a minimal example with inputs and outputs provided just to make sure that I’m able to reproduce your error without fail!

Oh, sorry. Here is the error:
TypeError Traceback (most recent call last)
in
11 model.add=Dense(units=10, activation= ‘relu’)
12 model(X_test[:2])
—> 13 model.add(qlayer)
14 model.add(Dense(units=24, activation= ‘linear’))
15 model.add(Dense(units=24, activation= ‘linear’))

~\anaconda3\lib\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~\anaconda3\lib\site-packages\keras\engine\input_spec.py in assert_input_compatibility(input_spec, inputs, layer_name)
    194     # have a `shape` attribute.
    195     if not hasattr(x, 'shape'):
--> 196       raise TypeError(f'Inputs to a layer should be tensors. Got: {x}')
    197 
    198   if len(inputs) != len(input_spec):

TypeError: Inputs to a layer should be tensors. Got: <Quantum Keras Layer: func=qnode>

And for the input, you can use the Iris Dataset and this is how I split it:

def make_timeseries_instances(timeseries, window_size):
timeseries = np.asarray(timeseries)
irnd = np.random.permutation(timeseries.shape[0]-window_size-24)
irnd = np.asarray(irnd)
print(irnd)
assert 0 < window_size < timeseries.shape[0]
X = np.atleast_3d(np.array([timeseries[start:start + window_size] for start in irnd]))
y = np.atleast_2d(np.array([timeseries[start+window_size+1:start+window_size+25,0] for start in irnd]))
q = np.atleast_3d([timeseries[-window_size:]])
print(“X shape”,X.shape)
print(“y shape”,y.shape)
print(“q shape”,q.shape)
return X,y,q
X,y,q = make_timeseries_instances(dataframe,85)
test_size = int(0.2 * X.shape[0])
X_train, X_test, y_train, y_test = X[:-test_size], X[-test_size:], y[:-test_size], y[-test_size:]
Thank you!!!

Hey @aouie! What version of all of your packages are you running? I’m running keras==2.9.0 and pennylane==0.24.0, but I can’t import keras.layers.merge (ModuleNotFoundError).

Make sure you’re using the right package versions that pennylane requires! I recommend that you make a virtual environment for your project (e.g., with conda: conda create -n my_environment_name python=3.8), then reinstall your packages :slight_smile:

Hello! I already updated all my packages but I encountered this error when I tried to run the lightning.qubit, but this error didn’t occur before.
Here is the code:

n_qubits =5
dev = qml.device("lightning.qubit", wires=n_qubits)
tf.keras.backend.set_floatx('float64')
@qml.qnode(dev, diff_method="adjoint")
def qnode(inputs, weights):
    QAOAEmbedding(features=inputs, weights=weights, local_field='Z', wires=range(n_qubits))
    return [qml.expval(qml.PauliZ(wires=i)) for i in range(n_qubits)]

And this is the error message:

ImportError                               Traceback (most recent call last)
<ipython-input-14-b074ffbcc871> in <module>
  1 n_qubits =10
----> 2 dev = qml.device("lightning.qubit", wires=n_qubits)
  3 tf.keras.backend.set_floatx('float64')
  4 @qml.qnode(dev, diff_method="adjoint")
  5 def qnode(inputs, weights):

~\anaconda3\lib\site-packages\pennylane\__init__.py in device(name, *args, **kwargs)
305 
306         # loads the device class
--> 307         plugin_device_class = plugin_devices[name].load()
308 
309         if Version(version()) not in Spec(plugin_device_class.pennylane_requires):

~\anaconda3\lib\site-packages\pkg_resources\__init__.py in load(self, require, *args, **kwargs)
   2448         if require:
   2449             self.require(*args, **kwargs)
-> 2450         return self.resolve()
   2451 
   2452     def resolve(self):

~\anaconda3\lib\site-packages\pkg_resources\__init__.py in resolve(self)
   2454         Resolve the entry point from its module and attrs.
   2455         """
-> 2456         module = __import__(self.module_name, fromlist=['__name__'], level=0)
   2457         try:
   2458             return functools.reduce(getattr, self.attrs, module)

~\anaconda3\lib\site-packages\pennylane_lightning\__init__.py in <module>
 15 
 16 from ._version import __version__
---> 17 from .lightning_qubit import LightningQubit

~\anaconda3\lib\site-packages\pennylane_lightning\lightning_qubit.py in <module>
 46 
 47 try:
---> 48     from .lightning_qubit_ops import (
 49         adjoint_diff,
 50         MeasuresC64,

ImportError: DLL load failed while importing lightning_qubit_ops: The specified module could not be found.

Hey @aouie! I can’t reproduce your error. I’m able to define dev = qml.device("lightning.qubit", wires=n_qubits) no problem. Can you tell me which version of PennyLane you’re using? You simply need to print qml.__version__.

Hi! it is ‘0.24.0’. Same as yours but I don’t know why I get this error.

Make sure you’re typing qml.__version__ (with the two underscores before and after version) :slight_smile:

Hi! I am using pennylane==0.24.0 and keras 2.9.0

Hi @aouie, from this error it looks like an installation issue. I suggest you follow the following steps on your terminal :

  1. Create a new virtual environment using: conda create --name my_environment_name python=3.9
  2. Make sure to activate your new environment: activate my_environment_name
  3. Check your Python version: python3 --version
  4. Upgrade pip: python -m pip install --upgrade pip
  5. Install Jupyter: python -m pip install jupyter
  6. Install PennyLane: python -m pip install pennylane
  7. Install Tensorflow: python -m pip install tensorflow

Now open a new Jupyter notebook and code:

import pennylane as qml
import tensorflow as tf
qml.about()
n_qubits =5
dev = qml.device("lightning.qubit", wires=n_qubits)

If you get no errors then your problem should be fixed. You should see some version and platform info printed out. If you see an error please post the printed info here, as well as the error. Hopefully this should fix your problem!

Hi! Thank you for the help! I followed the steps but I still got the same error.

>     Name: PennyLane
>         Version: 0.25.1
>         Summary: PennyLane is a Python quantum machine learning library by Xanadu Inc.
>         Home-page: https://github.com/XanaduAI/pennylane
>         Author: 
>         Author-email: 
>         License: Apache License 2.0
>         Location: c:\users\user\anaconda3\envs\nvdia_trial\lib\site-packages
>         Requires: appdirs, autograd, autoray, cachetools, networkx, numpy, pennylane-lightning, retworkx, scipy, semantic-version, toml
>         Required-by: PennyLane-Lightning
> 
>         Platform info:           Windows-10-10.0.19043-SP0
>         Python version:          3.9.12
>         Numpy version:           1.23.2
>         Scipy version:           1.9.0
>         Installed devices:
>         - default.gaussian (PennyLane-0.25.1)
>         - default.mixed (PennyLane-0.25.1)
>         - default.qubit (PennyLane-0.25.1)
>         - default.qubit.autograd (PennyLane-0.25.1)
>         - default.qubit.jax (PennyLane-0.25.1)
>         - default.qubit.tf (PennyLane-0.25.1)
>         - default.qubit.torch (PennyLane-0.25.1)
>         - default.qutrit (PennyLane-0.25.1)
>         - lightning.qubit (PennyLane-Lightning-0.25.0)
>         ---------------------------------------------------------------------------
>         ImportError                               Traceback (most recent call last)
>         Input In [1], in <cell line: 5>()
>               3 qml.about()
>               4 n_qubits =5
>         ----> 5 dev = qml.device("lightning.qubit", wires=n_qubits)
> 
>         File ~\anaconda3\envs\nvdia_trial\lib\site-packages\pennylane\__init__.py:309, in device(name, *args, **kwargs)
>             306 options.update(kwargs)
>             308 # loads the device class
>         --> 309 plugin_device_class = plugin_devices[name].load()
>             311 if Version(version()) not in SimpleSpec(plugin_device_class.pennylane_requires):
>             312     raise DeviceError(
>             313         f"The {name} plugin requires PennyLane versions {plugin_device_class.pennylane_requires}, "
>             314         f"however PennyLane version {__version__} is installed."
>             315     )
> 
>         File ~\anaconda3\envs\nvdia_trial\lib\site-packages\pkg_resources\__init__.py:2458, in EntryPoint.load(self, require, *args, **kwargs)
>            2456 if require:
>            2457     self.require(*args, **kwargs)
>         -> 2458 return self.resolve()
> 
>         File ~\anaconda3\envs\nvdia_trial\lib\site-packages\pkg_resources\__init__.py:2464, in EntryPoint.resolve(self)
>            2460 def resolve(self):
>            2461     """
>            2462     Resolve the entry point from its module and attrs.
>            2463     """
>         -> 2464     module = __import__(self.module_name, fromlist=['__name__'], level=0)
>            2465     try:
>            2466         return functools.reduce(getattr, self.attrs, module)
> 
>         File ~\anaconda3\envs\nvdia_trial\lib\site-packages\pennylane_lightning\__init__.py:17, in <module>
>              14 """Top level PennyLane-Lightning module."""
>              16 from ._version import __version__
>         ---> 17 from .lightning_qubit import LightningQubit
> 
>         File ~\anaconda3\envs\nvdia_trial\lib\site-packages\pennylane_lightning\lightning_qubit.py:48, in <module>
>              45 from ._version import __version__
>              47 try:
>         ---> 48     from .lightning_qubit_ops import (
>              49         adjoint_diff,
>              50         MeasuresC64,
>              51         StateVectorC64,
>              52         MeasuresC128,
>              53         StateVectorC128,
>              54         Kokkos_info,
>              55         allocate_aligned_array,
>              56         get_alignment,
>              57         best_alignment,
>              58     )
>              60     from ._serialize import _serialize_observables, _serialize_ops
>              62     CPP_BINARY_AVAILABLE = True
> 
>         ImportError: DLL load failed while importing lightning_qubit_ops: The specified module could not be found.

Hi @aouie. It looks like this is a bug with PennyLane-Lightning under Windows. We are trying to identify the cause now. Thank you for sharing these details!

If you do the following, does the problem get fixed?

python -m pip install pennylane==0.24
python -m pip install pennylane-lightning==0.24

Hi @aouie, my colleague Lee has found the source of the problem! Unfortunately the problem is Windows.

It seems that the bug was present in v0.24 as well so my previous comment won’t work.

To solve the problem you have 2 alternatives:

A: Revert back to v0.23 using:

python -m pip install pennylane==0.23
python -m pip install pennylane-lightning==0.23

B: Shift to Linux by using WSL on your Windows and using Visual Studio Code.
Follow the guide here
and use the Linux environment, they can pip install pennylane and it should work, getting around the Windows issue for now.

I hope this helps!

Thank you for this! I am using Linux now and it is working with pennylane v0.23. I will get back to you regarding my problem with my hybrid QNN. I really appreciate your help!!

Hi! When I try to augment the QNN, I get an error.
This is my code:
if name == ‘main’:
from keras.layers import Input
from keras.models import Model
from keras.layers.merge import add
from keras.layers import Lambda
input = Input(shape=(X_train.shape[1], X_train.shape[2]))
output = make_residual_lstm_layers(input, rnn_width=24, rnn_depth=3, rnn_dropout=0)
model = Sequential(Model(inputs=input, outputs=output))
model.add(Dense(units=10, activation= ‘relu’))
model.add(qlayer)
model.add(Dense(units=24, activation= ‘linear’))
model.add(Dense(units=24, activation= ‘linear’))
model(X_test[:2])
model.summary()
opt = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(opt, loss=“mse”, metrics=[“accuracy”])
history = model.fit(X_train, y_train, epochs=100,validation_data=(X_test, y_test), batch_size=5000, shuffle= True)
And this is the error:

UnknownError Traceback (most recent call last)
Input In [14], in <cell line: 1>()
11 model.add(Dense(units=24, activation= ‘linear’))
12 model.add(Dense(units=24, activation= ‘linear’))
—> 13 model(X_test[:2])
14 model.summary()
15 opt = tf.keras.optimizers.Adam(learning_rate=0.01)

File ~/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     65 except Exception as e:  # pylint: disable=broad-except
     66   filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67   raise e.with_traceback(filtered_tb) from None
     68 finally:
     69   del filtered_tb

File ~/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52 try:
     53   ctx.ensure_initialized()
---> 54   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                       inputs, attrs, num_outputs)
     56 except core._NotOkStatusException as e:
     57   if name is not None:

UnknownError: Exception encountered when calling layer "lstm" (type LSTM).

Fail to find the dnn implementation. [Op:CudnnRNN]

Call arguments received:
  • inputs=tf.Tensor(shape=(2, 85, 4), dtype=float64)
  • mask=None
  • training=None
  • initial_state=None

Hi @aouie, I’m glad you can use lightning now!
If you’re on Linux you should be able to use v0.25 if you want.

Your new error looks like a problem with your inputs.
Are you able to run this tutorial?

If the tutorial runs but not your code then I would suggest making a minimum working example. This involves making a super simple version of your code that runs (or doesn’t run) and that is stripped of anything non essential.

In many cases this will lead you to finding the error yourself, and otherwise it can allow us to concentrate on a simpler version of your program. Be sure to share the data you’re using as test and train data so that we can run your example too.

Hi @aouie, I have great news!

The bug that you found has now been fixed in Lightning! :star_struck:

Upgrading the package should bring in the latest release and enable you to continue.

Thank you for bringing this up!

Thank you! I will check into that!