Hi @_risto,
As a general approach, that could be done indeed.
When dealing with image datasets, as was hinted previously, feeding the input data into a convolutional neural network (CNN) is also very popular for image classification. Several architectures use convolutional networks (including ResNet).
For example in the Training a classifier tutorial includes the creation of a CNN which is then put as a layer before calculating the loss in each step.
Hi,
I am trying out the same approach of changing the last layer to a Quantum layer in NSPnet pretrained model which I got it from gluoncv in which the layers are defined using mxnet. As mxnet is not added to the interface list of Pennylane how to move forward?
for your reference I have added the models with respective changes.
Hi @antalszava
Thank you for explanation regarding CNN. I understand in theory what has to be done, but am having hard time writing the code. Is there a code written in once place, which was used to get this: https://raw.githubusercontent.com/PennyLaneAI/qml/master/demonstrations/embedding_metric_learning/X_antbees.txt data? Does simply removing the last layer of ResNet512 (or any other ResNet) give you the needed features vectors, or is there an additional modifications that need to be done?
Hi @Qudsiaamir,
Welcome to the forum! 
That’s an interesting approach. Indeed, mxnet is not a supported interface by PennyLane at the moment. For differentiability purposes the QNode that gets created internally is specifically tailored to the framework that is being used (in this case that would be mxnet). This is needed so that trainable parameters are registered inside of PennyLane.
Could perhaps the layers defined in mxnet be ported to TensorFlow using Keras or to Torch? Having interfaces and quantum layers for those frameworks (KerasLayer and TorchLayer), PennyLane could be used to define a quantum layer seamlessly.
Another, more experimental approach would be creating a function that changes a QNode to be compatible with mxnet (see the Torch equivalent here). Although a bit more advanced, this would be a fun way of integrating PennyLane with mxnet and could even result in a new interface!
In either case, feel free to share more details on the specific task you’re looking at (e.g. code snippets) and feel free to ask questions that might come up!
Hi @_risto,
We’ve uploaded the specific script that was being used, this is similar method as the one used in transfer learning. Credits to @andreamari!
Hi @antalszava.
After running the code, I get TypeError: ‘dict’ object is not callable.
Hi @_risto,
There was an issue with the script that has been fixed now. Just download the latest version of the script and it should hopefully work fine.
If you have any other issues, feel free to message us here again!
Hi @theodor
Thank you for the updated code. So, after getting the feature vectors, I have stored them in a specific subfolder (4 different subfolder for each txt file), as I will have to put my specific path in the code. When I try to run the code:
X = np.loadtxt("embedding_metric_learning/X_antbees.txt", ndmin=2) #1 pre-extracted inputs
Y = np.loadtxt("embedding_metric_learning/Y_antbees.txt") # labels
X_val = np.loadtxt(
"embedding_metric_learning/X_antbees_test.txt", ndmin=2
) # pre-extracted validation inputs
Y_val = np.loadtxt("embedding_metric_learning/Y_antbees_test.txt") # validation labels
# split data into two classes
A = X[Y == -1]
B = X[Y == 1]
A_val = X_val[Y_val == -1]
B_val = X_val[Y_val == 1]
print(A.shape)
print(B.shape)
where I have changed the loading txt data with my specific path, I get that permission to that folder is denied. I have tried to run jupyter as an admin, also run jupyter in conda as an admin, but it did not solve the problem.
Hi @_risto. That seems to be an issue with your local folder permissions. I would recommend that you move the files into a folder where you have read/write permissions (e.g. your local Documents folder, depending on OS/distro). You could also look at how to change permissions for that specific folder.
Hi @theodor
After extracting the feature vectors from hymenoptera_data, the number of lines in antbees.txt file that I get doesn’t match the number 153 (as it does in the antbees.txt file on github page).
Hi @_risto, thanks for pointing this out — it turns out there was a bug in the script, that has now been corrected. Have a go using the updated script, and let us know if it works!
Hi @josh
Thanks for the reply. Now I get 5 files (X_antbees_test.txt (1.9 MB) Y_antbees.txt (6.3 KB) Y_antbees_test.txt (4.0 KB) Y_antbees_train.txt (6.3 KB) and X_antbees.txt, which was not uploaded, as it is empty) and error:
“boolean index did not match indexed array along dimension 0; dimension is 0 but corresponding boolean dimension is 244”.
Hi @_risto.
The fifth file you are getting, Y_antbees_train.txt, is generated by the old script. Please note the the new script generates the following files:
X_antbees.txt containing X_train
Y_antbees.txt containing Y_train
X_antbees_test.txt containing X_val
Y_antbees_test.txt containing Y_val
Please also note that, in the new script, Y_train is computed using the train_data as:
Y_train = [1 if inp[1] == 0 else -1 for inp in train_data] which gives you the correct number of data points. The file Y_antbees.txt in the github folder is also updated with the new data.
Could you please do the following steps and let us know about the results?
Create a new folder.
Download the hymenoptera_data into your new folder.
Run the new script image_to_resnet_output.py, downloaded from here, in your new folder.
You can run the new script either by copying its content to a new notebook or using python image_to_resnet_output.py in a terminal. Please make sure that you run the script in the same folder that you have your hymenoptera_data. Then you will get 4 files: X_antbees.txt and Y_antbees.txt with 245 lines; X_antbees_test.txt and Y_antbees_test.txt with 153 lines.
Please let us know if you have any questions. Thanks.
Hi @sjahangiri.
After the recommended changes I get
(121, 512)
(122, 512)
instead of
(83, 512)
(70, 512)
Consequently the gram matrix does not separates the two clases:
Hey _risto,
Let me take over from @sjahangiri, since the bug is my fault 
You are totally right, somehow the validation set is perfectly learnt by the pre-trained parameters, but not the training set 
Very strange, and smells like a silly mistake somewhere. I will try to find it!
Update: I just cannot find why the model works so well on the (new version of the) test data but not the training data. That is so odd, even if we accidentally trained on the test data before.
I will have to retrain, which can take a day or two…
And another update: The bug with the data preparation uncovered a problem that will require a bit of research to fix.
I took the demo off the webpage in the meantime to avoid people running incorrect code. Will post here once this is fixed.
In the meantime, may I ask, would it be possible to provide the code for the exactly the same ResNet18, which was used in Quantum Transfer & Embeddings learning algorithm, before the last layer was removed and replaced by the quantum circuit? (so one can compare the two - original : quantum)?
Hi @_risto,
I’m joining this thread a bit late, so apologies if repeating something you’ve already discussed.
Have you checked out the quantum transfer learning repo? This is a great place to start if you want to dig deeper. I believe a relevant line in the tutorial would be:
model_hybrid = torchvision.models.resnet18(pretrained=True)
Since the model is not quantum, you can also check out the Torch documentation.
Hello. Just dropping by to ask, how the quantum embeddings demo is going on?
Cheers,
Risto