Error running make test targets in frontend for Catalyst

Hi, I am trying to make a first contribution to Catalyst, in test_template.py in frontend.
I am following the installing guide of catalyst.
I am running in the directory catalyst. I am not sure if I am running the commands in the correct directory.
I created a python virtual environment.
I executed:
pip install -r requirements.txt
And:
git submodule update --unit --depth=1
But when I execute make test-frontend, I get the following error:
check the Catalyst lit test suite

cmake --build /home/nick-test/Documents/catalyst//mlir/build --target check-frontend
Error: /home/nick-test/Documents/catalyst/mlir/build is not a directory
make: *** [Makefile:71: lit] Error 1

Then I create the build directory.
When I execute make test-frontend again, I get the following error:

check the Catalyst lit test suite
cmake --build /home/nick-test/Documents/catalyst//mlir/build --target check-frontend
Error: could not load cache
make: *** [Makefile:71: lit] Error 1

Hello @ndcroos ! For contributing to Catalyst I would suggest you to install the project from source: Installation — Catalyst 0.2.2-dev documentation You will need to run make all first and then it will populate the mlir build folder. After that you should be able to run make test-frontend

Let me know if it helps ?

1 Like

@ndcroos this is exciting, let us know what you are thinking and what we can help with once you have the tests up and running :slight_smile:

In addition, I believe our CI on GitHub will automatically run the tests, so a draft PR could be another avenue to test your work.

1 Like

Thanks @Romain_Moyard and @josh for answering. However, it seems like I also have to solve another issue with the cmake version, before I can continue.

But meanwhile, I can use the CI for running tests. Thanks for the suggestion.

I tried @Romain_Moyard’s suggestions. I was able to run make all.
When I run make test-frontend, test_autograph.py fails because of ModuleNotFoundError: No module named 'tensorflow'
tensorflow needs to be added to requirements.txt. There is another open PR about this issue:

After running pip install tensorflow, I run make test-frontend, but I get this error:

[0/1] Run the frontend tests
FAIL: Frontend Tests :: test_autograph.py (16 of 16)
******************** TEST 'Frontend Tests :: test_autograph.py' FAILED ********************
Script:
--
: 'RUN: at line 17';   /home/nick-test/Documents/venv_catalyst/bin/python3 /home/nick-test/Documents/catalyst/frontend/test/lit/test_autograph.py | FileCheck /home/nick-test/Documents/catalyst/frontend/test/lit/test_autograph.py
--
Exit Code: 1

Command Output (stderr):
--
2023-08-31 18:08:46.149415: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/nick-test/Documents/catalyst/frontend/test/lit/test_autograph.py:253:11: error: CHECK: expected string not found in input
 # CHECK: (y, z) = (0, False)
          ^
<stdin>:135:24: note: scanning from here
 def if_assign_multiple(x: float):
                       ^
<stdin>:138:2: note: possible intended match here
 y, z = (0, False)
 ^

Input file: <stdin>
Check file: /home/nick-test/Documents/catalyst/frontend/test/lit/test_autograph.py

-dump-input=help explains the following input dump.

Input was:
<<<<<<
             .
             .
             .
           130:  
           131: Conditional requires consistent return types across all branches, got: 
           132:  - Branch at index 0: [ShapedArray(float64[], weak_type=True)] 
           133:  - Branch at index 1: [ShapedArray(int64[], weak_type=True)] 
           134: Please specify an else branch if none was specified. 
           135:  def if_assign_multiple(x: float): 
check:253'0                            X~~~~~~~~~~~ error: no match found
           136:  """Test a conditional that assigns to multiple existing variables.""" 
check:253'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           137:  with ag__.FunctionScope('if_assign_multiple', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope: 
check:253'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           138:  y, z = (0, False) 
check:253'0     ~~~~~~~~~~~~~~~~~~~
check:253'1      ?                  possible intended match
           139:  
check:253'0     ~
           140:  def get_state(): 
check:253'0     ~~~~~~~~~~~~~~~~~~
           141:  return (y, z) 
check:253'0     ~~~~~~~~~~~~~~~
           142:  
check:253'0     ~
           143:  def set_state(vars_): 
check:253'0     ~~~~~~~~~~~~~~~~~~~~~~~
             .
             .
             .
>>>>>>

--

********************
********************
Failed Tests (1):
  Frontend Tests :: test_autograph.py


Testing Time: 131.58s
  Passed: 15
  Failed:  1

Thanks @ndcroos for your feedback, I think there might be an issue with tensorflow version. Could you share what version was installed? (On the CI tests we install python3 -m pip install tensorflow-cpu which is currently 2.13.0).

We will update the test suite to skip the test if TF is not installed. In the meantime you should not worry about this issue as the CI tests should run without an issue.

Thanks for reaching out and let me know if it helps :slight_smile:

2 Likes

When I run pip show tensorflow, I get:

Name: tensorflow
Version: 2.13.0

Hi @ndcroos, thanks for reaching out. Our team will take a look at this and come back with an answer.

Hi @ndcroos , is the issue persisting? There is probably some version mismatch that removes the parenthesis but it should not be a blocker for your PR. Can you share your full environment?

Hi, I haven’t found the time to further on this, but I did run the tests again now, to see what the results are.
I work on a Debian 12 system. I just fetched the updates upstream and merged with my local repository. I did pip install -r requirements.txt again. pip list gives this (I installed tensorflow instead of tensorflow-cpu:

Package                      Version     Editable project location
---------------------------- ----------- ----------------------------------
absl-py                      1.4.0
appdirs                      1.4.4
astroid                      2.15.6
asttokens                    2.2.1
astunparse                   1.6.3
attrs                        23.1.0
autograd                     1.5
autoray                      0.6.6
backcall                     0.2.0
black                        23.7.0
cachetools                   5.3.1
certifi                      2023.7.22
charset-normalizer           3.2.0
clang-format                 14.0.6
clang-tidy                   14.0.6
click                        8.1.7
comm                         0.1.4
coverage                     7.3.0
debugpy                      1.6.7.post1
decorator                    5.1.1
dill                         0.3.7
execnet                      2.0.2
executing                    1.2.0
fastjsonschema               2.18.0
flatbuffers                  23.5.26
future                       0.18.3
gast                         0.4.0
google-auth                  2.22.0
google-auth-oauthlib         1.0.0
google-pasta                 0.2.0
grpcio                       1.57.0
h5py                         3.9.0
idna                         3.4
iniconfig                    2.0.0
ipykernel                    6.25.1
ipython                      8.14.0
isort                        5.12.0
jax                          0.4.13
jaxlib                       0.4.13
jedi                         0.19.0
jsonschema                   4.19.0
jsonschema-specifications    2023.7.1
jupyter_client               8.3.1
jupyter_core                 5.3.1
keras                        2.13.1
lazy-object-proxy            1.9.0
libclang                     16.0.6
lit                          16.0.6
Markdown                     3.4.4
MarkupSafe                   2.1.3
matplotlib-inline            0.1.6
mccabe                       0.7.0
ml-dtypes                    0.2.0
mypy-extensions              1.0.0
nbclient                     0.6.8
nbformat                     5.9.2
nbmake                       1.4.3
nest-asyncio                 1.5.7
networkx                     3.1
numpy                        1.23.5
oauthlib                     3.2.2
opt-einsum                   3.3.0
packaging                    23.1
parso                        0.8.3
pathspec                     0.11.2
PennyLane                    0.32.0
pennylane-catalyst           0.2.2.dev0  /home/nick-test/Documents/catalyst
PennyLane-Lightning          0.32.0
pexpect                      4.8.0
pickleshare                  0.7.5
pip                          23.0.1
platformdirs                 3.10.0
pluggy                       1.3.0
prompt-toolkit               3.0.39
protobuf                     4.24.2
psutil                       5.9.5
ptyprocess                   0.7.0
pure-eval                    0.2.2
pyasn1                       0.5.0
pyasn1-modules               0.3.0
pybind11                     2.11.1
Pygments                     2.16.1
pylint                       2.17.5
pytest                       7.4.0
pytest-cov                   4.1.0
pytest-mock                  3.11.1
pytest-xdist                 3.3.1
python-dateutil              2.8.2
PyYAML                       6.0.1
pyzmq                        25.1.1
referencing                  0.30.2
requests                     2.31.0
requests-oauthlib            1.3.1
rpds-py                      0.10.0
rsa                          4.9
rustworkx                    0.13.1
scipy                        1.11.2
semantic-version             2.10.0
setuptools                   66.1.1
six                          1.16.0
stack-data                   0.6.2
tensorboard                  2.13.0
tensorboard-data-server      0.7.1
tensorflow                   2.13.0
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.33.0
termcolor                    2.3.0
tokenize-rt                  5.2.0
toml                         0.10.2
tomlkit                      0.12.1
tornado                      6.3.3
traitlets                    5.9.0
typing_extensions            4.5.0
urllib3                      1.26.16
wcwidth                      0.2.6
Werkzeug                     2.3.7
wheel                        0.41.2
wrapt                        1.15.0

I am not sure if I need to run make runtime again. It takes quite some time on my system. I did not run it this time, but I did run make test-frontend. The text output gives errors relating to a wrong version of jaxlib.

Command Output (stderr):
--
/home/nick-test/Documents/catalyst/frontend/catalyst/__init__.py:29: UserWarning: Catalyst detected a version mismatch for the installed 'jaxlib' package. Please make sure to install the exact version required by Catalyst to avoid undefined behaviour.
Expected: 0.4.14 Found: 0.4.13

Hi @ndcroos , we just released a new version of Catalyst and it requires a new version of Jax. Unfortunately you have to git submodule update and after that you need to rebuild Catalyst with make all. The last step will take some time.

Thanks for your help. I will continue with this, but it will not be during the first few days.