Yep, definitely numerical instability then. If your trace is not in the range ~0.95-1 or so, the simulation is likely unphysical. Try repeating your experiment with reduced squeezing amplitudes until you can get the trace near 1

Squeezing and displacement are both operations that increase the energy. Higher energies require higher cutoffs.

You always need to make sure you’re in a regime where the energy isn’t too high compared to the cutoff (trace ~1). As long as that holds true, you can use any parameters you wish

If I may ask you sir,
How exactly should I calculate the proper cutoff dimension value?
And for the above values, I guess 60 is good enough but still strange results come up.
Besides that, when I operate on a single mode with two different states and then compute the scalar product the outcome does obey the classical equation
Is it because it is a mixed state?
I will be more than happy if you set me up on the road to understand quantum optics. My basic background is microwave engineering and 8.05 courses about quantum mechanics from MIT besides qml.
Thanks again

Only for the simplest states are there easy formulas to determine the overlap of the state with various Fock-basis states. For example, the overlap of a coherent state \langle \alpha | n \rangle (displacement applied to vacuum) with the Fock state |n\rangle is e^{-\frac{|\alpha|^2}{2}}\frac{\alpha}{\sqrt{n}}. With this, you can estimate what value of cutoff n would make this small.

However, for anything beyond the basic states, the formulas are either known but complex, or simply unknown. For example, check out the formula for a squeezed displaced state. Other than numerically calculating, I don’t know a way to easily invert this and estimate a safe value of cutoff n.

Honestly, the best approach is to experiment numerically with the simulator, since it can give you quick feedback. Start with small values and circuits, and only if you are confident that the cutoff is not an issue should you increase parameter values and circuit depth. A preferred approach is to regularize values so they can’t become too big.