Fast encoder search for MTS defect

[chollmann]

For MTS there is an additional command line parameter MTSIntraMaxCand which is used to limit the number of completely evaluated MTS transform candidates. This worked fine in VTM-4.
In VTM-5, however, this part is broken. Setting the parameter to 1, 2, 4 and 5 (3 is CTC) gives identical results for both bit rate and PSNR. Only setting it to 0 shows some loss (but significantly less than expected as this should resemble the case where MTS = 0).

There seems to be a connection to the implementation of the LFNST, as with LFNST enabled the trModes vector which is passed to the transformNxN function in line 2538 of IntraSearch.cpp never seems to contain more than two modes - (DCT2, DCT2) and TransformSkip. This vector should include the MTS modes (between DST7 and DCT8) as well.

A further test shows that when LFNST is disabled, MTSIntraMaxCand works as expected (different run times, different results).

Maybe those who implemented LFNST can help with the bug fix.

[moonmo.koo]

In VTM 5.0, the cfg parameter, MTSIntraMaxCand, is not used when LFNST is enabled,
because the previous encoding algorithm is considering only MTS and could not be reused.

And the "trModes" vector construction in line 2538 of IntraSearch.cpp is performed only for deciding whether transform skip should be tried or not, which was borrowed from the existing MTS encoding for preventing encoding time increase due to the large size transform skip.

Therefore, the phenomenon you observed is due to not the bug but the newly integrated encoding algorithm. In addition, when LFNST is disabled, the existing MTS encoding routine is performed instead, which is the reason why the value change of MTSIntraMaxCand affected performance and time in your experiments.

[chollmann]

Oh, it is a feature, not a bug? Then you might want to remove the MTSIntraMaxCand parameter from the config file as otherwise more people might get confused.

[chollmann]

Fixed by JVET-P0273 (leftover ticket)