Issue with transform skip encoder threshold under non-CTC conditions

[jani.lainema]

TrQuant::transformNxN function (variant that takes transform candidate list as an input and performs forward transforms for all candidates) has an issue in the final loop that compares costs to thresholds and selects which transforms survive to full RDO.

In that final loop a threshold value intended to be used for transform skip case (thrTS) is selected if itC.second equals to 1. itC.second parameters are set with the position of the transform candidate in the input candidate list in the earlier iteration over the candidates.

Under CTC conditions transform skip is always checked as the second candidate (and itC.second equals then to 1) and thus there is no issue within CTC. However, depending on the encoder configuration, especially if transform skip is disabled, itC.second can equal to 1 for other transform modes and the threshold value apparently intended specifically for the transform skip is then used for whatever transform mode happens to be on the second position in the candidate list.

Merge request fixing the issue is provided. Fix improves all intra performance by roughly 0.2% when the fix is triggered (e.g. when LFNST and transform skip are disabled and only MTS modes are tested by the encoder). There is no change for the operation under CTC.

[fbossen]

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2676