TrQuant::transformNxN function (variant that takes transform candidate list as an input and performs forward transforms for all candidates) has an issue in the final loop that compares costs to thresholds and selects which transforms survive to full RDO.

In that final loop a threshold value intended to be used for transform skip case (thrTS) is selected if itC.second equals to 1. itC.second parameters are set with the position of the transform candidate in the input candidate list in the earlier iteration over the candidates.

Under CTC conditions transform skip is always checked as the second candidate (and itC.second equals then to 1) and thus there is no issue within CTC. However, depending on the encoder configuration, especially if transform skip is disabled, itC.second can equal to 1 for other transform modes and the threshold value apparently intended specifically for the transform skip is then used for whatever transform mode happens to be on the second position in the candidate list.

Merge request fixing the issue is provided. Fix improves all intra performance by roughly 0.2% when the fix is triggered (e.g. when LFNST and transform skip are disabled and only MTS modes are tested by the encoder). There is no change for the operation under CTC.

pu.cu->qp could be negative when bitdepth > 8. This causes an issue here: ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/commit/c4d9847ee887a5b9be1a3729a60dfc4b8f8a79bf#2181438aeeb6ddc369c442bd8d07ae622e76372f_1886_1902

In Figure 16 - Flowchart for decoding a decision of the ITU text there is a wrong sign while the ISO text is correct: ivlOffset = ivlOffset + ivlCurrRange should be ivlOffset = ivlOffset – ivlCurrRange