Inconsistency between two rate-distortion costs in the case of a I-slice in 4:0:0

[dumas]

Change in the code of VTM-8.0

---

In the file "IntraSearch.cpp", in the method IntraSearch::xGetIntraFracBitsQT, at line 3299, cu.isSepTree() should be replaced by cs.slice->isIntra().

Explanation

---

For a given luminance Coding Block (CB), for a given transform to be tested (e.g. trGrpIdx = 0, cu.mtsFlag = 1, cu.lfnstIdx = 0), in IntraSearch::estIntraPredLumaQT, once a list of intra prediction modes to be tested for full rate-distortion is created, the best intra prediction mode in terms of rate-distortion among those in this list is searched. The rate-distortion cost involved in the search is named csTemp->cost at line 1145 in the file "IntraSearch.cpp" (this rate-distortion cost is called "first" rate-distortion cost). The rate component of the first rate-distortion cost is computed via the method IntraSearch::xGetIntraFracBitsQT in the file "IntraSearch.cpp".
Now, let us consider another rate-distortion cost to complete the explanation. For a given luminance CB, in EncCu::xCheckRDCostIntra, the best set of transforms in terms of rate-distortion is searched. In the search, each test corresponds to a different set of transforms, each test already having the best intra prediction mode in terms of rate-distortion for its set of transform. The rate-distortion cost involved in the search is named tempCS->cost at line 1906 in the file "EncCu.cpp" (this rate-distortion cost is called "second" rate-distortion cost). The rate component of the second rate-distortion cost is computed from line 1878 to line 1905 in the file "EncCu.cpp". Note that, in the explanation, we omit that the rate component of the second rate-distortion cost can be modified by the subsequent calls to EncCu::xEncodeDontSplit, EncCu::xCheckDQP, and EncCu::xCheckChromaQPOffset as this does not impact the purpose of the ticket.
The problem is that, for a I-slice in 4:0:0, the rate component of the first rate-distortion cost is sometimes not equal to the rate component of the second rate-distortion cost. Indeed, at line 3299 in the file "IntraSearch.cpp", cu.isSepTree() is false, hence removing the cost of LFNST from the rate component of the first rate-distortion cost.
The problem does not appear in the case of a I-slice in 4:2:0 as cu.isSepTree() is true (using the standard configuration file), thus including the cost of LFNST in the rate component of the first rate-distortion cost.
For the same reason, the problem does not appear in 4:4:4.

Rate-distortion performance

---

On the first frame of each sequence of the CTC, in 4:0:0, in all-intra, VTM-8.0 with the above-mentioned modification gives -0.023% with respect to VTM-8.0. There is no change in running time, neither the encoder nor the decoder.