id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
1104	Inconsistency between two rate-distortion costs in the case of a I-slice in 4:0:0	dumas		"Change in the code of VTM-8.0
-----------------------------
In the file ""IntraSearch.cpp"", in the method `IntraSearch::xGetIntraFracBitsQT`, at line 3299, `cu.isSepTree()` should be replaced by `cs.slice->isIntra()`.

Explanation
-----------
For a given luminance Coding Block (CB), for a given transform to be tested (e.g. `trGrpIdx` = 0, `cu.mtsFlag` = 1, `cu.lfnstIdx` = 0), in `IntraSearch::estIntraPredLumaQT`, once a list of intra prediction modes to be tested for full rate-distortion is created, the best intra prediction mode in terms of rate-distortion among those in this list is searched. The rate-distortion cost involved in the search is named `csTemp->cost` at line 1145 in the file ""IntraSearch.cpp"" (this rate-distortion cost is called ""first"" rate-distortion cost). The rate component of the first rate-distortion cost is computed via the method `IntraSearch::xGetIntraFracBitsQT` in the file ""IntraSearch.cpp"".
Now, let us consider another rate-distortion cost to complete the explanation. For a given luminance CB, in `EncCu::xCheckRDCostIntra`, the best set of transforms in terms of rate-distortion is searched. In the search, each test corresponds to a different set of transforms, each test already having the best intra prediction mode in terms of rate-distortion for its set of transform. The rate-distortion cost involved in the search is named `tempCS->cost` at line 1906 in the file ""EncCu.cpp"" (this rate-distortion cost is called ""second"" rate-distortion cost). The rate component of the second rate-distortion cost is computed from line 1878 to line 1905 in the file ""EncCu.cpp"". Note that, in the explanation, we omit that the rate component of the second rate-distortion cost can be modified by the subsequent calls to `EncCu::xEncodeDontSplit`, `EncCu::xCheckDQP`, and `EncCu::xCheckChromaQPOffset` as this does not impact the purpose of the ticket.
The problem is that, for a I-slice in 4:0:0, the rate component of the first rate-distortion cost is sometimes not equal to the rate component of the second rate-distortion cost. Indeed, at line 3299 in the file ""IntraSearch.cpp"", `cu.isSepTree()` is false, hence removing the cost of LFNST from the rate component of the first rate-distortion cost.
The problem does not appear in the case of a I-slice in 4:2:0 as `cu.isSepTree()` is true (using the standard configuration file), thus including the cost of LFNST in the rate component of the first rate-distortion cost.
For the same reason, the problem does not appear in 4:4:4.

Rate-distortion performance
---------------------------
On the first frame of each sequence of the CTC, in 4:0:0, in all-intra, VTM-8.0 with the above-mentioned modification gives -0.023% with respect to VTM-8.0. There is no change in running time, neither the encoder nor the decoder.

"	defect	new	minor		VTM	VTM-8.0			ksuehring XiangLi fbossen jvet@…