Rate control in current VTM does not work

[lzz8246]

Rate control in current VTM does not work due to new coding configurations and new coding tools. These new coding tools may not take care of rate control since it is non-CTC test. Issues are pointed out below. We also provide MR to fix it.

1. m_sliceMap is set after frame level rate control

In thr current VTM, the infomation of each CTU in pcSlice will be acquired from the variable, m_sliceMap. This variable would be set in the following code:

for(uint32_t sliceIdx = 0; sliceIdx < pcPic->cs->pps->getNumSlicesInPic(); sliceIdx++ )
{
  pcSlice->setSliceMap( pcPic->cs->pps->getSliceMap( sliceIdx ) );

while when rate control is enabled, the parameters in rate control will be calculated before encoding this picture, in the function "xPicInitRateControl()". Current not working of rate control is caused by the function "m_pcSliceEncoder->calCostSliceI(pic);" called in "xPicInitRateControl()", which has used the information in m_sliceMap. Since the variable m_sliceMap has not been assigned, the function m_pcSliceEncoder->calCostSliceI() will obtain a wrong value of the HAD cost for the first frame.

2. In EncRCGop::xSolvEqa() which is called in EncRCGop:create(), two variables, i.e. EncRCGop::m_minEstLambda and EncRCGop::m_maxEstLambda, are initialized after executing the EncRCGop:xSolveEqa(), leading to a wrong lambda extimation before frame level bit allocation.

3. The GOP size of lowdelay configuration has been extended to 8 from 4, while the current rate control does not support this case.

4. Besides, the ticket #1050 mentioned that the summation of fracbits in useSubStructure() will lead to a wrong value of actual bits used in rate control parameters updating.

I have submitted a MR to fix the above issues 1,2,3. Another MR !1612 is to fix isssue 4 (which may not only influence rate control).
Following is the test results for BQSQuare in Class D.
The current rate control:
sequence bitrate Y-PSNR U-PSNR V-PSNR
BQSquare.Q22.jvet10.ra 192.7232 21.644 35.4786 33.9475
BQSquare.Q27.jvet10.ra 192.6112 21.3276 35.2614 33.688
BQSquare.Q32.jvet10.ra 192.5528 21.401 35.2997 33.7218
BQSquare.Q37.jvet10.ra 188.6208 21.3021 35.1642 33.6398

BQSquare.Q22.jvet10.lb 1830.4272 36.929 41.9686 43.0747
BQSquare.Q27.jvet10.lb 534.2264 29.0829 39.1437 39.51
BQSquare.Q32.jvet10.lb 213.1648 20.9342 35.7949 34.4646
BQSquare.Q37.jvet10.lb 212.8408 20.8209 35.8606 34.6728

Rate control with bug fix issue 1,2,3:
sequence bitrate Y-PSNR U-PSNR V-PSNR
BQSquare.Q22.jvet10.ra 1302.3992 38.4243 44.7958 46.3989
BQSquare.Q27.jvet10.ra 464.8968 34.6851 43.1581 44.4025
BQSquare.Q32.jvet10.ra 218.7384 31.7235 41.0754 42.2978
BQSquare.Q37.jvet10.ra 119.7456 29.0058 39.4253 40.4123

BQSquare.Q22.jvet10.lb 1830.4336 38.5727 43.2708 44.4245
BQSquare.Q27.jvet10.lb 534.1968 34.4014 40.6308 41.8491
BQSquare.Q32.jvet10.lb 205.3488 31.2706 39.0674 40.0996
BQSquare.Q37.jvet10.lb 94.2712 28.4893 37.8981 38.8527

Rate control with bug fix issue 1,2,3,4: About 6.34%/1.65% coding performance gain in classD for RA/LB would be obtained.
sequence bitrate Y-PSNR U-PSNR V-PSNR
BQSquare.Q22.jvet10.ra 1302.5584 38.6761 44.6346 46.115
BQSquare.Q27.jvet10.ra 462.8208 35.2285 42.4159 43.6698
BQSquare.Q32.jvet10.ra 217.6376 32.5367 40.3215 41.5436
BQSquare.Q37.jvet10.ra 117.288 30.1481 38.9837 40.0292

BQSquare.Q22.jvet10.lb 1830.6104 38.6445 43.5774 44.79
BQSquare.Q27.jvet10.lb 534.2432 34.4876 40.7271 42.0187
BQSquare.Q32.jvet10.lb 205.3168 31.3549 39.1067 40.2599
BQSquare.Q37.jvet10.lb 94.2248 28.5916 38.0758 38.881

[lzz8246]

The MR number is !1614.

[ksuehring]

What bitrate is the target bitrate? The very first block seems to be the only one in which the bit rate seems to be constrained to a certain limit?

[lzz8246]

The target bitrates come from the encoding results of VTM 7 using CTC configurations, which are:
1299.566 459.464 210.129 112.926 (Kbps) for RA cfg.
1828.344 532.094 203.279 92.222 (Kbps) for LB cfg, respectively.
Due to the issues mentioned above, the rate control in current VTM does not work, leading to the abnormal results shown in the first block.

[XiangLi]

Fixed in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1614.