Opened 5 years ago
#666 new defect
FIx the wrong QP and lambda clipping in rate control
Reported by: | lzz8246 | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | VTM | Version: | VTM-7.0rc1 |
Keywords: | rate control | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
In current VTM, the lambda has been scaled by "pow(2,bitdepth_luma_scale)" where
int bitdepth_luma_scale = 2 * (m_encRCSeq->getbitDepth() - 8 - DISTORTION_PRECISION_ADJUSTMENT(m_encRCSeq->getbitDepth()));
The scale parameter is 16 when the internal bitdepth is set as 10.
But when rate control is applied. The lambda will be estimated by target bits and clipped by the neighbor lambdas. The clip bounds are set manually, which have not been scaled by the bitdepth_luma_scale, leading to wrong QP and lambda clipping.
As the example code in "EncRCPic::estimatePicLambda()" shown:
lastLevelLambda = Clip3( 0.1, 10000.0, lastLevelLambda ); estLambda = Clip3( lastLevelLambda * pow( 2.0, -3.0/3.0 ), lastLevelLambda * pow( 2.0, 3.0/3.0 ), estLambda );
which means that the estLambda will be clipped by 20000, while the QP will be calculated from estLambda as following:
QP=int(4.2005 * log(lambda / pow(2.0, bitdepth_luma_scale)) + 13.7122 + 0.5);
the max QP will be no more than 44 due to the wrong clip bound at the low bitrate cases.
Fix it by scaling all the clip bounds by the pow(2,bitdepth_luma_scale).