Opened 4 years ago

#666 new defect

FIx the wrong QP and lambda clipping in rate control

Reported by: lzz8246 Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-7.0rc1
Keywords: rate control Cc: ksuehring, XiangLi, fbossen, jvet@…


In current VTM, the lambda has been scaled by "pow(2,bitdepth_luma_scale)" where

int bitdepth_luma_scale =
   2 * (m_encRCSeq->getbitDepth() - 8
     - DISTORTION_PRECISION_ADJUSTMENT(m_encRCSeq->getbitDepth()));

The scale parameter is 16 when the internal bitdepth is set as 10.
But when rate control is applied. The lambda will be estimated by target bits and clipped by the neighbor lambdas. The clip bounds are set manually, which have not been scaled by the bitdepth_luma_scale, leading to wrong QP and lambda clipping.
As the example code in "EncRCPic::estimatePicLambda()" shown:

lastLevelLambda = Clip3( 0.1, 10000.0, lastLevelLambda );
estLambda = Clip3( lastLevelLambda * pow( 2.0, -3.0/3.0 ), lastLevelLambda * pow( 2.0, 3.0/3.0 ), estLambda );

which means that the estLambda will be clipped by 20000, while the QP will be calculated from estLambda as following:

QP=int(4.2005 * log(lambda / pow(2.0, bitdepth_luma_scale)) + 13.7122 + 0.5);

the max QP will be no more than 44 due to the wrong clip bound at the low bitrate cases.

Fix it by scaling all the clip bounds by the pow(2,bitdepth_luma_scale).

Change history (0)

Note: See TracTickets for help on using tickets.