id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
676	Fix rate control with WCG_EXT	lzz8246		"In the WCG_EXT macro, the RD cost in the RDO process will be calculated by the unadjusted lambda and lambda-derived distortion scale factor as shown in the code:

{{{
#if WCG_EXT
double RdCost::calcRdCost( uint64_t fracBits, Distortion distortion, bool useUnadjustedLambda )
#else
double RdCost::calcRdCost( uint64_t fracBits, Distortion distortion )
#endif
{
#if JVET_AHG14_LOSSLESS
  if( m_costMode == COST_LOSSLESS_CODING && 0 != distortion )
  {
    return MAX_DOUBLE;
  }
#endif
#if WCG_EXT
  return ( useUnadjustedLambda ? m_DistScaleUnadjusted : m_DistScale ) * double( distortion ) + double( fracBits );
#else
  return m_DistScale * double( distortion ) + double( fracBits );
#endif
}
}}}

These unadjusted parameters are saved before encoding a slice. But when rate control is applied, the frame level lambda will be determined by the allocated bits after ""m_pcRdCost->saveUnadjustedLambda();"". The lambda used in the RDO process is not the calculated lambda but the unadjusted lambda saved before bit allocation, leading to a not working of frame level rate control. Fix it by adding a ""m_pcRdCost->saveUnadjustedLambda();"" in ""EncSlice::resetQP()"". This function ""EncSlice::resetQP()"" will be executed only when rate control is enabled.

Another modification is change the order of ""pRdCost->setLambda()"" and ""pRdCost->saveUnadjustedLambda();"" in the CTU level rate control.

Following is the encoding results of RaceHorses before/after fixation.
Configuration: randomaccess, RateControl=1, TargetBitrate=400000, LCULevelRateControl=0
Before fixation:
Total Frames |   Bitrate     Y-PSNR    U-PSNR    V-PSNR    YUV-PSNR   
	    300    a     279.3752   29.2691   34.6837   35.7230   30.4033
After fixation:
Total Frames |   Bitrate     Y-PSNR    U-PSNR    V-PSNR    YUV-PSNR   
	    300    a     403.0296   35.2344   38.8569   40.0848   35.8540


"	defect	closed	minor	VTM-7.1	VTM	VTM-7.0rc1	fixed	rate control	ksuehring XiangLi fbossen jvet@…