Fix the rate control with WCG_EXT

[lzz8246]

In the WCG_EXT macro enabled by the tool In-Loop Rashaping, the RD cost in the RDO process will be calculated by the unadjusted lambda and lambda-derived distortion scale factor as shown in the code:

#if WCG_EXT
double RdCost::calcRdCost( uint64_t fracBits, Distortion distortion, bool useUnadjustedLambda )
#else
double RdCost::calcRdCost( uint64_t fracBits, Distortion distortion )
#endif
{

#if WCG_EXT
  return ( useUnadjustedLambda ? m_DistScaleUnadjusted : m_DistScale ) * double( distortion ) + double( fracBits );
#else
  return m_DistScale * double( distortion ) + double( fracBits );
#endif
}

These unadjusted parameters are saved before encoding a slice and been applied for all CTUs in this slice since the CTUs share the same initial lambda and QP. But when rate control is applied, the lambda and QP of each CTU will be determined dynamically by the allocated rate to satisfy the rate constraint, varying from the parameters of the current slice. The RD cost will be improperly calculated by the unadjusted lambda and distortion scale parameters, leading to a huge quality decrease.

So it is proposed to assign the unadjusted lambda and unadjusted lambda-derived distortion scale factor from the actual lambda allocated to each CTU before encoding the current CTU rather than the lambda allocated to the slice. Then the RDO process will employ the proper lambda and distortion scale factor to calculate the RD cost.

This issue was proposed in JVET-N0741 and discussed in the Geneva meeting and was suggested to submit a bug ticket.

A fix is proposed in merge request !506

[ksuehring]

MR506 is merged