Opened 2 months ago

Closed 4 weeks ago

#239 closed defect (fixed)

Fix the rate control with WCG_EXT

Reported by: lzz8246 Owned by:
Priority: minor Milestone: VTM-5.1
Component: VTM Version: VTM-4.2
Keywords: rate control Cc: ksuehring, XiangLi, fbossen, jvet@…

Description (last modified by ksuehring)

In the WCG_EXT macro enabled by the tool In-Loop Rashaping, the RD cost in the RDO process will be calculated by the unadjusted lambda and lambda-derived distortion scale factor as shown in the code:

#if WCG_EXT
double RdCost::calcRdCost( uint64_t fracBits, Distortion distortion, bool useUnadjustedLambda )
#else
double RdCost::calcRdCost( uint64_t fracBits, Distortion distortion )
#endif
{

#if WCG_EXT
  return ( useUnadjustedLambda ? m_DistScaleUnadjusted : m_DistScale ) * double( distortion ) + double( fracBits );
#else
  return m_DistScale * double( distortion ) + double( fracBits );
#endif
}

These unadjusted parameters are saved before encoding a slice and been applied for all CTUs in this slice since the CTUs share the same initial lambda and QP. But when rate control is applied, the lambda and QP of each CTU will be determined dynamically by the allocated rate to satisfy the rate constraint, varying from the parameters of the current slice. The RD cost will be improperly calculated by the unadjusted lambda and distortion scale parameters, leading to a huge quality decrease.

So it is proposed to assign the unadjusted lambda and unadjusted lambda-derived distortion scale factor from the actual lambda allocated to each CTU before encoding the current CTU rather than the lambda allocated to the slice. Then the RDO process will employ the proper lambda and distortion scale factor to calculate the RD cost.

This issue was proposed in JVET-N0741 and discussed in the Geneva meeting and was suggested to submit a bug ticket.

A fix is proposed in merge request !506

Change history (2)

comment:1 Changed 2 months ago by ksuehring

  • Description modified (diff)

comment:2 Changed 4 weeks ago by ksuehring

  • Milestone set to VTM-5.1
  • Resolution set to fixed
  • Status changed from new to closed

MR506 is merged

Note: See TracTickets for help on using tickets.