Opened 4 years ago

Closed 4 years ago

#42 closed defect (fixed)

CTU bits/distortion calculation mistake for I slice when QTBT is enabled

Reported by: FangliangSong Owned by:
Priority: minor Milestone: HM-16.6-JEM-4.2
Component: JEM Version: HM-16.6-JEM-4.0
Keywords: CTU, bits, distortion, I slice, QTBT Cc: ksuehring, XiangLi, jvet@…

Description

Because of separated luma CTB and chroma CTB for I Slice, TEncCu::xCompressCU is invoked twice in TEncCu::compressCtu, the first time for the luma CTB encoding and the second time for the chroma CTB encoding. On the second invocation, four bits/distortion related values of CTU in pCtu (function parameter of TEncCu::compressCtu): m_dTotalCost, m_uiTotalDistortion, m_uiTotalBits, and m_uiNumPartition, are overwritten. It leads to the four values of a CTU are that of chroma CTB in the CTU, and that of luma CTB in the CTU are excluded since JEM-3.0. The mistake has two influences:

  1. R-D related values of I slice are that of chroma components, and the slice level delta QP RDO process does wrong QP selection when DeltaQpRD is on.
  2. The remaining bit number of current I frame is updated incorrectly when CTU level rate control is on. More specially, the remaining bit number of current I frame is only subtracted by chroma CTB bits.

The original code of encoding chroma CTB is as below:
#if JVET_C0024_QTBT
if (pCtu->getSlice()->isIntra())
{


xCompressCU(/*long arguments*/);

}
#endif
A simple solution to this bug is as below:
#if JVET_C0024_QTBT
if (pCtu->getSlice()->isIntra())
{


Double dLumaTotalCost = pCtu->getTotalCost();
Distortion dLumaTotalDistortion = pCtu->getTotalDistortion();
UInt uiLumaTotalBits = pCtu->getTotalBits();
UInt uiLumaTotalBins = pCtu->getTotalBins();
xCompressCU(/*long arguments*/);
pCtu->getTotalCost() += dLumaTotalCost;
pCtu->getTotalDistortion() += dLumaTotalDistortion;
pCtu->getTotalBits() += uiLumaTotalBits;
pCtu->getTotalBins() += uiLumaTotalBins;

}
#endif
The bug still exit in latest software version (JEM-4.1), and has no impact under CTC.

Attachments (2)

CTU_bit_distortion_calculation_fix.patch (1.7 KB) - added by FangliangSong 4 years ago.
CTU_bit_distortion_calculation_fix_v2.patch (1.7 KB) - added by FangliangSong 4 years ago.
replace variable name "dLumaCTBTotalDistortion" with "uiLumaCTBTotalDistortion"

Download all attachments as: .zip

Change history (4)

comment:1 Changed 4 years ago by FangliangSong

Four related variables is TComDataCU::m_dTotalCost, TComDataCU::m_uiTotalDistortion, TComDataCU::m_uiTotalBits, and TComDataCU::m_uiTotalBins. "m_uiNumPartition" in above description should be m_uiTotalBins. I'm sorry for miswriting it.

Changed 4 years ago by FangliangSong

Changed 4 years ago by FangliangSong

replace variable name "dLumaCTBTotalDistortion" with "uiLumaCTBTotalDistortion"

comment:2 Changed 4 years ago by XiangLi

  • Milestone set to HM-16.6-JEM-4.2
  • Resolution set to fixed
  • Status changed from new to closed

Fixed in r404.

Note: See TracTickets for help on using tickets.