Opened 5 years ago

Closed 5 years ago

#31 closed defect (fixed)

Mismatch between encoder and decoder when 4x4 coding with large QP

Reported by: kkawamura Owned by:
Priority: minor Milestone: HM-16.6-JEM-3.1
Component: JEM Version: HM-16.6-JEM-3.0
Keywords: Cc: ksuehring, XiangLi, jvet@…

Description

Encoder and decoder mismatch is occurred due to the overflow of iAdd.

TComTrQuant.cpp l.5535 xQuant()

const TCoeff quantisedMagnitude = TCoeff((tmpLevel * iWHScale + iAdd ) >> iQBits);

When iAdd is negative value, quantisedMagnitude becomes negative value. (It shoul be positive value.)
sumAbs becomes negative value and cbf decision (subAbs > 0) is wrong.
Encoder has coefficients, while decoder has no coefficients due to cbf = 0.

The reason of negative value in iAdd is overflow as following mechanism.

const Int iQBits = QUANT_SHIFT + cQP.per + iTransformShift;
const Int iAdd = (pcCU->getSlice()->getSliceType()==I_SLICE ? 171 : 85) << (iQBits-9);

As my case, QUANT_SHIFT = 14, cQP.per = 9, and iTransformShift = 11, then
iQBits = 34, and
iAdd = 85 << (34 - 9) which is larger than maximum value of signed int.

It is noted that overflow risk becomes large since iTransformShift += 7 in l.5494 is introduced by QTBT

If we can introduce the int64 variable, we can easily avoid this error. But, I'm not sure whether I introduce such variable.

Change history (1)

comment:1 Changed 5 years ago by XiangLi

  • Milestone set to HM-16.6-JEM-3.1
  • Resolution set to fixed
  • Status changed from new to closed

Already fixed in JEM-3.1.

Note: See TracTickets for help on using tickets.