Opened 5 years ago
Closed 5 years ago
#31 closed defect (fixed)
Mismatch between encoder and decoder when 4x4 coding with large QP
Reported by: | kkawamura | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | HM-16.6-JEM-3.1 |
Component: | JEM | Version: | HM-16.6-JEM-3.0 |
Keywords: | Cc: | ksuehring, XiangLi, jvet@… |
Description
Encoder and decoder mismatch is occurred due to the overflow of iAdd.
TComTrQuant.cpp l.5535 xQuant()
const TCoeff quantisedMagnitude = TCoeff((tmpLevel * iWHScale + iAdd ) >> iQBits);
When iAdd is negative value, quantisedMagnitude becomes negative value. (It shoul be positive value.)
sumAbs becomes negative value and cbf decision (subAbs > 0) is wrong.
Encoder has coefficients, while decoder has no coefficients due to cbf = 0.
The reason of negative value in iAdd is overflow as following mechanism.
const Int iQBits = QUANT_SHIFT + cQP.per + iTransformShift;
const Int iAdd = (pcCU->getSlice()->getSliceType()==I_SLICE ? 171 : 85) << (iQBits-9);
As my case, QUANT_SHIFT = 14, cQP.per = 9, and iTransformShift = 11, then
iQBits = 34, and
iAdd = 85 << (34 - 9) which is larger than maximum value of signed int.
It is noted that overflow risk becomes large since iTransformShift += 7 in l.5494 is introduced by QTBT
If we can introduce the int64 variable, we can easily avoid this error. But, I'm not sure whether I introduce such variable.
Change history (1)
comment:1 Changed 5 years ago by XiangLi
- Milestone set to HM-16.6-JEM-3.1
- Resolution set to fixed
- Status changed from new to closed
Already fixed in JEM-3.1.