id summary reporter owner description type status priority milestone component version resolution keywords cc 31 Mismatch between encoder and decoder when 4x4 coding with large QP kkawamura "Encoder and decoder mismatch is occurred due to the overflow of iAdd. TComTrQuant.cpp l.5535 xQuant() const TCoeff quantisedMagnitude = TCoeff((tmpLevel * iWHScale + iAdd ) >> iQBits); When iAdd is negative value, quantisedMagnitude becomes negative value. (It shoul be positive value.) sumAbs becomes negative value and cbf decision (subAbs > 0) is wrong. Encoder has coefficients, while decoder has no coefficients due to cbf = 0. The reason of negative value in iAdd is overflow as following mechanism. const Int iQBits = QUANT_SHIFT + cQP.per + iTransformShift; const Int iAdd = (pcCU->getSlice()->getSliceType()==I_SLICE ? 171 : 85) << (iQBits-9); As my case, QUANT_SHIFT = 14, cQP.per = 9, and iTransformShift = 11, then iQBits = 34, and iAdd = 85 << (34 - 9) which is larger than maximum value of signed int. It is noted that overflow risk becomes large since iTransformShift += 7 in l.5494 is introduced by QTBT If we can introduce the int64 variable, we can easily avoid this error. But, I'm not sure whether I introduce such variable. " defect closed minor HM-16.6-JEM-3.1 JEM HM-16.6-JEM-3.0 fixed ksuehring XiangLi jvet@…