id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
31	Mismatch between encoder and decoder when 4x4 coding with large QP	kkawamura		"Encoder and decoder mismatch is occurred due to the overflow of iAdd.

TComTrQuant.cpp l.5535 xQuant()
       const TCoeff quantisedMagnitude = TCoeff((tmpLevel * iWHScale +  iAdd ) >> iQBits);
When iAdd is negative value, quantisedMagnitude becomes negative value. (It shoul be positive value.)
sumAbs becomes negative value and cbf decision (subAbs > 0) is wrong.
Encoder has coefficients, while decoder has no coefficients due to cbf = 0.

The reason of negative value in iAdd is overflow as following mechanism.

     const Int iQBits = QUANT_SHIFT + cQP.per + iTransformShift;
     const Int iAdd   = (pcCU->getSlice()->getSliceType()==I_SLICE ? 171 : 85) << (iQBits-9);

As my case, QUANT_SHIFT = 14, cQP.per = 9, and iTransformShift = 11, then
iQBits = 34, and
iAdd = 85 << (34 - 9) which is larger than maximum value of signed int.

It is noted that overflow risk becomes large since iTransformShift += 7 in l.5494 is introduced by QTBT

If we can introduce the int64 variable, we can easily avoid this error. But, I'm not sure whether I introduce such variable. "	defect	closed	minor	HM-16.6-JEM-3.1	JEM	HM-16.6-JEM-3.0	fixed		ksuehring XiangLi jvet@…