Mismatch between encoder and decoder when 4x4 coding with large QP

[kkawamura]

Encoder and decoder mismatch is occurred due to the overflow of iAdd.

TComTrQuant.cpp l.5535 xQuant()

const TCoeff quantisedMagnitude = TCoeff((tmpLevel * iWHScale + iAdd ) >> iQBits);

When iAdd is negative value, quantisedMagnitude becomes negative value. (It shoul be positive value.)
sumAbs becomes negative value and cbf decision (subAbs > 0) is wrong.
Encoder has coefficients, while decoder has no coefficients due to cbf = 0.

The reason of negative value in iAdd is overflow as following mechanism.

const Int iQBits = QUANT_SHIFT + cQP.per + iTransformShift;
const Int iAdd = (pcCU->getSlice()->getSliceType()==I_SLICE ? 171 : 85) << (iQBits-9);

As my case, QUANT_SHIFT = 14, cQP.per = 9, and iTransformShift = 11, then
iQBits = 34, and
iAdd = 85 << (34 - 9) which is larger than maximum value of signed int.

It is noted that overflow risk becomes large since iTransformShift += 7 in l.5494 is introduced by QTBT

If we can introduce the int64 variable, we can easily avoid this error. But, I'm not sure whether I introduce such variable.

[XiangLi]

Already fixed in JEM-3.1.