Encoder and decoder mismatch is occurred due to the overflow of iAdd.

TComTrQuant.cpp l.5535 xQuant()

const TCoeff quantisedMagnitude = TCoeff((tmpLevel * iWHScale + iAdd ) >> iQBits);

When iAdd is negative value, quantisedMagnitude becomes negative value. (It shoul be positive value.) sumAbs becomes negative value and cbf decision (subAbs > 0) is wrong. Encoder has coefficients, while decoder has no coefficients due to cbf = 0.

The reason of negative value in iAdd is overflow as following mechanism.

const Int iQBits = QUANT_SHIFT + cQP.per + iTransformShift; const Int iAdd = (pcCU->getSlice()->getSliceType()==I_SLICE ? 171 : 85) << (iQBits-9);

As my case, QUANT_SHIFT = 14, cQP.per = 9, and iTransformShift = 11, then iQBits = 34, and iAdd = 85 << (34 - 9) which is larger than maximum value of signed int.

It is noted that overflow risk becomes large since iTransformShift += 7 in l.5494 is introduced by QTBT

If we can introduce the int64 variable, we can easily avoid this error. But, I'm not sure whether I introduce such variable.

INIT_LAST[NUMBER_OF_SLICE_TYPES][NUM_CTX_LAST_FLAG_SETS * NUM_CTX_LAST_FLAG_XY] = {

{ BSLICE_LUMA_LAST_POSITION_CONTEXT, 126, 111, 111, 79,

BSLICE_CHROMA_LAST_POSITION_CONTEXT, CNU, CNU, CNU, CNU,

}, { PSLICE_LUMA_LAST_POSITION_CONTEXT, 111, 111, 95, 94,

PSLICE_CHROMA_LAST_POSITION_CONTEXT, CNU, CNU, CNU, CNU,

}, { ISLICE_LUMA_LAST_POSITION_CONTEXT, 143, 127, 111, 79,

ISLICE_CHROMA_LAST_POSITION_CONTEXT, CNU, CNU, CNU, CNU,

},

};

As NUM_CTX_LAST_FLAG_XY=25, there are no enough elements for the initialization. Following code would fix the issue

INIT_LAST[NUMBER_OF_SLICE_TYPES][NUM_CTX_LAST_FLAG_SETS * NUM_CTX_LAST_FLAG_XY] = {

{ BSLICE_LUMA_LAST_POSITION_CONTEXT, 126, 111, 111, 79, CNU, CNU, CNU, CNU, CNU, CNU,

BSLICE_CHROMA_LAST_POSITION_CONTEXT, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU,

}, { PSLICE_LUMA_LAST_POSITION_CONTEXT, 111, 111, 95, 94, CNU, CNU, CNU, CNU, CNU, CNU,

PSLICE_CHROMA_LAST_POSITION_CONTEXT, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU,

}, { ISLICE_LUMA_LAST_POSITION_CONTEXT, 143, 127, 111, 79, CNU, CNU, CNU, CNU, CNU, CNU,

ISLICE_CHROMA_LAST_POSITION_CONTEXT, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU, CNU,

},

};

In the following code, slice level IC flag is not checked so that block level IC flag may be on even when slice level IC flag is off.

#if VCEG_AZ06_IC

if( getSlice()->getSPS()->getICFlag() ) {

pbICFlag[iCount] = bAtmvpAva ? !bICFlag : false;

}

#endif

It is proposed to change as follows

#if VCEG_AZ06_IC

if( getSlice()->getApplyIC() ) {

pbICFlag[iCount] = bAtmvpAva ? !bICFlag : false;

}

#endif