id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
1261	mismatch on inverse transform shift in 1-D transform	zhaoyin		"In Spec vE 8.7.4.1, the bdShift after inverse transform, i.e., equation (1154), is ""20 - BitDepth"", no matter whether it is 1-D or 2-D transform.


{{{
5.	The (nTbW)x(nTbH) array res[ x ][ y ] of residual samples with x = 0..nTbW − 1, y = 0..nTbH − 1 is derived as follows:
bdShift = Max( 20 − BitDepth, 0 )		(1154)
res[ x ][ y ] = ( r[ x ][ y ] + ( 1  <<  ( bdShift − 1 ) ) )  >>  bdShift	(1155)

}}}


In VTM software, function xIT(), shift = shift_2nd = 20 - BitDepth. But in the cases of 1-D transform, the shift value input to the fastInvTrans function is “shift + 1”, while in 2-D transform, the input value is shift_2nd.


{{{
  if( width > 1 && height > 1 ) //2-D transform
  {
    const int      shift_1st              =   TRANSFORM_MATRIX_SHIFT + 1 + COM16_C806_TRANS_PREC; // 1 has been added to shift_1st at the expense of shift_2nd
    const int      shift_2nd              = ( TRANSFORM_MATRIX_SHIFT + maxLog2TrDynamicRange - 1 ) - bitDepth + COM16_C806_TRANS_PREC;
    CHECK( shift_1st < 0, ""Negative shift"" );
    CHECK( shift_2nd < 0, ""Negative shift"" );
    TCoeff *tmp = ( TCoeff * ) alloca( width * height * sizeof( TCoeff ) );
    fastInvTrans[trTypeVer][transformHeightIndex](pCoeff.buf, tmp, shift_1st, width, skipWidth, skipHeight, clipMinimum, clipMaximum);
    fastInvTrans[trTypeHor][transformWidthIndex] (tmp,      block, shift_2nd, height,         0, skipWidth, clipMinimum, clipMaximum);
  }
  else if( width == 1 ) //1-D vertical transform
  {
    int shift = ( TRANSFORM_MATRIX_SHIFT + maxLog2TrDynamicRange - 1 ) - bitDepth + COM16_C806_TRANS_PREC;
    CHECK( shift < 0, ""Negative shift"" );
    CHECK( ( transformHeightIndex < 0 ), ""There is a problem with the height."" );
    fastInvTrans[trTypeVer][transformHeightIndex]( pCoeff.buf, block, shift + 1, 1, 0, skipHeight, clipMinimum, clipMaximum );
  }
  else //if(iHeight == 1) //1-D horizontal transform
  {
    const int      shift              = ( TRANSFORM_MATRIX_SHIFT + maxLog2TrDynamicRange - 1 ) - bitDepth + COM16_C806_TRANS_PREC;
    CHECK( shift < 0, ""Negative shift"" );
    CHECK( ( transformWidthIndex < 0 ), ""There is a problem with the width."" );
    fastInvTrans[trTypeHor][transformWidthIndex]( pCoeff.buf, block, shift + 1, 1, 0, skipWidth, clipMinimum, clipMaximum );
  }
}}}

Therefore, there seems a mismatch between the spec and VTM on the shift value in 1-D transform.


"	defect	closed	major		spec	VVC D10 vE	fixed		ksuehring bbross XiangLi fbossen jvet@…