residual_ts_coding syntax issue (mismatch with VTM)

[fbarbier]

In the Greater than X scan pass :

		for( n = 0; n  <=  numSbCoeff − 1; n++ ) {
			xC = ( xS << log2SbSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ n ][ 0 ] 
			yC = ( yS << log2SbSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ n ][ 1 ]
			for( j = 1; j  <  5; j++ ) {
				if( abs_level_gtx_flag[ n ][ j − 1 ] )
					abs_level_gtx_flag[ n ][ j ]
				MaxCcbs− −
				AbsLevelPassX[ xC ][ yC ] + = 2 * abs_level_gtx_flag[ n ][ j ]
			}
		}

I guess it should be like in VTM implementation (MaxCcbs decrementation):

		for( n = 0; n  <=  numSbCoeff − 1; n++ ) {
			xC = ( xS << log2SbSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ n ][ 0 ] 
			yC = ( yS << log2SbSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ n ][ 1 ]
			for( j = 1; j  <  5; j++ ) {
				if( abs_level_gtx_flag[ n ][ j − 1 ] ) {          
					abs_level_gtx_flag[ n ][ j ]
				        MaxCcbs− −
				        AbsLevelPassX[ xC ][ yC ] + = 2 * abs_level_gtx_flag[ n ][ j ]
                                }
			}
		}

[fbarbier]

Another issue in the same function is that abs_remainder is not multiplied by 2 whereas in VTM and also in residual_coding function, it is the case.

[fbarbier]

Another proposed changed in the same function is the following :

                        if( intra_bdpcm_flag  = =  0 )  {
-                               absRightCoeff  =  abs( TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC − 1 ][ yC ] )
+                               absRightCoeff  =  xC > 0 ? abs( TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC − 1 ][ yC ] ) : 0
-                               absBelowCoeff  =  abs( TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC − 1 ] )
+                               absBelowCoeff  =  yC > 0 ? abs( TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC − 1 ] ) : 0

[fbarbier]

A last fix in this function may be the missing assignation of AbsLevel array with the TransCoeffLevel absolute value so that the 9.3.3.2 Rice parameter derivation process for abs_remainder is based on the correct array.

        /* remainder scan pass */
                for( n = 0; n  <=  numSbCoeff − 1; n++ ) {
                        xC = ( xS << log2SbSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ n ][ 0 ] 
                        yC = ( yS << log2SbSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ n ][ 1 ]
                        if( abs_level_gtx_flag[ n ][ 4 ] )
                                abs_remainder[ n ]
                        if( intra_bdpcm_flag  = =  0 )  {
                                absRightCoeff  =  abs( TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC − 1 ][ yC ] )
                                absBelowCoeff  =  abs( TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC − 1 ] )
                                predCoeff = Max( absRightCoeff, absBelowCoeff )
                                if( AbsLevelPassX[ xC ][ yC ] + abs_remainder[ n ]  = =  1  &&  predCoeff > 0 )
                                        TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC ] = 
                                                                                        ( 1 − 2 * coeff_sign_flag[ n ] ) * predCoeff
                                else if( AbsLevelPassX[ xC ][ yC ] + abs_remainder[ n ]   <=  predCoeff )
                                        TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC ] = ( 1 − 2 * coeff_sign_flag[ n ] ) * 
                                                                                        ( AbsLevelPassX[ xC ][ yC ] + abs_remainder[ n ] − 1)
                                else
                                        TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC ] = ( 1 − 2 * coeff_sign_flag[ n ] ) * 
                                                                                        ( AbsLevelPassX[ xC ][ yC ] + abs_remainder[ n ] )
                        } else
                                TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC ]  = ( 1 − 2 * coeff_sign_flag[ n ] ) * 
                                                                                        ( AbsLevelPassX[ xC ][ yC ] + abs_remainder[ n ] )
+                       AbsLevel[xC][yC] = abs(TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC ])
                }
        }

[fbarbier]

Any comments / feedbacks on this one ?

[bbross]

Good catch! Thanks!
Fixed as follows:

1. put the decrementation within the if bracket so it is only decremented if the syntax is really parsed

2. added the multiplication by 2 for all abs_remainder cases

3. added the checks on xC and yC

4. Added the AbsLevel derivation

-> Currently I am checking whether 4. is need for the regular residual coding as well so in rice parameter derivation it always corresponds to the absolute value as in VTM.

[bbross]

The usage of AbsLevel for Rice parameter derivation in regular residual coding is correct.

So all issues are fixed in a document we are preparing to be submitted as an editorial input
to the 16th JVET meeting.

[fbarbier]

Thanks Benjamiin,
by "The usage of AbsLevel for Rice parameter derivation in regular residual coding is correct"
do you mean that in residual_ts_coding syntax, the update of AbsLevel[xC][yC] with the effective abs value of TransCoeffLevel is not needed ?

[bbross]

No that is needed in residual_ts_coding syntax as you suggested but it should not done in residual_coding syntax, which I just wanted to check. Since dependent quantization is already doing some scaling when deriving TransCoeffLevel so we should stick to AbsLevel as it is there.

[fbarbier]

Ok, thanks !