SW/text mismatch on sample clipping in LMCS

[peterchuang]

The SW/text has a mismatch that SW performs sample clipping after LMCS forward mapping for non-CIIP inter predictors but Spec skips this clipping after LMCS forward mapping.

It is suggested to align text with software because the design of the current LMCS was based on the results of software simulation. In addition, it is asserted that VTM8.0 has a more consistent design of sample clipping behavior between intra prediction, IBC prediction, CIIP inter prediction, and non-CIIP inter prediction.

In Spec. (JVET-Q2001-vE), in clause 8.7.5.2, the inter predictors are NOT clipped after LMCS forward mapping. The clipping operation is only performed on the reconstructed sample as shown as below.

– Otherwise (CuPredMode[ 0 ][ xCurr ][ yCurr ] is equal to MODE_INTER and ciip_flag[ xCurr ][ yCurr ] is equal to 0), the following applies:
idxY = predSamples[ i ][ j ] >> Log2( OrgCW )
predMapSamples[ i ][ j ] = LmcsPivot[ idxY ]

+ ( ScaleCoeff[ idxY ] * ( predSamples[ i ][ j ] − InputPivot[ idxY ] ) + ( 1 << 10 ) ) >> 11 (1232)

with i = 0..nCurrSw − 1, j = 0..nCurrSh − 1

The reconstructed luma picture sample recSamples is derived as follows:
recSamples[ xCurr + i ][ yCurr + j ] = Clip1( predMapSamples[ i ][ j ]+ resSamples[ i ][ j ] ] ) (1233)

with i = 0..nCurrSw − 1, j = 0..nCurrSh − 1

However, in VTM-8, the inter predictors are clipped after LMCS forward mapping. The software behavior is described as below:
– Otherwise (CuPredMode[ 0 ][ xCurr ][ yCurr ] is equal to MODE_INTER and ciip_flag[ xCurr ][ yCurr ] is equal to 0), the following applies:
idxY = predSamples[ i ][ j ] >> Log2( OrgCW )
predMapSamples[ i ][ j ] = Clip1(LmcsPivot[ idxY ]

+ ( ScaleCoeff[ idxY ] * ( predSamples[ i ][ j ] − InputPivot[ idxY ] ) +

( 1 << 10 ) ) >> 11)

with i = 0..nCurrSw − 1, j = 0..nCurrSh − 1

[peterchuang]

Sorry, this one should be categorized in "Spec" issue, not in "360 Lib".

[fbossen]

Doesn't the restriction defined in equation (100) guarantee that the result of forward LMCS is within range?

sum of lmcsCW[ i ] for i=0..15 <= ( 1 << BitDepth ) − 1

[peterchuang]

Thank you, Ben. It seems that the clipping after forward mapping is redundant due to the equation (100). Then, it would suggest to have a editorial change that removes the clipping in equation (1028) as follow.

When cIdx is equal to 0 and slice_lmcs_enabled_flag is equal to 1, predSamplesInter[ x ][ y ] with x = 0..cbWidth − 1 and y = 0..cbHeight − 1 are modified as follows:
idxY = predSamplesInter[ x ][ y ] >> Log2( OrgCW )
predSamplesInter [ x ][ y ] = Clip1( LmcsPivot[ idxY ] + (1028)

( ScaleCoeff[ idxY ] * ( predSamplesInter[ x ][ y ] − InputPivot[ idxY ] ) +
( 1 << 10 ) ) >> 11 )

Also, the clipping in SW (in line 273 in Reshape.cpp) can also be removed to align with spec.

[bbross]

Thanks for reporting!

This seems to be purely editorial and will be discussed at the next meeting according to JVET-S0232.