Mismatch on top/left neighbouring luma samples padding in CCLM

[wangyang.cs]

There is a mismatch on top/left neighbouring luma samples padding in CCLM between the spec and software.
In JVET-Q2001-vE, luma sample (-2, -1) will be used in (373) but is not derived with padding for 4:2:0 and 4:2:2 color format. Similarly, luma sample (-1, -2) will be used in (376) and is not derived with padding for 4:2:0 color format. However, these two luma samples are padded using (-2, 0) and (0, -2) in VTM-8.0.

pSelDsY[ idx ] = ( F3[ 1 ][ 0 ] * pY[ − SubWidthC ][ SubHeightC * y − 1 ] +

F3[ 0 ][ 1 ] * pY[ −1 − SubWidthC ][ SubHeightC * y ] +
F3[ 1 ][ 1 ] * pY[ −SubWidthC ][ SubHeightC * y ] + (373)
F3[ 2 ][ 1 ] * pY[ 1 − SubWidthC ][ SubHeightC * y ] +
F3[ 1 ][ 2 ] * pY[ −SubWidthC ][ SubHeightC * y + 1 ] + 4 ) >> 3

pSelDsY[ idx ] = ( F3[ 1 ][ 0 ] * pY[ SubWidthC * x ][ − 1 − SubHeightC ] +

F3[ 0 ][ 1 ] * pY[ SubWidthC * x − 1 ][ −SubHeightC ] +
F3[ 1 ][ 1 ] * pY[ SubWidthC * x ][ −SubHeightC] + (376)
F3[ 2 ][ 1 ] * pY[ SubWidthC * x + 1 ][ −SubHeightC] +
F3[ 1 ][ 2 ] * pY[ SubWidthC * x ][ 1 − SubHeightC ] + 4 ) >> 3

Suggest changes in JVET-Q2001-vE as follows:

8.4.5.2.13 Specification of INTRA_LT_CCLM, INTRA_L_CCLM and INTRA_T_CCLM intra prediction mode

The prediction samples predSamples[ x ][ y ] with x = 0..nTbW − 1, y = 0..nTbH − 1 are derived as follows:

– If both numSampL and numSampT are equal to 0, the following applies:

predSamples[ x ][ y ] = 1 << ( BitDepth − 1 ) (359)

• – Otherwise, the following ordered steps apply:
  1. The collocated luma samples pY[ x ][ y ] with x = 0..nTbW * SubWidthC − 1, y= 0..nTbH * SubHeightC − 1 are set equal to the reconstructed luma samples prior to the deblocking filter process at the locations ( xTbY + x, yTbY + y ).
  2. The neighbouring luma samples pY[ x ][ y ] are derived as follows:

– When numSampL is greater than 0, the neighbouring left luma samples pY[ x ][ y ] with x = −1..− 3, y = 0..SubHeightC * numSampL − 1, are set equal to the reconstructed luma samples prior to the deblocking filter process at the locations ( xTbY + x , yTbY +y ).

– When availT is equal to FALSE, the neighbouring top luma samples pY[ x ][ y ] with x = −1SubWidthC..SubWidthC * numSampTnTbW − 1, y = −1..−2, are set equal to the luma samples pY[ x ][ 0 ].

– When availL is equal to FALSE, the neighbouring left luma samples pY[ x ][ y ] with x = −1..−3, y = −1..SubHeightC * numSampL − 1, are set equal to the luma samples pY[ 0 ][ y ].

– When numSampT is greater than 0, the neighbouring top luma samples pY[ x ][ y ] with x = 0..SubWidthC * numSampT − 1, y = −1, −2, are set equal to the reconstructed luma samples prior to the deblocking filter process at the locations ( xTbY+ x, yTbY + y ).

– When availL is equal to FALSE, the neighbouring left luma samples pY[ x ][ y ] with x = −1..−3, y = −SubHeightC..SubHeightC * nTbH − 1, are set equal to the luma samples pY[ 0 ][ y ].

– When availTL is equal to TRUE, the neighbouring top-left luma samples pY[ x ][ y ] with x = −1, y = −1, −2, are set equal to the reconstructed luma samples prior to the deblocking filter process at the locations ( xTbY+ x, yTbY + y ).

[bbross]

Good catch, thanks!

This is fixed in vB of D9, ie. JVET-R2001-v11.