Coefficients outside the first CG are considered in 8x8 LFNST

[jason]

In 8.7.4.1 of the current spec text, input array for inverse LFNST is derived as follows:

8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...

log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2

...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:

– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:

xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ log2LfnstSize ][ log2LfnstSize ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ log2LfnstSize ][ log2LfnstSize ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]

Since the index for DiagScanOrder is based on LFNST size, not CG size, coefficients outside the first CG are assigned to u[ x ].
In 8x16 TB, for example, u[ 10 ] = d[ 0 ][ 4 ], u[ 14 ] = d [ 4 ][ 0 ], u[ 15 ] = d[ 0 ][ 5 ]

Suggested fix:
8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...

log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2
log2SbSize = 2

...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:

– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:

xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]

[moonmo.koo]

Thank you for your reporting the bug in the current LFNST spec text. Through my investigation, I agree that your modification should be correct.

I am going to report this to WD coordinators in order for this modification to be properly reflected in the next version of spec text.

[bbross]

Fixed in D5 v10