id,summary,reporter,owner,description,type,status,priority,milestone,component,version,resolution,keywords,cc
341,Coefficients outside the first CG are considered in 8x8 LFNST,jason,,"In 8.7.4.1 of the current spec text, input array for inverse LFNST is derived as follows:
8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...
log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2
...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:
– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:
xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ '''log2LfnstSize''' ][ '''log2LfnstSize''' ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ '''log2LfnstSize''' ][''' log2LfnstSize''' ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]
Since the index for DiagScanOrder is based on LFNST size, not CG size, coefficients outside the first CG are assigned to u[ x ].
In 8x16 TB, for example, u[ 10 ] = d[ 0 ][ 4 ], u[ 14 ] = d [ 4 ][ 0 ], u[ 15 ] = d[ 0 ][ 5 ]
Suggested fix:
8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...
log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2
'''log2SbSize = 2'''
...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:
– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:
xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ '''log2SbSize''' ][ '''log2SbSize''' ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ '''log2SbSize''' ][''' log2SbSize''' ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]",defect,closed,minor,,spec,VVC D5 v10,fixed,LFNST,ksuehring bbross XiangLi fbossen jvet@…