#341 closed defect (fixed)
Coefficients outside the first CG are considered in 8x8 LFNST
Reported by: | jason | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | spec | Version: | VVC D5 v10 |
Keywords: | LFNST | Cc: | ksuehring, bbross, XiangLi, fbossen, jvet@… |
Description
In 8.7.4.1 of the current spec text, input array for inverse LFNST is derived as follows:
8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...
log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2
...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:
– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:
xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ log2LfnstSize ][ log2LfnstSize ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ log2LfnstSize ][ log2LfnstSize ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]
Since the index for DiagScanOrder is based on LFNST size, not CG size, coefficients outside the first CG are assigned to u[ x ].
In 8x16 TB, for example, u[ 10 ] = d[ 0 ][ 4 ], u[ 14 ] = d [ 4 ][ 0 ], u[ 15 ] = d[ 0 ][ 5 ]
Suggested fix:
8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...
log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2
log2SbSize = 2
...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:
– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:
xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]
Change history (3)
comment:1 Changed 5 years ago by moonmo.koo
comment:2 Changed 5 years ago by bbross
- Resolution set to fixed
- Status changed from new to closed
Fixed in D5 v10
comment:3 Changed 5 years ago by bbross
- Version changed from VVC D5 v9 to VVC D5 v10
Thank you for your reporting the bug in the current LFNST spec text. Through my investigation, I agree that your modification should be correct.
I am going to report this to WD coordinators in order for this modification to be properly reflected in the next version of spec text.