Opened 5 months ago

Closed 4 months ago

Last modified 4 months ago

#341 closed defect (fixed)

Coefficients outside the first CG are considered in 8x8 LFNST

Reported by: jason Owned by:
Priority: minor Milestone:
Component: spec Version: VVC D5 v10
Keywords: LFNST Cc: ksuehring, bbross, XiangLi, fbossen, jvet@…

Description

In 8.7.4.1 of the current spec text, input array for inverse LFNST is derived as follows:

8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...

log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2

...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:

– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:

xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ log2LfnstSize ][ log2LfnstSize ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ log2LfnstSize ][ log2LfnstSize ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]

Since the index for DiagScanOrder is based on LFNST size, not CG size, coefficients outside the first CG are assigned to u[ x ].
In 8x16 TB, for example, u[ 10 ] = d[ 0 ][ 4 ], u[ 14 ] = d [ 4 ][ 0 ], u[ 15 ] = d[ 0 ][ 5 ]

Suggested fix:
8.7.4.1 General
...
– The variables predModeIntra, nLfnstOutSize, log2LfnstSize, nLfnstSize, numLfnstX, numLfnstY, and nonZeroSize are derived as follows:
...

log2LfnstSize = ( nTbW >= 8 && nTbH >= 8 ) ? 3 : 2
log2SbSize = 2

...
– For xSbIdx = 0..numLfnstX − 1 and ySbIdx = 0..numLfnstY − 1, the following applies:

– The values of the list u[ x ] with x = 0..nonZeroSize − 1 are derived as follows:

xC = ( xSbIdx << log2LfnstSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ x ][ 0 ] (8 975)
yC = ( ySbIdx << log2LfnstSize ) + DiagScanOrder[ log2SbSize ][ log2SbSize ][ x ][ 1 ] (8 976)
u[ x ] = d[ xC ][ yC ]

Change history (3)

comment:1 Changed 5 months ago by moonmo.koo

Thank you for your reporting the bug in the current LFNST spec text. Through my investigation, I agree that your modification should be correct.

I am going to report this to WD coordinators in order for this modification to be properly reflected in the next version of spec text.

comment:2 Changed 4 months ago by bbross

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in D5 v10

comment:3 Changed 4 months ago by bbross

  • Version changed from VVC D5 v9 to VVC D5 v10
Note: See TracTickets for help on using tickets.