Issue on ApplyLfnstFlag

[moonmo.koo]

In current VVC WD (JVET-R2001-v9), the variable ApplyLfnstFlag is defined in lfnst_idx semantics as the following:

'''lfnst_idx''' specifies whether and which one of the two low frequency non-separable transform kernels in a selected transform set is used. lfnst_idx equal to 0 specifies that the low frequency non-separable transform is not used in the current coding unit.

When lfnst_idx is not present, it is inferred to be equal to 0.

The variable ApplyLfnstFlag is derived as follows:
-  If treeType is equal to SINGLE_TREE, the following applies:
     ApplyLfnstFlag = ( lfnst_idx > 0 && cIdx  = =  0 ) ? 1 : 0
-  Otherwise, the following applies:
     ApplyLfnstFlag = ( lfnst_idx > 0 ) ? 1 : 0

However, the variable cIdx indicating a color component is not defined or passed as a parameter in coding_unit syntax table where lfnst_idx is signalled. Moreover, in order to enable LFNST only for Luma in single-tree case, ApplyLfnstFlag must be adjusted as cIdx varies from 0 to 2 (e.g. ApplyLfnstFlag must be zero for Cb/Cr.). But, any implication of this adjustment cannot be found in the above definition of ApplyLfnstFlag.

For resolving this ambiguity, the following fix can be suggested, which employs a 1-D array of ApplyLfnstFlag indexed by locally defined cIdx:

'''lfnst_idx''' specifies whether and which one of the two low frequency non-separable transform kernels in a selected transform set is used. lfnst_idx equal to 0 specifies that the low frequency non-separable transform is not used in the current coding unit.

When lfnst_idx is not present, it is inferred to be equal to 0.

The variable ApplyLfnstFlag[ cIdx ] is derived as follows for cIdx = 0..2:
    ApplyLfnstFlag[ 0 ] = ( lfnst_idx > 0  &&  treeType  !=  DUAL_TREE_CHROMA  ) ? 1 : 0
    ApplyLfnstFlag[ 1 ] = ApplyLfnstFlag[ 2 ] = ( lfnst_idx > 0  &&  treeType  = =  DUAL_TREE_CHROMA ) ? 1 : 0

With the above definition of the ApplyLfnstFlag array, the followig two spots referring ApplyLfnstFlag should be modified with ApplyLfnstFlag[cIdx] accordingly:

...
'''8.7.3   Scaling process for transform coefficients'''
...
For the derivation of the scaled transform coefficients d[ x ][ y ] with x = 0..nTbW − 1, y = 0..nTbH − 1, the following applies:
   - The intermediate scaling factor m[ x ][ y ] is derived as follows:
     - If one or more of the following conditions are true, m[ x ][ y ] is set equal to 16:
     - sh_explicit_scaling_list_used_flag is equal to 0.
     - transform_skip_flag[ xTbY ][ yTbY ][ cIdx ] is equal to 1.
     - sps_scaling_matrix_for_lfnst_disabled_flag is equal to 1 and ApplyLfnstFlag[ cIdx ] is equal to 1.
     - sps_scaling_matrix_for_alternative_colour_space_disabled_flag is equal to 1 and sps_scaling_matrix_designated_colour_space_flag is equal to cu_act_enabled_flag[ xTbY ][ yTbY ].
...
'''8.7.4 Transformation process for scaled transform coefficients
8.7.4.1 General
'''
...
When ApplyLfnstFlag[ cIdx ] is equal to 1, transform_skip_flag[ xTbY ][ yTbY ][ cIdx ] is equal to 0 and both nTbW and nTbH are greater than or equal to 4, the following applies:
...
The variables nonZeroW and nonZeroH are derived as follows:
  - If ApplyLfnstFlag[ cIdx ] is equal to 1 and nTbW is greater than or equal to 4 and nTbH is greater than or equal to 4, the following applies:
     nonZeroW = ( nTbW  = =  4  | |  nTbH  = =  4 ) ? 4 : 8
     nonZeroH = ( nTbW  = =  4  | |  nTbH  = =  4 ) ? 4 : 8
...

[T.Hashimoto]

I agree the problem.

But I think the following changes are more straight forward:

The variable ApplyLfnstFlag[ cIdx ] is derived as follows for cIdx = 0..2:
 -  If treeType is equal to SINGLE_TREE, the following applies:
      ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 && cIdx  = =  0 ) ? 1 : 0　
 -  Otherwise, the following applies:
      ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 ) ? 1 : 0

[moonmo.koo]

Replying to T.Hashimoto:

I agree the problem.

But I think the following changes are more straight forward:

The variable ApplyLfnstFlag[ cIdx ] is derived as follows for cIdx = 0..2:
 -  If treeType is equal to SINGLE_TREE, the following applies:
      ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 && cIdx  = =  0 ) ? 1 : 0　
 -  Otherwise, the following applies:
      ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 ) ? 1 : 0

@T.Hashimoto: Thank you for confirming the problem and suggesting another spec text.

According to my analysis, both fixes (mine and yours) work properly. But, there is one difference in default value of ApplyLfnstFlag[ cIdx ] between the two.

In more detail, for irrelevent components (Chroma for Luma separate tree, and Luma for Chroma separate tree), the default value in my fix is zero. In your fix, all components have the same value in separate tree case, i.e. lfnst_idx > 0, which means that ApplyLfnstFlag[ cIdx ] for the irrelevent components can be set as 1.

Of course, ApplyLfnstFlag[ cIdx ] for the irrelevant components does not affect processing at all. But, in my opinion, the default value set as zero seems to be clearer from the spec point of view.

[T.Hashimoto]

In my opinion, my suggestion would be more stable in that it has few changes.
This is just an editorial difference (editors could decide).

[Tomohiro Ikai]

The original intention is clear in JVET-Q0686 .

• - Chroma LFNST is disabled when single-tree partition mode is used to reduce worst-case latency.

And the text was agreed in the meeting.

Thus only issue is when the semantics inference rule is thought to be applied?. When decoding process, no change should be ok. but if it is applied in syntax parsing. Hashimoto-san's suggestion would be sufficient (minimum change).

`
The variable ApplyLfnstFlag[ cIdx ] is derived as follows for cIdx = 0..2:

• If treeType is equal to SINGLE_TREE, the following applies:

ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 && cIdx = = 0 ) ? 1 : 0　

• Otherwise, the following applies:

ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 ) ? 1 : 0

and
replace ApplyLfnstFlag with ApplyLfnstFlag[ cIdx ]
`

[moonmo.koo]

In my opinion, both fixes are okay for me, because they works well and are equivalent in behavior. Honestly, I prefer my fix because it has shorter length and assigns more rigorous default values for irrelevent components. But, I think that it would be better to let editors choose one according to their disrection.

[bbross]

Thanks for confirming, suggest the following equivalent also making sure that luma is not set in DUAL_TREE_CHROMA case:

The variable ApplyLfnstFlag[ cIdx ] is derived as follows:
– When treeType is equal to SINGLE_TREE or DUAL_TREE_LUMA, the following applies:
ApplyLfnstFlag[ 0 ] = ( lfnst_idx > 0 ) ? 1 : 0 (178)
– The following applies for cIdx = 1, 2:
ApplyLfnstFlag[ cIdx ] = ( lfnst_idx > 0 && treeType = = DUAL_TREE_CHROMA ) ? 1 : 0 (179)

[bbross]

This is fixed in JVET-S2001-v9.