Issues in LFNST index parsing

[bheng]

1.) LFNST with MIP condition

This treeType condition related to MIP seems incorrect:

( treeType != DUAL_TREE_CHROMA | | !intra_mip_flag[ x0 ][ y0 ] | |

Min( lfnstWidth, lfnstHeight ) >= 16 )

treeType != DUAL_TREE_CHROMA is TRUE for any tree that includes luma. Therefore anything with luma will ignore the following OR MIP conditions in that statement. And MIP only ever occurs with luma.

I assume it was meant to be treeType == DUAL_TREE_CHROMA.

The related condition in the VTM code appears to be correct already.

2.) LFNST with transform skip

I don't believe it was intended to allow combinations of LFNST with transform skip. The existing conditions include this check for luma transform skip ([ 0 ] component) to prevent this combination.

transform_skip_flag[ x0 ][ y0 ][ 0 ] = = 0

However, JVET-P0058 added transform skip for chroma. So, the condition above should probably be extended to include Cb and Cr checks as well ([ 1 ] and [ 2 ] components).

For this issue, VTM code appears to match the text, so if the text is updated, then the software should be as well.

[moonmo.koo]

Thank you for finding the current LFNST index parsing issues.

As you mentioned, due to the adoption of Chroma transform skip, the current checking of only Luma transform skip flag seems odd, even though it works well in VTM 7.0. Regarding this issue, currently we are testing proper handling schemes of it and will provide corresponding proposal in the upcoming meeting.

From my analysis, one spec text change is needed in order to align text with SW, irrespective of whether to check Luma transform skip flag or not. Currently, LFNST is applied in a componet-wise manner in VTM SW as follows:

void TrQuant::xInvLfnst( const TransformUnit &tu, const ComponentID compID )
{

......

#if JVET_P0058_CHROMA_TS

if (lfnstIdx && tu.mtsIdx[compID] != MTS_SKIP && width >= 4 && height >= 4)

#else

if( lfnstIdx && tu.mtsIdx != MTS_SKIP && width >= 4 && height >= 4 )

#endif

{

......

}

}

However, the section 8.7.4.1 does not seem to consider the above code correctly. Thus, the following change could be suggested:

When lfnst_idx is not equal to 0 and both nTbW and nTbH are greater than or equal to 4, the following applies:
=> When lfnst_idx is not equal to 0 and transform_skip_flag[ xTbY ][ yTbY ][ cIdx ] is equal to 0 and both nTbW and nTbH are greater than or equal to 4, the following applies:

[merkle]

The first issue "LFNST with MIP condition" is related to ticket #647, which suggested the following condition and it seems that something went wrong when integrating it to the spec.

( treeType == DUAL_TREE_CHROMA | | ( !intra_mip_flag[ x0 ][ y0 ] | | Min( lfnstWidth, lfnstHeight ) >= 16 ) )

[bbross]

Thanks for spotting this Brian and reviewing the fixes Moonmo and Philipp.
I fixed 1) as suggested and 2) as suggested by Monnmo in the editorial input to the next meeting.

This should match with VTM and regarding a potential gating of LFNST index parsing on chroma TS flags, my understanding is that there will be a contribution to the next meeting.