Interaction between M0140 (SBT) and M0297 (MTS Zero Out)

[bheng]

M0297 (MTS Zero Out) intends to prevent coefficients outside the low-frequency 16x16 region for MTS transform types (!= DCT2) by modifying the coefficient scanning syntax to make it impossible to send coefficients outside this area.

Part of M0140 assigns implicitly signaled MTS transform types (DCT8/DST7) to sub-block transform blocks.

Should the M0297 syntax modification also be applied to M0140 SBT blocks?

Currently the syntax modification is not applied to SBT blocks, so they can generate coefficients outside the 16x16 region for MTS transform types.

M0297 adds some logic later in an attempt to zero these coefficients out (see below), but the skipHeight doesn't have any effect (it’s not actually used in the inverse transform, for example fastInverseDST7_B32). So, the following code only zeros out the columns, not the rows.

const int skipWidth = ( trTypeHor != DCT2 && width == 32 ) ? 16 : ...;
const int skipHeight = ( trTypeVer != DCT2 && height == 32 ) ? 16 : ...;

[moonmo.koo]

According to my analysis of VTM 4.0rc1 code, M0140 (SBT) and M0297 are implemented in a harmonized way, which means that only top-left 16x16 coefficients in each partition for SBT are retained in the case of MTS. In other words, every time 32-point MTS is performed, the zero-out of M0297 is now applied. Could you explain what syntax modifications in M0140 and M0297 indicate?

Replying to bheng:

M0297 (MTS Zero Out) intends to prevent coefficients outside the low-frequency 16x16 region for MTS transform types (!= DCT2) by modifying the coefficient scanning syntax to make it impossible to send coefficients outside this area.

Part of M0140 assigns implicitly signaled MTS transform types (DCT8/DST7) to sub-block transform blocks.

Should the M0297 syntax modification also be applied to M0140 SBT blocks?

Currently the syntax modification is not applied to SBT blocks, so they can generate coefficients outside the 16x16 region for MTS transform types.

M0297 adds some logic later in an attempt to zero these coefficients out (see below), but the skipHeight doesn't have any effect (it’s not actually used in the inverse transform, for example fastInverseDST7_B32). So, the following code only zeros out the columns, not the rows.

const int skipWidth = ( trTypeHor != DCT2 && width == 32 ) ? 16 : ...;
const int skipHeight = ( trTypeVer != DCT2 && height == 32 ) ? 16 : ...;

[bheng]

M0297 syntax changes:

1.) Change maximum last scan position to make it impossible to set last scan position greater than 15 in mtsIdx = {2,3,4,5} blocks. Not applied to SBT here because SBT doesn't have mtsIdx.

if( tu.mtsIdx > 1 && !tu.cu->transQuantBypass && compID == COMPONENT_Y )
{

maxLastPosX = ( tu.blocks[ compID ].width == 32 ) ? g_uiGroupIdx[ 15 ] : maxLastPosX;
maxLastPosY = ( tu.blocks[ compID ].height == 32 ) ? g_uiGroupIdx[ 15 ] : maxLastPosY;

}

2.) Infer significant group flag to be zero when outside the 16x16 region in mtsIdx = {2,3,4,5} blocks. Again, not applied to SBT here because SBT doesn't have mtsIdx.

if( tu.mtsIdx > 1 && !tu.cu->transQuantBypass && compID == COMPONENT_Y )
{

sigGroup = ( ( tu.blocks[compID].height == 32 && cctx.cgPosY() >= ( 16 >> cctx.log2CGHeight() ) )

( tu.blocks[compID].width == 32 && cctx.cgPosX() >= ( 16 >> cctx.log2CGWidth() ) ) ) ? 0 : m_BinDecoder.decodeBin( cctx.sigGroupCtxId() );

}

M0297 zero out logic:

The following logic is also added to zero out anything outside 16x16 non-DCT2 blocks. This should be impossible in mtsIdx = {2,3,4,5} blocks because of the syntax changes above. But it is still possible in SBT.

But this zero-out code doesn't work. The skipHeight is never used and has no effect. See fastInverseDST7_B32 function for example.

const int skipWidth = ( trTypeHor != DCT2 && width == 32 ) ? 16 : width > JVET_C0024_ZERO_OUT_TH ? width - JVET_C0024_ZERO_OUT_TH : 0;
const int skipHeight = ( trTypeVer != DCT2 && height == 32 ) ? 16 : height > JVET_C0024_ZERO_OUT_TH ? height - JVET_C0024_ZERO_OUT_TH : 0;

[xinzhao]

In my understanding, the zero out parameters used in inverse transform, i.e., skipHeight, skipWidth, are only implementation related. As long as the forward transform guarantees the high-frequency transform coefficients are set as zero, whether inverse transform consider skipHeight (skipWidth) or not should have no impact on the bitstream.

[fbossen]

I think there is indeed an issue here with the entropy coding part. The coding of a block for a given size (such as 32x16) and a given transform combination (such as DST7/DCT8) should not depend on whether SBT is used or not. The conditions in residual_coding_subblock should probably be updated to take SBT into account.

[fbossen]

Ideally residual_coding_subblock should not be called for subblocks in the area that is zeroed out

[fbossen]

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/288