16x16 Coefficient Zero Out for MTS with JVET- O0538

[bheng]

In, residual_coding( ), the following condition:

( cu_sbt_flag && log2TbWidth < 6 && log2TbHeight < 6 )

zeros-out anything outsize the 16x16 region for MTS types. Previously, SBT blocks 32x32 and smaller used implicit MTS automatically.

After JVET-O0538 (disables MTS for SBT if MTS is disabled in the SPS), MTS is no longer always used with SBT.

Therefore, the above condition no longer aligns with the intended purpose (zero-out MTS blocks).

It is suggested to add the same SPS constraint to the above condition to realign this condition with the actual use of MTS. Specifically:

( sps_mts_enabled_flag && cu_sbt_flag && log2TbWidth < 6 && log2TbHeight < 6 )

[bbross]

Thanks Brian, I agree that what you suggests makes sense in the spirit of the JVET-O0538 adoption and I would fix it in the document we are preparing to be submitted as an editorial input to the 16th JVET meeting.

In order to avoid mismatches, how is it done in VTM?
Should that be fixed too?

[bheng]

Hi Ben, VTM was updated with the following merge request.

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/853/diffs

These changes were merged to the master branch on August 16. The specific changes can be seen in the above link, but basically the additional SPS constraint (tu.cs->sps->getUseMTS()) was added to the corresponding places in the VTM code.

Let me know if you need additional info. Thanks, Brian.

[bbross]

Fixed in JVET-P2001-v1