Align VTM clipping after inverse transform with specification

[adybrowne]

In section 8.7.4.1 (JVET-S2001vH), the equation (1163) specifies the clipping process for the inverse transform.

2. When nTbH and nTbW are both greater than 1, the intermediate sample values g[ x ][ y ] with x = 0..nonZeroW − 1, y = 0..nTbH − 1 are derived as follows:

g[ x ][ y ] = Clip3( CoeffMin, CoeffMax, ( e[ x ][ y ] + 64 ) >> 7 ) (1163)

Thus clipping is only specified at the intermediate stage for 2D transforms. However VTM (TrQuant::xIT) clips both the intermediate and final values to be between CoeffMin and CoeffMax for both 1D and 2D transforms.

This clipping leads to significant losses for 16 bit content when the ExtendedPrecision flag is not enabled.

It is suggested that VTM is brought into line with the spec.

This is most easily achieved by clipping the final value only to the numeric limits of the Pel data type (16 bits normally, 32 bits if the high bit depth macro is enabled). This is the solution adopted in HM.

[fbossen]

To clarify, the reported issue arises for the case where:

• ExtendedPrecision is set to 0
• InternalBitDepth is set to 16
• RExt__HIGH_BIT_DEPTH_SUPPORT is set to 1

Correct?

[adybrowne]

That is correct - the loss has only been seen under those conditions.

CrowdRun and ParkJoy have significant losses - many dB at some QPs, but all tested 16 bit sequences are affected.

I'm unsure if it might be possible to hit the clipping limits with other pathological data at lower bit depths.

[fbossen]

Setting ExtendedPrecision to 0 when InternalBitDepth is set to 16 seems silly. Is there a reason to support such a combination?

[adybrowne]

I can see your point, but the reason for raising this ticket was to remove a discrepancy between VTM and the specification, and I think this reason is still sound.

Perhaps the usage of 16 bit content without extended precision should be discussed at the next meeting, because, as you say, it does seem silly.

[fbossen]

The specification (JVET-S2001vH) says the following:
sps_bitdepth_minus8 shall be in the range of 0 to 2, inclusive.

The case that you describe (sps_bitdepth_minus8 = 8) is thus not relevant in the context of JVET-S2001vH. There is no discrepancy between VTM and the VVC v1 specification here.

I can nevertheless see the value in changing the software such as to enable experimentation outside the standardized parameter range and avoid possible future discrepancies between text and software.

As far as I can tell the suggested modification will not have any effect when RExt__HIGH_BIT_DEPTH_SUPPORT is set to 0, which is the default.
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1959