Issues with the modification of tC for bit depth less than 10

[bheng]

In Section 8.8.3.6.2 for luma, the "roundOffset" used for tC calculation is derived as:

roundOffset = 1 << ( 9 - BitDepth )

The right-shift function, "is defined only for non-negative integer values of y", so this behavior is undefined for BitDepth > 9.

This roundOffset is only required for BitDepth < 10, so I would suggest conditioning the derivation of roundOffset on "If BitDepth < 10 ..." rather than specifying an undefined operation.

Additionally, in Section 8.8.3.6.4 for chroma, the same calculation is incorrect:

"( tC′ + 2 ) >> ( 10 − BitDepth )"

here a fixed rounding value of "2" is used which doesn't match VTM behavior for 9-bit.

[siwamura]

Thank you for reporting this issue.

As a proponent of the modification of tC bit-adaptation proposed in JVET-R0130, I also agree with your suggestion.

In the software, the bit adaptation is done as follows:
Luma:

const int iTc = bitDepthLuma < 10 ? ((sm_tcTable[iIndexTC] + (1 << (9 - bitDepthLuma))) >> (10 - bitDepthLuma)) : ((sm_tcTable[iIndexTC]) << (bitDepthLuma - 10));

Chroma:

const int iTc = bitDepthChroma < 10 ? ((sm_tcTable[iIndexTC] + (1 << (9 - bitDepthChroma))) >> (10 - bitDepthChroma)) : ((sm_tcTable[iIndexTC]) << (bitDepthChroma - 10));

We have two options,
(1) Incorporate roundOffset into tC derivation instead of defining roundOffset, as done in VTM
(2) Define roundOffset with the suggested conditioning like "If BitDepth < 10 ..." for both luma and chroma.

[bbross]

Thanks for reporting that. Although this does not seems to impact the decoding process, we should avoid having variables undefined.
This will be fixed in JVET-S2001-vH as follows:

Luma and chroma (same since we only have one BitDepth for luma and chroma in the spec):

• If BitDepth is less than 10, the following applies:

tC = ( tC′ + ( 1 << ( 9 − BitDepth ) ) ) >> ( 10 − BitDepth ) (1256)

• Otherwise (BitDepth is greater than or equal to 10), the following applies:

tC = tC′ * ( 1 << ( BitDepth − 10 ) ) (1257)