Overflow in joint Cb/Cr transform

[bheng]

In TrQuant.cpp, invTransformCbCr( ), the value cb[x] = -32768 will overflow the 16-bit data type if the

Cr = -Cb

transformation is applied.

Specifically, the line below:

else if ( signedMode == -2 ) { cr[x] = -cb[x]; }

One way to fix it might be to add an extra condition (like below):

else if ( signedMode == -2 ) { cr[x] = (cb[x] == -32768) ? 32767 : -cb[x]; }

[xinzhao]

Is it a normative change or implementation issue? The fix seems to be identical to 16-bit range clipping, right?

In my understanding, for the case of cr[x] == 32768, after adding predictor, the resulting value will be anyway clipped to the internalBitdepth range, so whether cr[x] equal to 32767 or 32768 should not change the bitstream, right?

[bheng]

It is an implementation issue, not a normative change. The current implementation overflows and sets Cr = -32768 rather than +32768.

Correct. Setting the Cr residual to 32767 rather than 32768 would not make a difference after the residual is added to the prediction and clipped to the pixel bit depth. At least for bit depths up to 14-bit, I believe. So it is not a normative change.

[XiangLi]

Fixed in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1168