Opened 5 years ago

Closed 5 years ago

#744 closed defect (fixed)

Overflow in joint Cb/Cr transform

Reported by: bheng Owned by:
Priority: minor Milestone: VTM-7.2
Component: VTM Version: VTM-7.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In TrQuant.cpp, invTransformCbCr( ), the value cb[x] = -32768 will overflow the 16-bit data type if the

Cr = -Cb

transformation is applied.

Specifically, the line below:

else if ( signedMode == -2 ) { cr[x] = -cb[x]; }

One way to fix it might be to add an extra condition (like below):

else if ( signedMode == -2 ) { cr[x] = (cb[x] == -32768) ? 32767 : -cb[x]; }

Change history (3)

comment:1 Changed 5 years ago by xinzhao

Is it a normative change or implementation issue? The fix seems to be identical to 16-bit range clipping, right?

In my understanding, for the case of cr[x] == 32768, after adding predictor, the resulting value will be anyway clipped to the internalBitdepth range, so whether cr[x] equal to 32767 or 32768 should not change the bitstream, right?

comment:2 Changed 5 years ago by bheng

It is an implementation issue, not a normative change. The current implementation overflows and sets Cr = -32768 rather than +32768.

Correct. Setting the Cr residual to 32767 rather than 32768 would not make a difference after the residual is added to the prediction and clipped to the pixel bit depth. At least for bit depths up to 14-bit, I believe. So it is not a normative change.

comment:3 Changed 5 years ago by XiangLi

  • Milestone set to VTM-7.2
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.