Opened 5 years ago

Closed 5 years ago

#182 closed defect (fixed)

SSE reconstruction can overflow

Reported by: bheng Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-3.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…


The SSE implementation of reconstruction, reco_SSE(), can overflow 16-bit values. It is possible for prediction + residual to exceed 16-bit signed values, in which case they will wrap around and give the wrong result. For example, this is fairly easy to do with transform/quantization bypass.

I would suggest considering the "saturation" versions of the corresponding add intrinsics.

_mm256_add_epi16 --> _mm256_adds_epi16
_mm_add_epi16 --> _mm_adds_epi16

Change history (2)

comment:1 Changed 5 years ago by bheng

comment:2 Changed 5 years ago by fbossen

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.