Opened 5 years ago

Closed 5 years ago

#662 closed defect (fixed)

P0154 - Overflow in SIMD PROF

Reported by: bheng Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-7.0rc1
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…


For the !bi case, when the pixel value, PROF delta, and fixed offset are all added together, prior to the shift, the end result can overflow the 16-bit data type. The specific line from applyPROF_SSE is as follows:

mm_dI = _mm256_srai_epi16(_mm256_add_epi16(mm_dI, mm_offset), shiftNum);

Change history (1)

comment:1 Changed 5 years ago by fbossen

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.