Opened 4 years ago
Closed 4 years ago
#662 closed defect (fixed)
P0154 - Overflow in SIMD PROF
| Reported by: | bheng | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | VTM | Version: | VTM-7.0rc1 |
| Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
For the !bi case, when the pixel value, PROF delta, and fixed offset are all added together, prior to the shift, the end result can overflow the 16-bit data type. The specific line from applyPROF_SSE is as follows:
mm_dI = _mm256_srai_epi16(_mm256_add_epi16(mm_dI, mm_offset), shiftNum);
Change history (1)
comment:1 Changed 4 years ago by fbossen
- Resolution set to fixed
- Status changed from new to closed
Note: See TracTickets for help on using tickets.
Fixed in
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1116