Opened 5 years ago
Closed 5 years ago
#182 closed defect (fixed)
SSE reconstruction can overflow
| Reported by: | bheng | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | VTM | Version: | VTM-3.0 |
| Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
The SSE implementation of reconstruction, reco_SSE(), can overflow 16-bit values. It is possible for prediction + residual to exceed 16-bit signed values, in which case they will wrap around and give the wrong result. For example, this is fairly easy to do with transform/quantization bypass.
I would suggest considering the "saturation" versions of the corresponding add intrinsics.
_mm256_add_epi16 --> _mm256_adds_epi16
_mm_add_epi16 --> _mm_adds_epi16
Change history (2)
comment:1 Changed 5 years ago by bheng
comment:2 Changed 5 years ago by fbossen
- Resolution set to fixed
- Status changed from new to closed
Note: See TracTickets for help on using tickets.
One possible fix has been provided in:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/289