Opened 5 years ago

Closed 5 years ago

#363 closed defect (fixed)

Multiplication Overflow in ALF Classification

Reported by: bheng Owned by:
Priority: minor Milestone: VTM-6.0
Component: VTM Version: VTM-5.2
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

The following multiplication in ALF deriveClassificationBlk( ) can overflow the signed 32-bit data type being used:

if( d1*hv0 > hv1*d0 )

These d and hv values are always non-negative, so suggest mapping these to unsigned uint32_t to prevent overflow.

Similarly, the simdDeriveClassificationBlk( ) version of the function has the same problem. Suggest mapping the following SSE signed-comparison to equivalent unsigned-comparison operation(s).

_mm_cmpgt_epi32( xmm6, xmm9 );

Change history (2)

comment:1 Changed 5 years ago by bheng

  • Version changed from VTM-5.0rc1 to VTM-5.2

Possible solution provided here:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/679

comment:2 Changed 5 years ago by bheng

  • Milestone set to VTM-6.0
  • Resolution set to fixed
  • Status changed from new to closed

MR 679 was merged containing the suggested fixes.

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/679

Note: See TracTickets for help on using tickets.