Opened 3 years ago

Closed 3 years ago

#1515 closed defect (fixed)

Overflow in ALF derivation process when bitdepth > 10

Reported by: marie-pierre.gallasso Owned by:
Priority: minor Milestone: VTM-15.0
Component: VTM Version: VTM-14.2
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In the following condition (deriveClassificationBlk function in AdaptiveLoopFilter.cpp),

      if( (uint32_t)d1 * (uint32_t)hv0 > (uint32_t)hv1 * (uint32_t)d0 )

the result of (uint32_t)d1 * (uint32_t)hv0 or (uint32_t)hv1 * (uint32_t)d0 may need more than 32 bits when bitdepth > 10.
The multiplication may overflow and the comparison of the two results may be wrong.

Change history (2)

comment:1 Changed 3 years ago by dmytro.rusanovskyy

Software fix to address bug-report #1515 was produced. Suggested changes are the following:
#if RExtHIGH_BIT_DEPTH_SUPPORT

if ((uint64_t)d1 * (uint64_t)hv0 > (uint64_t)hv1 * (uint64_t)d0)

#else

if( (uint32_t)d1 * (uint32_t)hv0 > (uint32_t)hv1 * (uint32_t)d0 )

#endif

comment:2 Changed 3 years ago by XiangLi

  • Milestone changed from VTM-14.1 to VTM-15.0
  • Resolution set to fixed
  • Status changed from new to closed
  • Version set to VTM-14.2

Fixed as suggested.

Note: See TracTickets for help on using tickets.