Opened 4 years ago

Closed 4 years ago

#1453 closed defect (fixed)

Overflow in RPR scaling calculation

Reported by: audrey.turquin Owned by:
Priority: minor Milestone: VTM-12.0
Component: VTM Version: VTM-11.2
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In source/Lib/CommonLib/UnitTools.cpp, in getRprScaling function, there are the following calculations :

xScale = ( ( refPicWidth << SCALE_RATIO_BITS ) + ( curPicWidth >> 1 ) ) / curPicWidth;
yScale = ( ( refPicHeight << SCALE_RATIO_BITS ) + ( curPicHeight >> 1 ) ) / curPicHeight;

All the variables are int.
However, in case of 8K stream, with high values of scaling window, the calculation "(refPicWidth << SCALE_RATIO_BITS) + (curPicWidth >> 1)" can exceed the max int value, and thus, lead to an overflow.

Change history (2)

comment:2 Changed 4 years ago by ksuehring

  • Milestone set to VTM-12.0
  • Resolution set to fixed
  • Status changed from new to closed

Fixed as suggested

Note: See TracTickets for help on using tickets.