Opened 5 years ago
Closed 5 years ago
#562 closed defect (fixed)
RPR - Calculation of refHeight
Reported by: | bheng | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | VTM-7.0 |
Component: | VTM | Version: | VTM-6.1 |
Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
In xPredInterBlkRPR( ), the calculation of refHeight (the number of rows of intermediate horizontal filtered results required for vertical interpolation), is calculated as follows:
int refHeight = height * scalingRatio.second >> SCALE_RATIO_BITS;
However, during the actual interpolation process, the actual rows used are determined incrementally using (row *stepY), as follows:
int posY = (int32_t)y0Int + row * stepY;
yInt = ( posY + offY ) >> posShift;
Given that there is some rounding error in stepY, the accumulated error from (row * stepY) can cause the interpolation use rows beyond the the rows generated by using (refHeight + vFilterSize - 1 + extSize). This can cause the decoder to use uninitialized memory and/or crash.
I believe something like the following could be used to determine which rows the interpolation process will actually require.
int refHeight = ((((int32_t)y0Int + (height-1) * stepY) + offY ) >> posShift) - ((((int32_t)y0Int) + offY ) >> posShift) + 1;
Change history (2)
comment:1 Changed 5 years ago by bheng
comment:2 Changed 5 years ago by XiangLi
- Milestone set to VTM-7.0
- Resolution set to fixed
- Status changed from new to closed
Potential fixes for this issue are available in the following merge request:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/997