Opened 5 years ago

Closed 5 years ago

#199 closed defect (fixed)

bug in shaper for high bitdepth

Reported by: LGE_VCC Owned by:
Priority: minor Milestone: VTM-4.0.1
Component: VTM Version: VTM-4.0
Keywords: reshaper Cc: ksuehring, XiangLi, fbossen, jvet@…


If running VTM4.0 with bitdepth 12 ( warning regarding m_bitDepthConstraint in EncAppCfg is disabled). The resulted luma PSNR is very low, less than 10dB.

Example command is
-c ../cfg/encoder_randomaccess_vtm.cfg -c BasketballPass.cfg -f 5 -q 32 -ip 48 -v 6 --InternalBitDepth=12 --MaxBitDepthConstraint=16

The problem is because of the following assignments to get log2 of the reshaper segment.
int log2PwlFwdBinLen = g_aucLog2[pwlFwdBinLen];
int log2HistLenth = g_aucLog2[histLenth];

in VTM4, size of g_aucLog2 is only MAX_CU_SIZE + 1, but pwlFwdBinLen and histLenth are 256 when bitdepth is 12. So it resulted an out of bound array access.

Suggested fixes are
int log2PwlFwdBinLen = (pwlFwdBinLen>=MAX_CU_SIZE)? g_aucLog2[pwlFwdBinLen>> MAX_CU_DEPTH] + MAX_CU_DEPTH : g_aucLog2[pwlFwdBinLen];

int log2HistLenth = (histLenth >= MAX_CU_SIZE) ? g_aucLog2[histLenth >> MAX_CU_DEPTH] + MAX_CU_DEPTH : g_aucLog2[histLenth];

There are total 4 places need the fix. Each assignment occurred twice.


Attachments (1)

0001-Reshaper-High-bitdepth-bug-fix.patch (4.9 KB) - added by LGE_VCC 5 years ago.
patch for bug fix (LGE_BUG_FIX)

Download all attachments as: .zip

Change history (4)

Changed 5 years ago by LGE_VCC

patch for bug fix (LGE_BUG_FIX)

comment:1 Changed 5 years ago by taoranlu

thank you for reporting, we'll provide a more general fix that remove the usage of g_auLog2[] to make it more general.

comment:3 Changed 5 years ago by XiangLi

  • Milestone set to VTM-4.0.1
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.