bug in shaper for high bitdepth

[LGE_VCC]

If running VTM4.0 with bitdepth 12 ( warning regarding m_bitDepthConstraint in EncAppCfg is disabled). The resulted luma PSNR is very low, less than 10dB.

Example command is
-c ../cfg/encoder_randomaccess_vtm.cfg -c BasketballPass.cfg -f 5 -q 32 -ip 48 -v 6 --InternalBitDepth=12 --MaxBitDepthConstraint=16

The problem is because of the following assignments to get log2 of the reshaper segment.
int log2PwlFwdBinLen = g_aucLog2[pwlFwdBinLen];
int log2HistLenth = g_aucLog2[histLenth];

in VTM4, size of g_aucLog2 is only MAX_CU_SIZE + 1, but pwlFwdBinLen and histLenth are 256 when bitdepth is 12. So it resulted an out of bound array access.

Suggested fixes are
int log2PwlFwdBinLen = (pwlFwdBinLen>=MAX_CU_SIZE)? g_aucLog2[pwlFwdBinLen>> MAX_CU_DEPTH] + MAX_CU_DEPTH : g_aucLog2[pwlFwdBinLen];

int log2HistLenth = (histLenth >= MAX_CU_SIZE) ? g_aucLog2[histLenth >> MAX_CU_DEPTH] + MAX_CU_DEPTH : g_aucLog2[histLenth];

There are total 4 places need the fix. Each assignment occurred twice.
Best.

Jane

[LGE_VCC]

patch for bug fix (LGE_BUG_FIX)

[taoranlu]

thank you for reporting, we'll provide a more general fix that remove the usage of g_auLog2[] to make it more general.

[taoranlu]

suggested fix:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/330