id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
199	bug in shaper for high bitdepth	LGE_VCC		"If running VTM4.0 with bitdepth 12 ( warning regarding m_bitDepthConstraint in EncAppCfg is disabled). The resulted luma PSNR is very low, less than 10dB. 

Example command is
-c ../cfg/encoder_randomaccess_vtm.cfg -c BasketballPass.cfg -f 5 -q 32 -ip 48 -v 6 --InternalBitDepth=12 --MaxBitDepthConstraint=16


The problem is because of the following assignments to get log2 of the reshaper segment. 
int log2PwlFwdBinLen = g_aucLog2[pwlFwdBinLen];
int log2HistLenth = g_aucLog2[histLenth];

in VTM4, size of g_aucLog2 is only MAX_CU_SIZE + 1, but pwlFwdBinLen and histLenth are 256 when bitdepth is 12. So it resulted an out of bound array access. 

Suggested fixes are 
int log2PwlFwdBinLen = (pwlFwdBinLen>=MAX_CU_SIZE)? g_aucLog2[pwlFwdBinLen>> MAX_CU_DEPTH] + MAX_CU_DEPTH : g_aucLog2[pwlFwdBinLen];

int log2HistLenth = (histLenth >= MAX_CU_SIZE) ? g_aucLog2[histLenth >> MAX_CU_DEPTH] + MAX_CU_DEPTH : g_aucLog2[histLenth];

There are total 4 places need the fix. Each assignment occurred twice. 
Best.

Jane
"	defect	closed	minor	VTM-4.0.1	VTM	VTM-4.0	fixed	reshaper	ksuehring XiangLi fbossen jvet@…