id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
1497	Encoder crash for RGB content when TransformSkip and LFNST are disabled	adybrowne		"
The encoder crashes when processing RGB content with TransformSkip and LFNST disabled, for example using the command line below: 

bin/EncoderAppStatic -c cfg/encoder_intra_vtm.cfg -c cfg/per-class/formatRGB.cfg -c cfg/per-sequence-non-420/Map_RGB.cfg -o /dev/null  -q 37 -b out.bin --TransformSkip=0 --LFNST=0

Problem is caused by xRecurIntraCodingACTQT accessing beyond last element in trModes:

  tu.mtsIdx[COMPONENT_Y] = trModes[modeId].first;

which in turn is caused by using the wrong upper bound in the modeId loop 

  for (int modeId = firstCheckId; modeId <= ((m_pcEncCfg->getCostMode() == COST_LOSSLESS_CODING && slice.isLossless()) ? (nNumTransformCands - 1) : lastCheckId); modeId++)

lastCheckId can have a value greater than the length of trModes.

Fix is to use same upper bound calculation as that already used in the almost identical code in xRecurIntraCodingLumaQT:

  for( int modeId = firstCheckId; modeId <= ( sps.getUseLFNST() ? lastCheckId : ( nNumTransformCands - 1 ) ); modeId++ )

Tested on VTM13.0, VTM13.2 and recent git update (8168dafd)"	defect	closed	minor	VTM-14.0	VTM	VTM-13.0	fixed		ksuehring XiangLi fbossen jvet@…