Encoder crash for RGB content when TransformSkip and LFNST are disabled

[adybrowne]

The encoder crashes when processing RGB content with TransformSkip and LFNST disabled, for example using the command line below:

bin/EncoderAppStatic -c cfg/encoder_intra_vtm.cfg -c cfg/per-class/formatRGB.cfg -c cfg/per-sequence-non-420/Map_RGB.cfg -o /dev/null -q 37 -b out.bin --TransformSkip=0 --LFNST=0

Problem is caused by xRecurIntraCodingACTQT accessing beyond last element in trModes:

tu.mtsIdx[COMPONENT_Y] = trModes[modeId].first;

which in turn is caused by using the wrong upper bound in the modeId loop

for (int modeId = firstCheckId; modeId <= ((m_pcEncCfg->getCostMode() == COST_LOSSLESS_CODING && slice.isLossless()) ? (nNumTransformCands - 1) : lastCheckId); modeId++)

lastCheckId can have a value greater than the length of trModes.

Fix is to use same upper bound calculation as that already used in the almost identical code in xRecurIntraCodingLumaQT:

for( int modeId = firstCheckId; modeId <= ( sps.getUseLFNST() ? lastCheckId : ( nNumTransformCands - 1 ) ); modeId++ )

Tested on VTM13.0, VTM13.2 and recent git update (8168dafd)

[XiangLi]

Fixed as suggested ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2104