Segmentation fault occurred on VTM-21.0 with the two-layer-independent configuration on specific sequences under the lowdelay configuration.

[zixiang]

We ran VTM-21.0 in two-layer independent mode using the JVET VVC CTC sequences. The configuration and input of these two layers are identical.

We found that VTM only crashed with SlideShow and SlideEditing sequences under the LDB configuration.
All other sequences (including Class A, Class B, Class C, Class D, Class E, Class F), as well as this two sequences with the allintra or random-access configuration, can be encoded normally.

The shell commands we used are as follows:
EncoderApp -c cfg/encoder_lowdelay_vtm.cfg -c cfg/multi-layer/two_layers_independent.cfg -l0 -c cfg/per-class/classF.cfg -l0 -c cfg/per-sequence/SlideEditing.cfg -l0 -i SlideEditing_1280x720_30.yuv -l0 -q 37 -l1 -c cfg/per-class/classF.cfg -l1 -c cfg/per-sequence/SlideEditing.cfg -l1 -i SlideEditing_1280x720_30.yuv -l1 -q 37 --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --Verbosity=6 -o -b SlideEditing.bin

During our debugging process, we found that the crash occurred in the function EncGOP::xPicInitHashME (EncGOP.cpp line:1997). Specifically, when compressing the layer0 with POC equals to 9, the referenced flag of layer1 POC10 is true and this slice in rcListPic didn't initialized, thus triggering an error.

This issue exists on both windows and linux os with VTM21.0 and VTM21.2. We also found that other sequences which run well in release version on window os but crash with array
index out of bound error with debug mode using visual studio 2019 x64 under multi-layer configuration.

[zixiang]

coding log, vtm crashed at POC9 layer0

[furban]

This seems to be since the modification in merge request !2556 (
Fix: Related to HashME and sequential coding) (before VTM-20.0)
there is a loop on the picture list where layer is not checked. It may be cleaner to use a loop on the reference lists and check layers.