Encoding crash when DeltaQpRD is used

[XiangLi]

VTM-20 encoding crash was reported when slice level multi-qp optimization is used. The issue could be reproduced with following command line.

/EncoderAppStatic -c ../cfg/encoder_lowdelay_P_vtm.cfg -i ../../../sequences/RaceHorses_832x480_30.y4m -v 6 -dph 1 -q 37 -f 9 --DeltaQpRD=3

The encoding log is

VVCSoftware: VTM Encoder Version 20.0 [Linux][GCC 12.2.0][64 bit] [SIMD=AVX2]

Warning: Y4M file info is different from input setting. Using the info from Y4M file

*
WARNING: For conforming bitstreams a valid Level value must be set!
*
Input File : ../../../sequences/RaceHorses_832x480_30.y4m
Bitstream File : str.bin
Reconstruction File : rec.yuv
Real Format : 832x480 30Hz
Internal Format : 832x480 30Hz
Sequence PSNR output : Linear average only
Hexadecimal PSNR output : Disabled
Sequence MSE output : Disabled
Frame MSE output : Disabled
MS-SSIM output : Disabled
Cabac-zero-word-padding : Enabled
Frame/Field : Frame based coding
Frame index : 0 - 8 (9 frames)
Profile : main_10
AllRapPicturesFlag : 0
CTU size / min CU size : 128 / 4
subpicture info present flag : Disabled
subpicture ID present flag : Disabled
Max TB size : 64
Motion search range : 64
Intra period : -1
Decoding refresh type : 0
DRAP period : 0
EDRAP period : 0
QP : 37
Max dQP signaling subdiv : 0
Cb QP Offset (dual tree) : 0 (0)
Cr QP Offset (dual tree) : 0 (0)
QP adaptation : 0 (range=0)
GOP size : 8
Input bit depth : (Y:8, C:8)
MSB-extended bit depth : (Y:8, C:8)
Internal bit depth : (Y:10, C:10)
Chroma QP offset list : Disabled
extended_precision_processing_flag : Disabled
TSRC_Rice_present_flag : Disabled
reverse_last_sig_coeff_enabled_flag : Disabled
transform_skip_rotation_enabled_flag : Disabled
transform_skip_context_enabled_flag : Disabled
high_precision_offsets_enabled_flag : Disabled
rrc_rice_extension_flag : Disabled
persistent_rice_adaptation_enabled_flag: Disabled
cabac_bypass_alignment_enabled_flag : Disabled
Cost function: : Lossy coding (default)
RateControl : 0
WeightedPredMethod : 0
GDREnabled : 0
Max Num Merge Candidates : 6
Max Num Affine Merge Candidates : 5
Max Num Geo Merge Candidates : 5
Max Num IBC Merge Candidates : 6

TOOL CFG: IBD:1 HAD:1 RDQ:1 RDQTS:1 RDpenalty:0 LQP:0 SQP:3 ASR:0 MinSearchWindow:8 RestrictMESampling:0 FEN:1 ECU:0 FDM:1 ESD:0 TransformSkip:1 TransformSkipFast:1 TransformSkipLog2MaxSize:5 ChromaTS:1 BDPCM:0 Tiles: 1x1 Slices: 1 MCTS:0 SAO:1 ALF:1 CCALF:1 MaxNumALFAPS 8 AlfapsIDShift 0 ConstantJointCbCrSignFlagWPP:0 WPB:0 PME:2 WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1 DQ:1 SignBitHidingFlag:0 RecalQP:0
TOOL CFG: LFNST:0 MMVD:1 Affine:1 AffineType:1 AdaptBypassAffineMe:0 PROF:1 SbTMVP:1 DualITree:1 IMV:1 BIO:0 LMChroma:1 HorCollocatedChroma:1 VerCollocatedChroma:0 MTS:1(explicit intra) SBT:1 ISP:1 SMVD:0 CompositeLTReference:0 Bcw:0 BcwFast:0 LADF:0 CIIP:1 Geo:0 AllowDisFracMMVD:1 AffineAmvr:0 AffineAmvrEncOpt:0 AffineAmvp:1 DMVR:0 MmvdDisNum:6 JointCbCr:1 ACT:0 PLT:0 IBC:0 HashME:0 WrapAround:0 VirtualBoundariesEnabledFlag:0 VirtualBoundariesPresentInSPSFlag:1 vertical virtual boundaries:[ ] horizontal virtual boundaries:[ ] Reshape:1 (Signal:SDR Opt:0 CSoffset:1) MRL:1 MIP:0 EncDbOpt:1
FAST TOOL CFG: LCTUFast:1 FastMrg:1 MaxMergeRdCandNumTotal:5 MergeRdCandQuotaRegular:4 MergeRdCandQuotaRegularSmallBlk:4 MergeRdCandQuotaSubBlk:2 MergeRdCandQuotaCiip:1 MergeRdCandQuotaGpm:8 PBIntraFast:1 IMV4PelFast:1 MTSMaxCand: 3(intra) 4(inter) ISPFast:0 AMaxBT:1 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 TTFastSkip:31 TTFastSkipThr:1.075 FastLocalDualTree:2 RPR:0 TemporalFilter:4/0 SEI CTI:0 BIM:0 SEI FGC:0 SEI processing Order:0

Input ChromaFormatIDC = 4:2:0

Output (internal) ChromaFormatIDC = 4:2:0

Non-environment-variable-controlled macros set as follows:

RExtDECODER_DEBUG_BIT_STATISTICS = 0

RExtHIGH_BIT_DEPTH_SUPPORT = 0

RExtHIGH_PRECISION_FORWARD_TRANSFORM = 0

started @ Wed Apr 26 09:44:58 2023

POC 0 LId: 0 TId: 0 ( IDR_N_LP, I-SLICE, QP 33 ) 150880 bits [Y 33.9538 dB U 36.6393 dB V 38.0217 dB] [ET 295 ] [L0] [L1] [MD5:7401e26ecd7d5c7e37ba437e2cdd0b25,ade216e995f5f6d6fc4e4df5358d517a,14284139ce2ddf75292f21fbecb2b14d]
POC 1 LId: 0 TId: 0 ( TRAIL, P-SLICE, QP 44 ) 8168 bits [Y 30.6804 dB U 35.9144 dB V 37.3392 dB] [ET 80 ] [L0 0c] [L1] [MD5:42e6c4ab0612c22c3cc895a706afb8e7,fda9fcf32588525f94dd13a4b7b1aab0,137d350bf696a85e88a7051a7d0c9df0]
POC 2 LId: 0 TId: 0 ( TRAIL, P-SLICE, QP 41 ) 9816 bits [Y 30.1372 dB U 35.5666 dB V 36.8531 dB] [ET 106 ] [L0 1c 0] [L1] [MD5:692ad0b11529789da60fcc1aa14390a7,26b70afeed3eae00dced1a7de50a294e,73feca41cfa0e9201db1247b36e3cf02]
POC 3 LId: 0 TId: 0 ( TRAIL, P-SLICE, QP 44 ) 8736 bits [Y 29.5723 dB U 35.3917 dB V 36.5509 dB] [ET 97 ] [L0 2c 1 0] [L1] [MD5:8ab4bddfa0c1e9d2c2189783a2cb643f,adde8e3c6146457f41385033926a317c,ab78558a2f6b285a58796e93a0cddb6b]

ERROR: In function "intra_luma_pred_modes" in /usr/local/google/home/xlxiangli/program/g-vtm/source/Lib/EncoderLib/CABACWriter.cpp:1055: use of non-MPM

[XiangLi]

A fix was submitted as ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2579

[XiangLi]

The fix in !2579 was merged.