VTM 6.0 encoder crashes due to bestCS without CU

[tung.nguyen]

The VTM-6.0 encoder crashes due to access violation in EncCU.cpp.
Specifically, at several occurrences, the source code implements a direct access to the first object of the vector consisting of CUs, which is a member of the bestCS object, but the vector is empty.
Since the last lines of the method (xCompressCU) checks for the existence of a CU within the CS, the reporter assumes that the case where bestCS has an empty CU vector is not intended.
The issue seems to be solved by disabling the JVET_O0050_LOCAL_DUAL_TREE macro.

The issue can be reproduced using the following parameters, the encoder crashes after finishing POC 7:
-c cfg\encoder_randomaccess_vtm.cfg -c cfg\per-sequence\RaceHorses.cfg -i d:\Sequences\RaceHorses_416x240_30.yuv --TransquantBypassEnable=1 --CUTransquantBypassFlagForce=1 --CostMode=lossless --InternalBitDepth=8 -f 17 --ALF=0 --SBT=0 --LMCSEnable=0 --JointCbCr=0 --Log2MaxTbSize=5

[zhaoyin]

A merge request on this issue has been submitted. Please check whether the problem can be solved.
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/883

In xCompressCU(), several coding modes (e.g., skip/merge, affine, IMV, IBC, intra, modeSplit) are tried, but in some case all tried modes do not finish encoding because early termination (either from fast algorithm or normative checks) occurs inside encoding of each mode. In this case, both tempCS and bestCS have MAX_DOUBLE cost, and bestCS does not containing cu coding data.

Before the integration of JVET-O0050, any CU is allowed to do intra coding, and the RDO process does not skip intra coding mode (i.e., xCheckRDCostIntra()) if no inter coding mode gets complete coding results. Intra mode is like a backup plan of the RDO to ensure each CU has a valid coding mode. However, in an inter coding region (e.g., a 4x8 CU with modeTYpe == MODE_TYPE_INTER), only inter coding modes are tried, but the inter coding modes do not guarantee at least one valid coding mode.

So, if no valid coding mode after trying all coding modes in RDO, one way to avoid encoder crash is to return before saving best coding mode information at the end of xCompressCU(), as implemented by this merge request.

[tung.nguyen]

Dear zhaoyin,

Thank you for the patch, it looks good. The simulations are running without crashes so far.