AddressSanitizer: stack-buffer-overflow

[fbarbier]

When building VTM with ASAN, we now meet some issues.
Please reproduce with the following command lines:

git checkout e328a149e86611d8b77d4c0c891ac3e564148015

cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
make -j 8
cd ..

./bin/EncoderAppStaticd --SourceWidth=240 --SourceHeight=24 --InputChromaFormat=420 --InputBitDepth=8 --Profile=next --FrameRate=60 --FramesToBeEncoded=4 --GOPSize=4 --Frame1="B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 5 9 13 0 0" --Frame2="B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 2 6 10 0 0" --Frame3="B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 3 7 11 0 0" --Frame4="B 4 1  0.0 0.0    0 0 1.0 0 0 0 4 4 1 4 8 12 0 0" --FastSearch=2 --PBIntraFast=1 --CTUSize=128 --MaxPartitionDepth=5 --PCMEnabledFlag=0 --SAO=0 --LoopFilterDisable=0 --ALF=0 --DisableIntraInInter=1 --DepQuant=1 --DualITree=1 --LMChroma=1 --TransformSkip=0 --MaxDeltaQP=0 --MTS=3 --IBC=1 --IMV=1 --Affine=0 --AffineType=0 --AffineAmvr=1 --MHIntra=1 --Triangle=1 --MIP=1 --SMVD=1 --SBT=1 --RDPCM=1 -i source_leak.yuv -b encoded.vvc

The output:

=================================================================
==21052==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd3dba8888 at pc 0x555d6ded1fdc bp 0x7ffd3dba7610 sp 0x7ffd3dba7608
WRITE of size 1 at 0x7ffd3dba8888 thread T0
    #0 0x555d6ded1fdb in PU::addMergeHMVPCand(CodingStructure const&, MergeCtx&, bool, int const&, unsigned int, int&, int, bool, unsigned int, bool, bool) /home/fredb/projects/vtm/source/Lib/CommonLib/UnitTools.cpp:1081
    #1 0x555d6ded3574 in PU::getIBCMergeCandidates(PredictionUnit const&, MergeCtx&, int const&) /home/fredb/projects/vtm/source/Lib/CommonLib/UnitTools.cpp:1181
    #2 0x555d6e4a4cb4 in EncCu::xCheckRDCostIBCModeMerge2Nx2N(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:3328
    #3 0x555d6e47b216 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:828
    #4 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371
    #5 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838
    #6 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371
    #7 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838
    #8 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371
    #9 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838
    #10 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371
    #11 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838
    #12 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371
    #13 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838
    #14 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371
    #15 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838
    #16 0x555d6e473fda in EncCu::compressCtu(CodingStructure&, UnitArea const&, unsigned int, int const*, int const*) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:366
    #17 0x555d6e2bf54e in EncSlice::encodeCtus(Picture*, bool, bool, unsigned int, unsigned int, EncLib*) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncSlice.cpp:1760
    #18 0x555d6e2bb9ef in EncSlice::compressSlice(Picture*, bool, bool) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncSlice.cpp:1448
    #19 0x555d6e218528 in EncGOP::compressGOP(int, int, std::__cxx11::list<Picture*, std::allocator<Picture*> >&, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncGOP.cpp:2272
    #20 0x555d6e255d4e in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, int&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncLib.cpp:633
    #21 0x555d6ddad235 in EncApp::encode() /home/fredb/projects/vtm/source/App/EncoderApp/EncApp.cpp:749
    #22 0x555d6de664bb in main /home/fredb/projects/vtm/source/App/EncoderApp/encmain.cpp:153
    #23 0x7f9c0857409a in __libc_start_main ../csu/libc-start.c:308
    #24 0x555d6dda0db9 in _start (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x32ddb9)

[XiangLi]

Fixed in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/851