Undefined Behavior Sanitizer : multiple errors (encoding / decoding)

[fbarbier]

When building with UBSAN, many failures may be found.
Please reproduce with the following commands :

git checkout a5e1873a90f05a2eba9598401b07b12dd291aca4

cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=undefined" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=undefined"
make -j 8
cd ..

./bin/EncoderAppStaticd --SourceWidth=64 --SourceHeight=64 --InputChromaFormat=420 --InputBitDepth=8 --Profile=next --FrameRate=60 --FramesToBeEncoded=1 --GOPSize=4 '--Frame1=B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 5 9 13 0 0' '--Frame2=B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 2 6 10 0 0' '--Frame3=B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 3 7 11 0 0' '--Frame4=B 4 1  0.0 0.0    0 0 1.0 0 0 0 4 4 1 4 8 12 0 0' --FastSearch=2 --PBIntraFast=1 --LumaLevelToDeltaQPMode=0 --MaxCUWidth=4 --MaxCUHeight=12 --CTUSize=64 --MaxPartitionDepth=5 --PCMEnabledFlag=0 --SAO=1 --LoopFilterDisable=1 --ALF=0 --DisableIntraInInter=1 --DepQuant=0 --SignHideFlag=0 -i source.yuv -b encoded.vvc
./bin/DecoderAppStaticd -b encoded.vvc -o /dev/null

Here is the output :

VVCSoftware: VTM Encoder Version 5.0 [Linux][GCC 8.2.0][64 bit] [SIMD=AVX] 

TOOL CFG: IBD:0 HAD:1 RDQ:1 RDQTS:1 RDpenalty:0 LQP:0 SQP:0 ASR:0 MinSearchWindow:8 RestrictMESampling:0 FEN:0 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:0 TransformSkipFast:0 TransformSkipLog2MaxSize:5 Slice: M=0 Tiles:1x1 MCTS:0 CIP:0 SAO:1 ALF:0 PCM:0 TransQuantBypassEnabled:0 WPP:0 WPB:0 PME:2  WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1  DQ:0  SignBitHidingFlag:0 RecalQP:0 
NEXT TOOL CFG: LFNST:0 MMVD:1 Affine:0 SubPuMvp:0+0 DualITree:0 IMV:1 BIO:0 LMChroma:1 CclmCollocatedChroma:0 MTS: 0(intra) 0(inter) SBT:0 ISP:0 SMVD:0 CompositeLTReference:0 GBi:0 GBiFast:0 LADF:0 MHIntra:0 Triangle:0 AllowDisFracMMVD:0 AffineAmvr:0 AffineAmvrEncOpt:0 DMVR:0 MmvdDisNum:8 RDPCM:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:0 MIP:1 EncDbOpt:0 
FAST TOOL CFG: LCTUFast:0 FastMrg:0 PBIntraFast:1 IMV4PelFast:1 AMaxBT:0 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearALFLuma:1 UseNonLinearALFChroma:1 FastMIP:0 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 


 started @ Tue Jul  9 15:47:25 2019
source/Lib/CommonLib/Slice.cpp:2728:32: runtime error: index -1 out of bounds for type 'int [16]'
source/Lib/EncoderLib/EncLib.cpp:676:27: runtime error: reference binding to null pointer of type 'struct APS'
source/Lib/CommonLib/CodingStructure.cpp:1365:21: runtime error: left shift of negative value -1
source/Lib/CommonLib/CodingStructure.cpp:1408:21: runtime error: left shift of negative value -1
source/Lib/CommonLib/MatrixIntraPrediction.cpp:497:74: runtime error: left shift of negative value -1
source/Lib/CommonLib/Quant.cpp:530:82: runtime error: left shift of negative value -160
source/Lib/CommonLib/CodingStructure.cpp:1391:21: runtime error: left shift of negative value -1
source/Lib/CommonLib/MatrixIntraPrediction.cpp:318:40: runtime error: left shift of negative value -19
source/Lib/CommonLib/Slice.cpp:2101:54: runtime error: index -1 out of bounds for type 'WPScalingParam [16][3]'
source/Lib/EncoderLib/EncSampleAdaptiveOffset.cpp:508:67: runtime error: left shift of negative value -45
source/Lib/EncoderLib/EncSampleAdaptiveOffset.cpp:470:30: runtime error: left shift of negative value -2
source/Lib/CommonLib/BitStream.cpp:113:3: runtime error: left shift of negative value -1
POC    0 TId: 0 ( I-SLICE, QP 30 )      11936 bits [Y 39.8149 dB    U 42.4371 dB    V 37.4776 dB] [ET     9 ] [L0 ] [L1 ]

        Total Frames |   Bitrate     Y-PSNR    U-PSNR    V-PSNR    YUV-PSNR 
                1    a     716.1600   39.8149   42.4371   37.4776   39.6309

 finished @ Tue Jul  9 15:47:35 2019
 Total Time:        9.895 sec. [user]        9.895 sec. [elapsed]

VVCSoftware: VTM Decoder Version 5.0 [Linux][GCC 8.2.0][64 bit] [SIMD=AVX] 
source/Lib/DecoderLib/DecLib.cpp:887:23: runtime error: reference binding to null pointer of type 'struct APS'
source/Lib/CommonLib/CodingStructure.cpp:1391:21: runtime error: left shift of negative value -64
source/Lib/CommonLib/CodingStructure.cpp:1365:21: runtime error: left shift of negative value -1
source/Lib/CommonLib/CodingStructure.cpp:1408:21: runtime error: left shift of negative value -1
source/Lib/CommonLib/Quant.cpp:530:82: runtime error: left shift of negative value -240
source/Lib/CommonLib/MatrixIntraPrediction.cpp:497:74: runtime error: left shift of negative value -1
source/Lib/CommonLib/Slice.cpp:2101:54: runtime error: index -1 out of bounds for type 'WPScalingParam [16][3]'
POC    0 TId: 0 ( I-SLICE, QP 30 ) [DT  0.025] [L0 ] [L1 ] [:,(unk)] 

[fbossen]

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2191