Opened 5 years ago
Closed 5 years ago
#431 closed defect (fixed)
AddressSanitizer: stack-buffer-overflow
Reported by: | fbarbier | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | VTM-6.0 |
Component: | VTM | Version: | VTM-5.2 |
Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
When building VTM with ASAN, we now meet some issues.
Please reproduce with the following command lines:
git checkout e328a149e86611d8b77d4c0c891ac3e564148015 cd build cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address" make -j 8 cd .. ./bin/EncoderAppStaticd --SourceWidth=240 --SourceHeight=24 --InputChromaFormat=420 --InputBitDepth=8 --Profile=next --FrameRate=60 --FramesToBeEncoded=4 --GOPSize=4 --Frame1="B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 5 9 13 0 0" --Frame2="B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 2 6 10 0 0" --Frame3="B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 3 7 11 0 0" --Frame4="B 4 1 0.0 0.0 0 0 1.0 0 0 0 4 4 1 4 8 12 0 0" --FastSearch=2 --PBIntraFast=1 --CTUSize=128 --MaxPartitionDepth=5 --PCMEnabledFlag=0 --SAO=0 --LoopFilterDisable=0 --ALF=0 --DisableIntraInInter=1 --DepQuant=1 --DualITree=1 --LMChroma=1 --TransformSkip=0 --MaxDeltaQP=0 --MTS=3 --IBC=1 --IMV=1 --Affine=0 --AffineType=0 --AffineAmvr=1 --MHIntra=1 --Triangle=1 --MIP=1 --SMVD=1 --SBT=1 --RDPCM=1 -i source_leak.yuv -b encoded.vvc
The output:
================================================================= ==21052==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd3dba8888 at pc 0x555d6ded1fdc bp 0x7ffd3dba7610 sp 0x7ffd3dba7608 WRITE of size 1 at 0x7ffd3dba8888 thread T0 #0 0x555d6ded1fdb in PU::addMergeHMVPCand(CodingStructure const&, MergeCtx&, bool, int const&, unsigned int, int&, int, bool, unsigned int, bool, bool) /home/fredb/projects/vtm/source/Lib/CommonLib/UnitTools.cpp:1081 #1 0x555d6ded3574 in PU::getIBCMergeCandidates(PredictionUnit const&, MergeCtx&, int const&) /home/fredb/projects/vtm/source/Lib/CommonLib/UnitTools.cpp:1181 #2 0x555d6e4a4cb4 in EncCu::xCheckRDCostIBCModeMerge2Nx2N(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:3328 #3 0x555d6e47b216 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:828 #4 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371 #5 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838 #6 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371 #7 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838 #8 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371 #9 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838 #10 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371 #11 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838 #12 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371 #13 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838 #14 0x555d6e482549 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:1371 #15 0x555d6e47b33a in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:838 #16 0x555d6e473fda in EncCu::compressCtu(CodingStructure&, UnitArea const&, unsigned int, int const*, int const*) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncCu.cpp:366 #17 0x555d6e2bf54e in EncSlice::encodeCtus(Picture*, bool, bool, unsigned int, unsigned int, EncLib*) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncSlice.cpp:1760 #18 0x555d6e2bb9ef in EncSlice::compressSlice(Picture*, bool, bool) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncSlice.cpp:1448 #19 0x555d6e218528 in EncGOP::compressGOP(int, int, std::__cxx11::list<Picture*, std::allocator<Picture*> >&, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncGOP.cpp:2272 #20 0x555d6e255d4e in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, int&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncLib.cpp:633 #21 0x555d6ddad235 in EncApp::encode() /home/fredb/projects/vtm/source/App/EncoderApp/EncApp.cpp:749 #22 0x555d6de664bb in main /home/fredb/projects/vtm/source/App/EncoderApp/encmain.cpp:153 #23 0x7f9c0857409a in __libc_start_main ../csu/libc-start.c:308 #24 0x555d6dda0db9 in _start (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x32ddb9)
Attachments (1)
Change history (2)
Changed 5 years ago by fbarbier
comment:1 Changed 5 years ago by XiangLi
- Milestone set to VTM-6.0
- Resolution set to fixed
- Status changed from new to closed
Note: See TracTickets for help on using tickets.
Fixed in https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/851