Opened 5 years ago

Closed 5 years ago

#326 closed defect (fixed)

AddressSanitizer: heap-buffer-overflow

Reported by: fbarbier Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-5.0
Keywords: Cc: vzakharc, yuwenhe, jvet@…

Description (last modified by ksuehring)

When building with ASAN, failures may be found.
Please reproduce with the following commands :

git checkout 04d626a060dd0140ae97b7cd0e4efafd0fcf301b

cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
make -j 8
cd ..


./bin/EncoderAppStaticd --SourceWidth=192 --SourceHeight=192 --InputChromaFormat=420 --InputBitDepth=10 --Profile=next --FrameRate=60 --FramesToBeEncoded=1 --GOPSize=4 '--Frame1=B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -5 -9 -13 0' '--Frame2=B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -2 -6 -10 1 -1 5 1 1 1 0 1' '--Frame3=B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -3 -7 -11 1 -1 5 0 1 1 1 1' '--Frame4=B 4 1  0.0 0.0    0 0 1.0 0 0 0 4 4 -1 -4 -8 -12 1 -1 5 0 1 1 1 1' --FastSearch=2 --LCTUFast=1 --LumaLevelToDeltaQPMode=1 --MaxCUWidth=16 --MaxCUHeight=24 --CTUSize=64 --MaxPartitionDepth=5 --PCMEnabledFlag=1 --SAO=0 --LoopFilterDisable=1 --ALF=1 --DisableIntraInInter=0 --DepQuant=1 --DualITree=0 --LMChroma=1 --TransformSkip=0 --MaxDeltaQP=0 --MTS=3 --IBC=0 --IMV=1 --Affine=1 --AffineType=1 --AffineAmvr=0 --MHIntra=0 --Triangle=1 --MIP=1 --SMVD=0 --SBT=0 --RDPCM=0 -i source.yuv -b encoded.vvc


=================================================================
==3913==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x63100012ba4b at pc 0x000000925935 bp 0x7ffc02255690 sp 0x7ffc02255688
READ of size 16 at 0x63100012ba4b thread T0
==3913==WARNING: invalid path to external symbolizer!
==3913==WARNING: Failed to use and restart external symbolizer!
    #0 0x925934  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x925934)
    #1 0xf622b9  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xf622b9)
    #2 0xdc1824  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xdc1824)
    #3 0xb3146b  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xb3146b)
    #4 0xb8507d  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xb8507d)
    #5 0x53d4d1  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x53d4d1)
    #6 0x6178cc  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x6178cc)
    #7 0x7f8714c4109a  (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
    #8 0x4540e9  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x4540e9)

0x63100012ba4b is located 3 bytes to the right of 78408-byte region [0x631000118800,0x63100012ba48)
allocated by thread T0 here:
    #0 0x4fcc09  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x4fcc09)
    #1 0x638e31  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x638e31)
    #2 0x630462  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x630462)
    #3 0xf65ef7  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xf65ef7)
    #4 0xdbbf95  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xdbbf95)
    #5 0xb7735c  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0xb7735c)
    #6 0x53bd15  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x53bd15)
    #7 0x53c626  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x53c626)
    #8 0x6178cc  (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x6178cc)
    #9 0x7f8714c4109a  (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x925934) 
Shadow bytes around the buggy address:
  0x0c628001d6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c628001d700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c628001d710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c628001d720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c628001d730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c628001d740: 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa
  0x0c628001d750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c628001d760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c628001d770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c628001d780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c628001d790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc

Attachments (2)

source.yuv​ (108.0 KB) - added by fbarbier 5 years ago.
encoded.vvc​ (211 bytes) - added by fbarbier 5 years ago.

Download all attachments as: .zip

Change history (10)

Changed 5 years ago by fbarbier

comment:1 Changed 5 years ago by fbarbier

  • Component changed from 360Lib to VTM
  • Version set to VVC D5 v8

comment:2 Changed 5 years ago by fbarbier

  • Version changed from VVC D5 v8 to VTM-5.0

comment:3 Changed 5 years ago by ksuehring

  • Description modified (diff)

comment:4 Changed 5 years ago by ksuehring

Seems to be ALF SIMD code

/projects/git/VTM-Integration/VVCSoftware_VTM_ksuehring/run$ ../bin/umake/clang-10.0/x86_64/debug/EncoderApp --SourceWidth=192 --SourceHeight=192 --InputChromaFormat=420 --InputBitDepth=10 --Profile=next --FrameRate=60 --FramesToBeEncoded=1 --GOPSize=4 '--Frame1=B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -5 -9 -13 0' '--Frame2=B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -2 -6 -10 1 -1 5 1 1 1 0 1' '--Frame3=B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -3 -7 -11 1 -1 5 0 1 1 1 1' '--Frame4=B 4 1  0.0 0.0    0 0 1.0 0 0 0 4 4 -1 -4 -8 -12 1 -1 5 0 1 1 1 1' --FastSearch=2 --LCTUFast=1 --LumaLevelToDeltaQPMode=1 --MaxCUWidth=16 --MaxCUHeight=24 --CTUSize=64 --MaxPartitionDepth=5 --PCMEnabledFlag=1 --SAO=0 --LoopFilterDisable=1 --ALF=1 --DisableIntraInInter=0 --DepQuant=1 --DualITree=0 --LMChroma=1 --TransformSkip=0 --MaxDeltaQP=0 --MTS=3 --IBC=0 --IMV=1 --Affine=1 --AffineType=1 --AffineAmvr=0 --MHIntra=0 --Triangle=1 --MIP=1 --SMVD=0 --SBT=0 --RDPCM=0 -i source.yuv -b encoded.vvc

VVCSoftware: VTM Encoder Version 5.0 [Mac OS X][clang 10.0.1][64 bit] [SIMD=AVX2] 

TOOL CFG: IBD:0 HAD:1 RDQ:1 RDQTS:1 RDpenalty:0 LQP:1 SQP:0 ASR:0 MinSearchWindow:8 RestrictMESampling:0 FEN:0 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:0 TransformSkipFast:0 TransformSkipLog2MaxSize:5 Slice: M=0 Tiles:1x1 MCTS:0 CIP:0 SAO:0 ALF:1 PCM:1 TransQuantBypassEnabled:0 WPP:0 WPB:0 PME:2  WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1  DQ:1  SignBitHidingFlag:0 RecalQP:0 
NEXT TOOL CFG: LFNST:0 MMVD:1 Affine:1 AffineType:1 SubPuMvp:0+0 DualITree:0 IMV:1 BIO:0 LMChroma:1 CclmCollocatedChroma:0 MTS: 1(intra) 1(inter) SBT:0 ISP:0 SMVD:0 CompositeLTReference:0 GBi:0 GBiFast:0 LADF:0 MHIntra:0 Triangle:1 AllowDisFracMMVD:0 AffineAmvr:0 AffineAmvrEncOpt:0 DMVR:0 MmvdDisNum:8 RDPCM:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:0 MIP:1 EncDbOpt:0 
FAST TOOL CFG: LCTUFast:1 FastMrg:0 PBIntraFast:0 IMV4PelFast:1 MTSMaxCand: 3(intra) 4(inter) AMaxBT:0 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearALFLuma:1 UseNonLinearALFChroma:1 FastMIP:0 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 


 started @ Wed Jun 26 15:41:16 2019
=================================================================
==62335==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x631000117a4b at pc 0x000100e434ef bp 0x7ffeef5a5f90 sp 0x7ffeef5a5f88
READ of size 16 at 0x631000117a4b thread T0
    #0 0x100e434ee in void simdDeriveClassificationBlk<(X86_VEXT)4>(AlfClassifier**, int***, AreaBuf<short const> const&, Area const&, Area const&, int, int, int) AdaptiveLoopFilterX86.h:135
    #1 0x10076a843 in AdaptiveLoopFilter::deriveClassification(AlfClassifier**, AreaBuf<short const> const&, Area const&, Area const&) AdaptiveLoopFilter.cpp:1078
    #2 0x101004641 in EncAdaptiveLoopFilter::ALFProcess(CodingStructure&, double const*, double) EncAdaptiveLoopFilter.cpp:786
    #3 0x1010f2da5 in EncGOP::compressGOP(int, int, std::__1::list<Picture*, std::__1::allocator<Picture*> >&, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) EncGOP.cpp:2533
    #4 0x1011729af in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, int&) EncLib.cpp:707
    #5 0x100670b5e in EncApp::encode() EncApp.cpp:825
    #6 0x100745ffd in main encmain.cpp:153
    #7 0x7fff770393d4 in start (libdyld.dylib:x86_64+0x163d4)

0x631000117a4b is located 3 bytes to the right of 78408-byte region [0x631000104800,0x631000117a48)
allocated by thread T0 here:
    #0 0x101f2a9c4 in wrap_posix_memalign (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c9c4)
    #1 0x1007adce9 in short* detail::aligned_malloc<short>(unsigned long, unsigned long) CommonDef.h:570
    #2 0x1007ad131 in PelStorage::create(ChromaFormat const&, Area const&, unsigned int, unsigned int, unsigned int, bool) Buffer.cpp:677
    #3 0x10076e961 in AdaptiveLoopFilter::create(int, int, ChromaFormat, int, int, int, int const*) AdaptiveLoopFilter.cpp:952
    #4 0x100ffe2f7 in EncAdaptiveLoopFilter::create(EncCfg const*, int, int, ChromaFormat, int, int, int, int const*, int const*) EncAdaptiveLoopFilter.cpp:442
    #5 0x101160dd5 in EncLib::create() EncLib.cpp:150
    #6 0x10066e61d in EncApp::xCreateLib(std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&) EncApp.cpp:715
    #7 0x10066fb92 in EncApp::encode() EncApp.cpp:762
    #8 0x100745ffd in main encmain.cpp:153
    #9 0x7fff770393d4 in start (libdyld.dylib:x86_64+0x163d4)

SUMMARY: AddressSanitizer: heap-buffer-overflow AdaptiveLoopFilterX86.h:135 in void simdDeriveClassificationBlk<(X86_VEXT)4>(AlfClassifier**, int***, AreaBuf<short const> const&, Area const&, Area const&, int, int, int)
Shadow bytes around the buggy address:
  0x1c6200022ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6200022f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6200022f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6200022f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6200022f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1c6200022f40: 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa
  0x1c6200022f50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c6200022f60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c6200022f70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c6200022f80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c6200022f90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==62335==ABORTING
Abort trap: 6
Version 0, edited 5 years ago by ksuehring (next)

comment:5 Changed 5 years ago by fbarbier

Please find another error reported by ASAN (At decoder side) on recent version:

First build a decoder using address sanitizer:

git checkout a5e1873a90f05a2eba9598401b07b12dd291aca4

cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
make -j 8
cd ..


./bin/DecoderAppStaticd -b encoded.vvc -o /dev/null

This is the output at runtime :

VVCSoftware: VTM Decoder Version 5.0 [Linux][GCC 8.2.0][64 bit] [SIMD=AVX] 
=================================================================
==28477==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000a34 at pc 0x55b3ee1a94f4 bp 0x7ffc52fe2c10 sp 0x7ffc52fe2c08
READ of size 4 at 0x602000000a34 thread T0
    #0 0x55b3ee1a94f3 in BrickMap::getBrickIdxBsMap(unsigned int) const /home/fredb/projects/vtm/source/Lib/CommonLib/../CommonLib/Picture.h:212
    #1 0x55b3ee1a3c5e in DecLib::xDecodeSlice(InputNALUnit&, int&, int) /home/fredb/projects/vtm/source/Lib/DecoderLib/DecLib.cpp:1352
    #2 0x55b3ee1a57b6 in DecLib::decode(InputNALUnit&, int&, int&) /home/fredb/projects/vtm/source/Lib/DecoderLib/DecLib.cpp:1723
    #3 0x55b3edde995c in DecApp::decode() /home/fredb/projects/vtm/source/App/DecoderApp/DecApp.cpp:166
    #4 0x55b3eddfb62b in main /home/fredb/projects/vtm/source/App/DecoderApp/decmain.cpp:91
    #5 0x7ff9ed34b09a in __libc_start_main ../csu/libc-start.c:308
    #6 0x55b3edde8ae9 in _start (/home/fredb/projects/vtm/bin/DecoderAppStaticd+0x21cae9)

0x602000000a34 is located 0 bytes to the right of 4-byte region [0x602000000a30,0x602000000a34)
allocated by thread T0 here:
    #0 0x7ff9ed947f40 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xeaf40)
    #1 0x55b3ede2c8c0 in BrickMap::create(SPS const&, PPS const&) /home/fredb/projects/vtm/source/Lib/CommonLib/Picture.cpp:754
    #2 0x55b3ede33cfe in Picture::finalInit(SPS const&, PPS const&, APS**, APS&) /home/fredb/projects/vtm/source/Lib/CommonLib/Picture.cpp:1262
    #3 0x55b3ee19eed7 in DecLib::xActivateParameterSets() /home/fredb/projects/vtm/source/Lib/DecoderLib/DecLib.cpp:887
    #4 0x55b3ee1a35d3 in DecLib::xDecodeSlice(InputNALUnit&, int&, int) /home/fredb/projects/vtm/source/Lib/DecoderLib/DecLib.cpp:1308
    #5 0x55b3ee1a57b6 in DecLib::decode(InputNALUnit&, int&, int&) /home/fredb/projects/vtm/source/Lib/DecoderLib/DecLib.cpp:1723
    #6 0x55b3edde995c in DecApp::decode() /home/fredb/projects/vtm/source/App/DecoderApp/DecApp.cpp:166
    #7 0x55b3eddfb62b in main /home/fredb/projects/vtm/source/App/DecoderApp/decmain.cpp:91
    #8 0x7ff9ed34b09a in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/fredb/projects/vtm/source/Lib/CommonLib/../CommonLib/Picture.h:212 in BrickMap::getBrickIdxBsMap(unsigned int) const
Shadow bytes around the buggy address:
  0x0c047fff80f0: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa fd fa
  0x0c047fff8100: fa fa fd fa fa fa 04 fa fa fa 04 fa fa fa fd fa
  0x0c047fff8110: fa fa 04 fa fa fa 00 fa fa fa fd fa fa fa fd fa
  0x0c047fff8120: fa fa fd fa fa fa fd fa fa fa fd fd fa fa 04 fa
  0x0c047fff8130: fa fa 04 fa fa fa 04 fa fa fa 00 fa fa fa 00 fa
=>0x0c047fff8140: fa fa 04 fa fa fa[04]fa fa fa 00 fa fa fa 00 fa
  0x0c047fff8150: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c047fff8160: fa fa 04 fa fa fa 04 fa fa fa 04 fa fa fa 04 fa
  0x0c047fff8170: fa fa 04 fa fa fa 04 fa fa fa 04 fa fa fa 04 fa
  0x0c047fff8180: fa fa 04 fa fa fa 04 fa fa fa 04 fa fa fa 04 fa
  0x0c047fff8190: fa fa 04 fa fa fa 04 fa fa fa 04 fa fa fa 04 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==28477==ABORTING

Changed 5 years ago by fbarbier

comment:6 Changed 5 years ago by ksuehring

The fix for accessing BrickMap::getBrickIdxBsMap was submitted as

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/780

comment:7 Changed 5 years ago by fbarbier

Thanks Karsten ! its ok now.

comment:8 Changed 5 years ago by fbarbier

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.