Opened 2 months ago

Last modified 6 weeks ago

#468 new defect

ERROR: AddressSanitizer: unknown-crash

Reported by: fbarbier Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-6.1
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

When building with ASAN the VTM 6.0, a crash may be found using the following command lines :

git checkout cb45f4386cf08d5b209333be61965a4beb4ff305

cd build
readonly sanitizer="-fsanitize=address"
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS=$sanitizer -DCMAKE_EXE_LINKER_FLAGS=$sanitizer
make -j 8
cd ..

./bin/EncoderAppStaticd --SourceWidth=192 --SourceHeight=128 --InputChromaFormat=420 --InputBitDepth=8 --Profile=next --FrameRate=60 --FramesToBeEncoded=1 --GOPSize=4 --Frame1="B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 5 9 13 0 0" --Frame2="B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 2 6 10 0 0" --Frame3="B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 1 3 7 11 0 0" --Frame4="B 4 1  0.0 0.0    0 0 1.0 0 0 0 4 4 1 4 8 12 0 0" --FastSearch=2 --LCTUFast=1 --PBIntraFast=1 --CTUSize=64 --MaxPartitionDepth=5 --PCMEnabledFlag=1 --SAO=1 --ALF=1 --DepQuant=1 --DualITree=1 --MaxDeltaQP=2 --MTS=3 --IMV=1 --Affine=1 --AffineType=1 --Triangle=1 --SBT=1 -i source.yuv -b encoded.vvc

Output is :

VVCSoftware: VTM Encoder Version 6.0 [Linux][GCC 8.2.0][64 bit] [SIMD=AVX] 

TOOL CFG: IBD:0 HAD:1 RDQ:1 RDQTS:1 RDpenalty:0 LQP:0 SQP:0 ASR:0 MinSearchWindow:8 RestrictMESampling:0 FEN:0 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:0 TransformSkipFast:0 TransformSkipLog2MaxSize:5 BDPCM:0 Slice: M=0 Tiles:1x1 MCTS:0 CIP:0 SAO:1 ALF:1 PCM:1 TransQuantBypassEnabled:0 WPP:0 WPB:0 PME:2  WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1  DQ:1  SignBitHidingFlag:0 RecalQP:0 
NEXT TOOL CFG: LFNST:0 MMVD:1 Affine:1 AffineType:1 PROF:0 SubPuMvp:0+0 DualITree:1 IMV:1 BIO:0 LMChroma:1 CclmCollocatedChroma:0 MTS: 1(intra) 1(inter) SBT:1 ISP:0 SMVD:0 CompositeLTReference:0 GBi:0 GBiFast:0 LADF:0 MHIntra:0 Triangle:1 AllowDisFracMMVD:0 AffineAmvr:0 AffineAmvrEncOpt:0 DMVR:0 MmvdDisNum:8 JointCbCr:0 PLT:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:0 MIP:1 EncDbOpt:0 
FAST TOOL CFG: LCTUFast:1 FastMrg:0 PBIntraFast:1 IMV4PelFast:1 MTSMaxCand: 3(intra) 4(inter) AMaxBT:0 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 FastMIP:0 FastLocalDualTree:0 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 


 started @ Mon Aug 19 17:12:37 2019






=================================================================
==28801==ERROR: AddressSanitizer: unknown-crash on address 0x62f00002933c at pc 0x55c7f6d1982a bp 0x7ffdbd761fc0 sp 0x7ffdbd761fb8
READ of size 16 at 0x62f00002933c thread T0
    #0 0x55c7f6d19829 in _mm_loadu_si128(long long __vector(2) const*) /usr/lib/gcc/x86_64-linux-gnu/8/include/emmintrin.h:703
    #1 0x55c7f6d19829 in simdDeriveClassificationBlk<(X86_VEXT)3> /home/fredb/projects/vtm/source/Lib/CommonLib/x86/avx/../AdaptiveLoopFilterX86.h:100
    #2 0x55c7f721ced4 in AdaptiveLoopFilter::deriveClassification(AlfClassifier**, AreaBuf<short const> const&, Area const&, Area const&) /home/fredb/projects/vtm/source/Lib/CommonLib/AdaptiveLoopFilter.cpp:750
    #3 0x55c7f70bc929 in EncAdaptiveLoopFilter::ALFProcess(CodingStructure&, double const*, double) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncAdaptiveLoopFilter.cpp:770
    #4 0x55c7f6e86a95 in EncGOP::compressGOP(int, int, std::__cxx11::list<Picture*, std::allocator<Picture*> >&, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncGOP.cpp:2518
    #5 0x55c7f6ec2140 in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, int&) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncLib.cpp:637
    #6 0x55c7f6a17233 in EncApp::encode() /home/fredb/projects/vtm/source/App/EncoderApp/EncApp.cpp:779
    #7 0x55c7f6ad29f9 in main /home/fredb/projects/vtm/source/App/EncoderApp/encmain.cpp:153
    #8 0x7ffa43a1a09a in __libc_start_main ../csu/libc-start.c:308
    #9 0x55c7f6a0adb9 in _start (/home/fredb/projects/vtm/bin/EncoderAppStaticd+0x339db9)

0x62f000029348 is located 0 bytes to the right of 53064-byte region [0x62f00001c400,0x62f000029348)
allocated by thread T0 here:
    #0 0x7ffa44016038 in __interceptor_posix_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xea038)
    #1 0x55c7f6aec604 in short* detail::aligned_malloc<short>(unsigned long, unsigned long) /home/fredb/projects/vtm/source/Lib/CommonLib/CommonDef.h:604
    #2 0x55c7f6ae35ed in PelStorage::create(ChromaFormat const&, Area const&, unsigned int, unsigned int, unsigned int, bool) /home/fredb/projects/vtm/source/Lib/CommonLib/Buffer.cpp:799
    #3 0x55c7f721b5cd in AdaptiveLoopFilter::create(int, int, ChromaFormat, int, int, int, int const*) /home/fredb/projects/vtm/source/Lib/CommonLib/AdaptiveLoopFilter.cpp:652
    #4 0x55c7f70b71f9 in EncAdaptiveLoopFilter::create(EncCfg const*, int, int, ChromaFormat, int, int, int, int const*, int const*) /home/fredb/projects/vtm/source/Lib/EncoderLib/EncAdaptiveLoopFilter.cpp:423
    #5 0x55c7f6ebe72f in EncLib::create() /home/fredb/projects/vtm/source/Lib/EncoderLib/EncLib.cpp:148
    #6 0x55c7f6a16276 in EncApp::xCreateLib(std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&) /home/fredb/projects/vtm/source/App/EncoderApp/EncApp.cpp:666
    #7 0x55c7f6a1698f in EncApp::encode() /home/fredb/projects/vtm/source/App/EncoderApp/EncApp.cpp:713
    #8 0x55c7f6ad29f9 in main /home/fredb/projects/vtm/source/App/EncoderApp/encmain.cpp:153
    #9 0x7ffa43a1a09a in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: unknown-crash /usr/lib/gcc/x86_64-linux-gnu/8/include/emmintrin.h:703 in _mm_loadu_si128(long long __vector(2) const*)
Shadow bytes around the buggy address:
  0x0c5e7fffd210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5e7fffd220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5e7fffd230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5e7fffd240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5e7fffd250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c5e7fffd260: 00 00 00 00 00 00 00[00]00 fa fa fa fa fa fa fa
  0x0c5e7fffd270: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5e7fffd280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5e7fffd290: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5e7fffd2a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5e7fffd2b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==28801==ABORTING

Attachments (1)

source.yuv (36.0 KB) - added by fbarbier 2 months ago.

Download all attachments as: .zip

Change history (5)

Changed 2 months ago by fbarbier

comment:1 Changed 6 weeks ago by fbarbier

  • Version changed from VTM-6.0rc1 to VTM-6.1

still present on 6.1

comment:2 Changed 6 weeks ago by ksuehring

I tried reproducing with clang/Xcode. I had to remove EnablePCM from the command line. Now I'm getting a heap overflow in ALF SIMD code:

VVCSoftware: VTM Encoder Version 6.1 [Mac OS X][clang 10.0.1][64 bit] [SIMD=AVX2] 

TOOL CFG: IBD:0 HAD:1 RDQ:1 RDQTS:1 RDpenalty:0 LQP:0 SQP:0 ASR:0 MinSearchWindow:8 RestrictMESampling:0 FEN:0 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:0 TransformSkipFast:0 TransformSkipLog2MaxSize:5 BDPCM:0 Slice: M=0 Tiles:1x1 MCTS:0 CIP:0 SAO:1 ALF:1 TransQuantBypassEnabled:0 WPP:0 WPB:0 PME:2  WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1  DQ:1  SignBitHidingFlag:0 RecalQP:0 
NEXT TOOL CFG: LFNST:0 MMVD:1 Affine:1 AffineType:1 PROF:0 SubPuMvp:0+0 DualITree:1 IMV:1 BIO:0 LMChroma:1 CclmCollocatedChroma:0 MTS: 1(intra) 1(inter) SBT:1 ISP:0 SMVD:0 CompositeLTReference:0 GBi:0 GBiFast:0 LADF:0 MHIntra:0 Triangle:1 AllowDisFracMMVD:0 AffineAmvr:0 AffineAmvrEncOpt:0 DMVR:0 MmvdDisNum:8 JointCbCr:0 PLT:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:0 MIP:1 EncDbOpt:0 
FAST TOOL CFG: LCTUFast:1 FastMrg:0 PBIntraFast:1 IMV4PelFast:1 MTSMaxCand: 3(intra) 4(inter) AMaxBT:0 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 FastMIP:0 FastLocalDualTree:0 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 RPR:0


 started @ Thu Sep 12 16:43:44 2019
=================================================================
==45197==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0000c334b at pc 0x000100746b73 bp 0x7ffeefbf94d0 sp 0x7ffeefbf94c8
READ of size 16 at 0x62f0000c334b thread T0
2019-09-12 16:46:51.583453+0200 atos[45222:3599789] examining /Users/USER/*/EncoderApp [45197]
    #0 0x100746b72 in void simdDeriveClassificationBlk<(X86_VEXT)4>(AlfClassifier**, int***, AreaBuf<short const> const&, Area const&, Area const&, int, int, int) AdaptiveLoopFilterX86.h:94
    #1 0x100104f33 in AdaptiveLoopFilter::deriveClassification(AlfClassifier**, AreaBuf<short const> const&, Area const&, Area const&) AdaptiveLoopFilter.cpp:733
    #2 0x1009d545d in EncAdaptiveLoopFilter::ALFProcess(CodingStructure&, double const*, double) EncAdaptiveLoopFilter.cpp:780
    #3 0x100adc660 in EncGOP::compressGOP(int, int, std::__1::list<Picture*, std::__1::allocator<Picture*> >&, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) EncGOP.cpp:2757
    #4 0x100b568fe in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, int&) EncLib.cpp:807
    #5 0x10001ce7e in EncApp::encode() EncApp.cpp:818
    #6 0x1000e1ccd in main encmain.cpp:153
    #7 0x7fff6c9ef3d4 in start (libdyld.dylib:x86_64+0x163d4)

0x62f0000c334b is located 3 bytes to the right of 53064-byte region [0x62f0000b6400,0x62f0000c3348)
allocated by thread T0 here:
    #0 0x103b909c4 in wrap_posix_memalign (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c9c4)
    #1 0x10014a2a9 in short* detail::aligned_malloc<short>(unsigned long, unsigned long) CommonDef.h:592
    #2 0x1001496f1 in PelStorage::create(ChromaFormat const&, Area const&, unsigned int, unsigned int, unsigned int, bool) Buffer.cpp:799
    #3 0x100107a4e in AdaptiveLoopFilter::create(int, int, ChromaFormat, int, int, int, int const*) AdaptiveLoopFilter.cpp:663
    #4 0x1009ceaff in EncAdaptiveLoopFilter::create(EncCfg const*, int, int, ChromaFormat, int, int, int, int const*, int const*) EncAdaptiveLoopFilter.cpp:428
    #5 0x100adc0e0 in EncGOP::compressGOP(int, int, std::__1::list<Picture*, std::__1::allocator<Picture*> >&, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) EncGOP.cpp:2749
    #6 0x100b568fe in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, int&) EncLib.cpp:807
    #7 0x10001ce7e in EncApp::encode() EncApp.cpp:818
    #8 0x1000e1ccd in main encmain.cpp:153
    #9 0x7fff6c9ef3d4 in start (libdyld.dylib:x86_64+0x163d4)

SUMMARY: AddressSanitizer: heap-buffer-overflow AdaptiveLoopFilterX86.h:94 in void simdDeriveClassificationBlk<(X86_VEXT)4>(AlfClassifier**, int***, AreaBuf<short const> const&, Area const&, Area const&, int, int, int)
Shadow bytes around the buggy address:
  0x1c5e00018610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c5e00018620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c5e00018630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c5e00018640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c5e00018650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1c5e00018660: 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa
  0x1c5e00018670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c5e00018680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c5e00018690: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c5e000186a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c5e000186b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
2019-09-12 16:46:52.340189+0200 EncoderApp[45197:3593337] =================================================================
2019-09-12 16:46:52.340284+0200 EncoderApp[45197:3593337] ==45197==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0000c334b at pc 0x000100746b73 bp 0x7ffeefbf94d0 sp 0x7ffeefbf94c8
2019-09-12 16:46:52.340298+0200 EncoderApp[45197:3593337] READ of size 16 at 0x62f0000c334b thread T0
2019-09-12 16:46:52.340308+0200 EncoderApp[45197:3593337]     #0 0x100746b72 in void simdDeriveClassificationBlk<(X86_VEXT)4>(AlfClassifier**, int***, AreaBuf<short const> const&, Area const&, Area const&, int, int, int) AdaptiveLoopFilterX86.h:94
2019-09-12 16:46:52.340321+0200 EncoderApp[45197:3593337]     #1 0x100104f33 in AdaptiveLoopFilter::deriveClassification(AlfClassifier**, AreaBuf<short const> const&, Area const&, Area const&) AdaptiveLoopFilter.cpp:733
2019-09-12 16:46:52.340339+0200 EncoderApp[45197:3593337]     #2 0x1009d545d in EncAdaptiveLoopFilter::ALFProcess(CodingStructure&, double const*, double) EncAdaptiveLoopFilter.cpp:780
2019-09-12 16:46:52.340407+0200 EncoderApp[45197:3593337]     #3 0x100adc660 in EncGOP::compressGOP(int, int, std::__1::list<Picture*, std::__1::allocator<Picture*> >&, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) EncGOP.cpp:2757
2019-09-12 16:46:52.340552+0200 EncoderApp[45197:3593337]     #4 0x100b568fe in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, int&) EncLib.cpp:807
2019-09-12 16:46:52.340568+0200 EncoderApp[45197:3593337]     #5 0x10001ce7e in EncApp::encode() EncApp.cpp:818
2019-09-12 16:46:52.340578+0200 EncoderApp[45197:3593337]     #6 0x1000e1ccd in main encmain.cpp:153
2019-09-12 16:46:52.340616+0200 EncoderApp[45197:3593337]     #7 0x7fff6c9ef3d4 in start (libdyld.dylib:x86_64+0x163d4)
2019-09-12 16:46:52.340627+0200 EncoderApp[45197:3593337] 
2019-09-12 16:46:52.340633+0200 EncoderApp[45197:3593337] 0x62f0000c334b is located 3 bytes to the right of 53064-byte region [0x62f0000b6400,0x62f0000c3348)
2019-09-12 16:46:52.340640+0200 EncoderApp[45197:3593337] allocated by thread T0 here:
2019-09-12 16:46:52.340646+0200 EncoderApp[45197:3593337]     #0 0x103b909c4 in wrap_posix_memalign (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c9c4)
2019-09-12 16:46:52.340724+0200 EncoderApp[45197:3593337]     #1 0x10014a2a9 in short* detail::aligned_malloc<short>(unsigned long, unsigned long) CommonDef.h:592
2019-09-12 16:46:52.340740+0200 EncoderApp[45197:3593337]     #2 0x1001496f1 in PelStorage::create(ChromaFormat const&, Area const&, unsigned int, unsigned int, unsigned int, bool) Buffer.cpp:799
2019-09-12 16:46:52.340748+0200 EncoderApp[45197:3593337]     #3 0x100107a4e in AdaptiveLoopFilter::create(int, int, ChromaFormat, int, int, int, int const*) AdaptiveLoopFilter.cpp:663
2019-09-12 16:46:52.340784+0200 EncoderApp[45197:3593337]     #4 0x1009ceaff in EncAdaptiveLoopFilter::create(EncCfg const*, int, int, ChromaFormat, int, int, int, int const*, int const*) EncAdaptiveLoopFilter.cpp:428
2019-09-12 16:46:52.340801+0200 EncoderApp[45197:3593337]     #5 0x100adc0e0 in EncGOP::compressGOP(int, int, std::__1::list<Picture*, std::__1::allocator<Picture*> >&, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool) EncGOP.cpp:2749
2019-09-12 16:46:52.340922+0200 EncoderApp[45197:3593337]     #6 0x100b568fe in EncLib::encode(bool, PelStorage*, PelStorage*, InputColourSpaceConversion, std::__1::list<UnitBuf<short>*, std::__1::allocator<UnitBuf<short>*> >&, int&) EncLib.cpp:807
2019-09-12 16:46:52.340943+0200 EncoderApp[45197:3593337]     #7 0x10001ce7e in EncApp::encode() EncApp.cpp:818
2019-09-12 16:46:52.340964+0200 EncoderApp[45197:3593337]     #8 0x1000e1ccd in main encmain.cpp:153
2019-09-12 16:46:52.340977+0200 EncoderApp[45197:3593337]     #9 0x7fff6c9ef3d4 in start (libdyld.dylib:x86_64+0x163d4)
2019-09-12 16:46:52.340995+0200 EncoderApp[45197:3593337] 
2019-09-12 16:46:52.341009+0200 EncoderApp[45197:3593337] SUMMARY: AddressSanitizer: heap-buffer-overflow AdaptiveLoopFilterX86.h:94 in void simdDeriveClassificationBlk<(X86_VEXT)4>(AlfClassifier**, int***, AreaBuf<short const> const&, Area const&, Area const&, int, int, int)
2019-09-12 16:46:52.341023+0200 EncoderApp[45197:3593337] Shadow bytes around the buggy address:
2019-09-12 16:46:52.341096+0200 EncoderApp[45197:3593337]   0x1c5e00018610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-09-12 16:46:52.341110+0200 EncoderApp[45197:3593337]   0x1c5e00018620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-09-12 16:46:52.341124+0200 EncoderApp[45197:3593337]   0x1c5e00018630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-09-12 16:46:52.341153+0200 EncoderApp[45197:3593337]   0x1c5e00018640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-09-12 16:46:52.341163+0200 EncoderApp[45197:3593337]   0x1c5e00018650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2019-09-12 16:46:52.341196+0200 EncoderApp[45197:3593337] =>0x1c5e00018660: 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa
2019-09-12 16:46:52.341205+0200 EncoderApp[45197:3593337]   0x1c5e00018670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2019-09-12 16:46:52.341213+0200 EncoderApp[45197:3593337]   0x1c5e00018680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2019-09-12 16:46:52.341339+0200 EncoderApp[45197:3593337]   0x1c5e00018690: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2019-09-12 16:46:52.341358+0200 EncoderApp[45197:3593337]   0x1c5e000186a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2019-09-12 16:46:52.341372+0200 EncoderApp[45197:3593337]   0x1c5e000186b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2019-09-12 16:46:52.341384+0200 EncoderApp[45197:3593337] Shadow byte legend (one shadow byte represents 8 application bytes):
2019-09-12 16:46:52.341394+0200 EncoderApp[45197:3593337]   Addressable:           00
2019-09-12 16:46:52.341405+0200 EncoderApp[45197:3593337]   Partially addressable: 01 02 03 04 05 06 07
2019-09-12 16:46:52.341416+0200 EncoderApp[45197:3593337]   Heap left redzone:       fa
2019-09-12 16:46:52.341428+0200 EncoderApp[45197:3593337]   Freed heap region:       fd
2019-09-12 16:46:52.341440+0200 EncoderApp[45197:3593337]   Stack left redzone:      f1
2019-09-12 16:46:52.341452+0200 EncoderApp[45197:3593337]   Stack mid redzone:       f2
2019-09-12 16:46:52.341517+0200 EncoderApp[45197:3593337]   Stack right redzone:     f3
2019-09-12 16:46:52.341538+0200 EncoderApp[45197:3593337]   Stack after return:      f5
2019-09-12 16:46:52.341566+0200 EncoderApp[45197:3593337]   Stack use after scope:   f8
2019-09-12 16:46:52.341580+0200 EncoderApp[45197:3593337]   Global redzone:          f9
2019-09-12 16:46:52.341587+0200 EncoderApp[45197:3593337]   Global init order:       f6
2019-09-12 16:46:52.341593+0200 EncoderApp[45197:3593337]   Poisoned by user:        f7
2019-09-12 16:46:52.341599+0200 EncoderApp[45197:3593337]   Container overflow:      fc
2019-09-12 16:46:52.341604+0200 EncoderApp[45197:3593337]   Array cookie:            ac
2019-09-12 16:46:52.341624+0200 EncoderApp[45197:3593337]   Intra object redzone:    bb
2019-09-12 16:46:52.341638+0200 EncoderApp[45197:3593337]   ASan internal:           fe
2019-09-12 16:46:52.341645+0200 EncoderApp[45197:3593337]   Left alloca redzone:     ca
2019-09-12 16:46:52.341683+0200 EncoderApp[45197:3593337]   Right alloca redzone:    cb
2019-09-12 16:46:52.341702+0200 EncoderApp[45197:3593337]   Shadow gap:              cc
2019-09-12 16:46:52.341716+0200 EncoderApp[45197:3593337] 
==45197==ABORTING
AddressSanitizer report breakpoint hit. Use 'thread info -s' to get extended information about the report.
(lldb) 

comment:3 Changed 6 weeks ago by ksuehring

Apparently this can be fixed by giving the ALF temporary buffer a bit bigger margin, e.g.:

AdaptiveLoopFilter::create()

change

  m_tempBuf.create( format, Area( 0, 0, picWidth, picHeight ), maxCUWidth, MAX_ALF_FILTER_LENGTH >> 1, 0, false );

to

  m_tempBuf.create( format, Area( 0, 0, picWidth, picHeight ), maxCUWidth, MAX_ALF_FILTER_LENGTH, 0, false );

comment:4 Changed 6 weeks ago by fbossen

The core issue is that the SIMD code processes data in bunches of 8 samples. However the number of elements that are processed is a multiple of 4. The SIMD code may thus attempt to read an extra 4 elements in some cases. Allocating an additional 4 samples for m_tempBuf resolves the ASAN issue.

The allocation of m_tempBuf is not super clean to start with. The classifier assumes a margin of 3 which is independent of the max ALF filter length, so the margin shouldn't simply depend on the max filter length. Increasing the margin from MAX_ALF_FILTER_LENGTH >> 1 to MAX_ALF_FILTER_LENGTH works, but it somewhat a hack.

I would suggest modifying the SIMD code to not read more data than required and modify the margin in the allocation to min(3, MAX_ALF_FILTER_LENGTH >> 1).

Note: See TracTickets for help on using tickets.