id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
1029	Encoder/Decoder Mismatch occurs when RDOQ is disabled	tsukuba.takeshi		"At VTM-8.0, encoder/Decoder mismatch occurs when following commands are used: --LFNST=1 --MTS=0 --ISP=0 --DepQuant=0 --RDOQ=0 --RDOQTS=0.

Mismatch comes from wrongly applying LFNST constraints on number of valid coefficients in '''raster scan order''' at Quant.cpp::quant():
...
    const uint32_t lfnstIdx = tu.cu->lfnstIdx;
    const int '''maxNumberOfCoeffs''' = lfnstIdx > 0 ? ((( uiWidth == 4 && uiHeight == 4 ) || ( uiWidth == 8 && uiHeight == 8) ) ? 8 : 16) : piQCoef.area();

    memset( piQCoef.buf, 0, sizeof(TCoeff) * piQCoef.area() );

    '''for (int uiBlockPos = 0; uiBlockPos < maxNumberOfCoeffs; uiBlockPos++ )'''
    {
      const TCoeff iLevel   = piCoef.buf['''uiBlockPos'''];
      const TCoeff iSign    = (iLevel < 0 ? -1: 1);

      const int64_t  tmpLevel = (int64_t)abs(iLevel) * (enableScalingLists ? piQuantCoeff['''uiBlockPos'''] : defaultQuantisationCoefficient);

      const TCoeff quantisedMagnitude = TCoeff((tmpLevel + iAdd ) >> iQBits);
      deltaU['''uiBlockPos'''] = (TCoeff)((tmpLevel - ((int64_t)quantisedMagnitude<<iQBits) )>> qBits8);

      uiAbsSum += quantisedMagnitude;
      const TCoeff quantisedCoefficient = quantisedMagnitude * iSign;

      piQCoef.buf['''uiBlockPos'''] = Clip3<TCoeff>( entropyCodingMinimum, entropyCodingMaximum, quantisedCoefficient );
    } // for n
...

Because the value of maxNumberOfCoeffs is set in coeff scan order, loop operation should be done in '''coeff scan order''' instead raster scan order.




-- encoder/decoder log --
EncoderApp -c encoder_randomaccess_vtm.cfg -q 22 -i RaceHorses_416x240_30.yuv -b str.bin --InputBitDepth=8 --InputChromaFormat=420 -fr 30 -wdt 416 -hgt 240 --Level=6.2 -f 2 -fs 0 -ip 32 --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --LFNST=1 --MTS=0 --ISP=0 --DepQuant=0 --RDOQ=0 --RDOQTS=0 --SEIDecodedPictureHash=1 --PrintHexPSNR=1 

VVCSoftware: VTM Encoder Version 8.0 [Windows][VS 1900][64 bit] [SIMD=AVX2] 

TOOL CFG: IBD:0 HAD:1 RDQ:0 RDQTS:0 RDpenalty:0 LQP:0 SQP:0 ASR:1 MinSearchWindow:96 RestrictMESampling:0 FEN:1 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:1 TransformSkipFast:1 TransformSkipLog2MaxSize:5 ChromaTS:1 BDPCM:0 Tiles: 1x1 Slices: 1 MCTS:0 SAO:1 ALF:1 CCALF:1 WPP:0 WPB:0 PME:2  WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1  DQ:0  SignBitHidingFlag:0 RecalQP:0 
TOOL CFG: LFNST:1 MMVD:1 Affine:1 AffineType:1 PROF:1 SubPuMvp:1+0 DualITree:1 IMV:1 BIO:1 LMChroma:1 HorCollocatedChroma:1 VerCollocatedChroma:0 MTS: 0(intra) 0(inter) SBT:1 ISP:0 SMVD:1 CompositeLTReference:0 Bcw:1 BcwFast:1 LADF:0 CIIP:1 Geo:1 AllowDisFracMMVD:1 AffineAmvr:1 AffineAmvrEncOpt:1 DMVR:1 MmvdDisNum:6 JointCbCr:1 ACT:0 PLT:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:1 (Signal:SDR Opt:0 CSoffset:6) MRL:1 MIP:1 EncDbOpt:0 
FAST TOOL CFG: LCTUFast:1 FastMrg:1 PBIntraFast:1 IMV4PelFast:1 FastLFNST:0 AMaxBT:1 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 FastMIP:0 FastLocalDualTree:1 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 RPR:0 TemporalFilter:0 


 started @ Mon Apr 27 17:32:16 2020
POC    0 LId:  0 TId: 0 ( I-SLICE, QP 19 )     611360 bits [Y 45.6834 dB    U 46.4681 dB    V 46.8535 dB] [xY 4046d778fff8c814 xU 40473beb1a46161d xV 40476d4105e09f55] [ET    30 ] [L0] [L1] [MD5:41451e12ea536555cb2f19e6c6623776,146bbd4675109aaad3b47a4ef007114b,20be956c0f1e4d55cfc97a6a34468d60]
POC    1 LId:  0 TId: 4 ( B-SLICE, QP 29 )     173976 bits [Y 36.9931 dB    U 36.5246 dB    V 37.2789 dB] [xY 40427f1c6074104f xU 40424325b1b2aaea xV 4042a3b1a45349ba] [ET    47 ] [L0 0] [L1 0c] [MD5:afbc700f2f83ac13223bd1d2a27c541d,ef3268ffa4b18d382b86534d571b9fdf,f2feaf0f4204479f2aa9d0b3148be617]

LayerId  0
	Total Frames |   Bitrate     Y-PSNR    U-PSNR    V-PSNR    YUV-PSNR   xY-PSNR           xU-PSNR           xV-PSNR           
	        2    a   11780.0400   41.3382   41.4964   42.0662   39.4578   4044ab4ab0366c32  4044bf8865fc6084  404508795519f488

 finished @ Mon Apr 27 17:33:34 2020
 Total Time:       77.473 sec. [user]       77.473 sec. [elapsed]
DecoderApp  -b str.bin

VVCSoftware: VTM Decoder Version 8.0 [Windows][VS 1900][64 bit] [SIMD=AVX2] 
POC    0 LId:  0 TId: 0 ( I-SLICE, QP 19 ) [DT  0.052] [L0] [L1] [MD5:41451e12ea536555cb2f19e6c6623776,146bbd4675109aaad3b47a4ef007114b,20be956c0f1e4d55cfc97a6a34468d60,(OK)] 
POC    1 LId:  0 TId: 4 ( B-SLICE, QP 29 ) [DT  0.030] [L0 0] [L1 0c] [MD5:31f1e319e40c199ff5e61a5394682e8b,5c9f8c6d4e380e45cef362caff413b4d,2d043b3f5e5d39dbd3a36657b704fed4,(***ERROR***)] [rxMD5:afbc700f2f83ac13223bd1d2a27c541d,ef3268ffa4b18d382b86534d571b9fdf,f2feaf0f4204479f2aa9d0b3148be617] 


***ERROR*** A decoding mismatch occured: signalled md5sum does not match

 Total Time:        0.103 sec.
"	defect	closed	minor	VTM-9.0	VTM	VTM-8.0	fixed		ksuehring XiangLi fbossen jvet@…