Encoder/Decoder Mismatch occurs when RDOQ is disabled

[tsukuba.takeshi]

At VTM-8.0, encoder/Decoder mismatch occurs when following commands are used: --LFNST=1 --MTS=0 --ISP=0 --DepQuant=0 --RDOQ=0 --RDOQTS=0.

Mismatch comes from wrongly applying LFNST constraints on number of valid coefficients in raster scan order at Quant.cpp::quant():
...

const uint32_t lfnstIdx = tu.cu->lfnstIdx;

const int maxNumberOfCoeffs = lfnstIdx > 0 ? ((( uiWidth == 4 && uiHeight == 4 )

( uiWidth == 8 && uiHeight == 8) ) ? 8 : 16) : piQCoef.area();

memset( piQCoef.buf, 0, sizeof(TCoeff) * piQCoef.area() );

for (int uiBlockPos = 0; uiBlockPos < maxNumberOfCoeffs; uiBlockPos++ )
{

const TCoeff iLevel = piCoef.buf[uiBlockPos];
const TCoeff iSign = (iLevel < 0 ? -1: 1);

const int64_t tmpLevel = (int64_t)abs(iLevel) * (enableScalingLists ? piQuantCoeff[uiBlockPos] : defaultQuantisationCoefficient);

const TCoeff quantisedMagnitude = TCoeff((tmpLevel + iAdd ) >> iQBits);
deltaU[uiBlockPos] = (TCoeff)((tmpLevel - ((int64_t)quantisedMagnitude<<iQBits) )>> qBits8);

uiAbsSum += quantisedMagnitude;
const TCoeff quantisedCoefficient = quantisedMagnitude * iSign;

piQCoef.buf[uiBlockPos] = Clip3<TCoeff>( entropyCodingMinimum, entropyCodingMaximum, quantisedCoefficient );

} for n

...

Because the value of maxNumberOfCoeffs is set in coeff scan order, loop operation should be done in coeff scan order instead raster scan order.

-- encoder/decoder log --
EncoderApp -c encoder_randomaccess_vtm.cfg -q 22 -i RaceHorses_416x240_30.yuv -b str.bin --InputBitDepth=8 --InputChromaFormat=420 -fr 30 -wdt 416 -hgt 240 --Level=6.2 -f 2 -fs 0 -ip 32 --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --LFNST=1 --MTS=0 --ISP=0 --DepQuant=0 --RDOQ=0 --RDOQTS=0 --SEIDecodedPictureHash=1 --PrintHexPSNR=1

VVCSoftware: VTM Encoder Version 8.0 [Windows][VS 1900][64 bit] [SIMD=AVX2]

TOOL CFG: IBD:0 HAD:1 RDQ:0 RDQTS:0 RDpenalty:0 LQP:0 SQP:0 ASR:1 MinSearchWindow:96 RestrictMESampling:0 FEN:1 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:1 TransformSkipFast:1 TransformSkipLog2MaxSize:5 ChromaTS:1 BDPCM:0 Tiles: 1x1 Slices: 1 MCTS:0 SAO:1 ALF:1 CCALF:1 WPP:0 WPB:0 PME:2 WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1 DQ:0 SignBitHidingFlag:0 RecalQP:0
TOOL CFG: LFNST:1 MMVD:1 Affine:1 AffineType:1 PROF:1 SubPuMvp:1+0 DualITree:1 IMV:1 BIO:1 LMChroma:1 HorCollocatedChroma:1 VerCollocatedChroma:0 MTS: 0(intra) 0(inter) SBT:1 ISP:0 SMVD:1 CompositeLTReference:0 Bcw:1 BcwFast:1 LADF:0 CIIP:1 Geo:1 AllowDisFracMMVD:1 AffineAmvr:1 AffineAmvrEncOpt:1 DMVR:1 MmvdDisNum:6 JointCbCr:1 ACT:0 PLT:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:1 (Signal:SDR Opt:0 CSoffset:6) MRL:1 MIP:1 EncDbOpt:0
FAST TOOL CFG: LCTUFast:1 FastMrg:1 PBIntraFast:1 IMV4PelFast:1 FastLFNST:0 AMaxBT:1 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 FastMIP:0 FastLocalDualTree:1 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 RPR:0 TemporalFilter:0

started @ Mon Apr 27 17:32:16 2020

POC 0 LId: 0 TId: 0 ( I-SLICE, QP 19 ) 611360 bits [Y 45.6834 dB U 46.4681 dB V 46.8535 dB] [xY 4046d778fff8c814 xU 40473beb1a46161d xV 40476d4105e09f55] [ET 30 ] [L0] [L1] [MD5:41451e12ea536555cb2f19e6c6623776,146bbd4675109aaad3b47a4ef007114b,20be956c0f1e4d55cfc97a6a34468d60]
POC 1 LId: 0 TId: 4 ( B-SLICE, QP 29 ) 173976 bits [Y 36.9931 dB U 36.5246 dB V 37.2789 dB] [xY 40427f1c6074104f xU 40424325b1b2aaea xV 4042a3b1a45349ba] [ET 47 ] [L0 0] [L1 0c] [MD5:afbc700f2f83ac13223bd1d2a27c541d,ef3268ffa4b18d382b86534d571b9fdf,f2feaf0f4204479f2aa9d0b3148be617]

LayerId 0

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR YUV-PSNR xY-PSNR xU-PSNR xV-PSNR

2 a 11780.0400 41.3382 41.4964 42.0662 39.4578 4044ab4ab0366c32 4044bf8865fc6084 404508795519f488

finished @ Mon Apr 27 17:33:34 2020
Total Time: 77.473 sec. [user] 77.473 sec. [elapsed]

DecoderApp -b str.bin

VVCSoftware: VTM Decoder Version 8.0 [Windows][VS 1900][64 bit] [SIMD=AVX2]
POC 0 LId: 0 TId: 0 ( I-SLICE, QP 19 ) [DT 0.052] [L0] [L1] [MD5:41451e12ea536555cb2f19e6c6623776,146bbd4675109aaad3b47a4ef007114b,20be956c0f1e4d55cfc97a6a34468d60,(OK)]
POC 1 LId: 0 TId: 4 ( B-SLICE, QP 29 ) [DT 0.030] [L0 0] [L1 0c] [MD5:31f1e319e40c199ff5e61a5394682e8b,5c9f8c6d4e380e45cef362caff413b4d,2d043b3f5e5d39dbd3a36657b704fed4,(***ERROR***)] [rxMD5:afbc700f2f83ac13223bd1d2a27c541d,ef3268ffa4b18d382b86534d571b9fdf,f2feaf0f4204479f2aa9d0b3148be617]

*ERROR* A decoding mismatch occured: signalled md5sum does not match

Total Time: 0.103 sec.

[tsukuba.takeshi]

Suggested fix is ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1528

[XiangLi]

Thanks for the report. I am using following settings and cannot duplicate the mismatch.

-c ..\cfg\encoder_randomaccess_vtm.cfg -q 22 -i ..\..\..\Sequences\RaceHorses_416x240_30.yuv -b str.bin --InputBitDepth=8 --InputChromaFormat=420 -fr 30 -wdt 416 -hgt 240 --Level=6.2 -f 2 -fs 0 -ip 32 --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --LFNST=1 --MTS=0 --ISP=0 --DepQuant=0 --RDOQ=0 --RDOQTS=0 --SEIDecodedPictureHash=1 --PrintHexPSNR=1

I compared encoding log, the settings look the same but the bits and PSNR I got are different from your log. Could you let me know the correct setting? Thanks.

VVCSoftware: VTM Encoder Version 8.0 [Windows][VS 1915][64 bit] [SIMD=AVX2]

TOOL CFG: IBD:0 HAD:1 RDQ:0 RDQTS:0 RDpenalty:0 LQP:0 SQP:0 ASR:1 MinSearchWindow:96 RestrictMESampling:0 FEN:1 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:1 TransformSkipFast:1 TransformSkipLog2MaxSize:5 ChromaTS:1 BDPCM:0 Tiles: 1x1 Slices: 1 MCTS:0 SAO:1 ALF:1 CCALF:1 WPP:0 WPB:0 PME:2 WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1 DQ:0 SignBitHidingFlag:0 RecalQP:0
TOOL CFG: LFNST:1 MMVD:1 Affine:1 AffineType:1 PROF:1 SubPuMvp:1+0 DualITree:1 IMV:1 BIO:1 LMChroma:1 HorCollocatedChroma:1 VerCollocatedChroma:0 MTS: 0(intra) 0(inter) SBT:1 ISP:0 SMVD:1 CompositeLTReference:0 Bcw:1 BcwFast:1 LADF:0 CIIP:1 Geo:1 AllowDisFracMMVD:1 AffineAmvr:1 AffineAmvrEncOpt:1 DMVR:1 MmvdDisNum:6 JointCbCr:1 ACT:0 PLT:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:1 (Signal:SDR Opt:0 CSoffset:6) MRL:1 MIP:1 EncDbOpt:0
FAST TOOL CFG: LCTUFast:1 FastMrg:1 PBIntraFast:1 IMV4PelFast:1 FastLFNST:0 AMaxBT:1 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 FastMIP:0 FastLocalDualTree:1 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 RPR:0 TemporalFilter:0

started @ Tue Apr 28 10:47:59 2020

POC 0 LId: 0 TId: 0 ( I-SLICE, QP 19 ) 243664 bits [Y 45.6784 dB U 46.3301 dB V 46.7326 dB] [xY 4046d6d6050eb466 xU 40472a3f3f4c8826 xV 40475dc66d23d4f1] [ET 17 ] [L0] [L1] [MD5:ac64ea962c02d4fd227368d6472dfb6f,b0ad4d0d42de4459758c4f0d3add97ec,3c9d809d5fa1b4cca3402778b1dbbe70]
POC 1 LId: 0 TId: 4 ( B-SLICE, QP 29 ) 23744 bits [Y 38.2873 dB U 41.8473 dB V 42.5038 dB] [xY 404324c54a02f2c3 xU 4044ec73c328dacc xV 4045407bc5f7bb0b] [ET 20 ] [L0 0] [L1 0c] [MD5:3a075b8eae873fad2d923043da503d97,957770f4061570c482fa1acf7d8d12b4,9f3819649e22529295b2117f67be3d47]

LayerId 0

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR YUV-PSNR xY-PSNR xU-PSNR xV-PSNR

2 a 4011.1200 41.9828 44.0887 44.6182 41.4085 4044fdcda788d394 40460b59813ab179 40464f21198dc7fe

finished @ Tue Apr 28 10:48:37 2020
Total Time: 37.976 sec. [user] 37.976 sec. [elapsed]

[tsukuba.takeshi]

Sorry for the inconvenience. I used wrong InputChromaFormat setting (--InputChromaFormat=444). In case using correct InputChormaFormat setting (420), our logs matched yours.

Here is another example to duplicate decoder mismatch.

EncoderAppStatic -c cfg-vtm8.0/encoder_randomaccess_vtm.cfg -q 22 -i RaceHorsesC.yuv -b str.bin --InputBitDepth=8 --InputChromaFormat=420 -fr 30 -wdt 832 -hgt 480 --Level=3 -f 17 -fs 0 -ip 32 --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --LFNST=1 --MTS=0 --ISP=0 --DepQuant=0 --RDOQ=0 --RDOQTS=0

-- enc/dec logs --

VVCSoftware: VTM Encoder Version 8.0 [Linux][GCC 6.3.1][64 bit] [SIMD=AVX2]

TOOL CFG: IBD:1 HAD:1 RDQ:0 RDQTS:0 RDpenalty:0 LQP:0 SQP:0 ASR:1 MinSearchWindow:96 RestrictMESampling:0 FEN:1 ECU:0 FDM:1 CFM:0 ESD:0 TransformSkip:1 TransformSkipFast:1 TransformSkipLog2MaxSize:5 ChromaTS:1 BDPCM:0 Tiles: 1x1 Slices: 1 MCTS:0 SAO:1 ALF:1 CCALF:1 WPP:0 WPB:0 PME:2 WaveFrontSynchro:0 WaveFrontSubstreams:1 ScalingList:0 TMVPMode:1 DQ:0 SignBitHidingFlag:0 RecalQP:0
TOOL CFG: LFNST:1 MMVD:1 Affine:1 AffineType:1 PROF:1 SubPuMvp:1+0 DualITree:1 IMV:1 BIO:1 LMChroma:1 HorCollocatedChroma:1 VerCollocatedChroma:0 MTS: 0(intra) 0(inter) SBT:1 ISP:0 SMVD:1 CompositeLTReference:0 Bcw:1 BcwFast:1 LADF:0 CIIP:1 Geo:1 AllowDisFracMMVD:1 AffineAmvr:1 AffineAmvrEncOpt:1 DMVR:1 MmvdDisNum:6 JointCbCr:1 ACT:0 PLT:0 IBC:0 HashME:0 WrapAround:0 LoopFilterAcrossVirtualBoundaries:0 Reshape:1 (Signal:SDR Opt:0 CSoffset:6) MRL:1 MIP:1 EncDbOpt:0
FAST TOOL CFG: LCTUFast:1 FastMrg:1 PBIntraFast:1 IMV4PelFast:1 FastLFNST:0 AMaxBT:1 E0023FastEnc:1 ContentBasedFastQtbt:0 UseNonLinearAlfLuma:1 UseNonLinearAlfChroma:1 MaxNumAlfAlternativesChroma:8 FastMIP:0 FastLocalDualTree:1 NumSplitThreads:1 NumWppThreads:1+0 EnsureWppBitEqual:0 RPR:0 TemporalFilter:0

started @ Wed Apr 29 19:56:40 2020

POC 0 LId: 0 TId: 0 ( I-SLICE, QP 19 ) 764648 bits [Y 45.0160 dB U 45.8678 dB V 46.4454 dB] [xY 4046820d4adf7eb4 xU 4046ef130a1771a4 xV 404739018a47e0b8] [ET 54 ] [L0] [L1] [MD5:c4b17ffa379a5152536c86194cef3098,d51afd9866a34eb90a26141f845ee997,86c7948edde6d492fcd226f2f43297ce]
POC 16 LId: 0 TId: 0 ( B-SLICE, QP 23 ) 408360 bits [Y 41.0229 dB U 43.0661 dB V 44.0449 dB] [xY 404482ede83301be xU 40458875704d83e3 xV 404605bea0cd08fa] [ET 131 ] [L0 0] [L1 0c] [MD5:7d7a95dbedb12fe81400f0b7ab266bfe,aae669e4811ddb7006814fd30bed25c8,f13290d18f415d2d6c2843a61ab9f369]
POC 8 LId: 0 TId: 1 ( B-SLICE, QP 23 ) 389416 bits [Y 40.8622 dB U 42.7752 dB V 43.8647 dB] [xY 40446e5ca98a1146 xU 4045633b3683ce69 xV 4045eeaec300b004] [ET 146 ] [L0 0 16] [L1 16c 0] [MD5:112b928a5a83dfc0a47a395e5c704406,f8770e5377d1cfe4633b9cbe9f631a1a,69a6945da96ed6d5723f4412de6e603e]
POC 4 LId: 0 TId: 2 ( B-SLICE, QP 26 ) 201072 bits [Y 38.9116 dB U 41.5165 dB V 42.8061 dB] [xY 404374afbfe06b60 xU 4044c21cb8fb12bd xV 4045672f0c33090e] [ET 129 ] [L0 0 8] [L1 8c 16]
...
LayerId 0

Total Frames | Bitrate Y-PSNR U-PSNR V-PSNR YUV-PSNR xY-PSNR xU-PSNR xV-PSNR

17 a 5262.3529 38.7282 41.7013 42.9162 39.3404 40435d3683a3675f 4044d9c31474eeec 40457546052edcf3

finished @ Wed Apr 29 20:22:03 2020
Total Time: 1522.570 sec. [user] 1522.743 sec. [elapsed]

VVCSoftware: VTM Decoder Version 8.0 [Linux][GCC 6.3.1][64 bit] [SIMD=AVX2]
POC 0 LId: 0 TId: 0 ( I-SLICE, QP 19 ) [DT 0.070] [L0] [L1] [MD5:c4b17ffa379a5152536c86194cef3098,d51afd9866a34eb90a26141f845ee997,86c7948edde6d492fcd226f2f43297ce,(OK)]
POC 16 LId: 0 TId: 0 ( B-SLICE, QP 23 ) [DT 0.060] [L0 0] [L1 0c] [MD5:47ac63f68ad84ee680d63823f8ffef3f,2b8ebd4c1c075dc68784e2912b2bbadb,4b8a7663b833084e6384a08cba0c130b,(***ERROR***)] [rxMD5:7d7a95dbedb12fe81400f0b7ab266bfe,aae669e4811ddb7006814fd30bed25c8,f13290d18f415d2d6c2843a61ab9f369]
POC 8 LId: 0 TId: 1 ( B-SLICE, QP 23 ) [DT 0.060] [L0 0 16] [L1 16c 0] [MD5:1bffeb89f3a0d8f68446b8cb94bfef6e,6c9c472539c19c2cb1ee1814d1d88a53,2a101e8e78d0a03462c7f595fe9265c9,(***ERROR***)] [rxMD5:112b928a5a83dfc0a47a395e5c704406,f8770e5377d1cfe4633b9cbe9f631a1a,69a6945da96ed6d5723f4412de6e603e]
POC 4 LId: 0 TId: 2 ( B-SLICE, QP 26 ) [DT 0.040] [L0 0 8] [L1 8c 16] [MD5:c14999f56863c7fa9dd265cc8ef9c6ac,bee17ff50621e42addc1662380d670ed,682334ee0077b51c2d847ae342a78f3c,(***ERROR***)] [rxMD5:a5839699bb1cf327cf2b8983e870177d,42a1a3e77fcfe9b824c516cd12d93b89,b9685dfa03c78baee5ac943c27d7b67a]
...
*ERROR* A decoding mismatch occured: signalled md5sum does not match

Total Time: 0.690 sec.

[XiangLi]

Fixed in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1528.