Encoder may produce uncompliant AlfCoeff for Chroma

[fbarbier]

Accordingly to specification, AlfCoeffC shall be in the range of −127 to 127, inclusive.

Please find a related merge request that checks AlfCoeff values for Luma and Chroma:

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/570

The following command lines show that the current VTM encoder may produce invalid bitstreams :

git fetch ​https://vcgit.hhi.fraunhofer.de/fbarbier/VVCSoftware_VTM.git alf_compliance_check
git checkout -b fbarbier/VVCSoftware_VTM-alf_compliance_check1 FETCH_HEAD

cd build
cmake ..
make -j 8
cd ..

$ ./bin/EncoderAppStaticd --SourceWidth=144 --SourceHeight=192 --InputChromaFormat=420 --InputBitDepth=8 --Profile=next --FrameRate=60 --FramesToBeEncoded=1 --GOPSize=4 '--Frame1=B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -5 -9 -13 0' '--Frame2=B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -2 -6 -10 1 -1 5 1 1 1 0 1' '--Frame3=B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -3 -7 -11 1 -1 5 0 1 1 1 1' '--Frame4=B 4 1 0.0 0.0 0 0 1.0 0 0 0 4 4 -1 -4 -8 -12 1 -1 5 0 1 1 1 1' --FastSearch=1 --MaxCUWidth=12 --MaxCUHeight=16 --CTUSize=128 --MaxPartitionDepth=5 --PCMEnabledFlag=0 --SAO=0 --LoopFilterDisable=1 --ALF=1 --DisableIntraInInter=0 --DepQuant=0 --SignHideFlag=0 --DualITree=1 --LMChroma=1 --TransformSkip=0 --MaxDeltaQP=0 --MTS=2 --Triangle=1 --RDPCM=1 -i alf_source.yuv -b alf_compliance_issue.vvc

$ ./bin/DecoderAppStaticd -b alf_compliance_issue.vvc -o decoded.yuv

ERROR: In function "alfFilter" in source/Lib/DecoderLib/VLCReader.cpp:3084: AlfCoeffC shall be in the range of −127 to 127, inclusive

[fbarbier]

Please find another failure with the following command lines :

git fetch ​https://vcgit.hhi.fraunhofer.de/fbarbier/VVCSoftware_VTM.git alf_compliance_check
git checkout -b fbarbier/VVCSoftware_VTM-alf_compliance_check1 FETCH_HEAD

cd build
cmake ..
make
cd ..

./bin/EncoderAppStaticd --SourceWidth=256 --SourceHeight=88 --InputChromaFormat=420 --InputBitDepth=8 --Profile=next --FrameRate=60 --FramesToBeEncoded=2 --GOPSize=4 '--Frame1=B 1 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -5 -9 -13 0' '--Frame2=B 2 4 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -2 -6 -10 1 -1 5 1 1 1 0 1' '--Frame3=B 3 5 -6.5 0.2590 0 0 1.0 0 0 0 4 4 -1 -3 -7 -11 1 -1 5 0 1 1 1 1' '--Frame4=B 4 1 0.0 0.0 0 0 1.0 0 0 0 4 4 -1 -4 -8 -12 1 -1 5 0 1 1 1 1' --FastSearch=1 --MaxCUWidth=16 --MaxCUHeight=24 --CTUSize=128 --MaxPartitionDepth=5 --PCMEnabledFlag=1 --SAO=1 --LoopFilterDisable=1 --ALF=1 --DisableIntraInInter=0 --DepQuant=0 --SignHideFlag=0 --DualITree=1 --LMChroma=0 --TransformSkip=1 --MaxDeltaQP=0 --MTS=3 --IBC=1 --IMV=1 --Affine=1 --AffineType=0 --AffineAmvr=1 --MHIntra=1 --Triangle=1 --MIP=1 --SMVD=0 --SBT=1 --RDPCM=1 -i alf_compliance_check1.yuv -b alf_compliance_issue.vvc

./bin/DecoderAppStaticd -b alf_compliance_issue.vvc -o decoded.yuv

ERROR: In function "alfFilter" in source/Lib/DecoderLib/VLCReader.cpp:3084: AlfCoeffC shall be in the range of −127 to 127, inclusive

[jennylai]

A potential fix is provided as the following shown:

In EncAdaptiveLoopFilter.cpp,
double EncAdaptiveLoopFilter::deriveCoeffQuant
{
...

  const int min_value = -factor + 1;

...
}

[fbarbier]

Thanks jennylai for answering. I'll test it and let you know.

Do the merge request ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/570
seems ok for you as a compliance check ?

[fbarbier]

Please find the merge request suggested by jennylai :
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/604

[ksuehring]

MR 604 was merged.

It's still open if MR 570 (compliance check) should be merged. I think it might be useful. But I would prefer, if somebody could check, that this does not fail in CTC.

[fbarbier]

Thanks Karsten. I agree with you, adding the compliance check should be merged. On my side, my tests are OK, but it would be good if anybody else may check.