Scaling list bugs (N0847)

[bheng]

1.) Wrong size index for regular dequant.

File: Quant.cpp
Function: void Quant::dequant(... )

The size index used to lookup the scaling list in the line below is incorrect. The size index should just be uiLog2TrWidth and uiLog2TrHeight. The "- 1" below will give the wrong result.

int *piDequantCoef = getDequantCoeff(scalingListType, QP_rem, uiLog2TrWidth - 1, uiLog2TrHeight - 1);

2.) Scaling factor incorrect for dependent quant.

File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )

When (shift < 0) the following code changes invQScale and shift for the first coefficient. But for all remaining coefficients, shift was already changed to 0, so the same adjustment (invQScale <<= -shift) will not get applied to all the remaining coefficients in the block.

if (enableScalingLists)

invQScale = piDequantCoef[rasterPos];

if (shift < 0)
{

invQScale <<= -shift;
shift = 0;
add = (1 << shift) >> 1;

}

3.) 32-bit variable overflow.

File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )

For extreme QP and scaling list values, the following operation can overflow 32-bit integer variables and return the wrong result. It is suggested to type-cast nomTCoeff, qIdx, and invQScale to 64-bit integers for this multiplication. Or possibly pre-clip the values to prevent overflow during the multiplication.

Intermediate_Int nomTCoeff = ( qIdx * invQScale + add ) >> shift;

4.) MTS_SKIP affects quantization scaling of chroma TUs .

File: DepQuant.cpp
Function: void DepQuant::dequant(... )

MTS_SKIP only applies for luma TUs. In the following code, an additional check should be added to make sure the tu.mtsIdx == MTS_SKIP condition is only used with luma TUs.

const bool enableScalingLists = getUseScalingList(width, height, (tu.mtsIdx == MTS_SKIP));

5.) pred_matrix_id_delta for chroma 2x2 needs to be adjusted for missing luma entries.

File: VLCReader.cpp
Function: void HLSyntaxReader::parseScalingList(...)

The matrix delta for Inter 2x2 chroma lists should take into account the fact that there are no luma 2x2 lists. For example, when listId=4, the value pred_matrix_id_delta=1 should point to the previous chroma 2x2 list, not a non-existent luma 2x2 list as it does now.

[bheng]

A potential fix is provided with the following merge request:

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/594

[jennylai]

Another bug fix solution is provided with the following merge request:
​​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/601
As a proponent, we suggest to fix bug with 601

[bheng]

Fixed with merge request:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/601