Opened 5 years ago

Closed 5 years ago

#311 closed defect (fixed)

Scaling list bugs (N0847)

Reported by: bheng Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-5.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…


1.) Wrong size index for regular dequant.

File: Quant.cpp
Function: void Quant::dequant(... )

The size index used to lookup the scaling list in the line below is incorrect. The size index should just be uiLog2TrWidth and uiLog2TrHeight. The "- 1" below will give the wrong result.

int *piDequantCoef = getDequantCoeff(scalingListType, QP_rem, uiLog2TrWidth - 1, uiLog2TrHeight - 1);

2.) Scaling factor incorrect for dependent quant.

File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )

When (shift < 0) the following code changes invQScale and shift for the first coefficient. But for all remaining coefficients, shift was already changed to 0, so the same adjustment (invQScale <<= -shift) will not get applied to all the remaining coefficients in the block.

if (enableScalingLists)

invQScale = piDequantCoef[rasterPos];

if (shift < 0)

invQScale <<= -shift;
shift = 0;
add = (1 << shift) >> 1;


3.) 32-bit variable overflow.

File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )

For extreme QP and scaling list values, the following operation can overflow 32-bit integer variables and return the wrong result. It is suggested to type-cast nomTCoeff, qIdx, and invQScale to 64-bit integers for this multiplication. Or possibly pre-clip the values to prevent overflow during the multiplication.

Intermediate_Int nomTCoeff = ( qIdx * invQScale + add ) >> shift;

4.) MTS_SKIP affects quantization scaling of chroma TUs .

File: DepQuant.cpp
Function: void DepQuant::dequant(... )

MTS_SKIP only applies for luma TUs. In the following code, an additional check should be added to make sure the tu.mtsIdx == MTS_SKIP condition is only used with luma TUs.

const bool enableScalingLists = getUseScalingList(width, height, (tu.mtsIdx == MTS_SKIP));

5.) pred_matrix_id_delta for chroma 2x2 needs to be adjusted for missing luma entries.

File: VLCReader.cpp
Function: void HLSyntaxReader::parseScalingList(...)

The matrix delta for Inter 2x2 chroma lists should take into account the fact that there are no luma 2x2 lists. For example, when listId=4, the value pred_matrix_id_delta=1 should point to the previous chroma 2x2 list, not a non-existent luma 2x2 list as it does now.

Change history (3)

comment:1 Changed 5 years ago by bheng

A potential fix is provided with the following merge request:

comment:2 Changed 5 years ago by jennylai

Another bug fix solution is provided with the following merge request:
As a proponent, we suggest to fix bug with 601

comment:3 Changed 5 years ago by bheng

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.