id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
311	Scaling list bugs (N0847)	bheng		"'''1.) Wrong size index for regular dequant.'''

File: Quant.cpp
Function: void Quant::dequant(... )

The size index used to lookup the scaling list in the line below is incorrect. The size index should just be uiLog2TrWidth and uiLog2TrHeight.  The ""- 1"" below will give the wrong result.

int *piDequantCoef        = getDequantCoeff(scalingListType, QP_rem, uiLog2TrWidth - 1, uiLog2TrHeight - 1);


'''2.) Scaling factor incorrect for dependent quant.'''

File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )

When (shift < 0) the following code changes invQScale and shift for the '''first''' coefficient.  But for all remaining coefficients, shift was already changed to 0, so the same adjustment (invQScale <<= -shift) will not get applied to all the remaining coefficients in the block. 

    if (enableScalingLists)
        invQScale = piDequantCoef[rasterPos];
    if (shift < 0)
    {
        invQScale <<= -shift;
        shift = 0;
        //add = (1 << shift) >> 1;
    }


'''3.) 32-bit variable overflow.'''

File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )

For extreme QP and scaling list values, the following operation can overflow 32-bit integer variables and return the wrong result.  It is suggested to type-cast nomTCoeff, qIdx, and invQScale to 64-bit integers for this multiplication. Or possibly pre-clip the values to prevent overflow during the multiplication.

    Intermediate_Int  nomTCoeff = ( qIdx * invQScale + add ) >> shift;
       
       
'''4.) MTS_SKIP affects quantization scaling of chroma TUs .'''

File: DepQuant.cpp
Function: void DepQuant::dequant(... )

MTS_SKIP only applies for luma TUs.  In the following code, an additional check should be added to make sure the tu.mtsIdx == MTS_SKIP condition is only used with luma TUs.

    const bool enableScalingLists = getUseScalingList(width, height, (tu.mtsIdx == MTS_SKIP));


'''5.) pred_matrix_id_delta for chroma 2x2 needs to be adjusted for missing luma entries.'''

File: VLCReader.cpp
Function: void HLSyntaxReader::parseScalingList(...)

The matrix delta for Inter 2x2 chroma lists should take into account the fact that there are no luma 2x2 lists.  For example, when listId=4, the value pred_matrix_id_delta=1 should point to the previous chroma 2x2 list, not a non-existent luma 2x2 list as it does now.

"	defect	closed	minor		VTM	VTM-5.0	fixed		ksuehring XiangLi fbossen jvet@…