id summary reporter owner description type status priority milestone component version resolution keywords cc 311 Scaling list bugs (N0847) bheng "'''1.) Wrong size index for regular dequant.''' File: Quant.cpp Function: void Quant::dequant(... ) The size index used to lookup the scaling list in the line below is incorrect. The size index should just be uiLog2TrWidth and uiLog2TrHeight. The ""- 1"" below will give the wrong result. int *piDequantCoef = getDequantCoeff(scalingListType, QP_rem, uiLog2TrWidth - 1, uiLog2TrHeight - 1); '''2.) Scaling factor incorrect for dependent quant.''' File: DepQuant.cpp Function: void Quantizer::dequantBlock(... ) When (shift < 0) the following code changes invQScale and shift for the '''first''' coefficient. But for all remaining coefficients, shift was already changed to 0, so the same adjustment (invQScale <<= -shift) will not get applied to all the remaining coefficients in the block. if (enableScalingLists) invQScale = piDequantCoef[rasterPos]; if (shift < 0) { invQScale <<= -shift; shift = 0; //add = (1 << shift) >> 1; } '''3.) 32-bit variable overflow.''' File: DepQuant.cpp Function: void Quantizer::dequantBlock(... ) For extreme QP and scaling list values, the following operation can overflow 32-bit integer variables and return the wrong result. It is suggested to type-cast nomTCoeff, qIdx, and invQScale to 64-bit integers for this multiplication. Or possibly pre-clip the values to prevent overflow during the multiplication. Intermediate_Int nomTCoeff = ( qIdx * invQScale + add ) >> shift; '''4.) MTS_SKIP affects quantization scaling of chroma TUs .''' File: DepQuant.cpp Function: void DepQuant::dequant(... ) MTS_SKIP only applies for luma TUs. In the following code, an additional check should be added to make sure the tu.mtsIdx == MTS_SKIP condition is only used with luma TUs. const bool enableScalingLists = getUseScalingList(width, height, (tu.mtsIdx == MTS_SKIP)); '''5.) pred_matrix_id_delta for chroma 2x2 needs to be adjusted for missing luma entries.''' File: VLCReader.cpp Function: void HLSyntaxReader::parseScalingList(...) The matrix delta for Inter 2x2 chroma lists should take into account the fact that there are no luma 2x2 lists. For example, when listId=4, the value pred_matrix_id_delta=1 should point to the previous chroma 2x2 list, not a non-existent luma 2x2 list as it does now. " defect closed minor VTM VTM-5.0 fixed ksuehring XiangLi fbossen jvet@…