Opened 5 years ago
Closed 5 years ago
#311 closed defect (fixed)
Scaling list bugs (N0847)
Reported by: | bheng | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | VTM | Version: | VTM-5.0 |
Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
1.) Wrong size index for regular dequant.
File: Quant.cpp
Function: void Quant::dequant(... )
The size index used to lookup the scaling list in the line below is incorrect. The size index should just be uiLog2TrWidth and uiLog2TrHeight. The "- 1" below will give the wrong result.
int *piDequantCoef = getDequantCoeff(scalingListType, QP_rem, uiLog2TrWidth - 1, uiLog2TrHeight - 1);
2.) Scaling factor incorrect for dependent quant.
File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )
When (shift < 0) the following code changes invQScale and shift for the first coefficient. But for all remaining coefficients, shift was already changed to 0, so the same adjustment (invQScale <<= -shift) will not get applied to all the remaining coefficients in the block.
if (enableScalingLists)
invQScale = piDequantCoef[rasterPos];
if (shift < 0)
{
invQScale <<= -shift;
shift = 0;
add = (1 << shift) >> 1;
}
3.) 32-bit variable overflow.
File: DepQuant.cpp
Function: void Quantizer::dequantBlock(... )
For extreme QP and scaling list values, the following operation can overflow 32-bit integer variables and return the wrong result. It is suggested to type-cast nomTCoeff, qIdx, and invQScale to 64-bit integers for this multiplication. Or possibly pre-clip the values to prevent overflow during the multiplication.
Intermediate_Int nomTCoeff = ( qIdx * invQScale + add ) >> shift;
4.) MTS_SKIP affects quantization scaling of chroma TUs .
File: DepQuant.cpp
Function: void DepQuant::dequant(... )
MTS_SKIP only applies for luma TUs. In the following code, an additional check should be added to make sure the tu.mtsIdx == MTS_SKIP condition is only used with luma TUs.
const bool enableScalingLists = getUseScalingList(width, height, (tu.mtsIdx == MTS_SKIP));
5.) pred_matrix_id_delta for chroma 2x2 needs to be adjusted for missing luma entries.
File: VLCReader.cpp
Function: void HLSyntaxReader::parseScalingList(...)
The matrix delta for Inter 2x2 chroma lists should take into account the fact that there are no luma 2x2 lists. For example, when listId=4, the value pred_matrix_id_delta=1 should point to the previous chroma 2x2 list, not a non-existent luma 2x2 list as it does now.
Change history (3)
comment:1 Changed 5 years ago by bheng
comment:2 Changed 5 years ago by jennylai
Another bug fix solution is provided with the following merge request:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/601
As a proponent, we suggest to fix bug with 601
comment:3 Changed 5 years ago by bheng
- Resolution set to fixed
- Status changed from new to closed
Fixed with merge request:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/601
A potential fix is provided with the following merge request:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/594