Opened 3 years ago
Closed 3 years ago
#1541 closed defect (fixed)
remRegBins calculation in xRateDistOptQuant looks not correct for 64x64 transform
| Reported by: | XiangLi | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | VTM-16.1 |
| Component: | VTM | Version: | VTM-16.0 |
| Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
In VTM-16.0, function xRateDistOptQuant() (when depQuant off), remRegBins calculation looks not correct for 64x64 transform. The code in current master is as follows
int remRegBins = (uiWidth * uiHeight * ctxBinSampleRatio) >> 4;
When we have 64x64 transform, zeroout has to be applied. It looks not correct to always use uiWidth and uiHeight here.
In function CABACWriter::residual_coding, we are using
cctx.regBinLimit = (tu.getTbAreaAfterCoefZeroOut(compID) * ctxBinSampleRatio) >> 4;
Using tu.getTbAreaAfterCoefZeroOut(compID) instead of "uiWidth * uiHeight" looks correct.
To summarize, the proposed change
int remRegBins = (uiWidth * uiHeight * ctxBinSampleRatio) >> 4;
=>
int remRegBins = (tu.getTbAreaAfterCoefZeroOut(compID) * ctxBinSampleRatio) >> 4;
Change history (1)
comment:1 Changed 3 years ago by XiangLi
- Milestone set to VTM-16.1
- Resolution set to fixed
- Status changed from new to closed
Fixed in https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2197