Opened 3 years ago

Closed 3 years ago

#1541 closed defect (fixed)

remRegBins calculation in xRateDistOptQuant looks not correct for 64x64 transform

Reported by: XiangLi Owned by:
Priority: minor Milestone: VTM-16.1
Component: VTM Version: VTM-16.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In VTM-16.0, function xRateDistOptQuant() (when depQuant off), remRegBins calculation looks not correct for 64x64 transform. The code in current master is as follows

int remRegBins = (uiWidth * uiHeight * ctxBinSampleRatio) >> 4;

When we have 64x64 transform, zeroout has to be applied. It looks not correct to always use uiWidth and uiHeight here.

In function CABACWriter::residual_coding, we are using
cctx.regBinLimit = (tu.getTbAreaAfterCoefZeroOut(compID) * ctxBinSampleRatio) >> 4;

Using tu.getTbAreaAfterCoefZeroOut(compID) instead of "uiWidth * uiHeight" looks correct.

To summarize, the proposed change
int remRegBins = (uiWidth * uiHeight * ctxBinSampleRatio) >> 4;
=>
int remRegBins = (tu.getTbAreaAfterCoefZeroOut(compID) * ctxBinSampleRatio) >> 4;

Change history (1)

comment:1 Changed 3 years ago by XiangLi

  • Milestone set to VTM-16.1
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.