Opened 4 months ago

Closed 3 months ago

Last modified 3 months ago

#420 closed defect (fixed)

Bug on RDOQ when DepQuant=0

Reported by: tsukuba.takeshi Owned by:
Priority: minor Milestone: VTM-6.0
Component: VTM Version: VTM-5.2
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

Segmentation fault occurs at QuantRDOQ.cpp::xRateDistOptQuant when uiLog2BlockWidth=0 or uiLog2BlockHeight=0 under DepQuant=0 as below:

VTM revision=f371d2473ddd5f2247bee2e674f76d506603157a

void QuantRDOQ::xRateDistOptQuant(TransformUnit &tu, ...)
{
...

const double defaultErrorScale = xGetErrScaleCoeffNoScalingList(scalingListType, (uiLog2BlockWidth-1), (uiLog2BlockHeight-1), cQP.rem(isTransformSkip));

...

}

Suggested fix is as below:
void QuantRDOQ::xRateDistOptQuant(TransformUnit &tu, ...)
{
...

const double defaultErrorScale = xGetErrScaleCoeffNoScalingList(scalingListType, uiLog2BlockWidth, uiLog2BlockHeight, cQP.rem(isTransformSkip));

...
}

Change history (3)

comment:1 Changed 3 months ago by ksuehring

  • Milestone VTM-6.0 deleted

comment:2 Changed 3 months ago by ksuehring

  • Milestone set to VTM-6.0
  • Resolution set to fixed
  • Status changed from new to closed

comment:3 Changed 3 months ago by deluxan

MR !861 was also part of the fix.

Note: See TracTickets for help on using tickets.