#420 closed defect (fixed)
Bug on RDOQ when DepQuant=0
| Reported by: | tsukuba.takeshi | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | VTM-6.0 |
| Component: | VTM | Version: | VTM-5.2 |
| Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
Segmentation fault occurs at QuantRDOQ.cpp::xRateDistOptQuant when uiLog2BlockWidth=0 or uiLog2BlockHeight=0 under DepQuant=0 as below:
VTM revision=f371d2473ddd5f2247bee2e674f76d506603157a
void QuantRDOQ::xRateDistOptQuant(TransformUnit &tu, ...)
{
...
const double defaultErrorScale = xGetErrScaleCoeffNoScalingList(scalingListType, (uiLog2BlockWidth-1), (uiLog2BlockHeight-1), cQP.rem(isTransformSkip));
...
}
Suggested fix is as below:
void QuantRDOQ::xRateDistOptQuant(TransformUnit &tu, ...)
{
...
const double defaultErrorScale = xGetErrScaleCoeffNoScalingList(scalingListType, uiLog2BlockWidth, uiLog2BlockHeight, cQP.rem(isTransformSkip));
...
}
Change history (3)
comment:1 Changed 5 years ago by ksuehring
- Milestone VTM-6.0 deleted
comment:2 Changed 5 years ago by ksuehring
- Milestone set to VTM-6.0
- Resolution set to fixed
- Status changed from new to closed
comment:3 Changed 5 years ago by deluxan
MR !861 was also part of the fix.
Note: See TracTickets for help on using tickets.
Fixed in
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/838