Opened 6 years ago

Closed 6 years ago

#330 closed defect (fixed)

Bugs in transform-skip RDOQ

Reported by: wanght99 Owned by:
Priority: minor Milestone: VTM-5.1
Component: VTM Version: VTM-5.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

Check of last CG in RDOQ

In RDOQ, the code to check whether the last CG is being processed used the variable 'sbSizeM1', which indicates the scan index of the last coefficient within a CG.

Related code: 

  else if( sbId != sbSizeM1 || anySigCG )
  {
    if( rdStats.iNNZbeforePos0 == 0 )
    {
      baseCost -= rdStats.d64SigCost_0;
      rdStats.d64SigCost -= rdStats.d64SigCost_0;
    }
    ...

Setting a CG to 0 in RDOQ

In RDOQ, the following code sets all coefficients within a CG to be 0. But according to the loop condition, the last coefficient ( with the index of sbSizeM1 ) is untouched.

Related code: 

      for( int scanPosInSB = 0; scanPosInSB < sbSizeM1; scanPosInSB++ )
      {
        scanPos = cctx.minSubPos() + scanPosInSB;
        uint32_t blkPos = cctx.blockPos( scanPos );

        if( dstCoeff[ blkPos ] )
        {
          dstCoeff[ blkPos ] = 0;
          costCoeff[ scanPos ] = costCoeff0[ scanPos ];
          costSig[ scanPos] = 0;
        }
      }

Handling of the last coefficient of a CG

A wrong position was considered as the last coefficient

The RDOQ code assumes that when the first coefficient is the only non-zero coefficient within a CG, the sigFlag of that coefficient is not encoded. But it should be the last coefficient instead of the first one that may have this special case.

The RDOQ for the last coefficient is inaccurate

The parameter isLast of xGetCodedLevelTS() was always set to 0, so inside xGetCodedLevelTS(), the cost of sigFlag will always be considered. Wrong output can be produced by such a comparison.

Related code: 

if( !isLast && maxAbsLevel < 3 )
{
  codedCostSig = xGetRateSigCoef( *fracBitsSig, 0 );
  codedCost    = codedCost0 + codedCostSig;
  if( maxAbsLevel == 0 )
  {
    return bestAbsLevel;
  }
}
else
{
  codedCost = MAX_DOUBLE;
}

if( !isLast )
{
  currCostSig = xGetRateSigCoef( *fracBitsSig, 1 );
}

Change history (1)

comment:1 Changed 6 years ago by XiangLi

  • Milestone set to VTM-5.1
  • Resolution set to fixed
  • Status changed from new to closed

Fixed in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/613

Note: See TracTickets for help on using tickets.