Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#267 closed defect (fixed)

Wrong SAD calculation in xGetSAD_NxN_SIMD(...) when size is 4x4

Reported by: chiaming Owned by:
Priority: minor Milestone: VTM-5.1
Component: VTM Version: VTM-5.0rc1
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In RdCost::xGetSAD_NxN_SIMD(...), if rcDtParam.subShift is not equal to zero and size is equal to 4x4, then the SAD calculation dose not consider the "subsampling" of SAD accumulation.
It is suggested to fix the code as followings:


template< int iWidth, X86_VEXT vext >
Distortion RdCost::xGetSAD_NxN_SIMD( const DistParam &rcDtParam )
{

...
if( iWidth == 4 )
{

if( iRows == 4 && iSubShift == 0)
{
}
...

}
...

}


Change history (4)

comment:1 Changed 3 years ago by XiangLi

Thanks for the report. Could you submit a merge request for the fix? The fix has no impact on CTC, right?

comment:2 Changed 3 years ago by chiaming

Yes, the fix does not change the CTC results. I will submit a merge request later. Thank you.

comment:3 Changed 3 years ago by XiangLi

  • Resolution set to fixed
  • Status changed from new to closed

comment:4 Changed 3 years ago by XiangLi

  • Milestone set to VTM-5.1
Note: See TracTickets for help on using tickets.