#267 closed defect (fixed)
Wrong SAD calculation in xGetSAD_NxN_SIMD(...) when size is 4x4
| Reported by: | chiaming | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | VTM-5.1 |
| Component: | VTM | Version: | VTM-5.0rc1 |
| Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
In RdCost::xGetSAD_NxN_SIMD(...), if rcDtParam.subShift is not equal to zero and size is equal to 4x4, then the SAD calculation dose not consider the "subsampling" of SAD accumulation.
It is suggested to fix the code as followings:
template< int iWidth, X86_VEXT vext >
Distortion RdCost::xGetSAD_NxN_SIMD( const DistParam &rcDtParam )
{
...
if( iWidth == 4 )
{
if( iRows == 4 && iSubShift == 0)
{
}
...
}
...
}
Change history (4)
comment:1 Changed 5 years ago by XiangLi
comment:2 Changed 5 years ago by chiaming
Yes, the fix does not change the CTC results. I will submit a merge request later. Thank you.
comment:3 Changed 5 years ago by XiangLi
- Resolution set to fixed
- Status changed from new to closed
comment:4 Changed 5 years ago by XiangLi
- Milestone set to VTM-5.1
Note: See TracTickets for help on using tickets.
Thanks for the report. Could you submit a merge request for the fix? The fix has no impact on CTC, right?