Opened 5 years ago

Closed 5 years ago

#305 closed defect (fixed)

MTS candidate ordering if LFNST is used

Reported by: chollmann Owned by:
Priority: minor Milestone: VTM-5.1
Component: VTM Version: VTM-5.0
Keywords: LFNST, MTS Cc: ksuehring, XiangLi, fbossen, jvet@…


When using LFNST, the encoder evaluates the MTS candidates (DST7, DCT8) and (DCT8, DST7) in an order based on the intra mode:

IntraSearch.cpp line 2919

                | first        | second
intraMode <  34 | (DST7, DCT8) | (DCT8, DST7)
intraMode >= 34 | (DCT8, DST7) | (DST7, DCT8)

This is done if a bool variable moreProbMTSIdxFirst is true. This bool is set based on whether the transform index is larger than 1 (see EncCu.cpp line 1459). However, transform index = 1 (which is (DCT8, DST7)) is one of the modes that are supposed to be sorted.

The fix corrects the comparison to transform index > 0.

Change history (2)

comment:1 Changed 5 years ago by ksuehring

Merge request 589 was submitted to address the issue.

comment:2 Changed 5 years ago by XiangLi

  • Milestone set to VTM-5.1
  • Resolution set to fixed
  • Status changed from new to closed

Fixed as suggested.

Note: See TracTickets for help on using tickets.