Opened 5 years ago

Last modified 5 years ago

#521 new defect

Platform-inconsistent results for palette, RA, qp=2

Reported by: gordon Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-6.1
Keywords: Cc: vzakharc, yuwenhe, jvet@…


With qp=2 RA encodes, the calculated qp for intra frames arrives as -1.

In IntraSearch::preCalcPLTIndex and IntraSearch::derivePLTLossy this value is used as an index into a table, which is out of bounds, causing different results on different platforms.

The attached patch resolves the problem, but might be considered a 'band aid' -- there is still the issue of a negative qp value arriving in the code, causing, possibly, other potential issues either now or later.

Should a merge request be made for this patch, or should we wait for a higher level fix?

Attachments (1)

ticket-pltqp.patch (1.0 KB) - added by gordon 5 years ago.

Download all attachments as: .zip

Change history (2)

Changed 5 years ago by gordon

comment:1 in reply to: ↑ description Changed 5 years ago by gordon

  • Component changed from 360Lib to VTM

Sorry, this is of course a VTM issue, not a 360lib issue.

Note: See TracTickets for help on using tickets.