Opened 5 years ago
Last modified 5 years ago
#521 new defect
Platform-inconsistent results for palette, RA, qp=2
| Reported by: | gordon | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | VTM | Version: | VTM-6.1 |
| Keywords: | Cc: | vzakharc, yuwenhe, jvet@… |
Description
With qp=2 RA encodes, the calculated qp for intra frames arrives as -1.
In IntraSearch::preCalcPLTIndex and IntraSearch::derivePLTLossy this value is used as an index into a table, which is out of bounds, causing different results on different platforms.
The attached patch resolves the problem, but might be considered a 'band aid' -- there is still the issue of a negative qp value arriving in the code, causing, possibly, other potential issues either now or later.
Should a merge request be made for this patch, or should we wait for a higher level fix?
Attachments (1)
Change history (2)
Changed 5 years ago by gordon
comment:1 in reply to: ↑ description Changed 5 years ago by gordon
- Component changed from 360Lib to VTM
Note: See TracTickets for help on using tickets.
Sorry, this is of course a VTM issue, not a 360lib issue.