Opened 3 months ago

Closed 3 months ago

#512 closed defect (fixed)

Palette crash with kimono-444, singletree, low-delay, lowqp (qp=2), PLT=1

Reported by: gordon Owned by:
Priority: minor Milestone: VTM-6.2
Component: VTM Version: VTM-6.1
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

A 64-bit 'max' limit in IntraSearch::PLTSearch is initialized to a 32-bit value which is too small. When coding the summary line a crash occurs the first time palette is tested because all of the attempts searched have a bits value greater than this max, leaving things uninitialized for later treatment.
Initializing to MAX_UINT64 solves the problem.

diff --git a/source/Lib/EncoderLib/IntraSearch.cpp b/source/Lib/EncoderLib/IntraSearch.cpp                                                                                    index d9c2b82..b3380a9 100644
--- a/source/Lib/EncoderLib/IntraSearch.cpp
+++ b/source/Lib/EncoderLib/IntraSearch.cpp
@@ -1789,7 +1789,7 @@ void IntraSearch::PLTSearch(CodingStructure &cs, Partitioner& partitioner, Compo
   //calculate palette index
   preCalcPLTIndex(cs, partitioner, compBegin, numComp);
   //derive run
-  uint64_t bits = MAX_UINT;
+  uint64_t bits = MAX_UINT64;
   deriveRunAndCalcBits(cs, partitioner, compBegin, numComp, PLT_SCAN_HORTRAV, bits);
   if ((cu.curPLTSize[compBegin] + cu.useEscape[compBegin]) > 1)
   {

Change history (1)

comment:1 Changed 3 months ago by XiangLi

  • Milestone set to VTM-6.2
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.