Floating-point QP not working

[yyasugi]

Version: HM-16.6-JEM-4.0

I noticed encoding failed with floating point QP in most cases.
The encoder was terminated by accessing an array with index -1.

It would seem that NUM_QP_PROB is not enough and TComStats::aaQPUsed runs out of space during encoding.

The issue seems to be fixed by increasing JVET_B0039_INC_NUM_QP_PROB from 7 to 14 as its comment says, but I'm not sure what value is appropriate.

---

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000004b5b36 in TEncSbac::loadContextsFromPrev (this=0x7ffe173f3980, apcStats=0x83b8670, eSliceType=B_SLICE, iQPIdx=-1, bFromGloble=false, iQPIdxRst=-1, bAfterLastISlice=false)

at /.../source/Lib/TLibEncoder/TEncSbac.cpp:3273

#1 0x0000000000456a2e in TEncSlice::encodeSlice (this=0x7ffe173f66e0, pcPic=0x2ba23b668010, pcSubstreams=0x11902a80, numBinsCoded=@0x7ffe173d0680: 0, alfParam=...)

at /.../source/Lib/TLibEncoder/TEncSlice.cpp:1290

[sier]

I notice that when we use the switch DeltaQpRD, the total number of appeared QP values will be larger than 7, and this will cause that TComStats::aaQPUsed runs out of space during encoding.

In order to solve this problem, it is necessary to take every QP value into consideration during the encoding process, especially for muliti QP optimization. Therefore, the parameter JVET_B0039_INC_NUM_QP_PROB should be set as 52.

I have tested the performance by setting JVET_B0039_INC_NUM_QP_PRO as 52,and the results show that there is no luma nor chroma BD-rate reduction over JEM-3.0 for RA cases with neglected encoding time increase.

[lizhang]

The total number of used QP values is changed to the maximum number of positive QP values allowed in HEVC, i.e., JVET_B0039_INC_NUM_QP_PRO is changed from 7 to 52. In addition, when JVET_B0039_INC_NUM_QP_PROB is not used, the macro NUM_QP_PROB is also modified to be 52 for potential non-CTC tests.

[XiangLi]

Fixed in r354.