Assertion on min_qp_prime_ts_minus4 failed at decoder when internalBitDepth is set to 8 and inputBitDepth is set to 10

[tsukuba.takeshi]

In VTM-8.0, min_qp_prime_ts_minus4 is set to (6*(internalBitDepth - inputBitDepth)) at encoder as below:

void EncLib::xInitSPS( SPS& sps )
{
...
    sps.setMinQpPrimeTsMinus4(ChannelType(channelType), (6 * (m_bitDepth[channelType] - m_inputBitDepth[channelType])));
...
}

When internalBitDepth is set to 8 and inputBitDepth is set to 10, the value of min_qp_prime_ts_minus4 becomes -12 (=6*(8-10)), and the signed value of -12 is coded by UVLC as below:

    WRITE_UVLC(pcSPS->getMinQpPrimeTsMinus4(CHANNEL_TYPE_LUMA),                                "min_qp_prime_ts_minus4");

The decoded value of min_qp_prime_ts_minus4 by UVLC becomes the unsigned value of 4294967284 at decoder.

Thus, following assertion on min_qp_prime_ts_minus4 failed at decoder:

    CHECK(uiCode > 48, "Invalid min_qp_prime_ts_minus4 signalled");

Suggested fix is to change inputBitDepth in min_qp_prime_ts_minus4 derivation to 8bit as below:

    sps.setMinQpPrimeTsMinus4(ChannelType(channelType), (6 * (m_bitDepth[channelType] - 8)));

i.e., min_qp_prime_ts_minus4 is set to (6*(internalBitDepth - 8)) at encoder.

[tsukuba.takeshi]

Suggested MR is ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1481

[XiangLi]

Thanks for the report. It might be better to clarify why internalBitDepth is set to 8 given inputBitDepth is 10? We are losing quality before encoding. In this case, should we print out an error message and stop the encoder to avoid confusion?

[yhchao]

Hi Xiang,
If the derivation is changed to (6*(internalBitDepth - 8)), it will cause quality loss at low QP if the inputBitDepth is 10. The detail can be found in our proposal JVET-O0919. I think the solution you provide, i.e., printing error message, makes sense. Another solution is to clip min_qp_prime_ts_minus4 to 0 when it goes to negative.

[fbossen]

It should be possible to encode a 10-bit source with 8-bit precision (or alternatively a 12-bit source with 10-bit precision). We shouldn't disallow such a combination. Printing a warning message may be helpful. Clipping min_qp_prime_ts_minus4 to 0 at the low end seems to make sense.

[tsukuba.takeshi]

Thanks a lot for your feedback.

I agree that clipping min_qp_prime_ts_minus4 to 0 at the low end is a better solution since it has no impacts to other conditions "inputBitDepth>=internalBitDepth".
So, I will withdraw MR!1481 and create a new MR including:

• Clipping min_qp_prime_ts_minus4 to 0 at the low end
• Print a warning message when intenalBitDepth is lower than inputBitDepth at xCheckParameter()

[tsukuba.takeshi]

Please take a loot at new MR!1483:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1483

[tsukuba.takeshi]

MR!1483 was already merged.