Opened 5 years ago

Closed 5 years ago

#987 closed defect (fixed)

Assertion on min_qp_prime_ts_minus4 failed at decoder when internalBitDepth is set to 8 and inputBitDepth is set to 10

Reported by: tsukuba.takeshi Owned by:
Priority: minor Milestone: VTM-8.1
Component: VTM Version: VTM-8.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In VTM-8.0, min_qp_prime_ts_minus4 is set to (6*(internalBitDepth - inputBitDepth)) at encoder as below:

void EncLib::xInitSPS( SPS& sps )
{
...
    sps.setMinQpPrimeTsMinus4(ChannelType(channelType), (6 * (m_bitDepth[channelType] - m_inputBitDepth[channelType])));
...
}

When internalBitDepth is set to 8 and inputBitDepth is set to 10, the value of min_qp_prime_ts_minus4 becomes -12 (=6*(8-10)), and the signed value of -12 is coded by UVLC as below:

    WRITE_UVLC(pcSPS->getMinQpPrimeTsMinus4(CHANNEL_TYPE_LUMA),                                "min_qp_prime_ts_minus4");

The decoded value of min_qp_prime_ts_minus4 by UVLC becomes the unsigned value of 4294967284 at decoder.

Thus, following assertion on min_qp_prime_ts_minus4 failed at decoder:

    CHECK(uiCode > 48, "Invalid min_qp_prime_ts_minus4 signalled");

Suggested fix is to change inputBitDepth in min_qp_prime_ts_minus4 derivation to 8bit as below:

    sps.setMinQpPrimeTsMinus4(ChannelType(channelType), (6 * (m_bitDepth[channelType] - 8)));

i.e., min_qp_prime_ts_minus4 is set to (6*(internalBitDepth - 8)) at encoder.

Change history (7)

comment:2 Changed 5 years ago by XiangLi

Thanks for the report. It might be better to clarify why internalBitDepth is set to 8 given inputBitDepth is 10? We are losing quality before encoding. In this case, should we print out an error message and stop the encoder to avoid confusion?

comment:3 Changed 5 years ago by yhchao

Hi Xiang,
If the derivation is changed to (6*(internalBitDepth - 8)), it will cause quality loss at low QP if the inputBitDepth is 10. The detail can be found in our proposal JVET-O0919. I think the solution you provide, i.e., printing error message, makes sense. Another solution is to clip min_qp_prime_ts_minus4 to 0 when it goes to negative.

comment:4 Changed 5 years ago by fbossen

It should be possible to encode a 10-bit source with 8-bit precision (or alternatively a 12-bit source with 10-bit precision). We shouldn't disallow such a combination. Printing a warning message may be helpful. Clipping min_qp_prime_ts_minus4 to 0 at the low end seems to make sense.

comment:5 Changed 5 years ago by tsukuba.takeshi

Thanks a lot for your feedback.

I agree that clipping min_qp_prime_ts_minus4 to 0 at the low end is a better solution since it has no impacts to other conditions "inputBitDepth>=internalBitDepth".
So, I will withdraw MR!1481 and create a new MR including:

  • Clipping min_qp_prime_ts_minus4 to 0 at the low end
  • Print a warning message when intenalBitDepth is lower than inputBitDepth at xCheckParameter()

comment:6 Changed 5 years ago by tsukuba.takeshi

comment:7 Changed 5 years ago by tsukuba.takeshi

  • Resolution set to fixed
  • Status changed from new to closed

MR!1483 was already merged.

Note: See TracTickets for help on using tickets.