id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
987	Assertion on min_qp_prime_ts_minus4 failed at decoder when internalBitDepth is set to 8 and inputBitDepth is set to 10	tsukuba.takeshi		"In VTM-8.0, min_qp_prime_ts_minus4 is set to '''(6*(internalBitDepth - inputBitDepth))''' at encoder as below:

{{{
void EncLib::xInitSPS( SPS& sps )
{
...
    sps.setMinQpPrimeTsMinus4(ChannelType(channelType), (6 * (m_bitDepth[channelType] - m_inputBitDepth[channelType])));
...
}
}}}

When internalBitDepth is set to 8 and inputBitDepth is set to 10, the value of min_qp_prime_ts_minus4 becomes '''-12''' (=6*(8-10)), and the signed value of '''-12''' is coded by UVLC as below:

{{{
    WRITE_UVLC(pcSPS->getMinQpPrimeTsMinus4(CHANNEL_TYPE_LUMA),                                ""min_qp_prime_ts_minus4"");
}}}

The decoded value of min_qp_prime_ts_minus4 by UVLC becomes the unsigned value of '''4294967284''' at decoder.

Thus, following assertion on min_qp_prime_ts_minus4 failed at decoder:
{{{
    CHECK(uiCode > 48, ""Invalid min_qp_prime_ts_minus4 signalled"");
}}}

Suggested fix is to change '''inputBitDepth''' in min_qp_prime_ts_minus4 derivation to '''8'''bit as below:
{{{
    sps.setMinQpPrimeTsMinus4(ChannelType(channelType), (6 * (m_bitDepth[channelType] - 8)));
}}}

i.e., min_qp_prime_ts_minus4 is set to '''(6*(internalBitDepth - 8))''' at encoder."	defect	closed	minor	VTM-8.1	VTM	VTM-8.0	fixed		ksuehring XiangLi fbossen jvet@…