missing conformance check on TransCoeffLevel for dependent quantization

[iole_moccagatta]

VTM is missing conformance check on TransCoeffLevel for dependent quantization.

This setting of TransCoeffLevel in the spec residual coding syntax for dependent quantization (sh_dep_quant_used_flag = 1):

TransCoeffLevel[ x0 ][ y0 ][ cIdx ][ xC ][ yC ] =
( 2 * AbsLevel[ xC ][ yC ] − ( QState > 1 ? 1 : 0 ) ) *
( 1 − 2 * coeff_sign_flag[ n ] )

is implemented in line 777 of DepQuant.cpp:

Intermediate_Int qIdx = 2 * level + (level > 0 ? -(state >> 1) : (state >> 1));

TransCoeffLevel must be in the range of CoeffMin to CoeffMax, inclusive.
So, a check on qIdx should be added after line 777 to catch non conformance bitstreams.

[ksuehring]

Fixed in

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2241