Performance of IBDI = 12 is worse than IBDI = 10 on top of VTM-6

[leolzhao]

When IBDI is set equal to 12 on top of VTM-6, there is 1.68% and 1.33% luma loss for class B and class C with RA configuration. This result is inconsistent with the result on top of VTM-5. In VTM-5, set IBDI = 12 brings minor coding gain when compared to IBDI = 10.

After some testing, it is found that setting macro JVET_O0432_LMCS_ENCODER = 0, then IBDI = 12 bings -0.18% BD-rate on average with RA configuration compared to IBDI = 10, which is consistent with results on top of VTM-5.

Btw, in order to run IBDI = 12 on top of VTM-6, the following two lines needs to be annotated.

const uint32_t maxBitDepth=(m_chromaFormatIDC==CHROMA_400) ? m_internalBitDepth[CHANNEL_TYPE_LUMA] : std::max(m_internalBitDepth[CHANNEL_TYPE_LUMA], m_internalBitDepth[CHANNEL_TYPE_CHROMA]);

xConfirmPara(m_bitDepthConstraint<maxBitDepth, "The internalBitDepth must not be greater than the bitDepthConstraint value");

[taoranlu]

Thanks for reporting. A fix is provided in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/942

[XiangLi]

Fixed as suggested.