VTM master branch encoder crash for YUV444 coding

[xyxiu]

Encoder crashed for the VTM master branch s/w for YUV444 coding. The attached cfg files and command line can be used to generate the crash point where happened at the first frame

EncoderApp -c Map_444.cfg -c encoder_randomaccess_vtm.cfg -c classF.cfg -c yuv444.cfg

It seems after disabling the macro JVET_N0278_FIXES the encoder can run but decoder still crash.

[Vadim]

There is currently a problem with JVET_P0410_CHROMA_QP_MAPPING integration for non 4:2:0 formats in VTM master branch.

Encoder sets
cfg_qpOutValCb.values = { 0 };
but later
m_chromaQpMappingTableParams.m_numPtsInCQPTableMinus1[0] = (int)cfg_qpOutValCb.values.size() - 2;

It gives negative value coded with UVLC, encoder crashes.

Probably the fix should be
m_chromaQpMappingTableParams.m_numPtsInCQPTableMinus1[0] = std::max<int>(0, (int)cfg_qpOutValCb.values.size() - 2);

[XiangLi]

Yes, chroma table is reset in the following code, where m_useIdentityTableForNon420Chroma (i.e., encoder cfg option UseIdentityTableForNon420Chroma) was introduced in JVET-O0650. Tests showed that the crash can be avoided by "--UseIdentityTableForNon420Chroma=0". It looks this encoder only parameter is not used anymore. An email was sent to the proponent on whether the parameter and related code could be removed.

if (m_useIdentityTableForNon420Chroma && m_chromaFormatIDC != CHROMA_420) {

m_chromaQpMappingTableParams.m_sameCQPTableForAllChromaFlag = true;
cfg_qpInValCb.values = { 0 };
cfg_qpInValCr.values = { 0 };
cfg_qpInValCbCr.values = { 0 };
cfg_qpOutValCb.values = { 0 };
cfg_qpOutValCr.values = { 0 };
cfg_qpOutValCbCr.values = { 0 };

}

[XiangLi]

A possible fix: ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1149

[delagrangep]

Thanks for raising the issue. Root cause is bad handling of single-value (=identity) tables.
Attached are two fixes (choose):

• one on top of MR1149 (ticket722-407dba6.patch)
• one two commits before, without removal of UseIdentityTableForNon420Chroma (ticket722-ea41699.patch)

Quickly tested with attached test.cfg, which still crashes in MR1149.
Please review and test.

About removal of UseIdentityTableForNon420Chroma:
I think this should be decided by AHG15. I remember discussions about bypass of mapping tables in 4:4:4 which was the way it worked before adoption of progammable tables (O0650). We decided to keep that behaviour for our tests. UseIdentityTableForNon420Chroma, which is 1 by default, mimics that behaviour without touching the cfg files. If we remove it, then we have to update cfg files for non-4:2:0 cases (no big deal).

[delagrangep]

Test config with identity tables (crash)

[delagrangep]

Fix on top of MR1149

[delagrangep]

Fix while keeping UseIdentityTableForNon420Chroma

[fbossen]

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1160