HDR Anchor encoding crashes with VTM4.0

[taoranlu]

Current VTM will crash during encoding for HDR Anchor test condition. Part of reason is caused by bug of reshaper implementation regarding RDCost. The fix is provided in merge request: ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/269/diffs

However, lumaDQP still have issue with (possibly) JVET_M0102_INTRA_SUBPARTITIONS. The encoder for HDR Anchor setting will crash even if we turn reshaper macro completely off ( JVET_M0427_INLOOP_RESHAPER=0) and enable WCG_EXT = 1 in typedef.h

The code will crash in EncCU.cpp, xCompressCU(): it enters the condition when DQP is used, and then the getTU() function will crash. Inside getTU(), it crashes in the while loop introduced by JVET_M0102_INTRA_SUBPARTITIONS.

if (tempCS->pps->getUseDQP() && CS::isDualITree(*tempCS) && isChroma(partitioner.chType))

{

const Position chromaCentral(tempCS->area.Cb().chromaPos().offset(tempCS->area.Cb().chromaSize().width >> 1, tempCS->area.Cb().chromaSize().height >> 1));
const Position lumaRefPos(chromaCentral.x << getComponentScaleX(COMPONENT_Cb, tempCS->area.chromaFormat), chromaCentral.y << getComponentScaleY(COMPONENT_Cb, tempCS->area.chromaFormat));
const CodingStructure* baseCS = bestCS->picture->cs;
const CodingUnit* colLumaCu = baseCS->getCU(lumaRefPos, CHANNEL_TYPE_LUMA);

const TransformUnit* tu = baseCS->getTU(lumaRefPos, CHANNEL_TYPE_LUMA);

if (colLumaCu && tu)
{

currTestMode.qp = colLumaCu->qp;

}

}

In CodingStructure.cpp, getTU()

#if JVET_M0102_INTRA_SUBPARTITIONS

if( idx != 0 )
{

unsigned extraIdx = 0;
if( isLuma( effChType ) )
{

const TransformUnit& tu = *tus[idx - 1];
if( tu.cu->ispMode ) Intra SubPartitions mode
{

we obtain the offset to index the corresponding sub-partition
if( subTuIdx != -1 )
{

extraIdx = subTuIdx;

}
else
{

while( pos != tus[idx - 1 + extraIdx]->blocks[effChType].pos() )

{

extraIdx++;

}

}

}

}
return tus[idx - 1 + extraIdx];

}

#else