Invalid encoder memory access for EncDbOpt=1 and CHROMA_400

[regensky]

In case the deblocking filter is applied during RDO ('EncDbOpt=1') and the chroma format is CHROMA_400, the encoder accesses invalid memory in RdCost::getDistPart (L. 429, L. 433)`. This leads to a segmentation fault (L. 438).

The issue originates in EncCu::xCalDebCost (L. 4613, L. 4629):

ComponentID compEnd = ( cu->isSepTree() &&  isLuma( partitioner.chType ) ) ? COMPONENT_Y : COMPONENT_Cr;

COMPONENT_Cr is selected although CHROMA_400 features no COMPONENT_Cr.

---

This can be solved by replacing the corresponding lines with

ComponentID compEnd = isChromaEnabled(cu->chromaFormat) && !(cu->isSepTree() && isLuma(partitioner.chType)) ? COMPONENT_Cr : COMPONENT_Y;

---

Command line to reproduce the issue:

EncoderAppStatic
-c cfg/encoder_randomaccess_vtm.cfg
-i Tarsier_1920x1080_30fps_8bit_400.yuv
-wdt 1920
-hgt 1080
--InputBitDepth=8
--InputChromaFormat=400
-fr 30
-f 1
-q 37

[regensky]

Test video file to reproduce the issue. (Source: unsplash.com)

[regensky]

Command line for attached file (reduced resolution):

EncoderAppStatic
-c cfg/encoder_randomaccess_vtm.cfg
-i Tarsier_256x128_30fps_8bit_400.yuv
-wdt 256
-hgt 128
--InputBitDepth=8
--InputChromaFormat=400
-fr 30
-f 1
-q 37

[XiangLi]

Thanks Nan for the fix ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2279

@regensky Could you confirm the fix resolve the issue?

[regensky]

I can confirm that merge request 2279 fixes the issue. Thanks a lot!