Opened 6 months ago

Closed 6 months ago

#1565 closed defect (fixed)

Invalid encoder memory access for EncDbOpt=1 and CHROMA_400

Reported by: regensky Owned by:
Priority: minor Milestone:
Component: VTM Version: VTM-17.0
Keywords: Cc: ksuehring, XiangLi, fbossen, jvet@…

Description

In case the deblocking filter is applied during RDO ('EncDbOpt=1') and the chroma format is CHROMA_400, the encoder accesses invalid memory in RdCost::getDistPart (L. 429, L. 433)`. This leads to a segmentation fault (L. 438).

The issue originates in EncCu::xCalDebCost (L. 4613, L. 4629):

ComponentID compEnd = ( cu->isSepTree() &&  isLuma( partitioner.chType ) ) ? COMPONENT_Y : COMPONENT_Cr;

COMPONENT_Cr is selected although CHROMA_400 features no COMPONENT_Cr.


This can be solved by replacing the corresponding lines with

ComponentID compEnd = isChromaEnabled(cu->chromaFormat) && !(cu->isSepTree() && isLuma(partitioner.chType)) ? COMPONENT_Cr : COMPONENT_Y;

Command line to reproduce the issue:

EncoderAppStatic
-c cfg/encoder_randomaccess_vtm.cfg
-i Tarsier_1920x1080_30fps_8bit_400.yuv
-wdt 1920
-hgt 1080
--InputBitDepth=8
--InputChromaFormat=400
-fr 30
-f 1
-q 37

Attachments (1)

Tarsier_256x128_30fps_8bit_400.yuv (32.0 KB) - added by regensky 6 months ago.
Test video file to reproduce the issue. (Source: unsplash.com)

Download all attachments as: .zip

Change history (4)

Changed 6 months ago by regensky

Test video file to reproduce the issue. (Source: unsplash.com)

comment:1 Changed 6 months ago by regensky

Command line for attached file (reduced resolution):

EncoderAppStatic
-c cfg/encoder_randomaccess_vtm.cfg
-i Tarsier_256x128_30fps_8bit_400.yuv
-wdt 256
-hgt 128
--InputBitDepth=8
--InputChromaFormat=400
-fr 30
-f 1
-q 37

comment:2 Changed 6 months ago by XiangLi

Thanks Nan for the fix https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2279

@regensky Could you confirm the fix resolve the issue?

comment:3 Changed 6 months ago by regensky

  • Resolution set to fixed
  • Status changed from new to closed

I can confirm that merge request 2279 fixes the issue. Thanks a lot!

Note: See TracTickets for help on using tickets.