Opened 4 years ago

Closed 4 years ago

#3 closed task (fixed)

Compilation can fail for TEncSearch.cpp

Reported by: gordon Owned by: XiangLi
Priority: minor Milestone:
Component: JEM Version: HM-14.0-KTA-2.0
Keywords: TEncSearch compilation failure Cc: ksuehring, XiangLi, vceg-experts@…

Description

Gcc compilation can fail in TEncSearch.cpp, in TEncSearch::estIntraPredQT (line 3144).

The indication is "array bounds exceeded". The compiler has partially followed the logic of

Int numModesAvailable = 67;
...
if (doFastSearch)
...
else
... problem here, with

for( Int i=0; i < numModesForFullRD; i++)

uiRdModeList[i] = i;

The compiler assumes that the array is being initialized up to numModesAvailable (deduced from previous tests), which is bigger than the array size.

From what I can see, doFastSearch is always true and the else case is never executed.
If that is true, the else case can be deleted as in the supplied patch.

If the else-case is to be left in for future-proofing any up coming logic, the array size will have to be increased.

Attachments (2)

TEncSearch-compilation-fix.patch (823 bytes) - added by gordon 4 years ago.
asserts doFastSearch is true, and deletes else case.
ticket_3.patch (729 bytes) - added by XiangLi 4 years ago.
bug fix

Download all attachments as: .zip

Change history (5)

Changed 4 years ago by gordon

asserts doFastSearch is true, and deletes else case.

comment:1 Changed 4 years ago by XiangLi

  • Owner set to XiangLi
  • Status changed from new to assigned

Thanks for the report. Yes, it is a bug with additional intra prediction modes (under macro QC_USE_65ANG_MODES). FAST_UDI_MAX_RDMODE_NUM should be set to 67. Please check whether the attached fix solve the problem.

Changed 4 years ago by XiangLi

bug fix

comment:2 Changed 4 years ago by gordon

Yes, the patch works and everything compiles OK.

comment:3 Changed 4 years ago by XiangLi

  • Resolution set to fixed
  • Status changed from assigned to closed

Thanks for the confirmation. The patch has been applied in /branches/HM-14.0-KTA-1.0-dev.

Note: See TracTickets for help on using tickets.