Uninitialized variables are being used in the latest master (b0a1f053d97b439c6ae55b5d3171fc8e0f584f8c)

[deluxan]

Valgrind detects a problem in line 563 in IntraSearch in the latest master (b0a1f053d97b439c6ae55b5d3171fc8e0f584f8c). The problem is apparently in the line

double cost = ( double ) minSadHad + (double)fracModeBits * sqrtLambdaForFirstPass;

The test is with BasketballPass, QP37, RA, 17 frames and the error already happens in the first frame.

This is beginning of the report:

==26406== Memcheck, a memory error detector
==26406== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==26406== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==26406== Command: ../bin/EncoderAppStaticp -c encoder_randomaccess_vtm.cfg -c per-sequence/BasketballPass.cfg -q 22 -f 17
==26406==
==26406== Warning: set address range perms: large range [0x14997040, 0x4345c840) (undefined)
==26406== Warning: set address range perms: large range [0x59e3f040, 0x713a1c40) (undefined)
==26406== Conditional jump or move depends on uninitialised value(s)
==26406== at 0x55D18D: IntraSearch::estIntraPredLumaQT(CodingUnit&, Partitioner&, double, bool, int, int, bool) (IntraSearch.cpp:563)
==26406== by 0x5A84DD: EncCu::xCheckRDCostIntra(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) (EncCu.cpp:1790)
==26406== by 0x5B6467: EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) (EncCu.cpp:816)
==26406== by 0x5B7D4F: EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, ModeType, bool&) (EncCu.cpp:1406)
==26406== by 0x5B5987: EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) (EncCu.cpp:874)
==26406== by 0x5B7D4F: EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, ModeType, bool&) (EncCu.cpp:1406)
==26406== by 0x5B5987: EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) (EncCu.cpp:874)
==26406== by 0x5B7D4F: EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, ModeType, bool&) (EncCu.cpp:1406)
==26406== by 0x5B5987: EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) (EncCu.cpp:874)
==26406== by 0x5B7D4F: EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, ModeType, bool&) (EncCu.cpp:1406)
==26406== by 0x5B5987: EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) (EncCu.cpp:874)
==26406== by 0x5BAE40: EncCu::compressCtu(CodingStructure&, UnitArea const&, unsigned int, int const*, int const*) (EncCu.cpp:380)
==26406==

[deluxan]

Sorry, I meant QP 22.

[fbossen]

Suggested fix:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1092

[afilippov]

Suggested fix:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1096