Infinite loops

[nanhu]

In xSolveAndQuant(), loops controlled by while(quantCoeffSum!=targetCoeffSumInt && count < 10) can be infinite

[ksuehring]

It looks like count is not incremented at all, so this part of the condition will never be true.

Can you provide an example where the loop actually fails to finish? Or is it just static code analysis?

The code is inherited from JEM ALF code in QuantizeIntegerFilterPP().

[nanhu]

No issue was found based on current code and CTC. But it may give rise to infinite loops. I advise to increase count to prevent it.

[XiangLi]

Proposed fix

[XiangLi]

I received a fix from Yiming (userlym@…) for this. Please find the patch in the attachment. The reporter also fixed the issue of equation solving in some special case (such as gray picture input).

[nanhu]

patch is added

[ksuehring]

I think this bug tracker issue now mixes up different problems. Those shouldn't be filed into one issue.

Can you please create a ticket, which describes the actual problem that is solved by CholFix_Bms_Tag1.patch​? Also the patch seems to replace three short functions by a large one. The three unused functions seem to be left behind.

Regarding the count issue: Incrementing seems to terminate the process early and will create different output. Does that have influence on coding performance? Did anybody run tests?

[yimingli]

Thanks, the modification in the CholFix_Bms_Tag1.patch is to solve the invalid value calculated by Cholesky method. "count" not increasing in the loop may actually causes the infinite loop. But the trigger or the main reason for the infinite loop is the invalid value of the calculated Wiener filter. So we mixed it in this bug fix. For more details of the Cholesky method, I will add some explanations below or in a new ticket.

The failure case of the Cholesky method is not very common, thus in the CTC sequences, this bug (infinite loop) is not triggered although count is not increasing in the loop. So, If we only fix the bug as count++ or ++ count, the performance will not be changed in CTC sequences.

But it should be mentioned, if we only fix it with ++count, the failure case of Cholesky method is also exist and may affect the performance of ALF, although the loop is finite. We have also prepared the CTC results for our fix. If it is considered that it's better to create a new ticket to discuss it, I will create it and give more details in the new ticket. Or, if it is considered this modification is not only a bug fix, we can prepare a proposal for the discussion in the meeting.

Besides, I have also seen the fix in the alf_fix.patch provided by Nan Hu. I think the fix about the constraint of minInd is not necessary, since in the code above the fix, there is:

if (error<errMin

minInd==-1)

{

errMin=error;
minInd=k;

}
where minInd=k is always not smaller than 0.

[ksuehring]

BMS is not maintained anymore. VTM is the currently supported reference software of VVC. If the report is still valid, please re-open and assign to VTM.