invAngle calculation issue

[audrey.turquin]

In section 8.4.5.2.12, the calculation of invAngle is :

invAngle = Round(512 * 32 / intraPredAngle)

However, intraPredAngle can be equal to 0. This leads to a division by zero, which is undefined.

Moreover, as the value of intraPredAngle can be negative (table 24), according to the spec invAngle can also be negative (because Round function keeps the sign). But in the VTM, the value of invAngle is always positive (invAngleTable array in function initPredIntraParams). So it seems there is a mismatch between VTM and spec.

[fbossen]

I don't see a mismatch between VTM and spec here, even though different conventions are used.
In JVET-R2001-v7, when intraPredAngle is negative, invAngle is negative and used in equations (334) and (344) as follows : ( x * invAngle + 256 ) >> 9. The VTM does (-k * invAngle + 256) >> 9 and I believe k == x. The result should thus be the same if the invAngle value in the spec is negative, and the one in the VTM is positive, and the two values have the same magnitude.

As for the division by 0, the value of invAngle is never used in such case. It may nevertheless be preferable to define invAngle only when intraPredAngle is nonzero:

Just before equation (332), change

The inverse angle parameter invAngle is derived based on intraPredAngle as follows:

to

When intraPredAngle is not equal to 0, the inverse angle parameter invAngle is derived based on intraPredAngle as follows:

[audrey.turquin]

Ok, I didn't see that.
I misunderstood the way it is done in VTM.

[bbross]

Good catch, thanks!

The division by 0 will be fixed in the first version of D10.