Opened 2 years ago

Closed 2 years ago

# invAngle calculation issue

Reported by: Owned by: audrey.turquin minor spec ksuehring, bbross, XiangLi, fbossen, jvet@…

### Description

In section 8.4.5.2.12, the calculation of invAngle is :

invAngle = Round(512 * 32 / intraPredAngle)

However, intraPredAngle can be equal to 0. This leads to a division by zero, which is undefined.

Moreover, as the value of intraPredAngle can be negative (table 24), according to the spec invAngle can also be negative (because Round function keeps the sign). But in the VTM, the value of invAngle is always positive (invAngleTable array in function initPredIntraParams). So it seems there is a mismatch between VTM and spec.

### comment:1 Changed 2 years ago by fbossen

I don't see a mismatch between VTM and spec here, even though different conventions are used.
In JVET-R2001-v7, when intraPredAngle is negative, invAngle is negative and used in equations (334) and (344) as follows : ( x * invAngle + 256 ) >> 9. The VTM does (-k * invAngle + 256) >> 9 and I believe k == x. The result should thus be the same if the invAngle value in the spec is negative, and the one in the VTM is positive, and the two values have the same magnitude.

As for the division by 0, the value of invAngle is never used in such case. It may nevertheless be preferable to define invAngle only when intraPredAngle is nonzero:

Just before equation (332), change

The inverse angle parameter invAngle is derived based on intraPredAngle as follows:

to

When intraPredAngle is not equal to 0, the inverse angle parameter invAngle is derived based on intraPredAngle as follows:

### comment:2 Changed 2 years ago by audrey.turquin

Ok, I didn't see that.
I misunderstood the way it is done in VTM.

### comment:3 Changed 2 years ago by bbross

• Resolution set to fixed
• Status changed from new to closed

Good catch, thanks!

The division by 0 will be fixed in the first version of D10.

Note: See TracTickets for help on using tickets.