Opened 3 years ago
Closed 3 years ago
#1057 closed defect (fixed)
invAngle calculation issue
Reported by: | audrey.turquin | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | spec | Version: | |
Keywords: | Cc: | ksuehring, bbross, XiangLi, fbossen, jvet@… |
Description
In section 8.4.5.2.12, the calculation of invAngle is :
invAngle = Round(512 * 32 / intraPredAngle)
However, intraPredAngle can be equal to 0. This leads to a division by zero, which is undefined.
Moreover, as the value of intraPredAngle can be negative (table 24), according to the spec invAngle can also be negative (because Round function keeps the sign). But in the VTM, the value of invAngle is always positive (invAngleTable array in function initPredIntraParams). So it seems there is a mismatch between VTM and spec.
Change history (3)
comment:1 Changed 3 years ago by fbossen
comment:2 Changed 3 years ago by audrey.turquin
Ok, I didn't see that.
I misunderstood the way it is done in VTM.
comment:3 Changed 3 years ago by bbross
- Resolution set to fixed
- Status changed from new to closed
Good catch, thanks!
The division by 0 will be fixed in the first version of D10.
I don't see a mismatch between VTM and spec here, even though different conventions are used.
In JVET-R2001-v7, when intraPredAngle is negative, invAngle is negative and used in equations (334) and (344) as follows : ( x * invAngle + 256 ) >> 9. The VTM does (-k * invAngle + 256) >> 9 and I believe k == x. The result should thus be the same if the invAngle value in the spec is negative, and the one in the VTM is positive, and the two values have the same magnitude.
As for the division by 0, the value of invAngle is never used in such case. It may nevertheless be preferable to define invAngle only when intraPredAngle is nonzero:
Just before equation (332), change
The inverse angle parameter invAngle is derived based on intraPredAngle as follows:
to
When intraPredAngle is not equal to 0, the inverse angle parameter invAngle is derived based on intraPredAngle as follows: