Incorrect shiftHor derivation of GEO

[XueweiMeng]

In JVET-Q2001-vE

In 8.5.7.2 "Weighted sample prediction process for geometric partitioning mode",

hwRatio = nH / nW (1034)
displacementX = angleIdx (1035)
displacementY = ( angleIdx + 8 ) % 32 (1036)
partFlip = ( angleIdx >= 13 && angleIdx <= 27 ) ? 0 : 1 (1037)
shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 0 ) ) ? 0 : 1 (1038)

hwRatio is always larger than 0. So I think the equ(1038) shoulb be

shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 1 ) ) ? 0 : 1 (1038)

[hangao]

Thank you for pointing it out, I think this is another integration typo,

Actually the equation of (1038) and (1053) should be:

shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio >= 1 ) ) ? 0 : 1 (1038)
shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio >= 1 )) ? 0 : 1 (1053)

Instead of the JVET-Q2001-vE or the "hwRatio > 1" version (btw, in 1053 also a parenthese missing)

Cause based on the provided and reviewed text in JVET-Q0806:

If one of the following conditons is true, variable shiftHor is set equal to 0:
	angleIdx % 16 is equal to 8
	angleIdx % 16 is not equal to 0 and hwRatio ≥ 1
Otherwise, shiftHor is set equal to 1.

and based on the VTM-8.0 software:

if( angle % 16 == 8 || (angle % 16 != 0 && height >= width) )

Replying to XueweiMeng:

In JVET-Q2001-vE

In 8.5.7.2 "Weighted sample prediction process for geometric partitioning mode",

hwRatio = nH / nW (1034)
displacementX = angleIdx (1035)
displacementY = ( angleIdx + 8 ) % 32 (1036)
partFlip = ( angleIdx >= 13 && angleIdx <= 27 ) ? 0 : 1 (1037)
shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 0 ) ) ? 0 : 1 (1038)

hwRatio is always larger than 0. So I think the equ(1038) shoulb be

shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 1 ) ) ? 0 : 1 (1038)

[bbross]

Thanks for pointing that out.

To align spec with SW, I actually changed both places in the draft by

• removing hwRatio
• changing the conditions to cbHeight > = cbWidth and nH > = nW

This will be fixed in the first version of D10.