Opened 4 years ago

Closed 4 years ago

# Incorrect shiftHor derivation of GEO

Reported by: Owned by: XueweiMeng minor spec ksuehring, bbross, XiangLi, fbossen, jvet@…

### Description

In JVET-Q2001-vE

In 8.5.7.2 "Weighted sample prediction process for geometric partitioning mode",

hwRatio = nH / nW (1034)
displacementX = angleIdx (1035)
displacementY = ( angleIdx + 8 ) % 32 (1036)
partFlip = ( angleIdx >= 13 && angleIdx <= 27 ) ? 0 : 1 (1037)
shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 0 ) ) ? 0 : 1 (1038)

hwRatio is always larger than 0. So I think the equ(1038) shoulb be

shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 1 ) ) ? 0 : 1 (1038)

### comment:1 in reply to: ↑ description Changed 4 years ago by hangao

Thank you for pointing it out, I think this is another integration typo,

Actually the equation of (1038) and (1053) should be:

shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio >= 1 ) ) ? 0 : 1 (1038)
shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio >= 1 )) ? 0 : 1 (1053)

Instead of the JVET-Q2001-vE or the "hwRatio > 1" version (btw, in 1053 also a parenthese missing)

Cause based on the provided and reviewed text in JVET-Q0806:

```If one of the following conditons is true, variable shiftHor is set equal to 0:
angleIdx % 16 is equal to 8
angleIdx % 16 is not equal to 0 and hwRatio ≥ 1
Otherwise, shiftHor is set equal to 1.

```

and based on the VTM-8.0 software:

```if( angle % 16 == 8 || (angle % 16 != 0 && height >= width) )
```

In JVET-Q2001-vE

In 8.5.7.2 "Weighted sample prediction process for geometric partitioning mode",

hwRatio = nH / nW (1034)
displacementX = angleIdx (1035)
displacementY = ( angleIdx + 8 ) % 32 (1036)
partFlip = ( angleIdx >= 13 && angleIdx <= 27 ) ? 0 : 1 (1037)
shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 0 ) ) ? 0 : 1 (1038)

hwRatio is always larger than 0. So I think the equ(1038) shoulb be

shiftHor = ( angleIdx % 16 = = 8 | | ( angleIdx % 16 != 0 && hwRatio > 1 ) ) ? 0 : 1 (1038)

### comment:2 Changed 4 years ago by bbross

• Resolution set to fixed
• Status changed from new to closed

Thanks for pointing that out.

To align spec with SW, I actually changed both places in the draft by

• removing hwRatio
• changing the conditions to cbHeight > = cbWidth and nH > = nW

This will be fixed in the first version of D10.

Note: See TracTickets for help on using tickets.