spatial merge candidate derivation process with wrong prunings

[LiZhang]

The pruning process is dependent on the enabling of triangular prediction mode which is not aligned with the SW.

8.5.2.3 Derivation process for spatial merging candidates
…

For the derivation of availableFlagB0, refIdxLXB0, predFlagLXB0 and mvLXB0 the following applies:
– …
– The variables availableFlagB0, refIdxLXB0, predFlagLXB0 and mvLXB0 are derived as follows:
– If one or more of the following conditions are true, availableFlagB0 is set equal to 0, both components of mvLXB0 are set equal to 0, refIdxLXB0 is set equal to −1 and predFlagLXB0 is set equal to 0, with X being 0 or 1, and bcwIdxB0 is set equal to 0:
– availableB0 is equal to FALSE.
– availableB1 is equal to TRUE and the luma locations ( xNbB1, yNbB1 ) and ( xNbB0, yNbB0 ) have the same motion vectors and the same reference indices.
– availableA1 is equal to TRUE, the luma locations ( xNbA1, yNbA1 ) and ( xNbB0, yNbB0 ) have the same motion vectors and the same reference indices and MergeTriangleFlag[ xCb ][ yCb ] is equal to 1.
– Otherwise, availableFlagB0 is set equal to 1 and the following assignments are made:
…

For the derivation of availableFlagA0, refIdxLXA0, predFlagLXA0 and mvLXA0 the following applies:
– …
– The variables availableFlagA0, refIdxLXA0, predFlagLXA0 and mvLXA0 are derived as follows:
– If one or more of the following conditions are true, availableFlagA0 is set equal to 0, both components of mvLXA0 are set equal to 0, refIdxLXA0 is set equal to −1 and predFlagLXA0 is set equal to 0, with X being 0 or 1, and bcwIdxA0 is set equal to 0:
– availableA0 is equal to FALSE.
– availableA1 is equal to TRUE and the luma locations ( xNbA1, yNbA1 ) and ( xNbA0, yNbA0 ) have the same motion vectors and the same reference indices.
– availableB1 is equal to TRUE, the luma locations ( xNbB1, yNbB1 ) and ( xNbA0, yNbA0 ) have the same motion vectors and the same reference indices and MergeTriangleFlag[ xCb ][ yCb ] is equal to 1.
– availableB0 is equal to TRUE, the luma locations ( xNbB0, yNbB0 ) and ( xNbA0, yNbA0 ) have the same motion vectors and the same reference indices and MergeTriangleFlag[ xCb ][ yCb ] is equal to 1.
– Otherwise, availableFlagA0 is set equal to 1 and the following assignments are made:
…

For the derivation of availableFlagB2, refIdxLXB2, predFlagLXB2 and mvLXB2 the following applies:
– …
– The variables availableFlagB2, refIdxLXB2, predFlagLXB2 and mvLXB2 are derived as follows:
– If one or more of the following conditions are true, availableFlagB2 is set equal to 0, both components of mvLXB2 are set equal to 0, refIdxLXB2 is set equal to −1 and predFlagLXB2 is set equal to 0, with X being 0 or 1, and bcwIdxB2 is set equal to 0:
– availableB2 is equal to FALSE.
– availableA1 is equal to TRUE and the luma locations ( xNbA1, yNbA1 ) and ( xNbB2, yNbB2 ) have the same motion vectors and the same reference indices.
– availableB1 is equal to TRUE and the luma locations ( xNbB1, yNbB1 ) and ( xNbB2, yNbB2 ) have the same motion vectors and the same reference indices.
– availableB0 is equal to TRUE, the luma locations ( xNbB0, yNbB0 ) and ( xNbB2, yNbB2 ) have the same motion vectors and the same reference indices and MergeTriangleFlag[ xCb ][ yCb ] is equal to 1.
– availableA0 is equal to TRUE, the luma locations ( xNbA0, yNbA0 ) and ( xNbB2, yNbB2 ) have the same motion vectors and the same reference indices and MergeTriangleFlag[ xCb ][ yCb ] is equal to 1.
– availableFlagA0 + availableFlagA1 + availableFlagB0 + availableFlagB1 is equal to 4 and MergeTriangleFlag[ xCb ][ yCb ] is equal to 0.
– Otherwise, availableFlagB2 is set equal to 1 and the following assignments are made:
…

[bbross]

Does that issue still exists in VTM-6.0 and D6?
If so should the draft be modified to match VTM or vice-versa?

[bbross]

I would fix it by aligning the spec to VTM if no one objects.

[bbross]

Ticket #547 has been marked as a duplicate of this ticket.

[jlchen]

Fixed in D7 vC to match the software.