Mismatch between spec and VTM in derivation of coding sub-block boundaries

[emora]

There seems to be a mismatch between the spec and VTM software regarding the derivation of max filtering length P/Q for coding sub-block boundaries. Section 8.8.3.4 states (for example, for vertical edges) that max filter length P/Q shall be equal to 2 (equations 1255 and 1256) if one or more of the following conditions are true:
– xEdge is equal to 1,
– xEdge is equal to ( nCbW / 8 ) − 1,
– edgeTbFlags[ x − sbW ][ y ] is equal to 1,
– edgeTbFlags[ x + sbW ][ y ] is equal to 1,

In VTM however, the first condition (xEdge == 1) is never checked. The code reads

"else if ( x > 0 && ( m_transformEdge[cIdx][ctuXOff+x-8][ctuYOff+y] | | ( (x + 8) >= areaPu.width ) | | m_transformEdge[cIdx][ctuXOff+x+8][ctuYOff+y] ) )"

I believe software is correct in this regards and so, first condition should be deleted. Same can be said for horizontal edges.

[abe.kiyo]

I agree with this suggestion. "xEdge is equal to 1" is not needed to check since it is covered by the condition "edgeTbFlags[ x − sbW ][ y ] is equal to 1".

Furthermore, by changing the description to the following, it becomes possible to align with the VTM code and the description of the condition for equations 1253 and 1254.

– ( x + 8 ) is greater than or equal to nCbW,
– edgeTbFlags[ x − 8 ][ y ] is equal to 1,
– edgeTbFlags[ x + 8 ][ y ] is equal to 1,

[bheng]

It is my understanding that the condition

• xEdge is equal to 1,

is in place to ensure that CU edges are still treated as transform edges (for the purposes of sub-block motion filter length derivation), even if the CU aligns with a disabled loop-filter-across slice / tile / subpicture / picture boundary.

The edgeTbFlags for such boundaries are effectively “erased” (set to 0) in 8.8.3.3. So, the condition “edgeTbFlags[ x − sbW ][ y ] is equal to 1” alone will fail to catch these cases (because the corresponding edgeTbFlag is 0).

Leaving the “xEdge is equal to 1” condition in place ensures that the sub-block motion filtering length derivation is consistent regardless of whether neighboring CTUs have loop-filter-across flags equal to 0 or not.

Therefore, for this mismatch, I think the software should be updated to add the “xEdge is equal to 1” case (x == 8).

else if ( x > 0 && ((x == 8) | | m_transformEdge[cIdx][ctuXOff+x-8][ctuYOff+y] | | ( (x + 8) >= areaPu.width ) | | m_transformEdge[cIdx][ctuXOff+x+8][ctuYOff+y] ) )

Additionally, for Virtual Boundaries, the text and software also mismatch but in exactly the opposite way. The text “erases” any edgeTbFlags that align with Virtual Boundaries in 8.8.3.3. Therefore, the text effectively ignores them for the purposes of sub-block motion filter length derivation in 8.8.3.4.

In contrast, the software treats TU boundaries as TU boundaries, regardless of whether or not they align with a Virtual Boundary, and computes the sub-block motion filter length accordingly.

So, in this case, I think the text should be modified.

For example, the process:

• If VirtualBoundariesPresentFlag equal to 1 and ( xCb + x ) is equal to VirtualBoundariesPosX[ n ] for any n = 0..NumVerVirtualBoundaries − 1, edgeFlags[ x ][ y ] is set equal to 0.

could be shifted to 8.8.3.5 and set bS equal to 0 rather than edgeFlags equal to 0

[Kenneth]

Thanks for finding the mismatch between spec and software.
I agree with Brian that it make sense to fix the software to match the specification such that the behaviour is consistent. Thus x==8 check should be added for vertical subblock edges and y==8 check for horizontal subblock edges.

[bbross]

Thanks for reporting and discussing the issue. It seems that no action on draft text is required here so I close it.
Please re-open if issues still exists.