Subblock edges and filtering across picture, subpicture, tile and slice boundaries (8.8.3.4)

[fbossen]

In 8.8.3.4, edgeFlags is unconditionally set to 2 at subblock boundary locations:

– The value of edgeFlags[ x ][ y ] is derived as follows:
edgeFlags[ x ][ y ] = 2 (1206)

This appears to happen regardless of whether the boundary is also a picture, subpicture, tile or slice boundary, leading to undesired filtering.

The assignment to edgeFlags should probably be conditioned on filterEdgeFlag (similarly to what is done in 8.8.3.3 for transform block edges):

– When x is larger than 0 or filterEdgeFlag is equal to 1, the value of edgeFlags[ x ][ y ] is modified as follows:
edgeFlags[ x ][ y ] = 2

In such case, filterEdgeFlag needs to be added as an input to 8.8.3.4.

[bbross]

Thanks for brining that up, that part indeed looks strange since after it is set to 2, there is a comndition "When edgeFlags[ x ][ y ] is equal to 1 or 2,..." which is always true then.

Can somebody confirm that Frank suggested fix is correct?

[nanhu]

I checked the spec, edgeFlags[][] should be conditionally set as Frank suggested.

[jlchen]

Thanks Frank for reporting the issue and Han for confirmation. The suggested fix will be included in the vF release.

[LiZhang]

Hi Frank, Nan,

Is it necessary to have 'filterEdgeFlag is equal to 1' in the suggested text below? I just checked 8.8.3.3 which doesn't have this condition. Also, when x is equal to 0 and filterEdgeFlag is equal to 1 (edgeFlags being 1), I think there is no difference between edgeFlags being 1 or 2.

– When x is larger than 0 or filterEdgeFlag is equal to 1, the value of edgeFlags[ x ][ y ] is modified as follows:
edgeFlags[ x ][ y ] = 2

[jlchen]

Hi Li,

edgeFlags are used in 8.8.3.5 as follows, not sure whether the behavior is same for edgeFlags being a or 2.

– Otherwise, if cIdx is equal to 0, edgeFlags[ xDi ][ yDj ] is equal to 2, and one or more of the following conditions are true, bS[ xDi ][ yDj ] is set equal to 1:

[chiaming]

It shall not include the condition "x is larger than 0". Otherwise, the edges of the current coding block coded by non-subblock coding mode will not check PB related conditions in deriving bS values. Then, it is not the intention of the current spec.
I suggest the text is fixed to:
– When filterEdgeFlag is equal to 1, the value of edgeFlags[ x ][ y ] is modified as follows:
edgeFlags[ x ][ y ] = 2

[nanhu]

In this case, when filterEdgeFlag is 0, does it mean that edgeFlags[][] cannot be set as 2 for the whole coding block? The condition suggested by Frank is only not filtering left/top boundary of current coding block for some undesired cases, while other sub-block (PB) edges of current coding block can still be filtered.

Replying to chiaming:

It shall not include the condition "x is larger than 0". Otherwise, the edges of the current coding block coded by non-subblock coding mode will not check PB related conditions in deriving bS values. Then, it is not the intention of the current spec.
I suggest the text is fixed to:
– When filterEdgeFlag is equal to 1, the value of edgeFlags[ x ][ y ] is modified as follows:
edgeFlags[ x ][ y ] = 2

[chiaming]

Yes, when filterEdgeFlag is 0, deblocking is disabled for the left/top boundary of the current coding block. It should not set edgeFlags[][] as 2, otherwise it cannot bypass the bS derivation process.

8.8.3.5	Derivation process of boundary filtering strength
...
For xDi with i = 0..xN and yDj with j = 0..yN, the following applies:
 – If edgeFlags[ xDi ][ yDj ] is equal to 0, the variable bS[ xDi ][ yDj ] is set equal to 0.

Replying to nanhu:

In this case, when filterEdgeFlag is 0, does it mean that edgeFlags[][] cannot be set as 2 for the whole coding block? The condition suggested by Frank is only not filtering left/top boundary of current coding block for some undesired cases, while other sub-block (PB) edges of current coding block can still be filtered.

[fbossen]

The condition "When x is larger than 0 or filterEdgeFlag is equal to 1" works as follows:

If x is equal to 0 and filterEdge is equal to 0, the condition is false, and edgeFlags[x][y] is unchanged.
Otherwise, the condition is true, and edgeFlags[x][y] is changed to 2.

Note that here x = xEdge * sbW takes only values that are aligned with subblock edges.

This is the same as in 8.8.3.3, where edgeFlags[x][y] is set to 1 for transform block edges unless x is equal to 0 and filterEdge is equal to 0:

• If x is equal to 0, edgeFlags[ x ][ y ] is set equal to filterEdgeFlag.
• Otherwise, if the location ( xCb + x , yCb + y ) is at a transform block edge, edgeFlags[ x ][ y ] is set equal to 1.

edgeFlags[x][y] being 1 or 2 is important, as noted in several other tickets.

@chiaming I don't understand your concern. The condition "x is larger than 0" is necessary here because we want to filter subblock edges that are not coding block boundaries, regardless of the value of filterEdgeFlag. The value of filterEdgeFlag is relevant only for edges that are also the left (or top) boundary of the current coding block. Having the condition "x is larger than 0" will not cause the left boundary of the current coding block to be filtered if filterEdgeFlag is 0.

[chiaming]

Sorry, I mistook the condition. The condition suggested by Frank is right.

[bbross]

Thanks everyone for checking and finally confirming the fix suggested by Frank which will be in JVET-S2001-vF.