Edge range in 8.8.3.4

[bheng]

For coding blocks with width of 4, the following range:

xEdge = 0..min( ( nCbW / 8 ) - 1, numSbX - 1)

will evaluate to xEdge = [0 .. -1], since (nCbW / 8) is 0.

Depending on how one interprets that backwards range, either the xEdge set is empty, or xEdge goes backwards outside the CU. In either case, the behavior is not correct.

For example, if one treats xEdge as empty, the edgeIdc[x][y] is never set to 2 for the CU edge, and any motion vector differences are ignored for deblocking.

I believe (nCbW / 8) should be replaced by something like Max(1,(nCbW / 8)) to prevent this issue.

The same is true for the yEdge range later in the same section.

[bbross]

Right, thanks! The will be fixed in JVET-S2001-vH by adding the Max(1,..) operation for both xEdge and yEdge as suggested.

[bheng]

Thanks for the fix!

The yEdge change ended up with an extra left-parenthesis:

For yEdge = 0..Min( ( Max (1, nCbH / 8 ) − 1, numSbY − 1 ), x = 0..nCbW − 1:

I would suggest:

For yEdge = 0..Min( Max (1, nCbH / 8 ) − 1, numSbY − 1 ), x = 0..nCbW − 1:

[bbross]

Right! That was a test to see whether you check the fix ;)
Extra left-parenthesis is removed now in JVET-S2001-vH.