Misalignment with spec when setting max filter length in xDeblockCU

[xinzhao]

In VTM-10, xDeblockCU(..), the setting of max filter length is skipped when luma TU position (x or y) is not multiples of 4. However, for chroma 4:4:4 color format, in single tree, for an 8x8 CU, when luma is coded by ISP and chroma has one single 8x8 TU, VTM-10 skips the setting of max filter length for chroma (i.e., max filter length is initialized value 0), but Spec text tells that the max filter length for chroma should be set as either 1 or 3. Suggest to fix VTM.

Related codes in VTM-10.0, where areaTu points to a luma TU that may be ISP coded such that areaTu.x or areaTu.y may be not multiples of 4:

if( edgeDir == EDGE_HOR && ((areaTu.y % 4) != 0) )
{

continue;

}
if( edgeDir == EDGE_VER && ((areaTu.x % 4) != 0) )
{

continue;

}

Related spec text in JVET-S2001:

8.8.3.3 Derivation process of transform block boundary
...

– The variable gridSize is set as follows:

gridSize = cIdx = = 0 ? 4 : 8 (1205)

– If edgeType is equal to EDGE_VER, the following applies:
– The variable numEdges is set equal to Max( 1, nCbW / gridSize ).
– For xEdge = 0..numEdges − 1 and y = 0..nCbH − 1, the following applies:
...

– Otherwise (cIdx is not equal to 0), the values of maxFilterLengthPs[ x][ y ] and maxFilterLengthQs[ x ][ y ] are derived as follows:
– If the width in chroma samples of the transform block at chroma location ( xCb + x, yCb + y ) and the width at chroma location ( xCb + x − 1, yCb + y ) are both equal to or greater than 8, maxFilterLengthPs[ x ][ y ] and maxFilterLengthQs[ x ][ y ] are set equal to 3.
– Otherwise, maxFilterLengthPs[ x ][ y ] and maxFilterLengthQs[ x ][ y ] are set equal to 1.

[fbossen]

Suggested fix:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/1885

It should be noted that currently available 4:2:2 and 4:4:4 conformance bitstreams may be incorrect because of this bug.

[XiangLi]

Fixed as suggested.