Incorrect use of collocated luma block location when deriving SbTMVP merge candidates

[wzeng]

In spec text D7 vE section 8.5.5.3 and 8.5.5.4, top-left luma sample location of collocated luma coding block (xColCtrCb, yColCtrCb) is used to derive SbTMVP base motion data.

However, in VTM7.0 function PU::getInterMergeSubPuMvpCand(), current processing CU's top-left position is used instead.

centerPos.x = puPos.x + (puSize.width >> 1) + tempX;
centerPos.y = puPos.y + (puSize.height >> 1) + tempY;
clipColPos(centerPos.x, centerPos.y, pu);
centerPos = Position{ PosType(centerPos.x & mask), PosType(centerPos.y & mask) };

Suggest to align the text with VTM. A quick fix is to remove the definition of (xColCtrCb, yColCtrCb) and replace with (xCtrCb, yCtrCb).

[zhou]

The spec text is totally wrong in this case, as it requires writing/storing/loading of top-left CU locations of the col-located pictures. Suggested fix:

in Section 8.5.5.3

1) replace

1. The location ( xCtb, yCtb ) of the top-left sample of the luma coding tree block that contains the current coding block and the location ( xCtr, yCtr ) of the below-right center sample of the current luma coding block are derived as follows:

xCtb = ( xCb >> CtuLog2Size ) << CtuLog2Size (8 529)
yCtb = ( yCb >> CtuLog2Size ) << CtuLog2Size (8 530)
xCtr = xCb + ( cbWidth / 2 ) (8 531)
yCtr = yCb + ( cbHeight / 2 )

with

1. The location ( xCtb, yCtb ) of the top-left sample of the luma coding tree block that contains the current coding block and the location ( xCtrCb, yCtrCb ) of the below-right center sample of the current luma coding block are derived as follows:

xCtb = ( xCb >> CtuLog2Size ) << CtuLog2Size (8 529)
yCtb = ( yCb >> CtuLog2Size ) << CtuLog2Size (8 530)
xCtrCb = xCb + ( cbWidth / 2 ) (8 531)
yCtrCb = yCb + ( cbHeight / 2 )

2) delete the following clause

2. The luma location ( xColCtrCb, yColCtrCb ) is set equal to the top-left sample of the collocated luma coding block covering the location given by ( xCtr, yCtr ) inside ColPic relative to the top-left luma sample of the collocated picture specified by ColPic.

3) replace

3. The derivation process for subblock-based temporal merging base motion data as specified in clause 8.5.5.4 is invoked with the location ( xCtb, yCtb ), the location ( xColCtrCb, yColCtrCb ), ...

with

2. The derivation process for subblock-based temporal merging base motion data as specified in clause 8.5.5.4 is invoked with the location ( xCtb, yCtb ), the location ( xCtrCb, yCtrCb ), ...

in section 8.5.5.4

4) replace
– the location ( xColCtrCb, yColCtrCb ) of the top-left sample of the collocated luma coding block that covers the below-right center sample.

with

– the location ( xCtrCb, yCtrCb ) of the top-left sample of the luma coding block that covers the below-right center sample.

5) replace

The location ( xColCb, yColCb ) of the collocated block inside ColPic is derived as follows.
– The following applies:
yColCb = Clip3( yCtb,

Min( pic_height_in_luma_samples − 1, yCtb + ( 1 << CtbLog2SizeY ) − 1 ), (8 547)
yColCtrCb + tempMv[1] )

– If subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 1, the following applies:
xColCb = Clip3( xCtb,

Min( SubPicRightBoundaryPos, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 548)
xColCtrCb + tempMv[0] )

– Otherwise ( subpic_treated_as_pic_flag[ SubPicIdx ] is equal to o, the following applies:
xColCb = Clip3( xCtb,

Min( pic_width_in_luma_samples − 1, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 549)
xColCtrCb + tempMv[0] )

with

The location ( xColCb, yColCb ) of the collocated block inside ColPic is derived as follows.
– The following applies:
yColCb = Clip3( yCtb,

Min( pic_height_in_luma_samples − 1, yCtb + ( 1 << CtbLog2SizeY ) − 1 ), (8 547)
yCtrCb + tempMv[1] )

– If subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 1, the following applies:
xColCb = Clip3( xCtb,

Min( SubPicRightBoundaryPos, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 548)
xCtrCb + tempMv[0] )

– Otherwise ( subpic_treated_as_pic_flag[ SubPicIdx ] is equal to o, the following applies:
xColCb = Clip3( xCtb,

Min( pic_width_in_luma_samples − 1, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 549)
xCtrCb + tempMv[0] )

Note that in the current spec text ( xCtrCb ,yCtrCb ) is undefined but is used in section 8.5.5.4.

[yucheng]

Dear experts,

In spec VVC Draft 7 vE version, in section 8.5.5.3 and 8.5.5.4, there is a variable colCb used to specify the luma coding block on the collocated picture covering the center sample location in 8x8 motion grid. What is the definition of luma coding block? If it is coding unit, and it is affine, then the collocated motion information for later motion vector computation will be wrong.

Another concern is that in 8.5.5.3 and 8.5.5.4, if sub-pictures exist, clipping operation is only considered in horizontal direction of sub-pictures, vertical direction is not considered.

The related sections are listed below.

8.5.5.3 Derivation process for subblock-based temporal merging candidates
…
– The location ( xColSb, yColSb ) of the collocated subblock inside ColPic is derived as follows.
– The following applies:
yColSb = Clip3( yCtb,

Min( pic_height_in_luma_samples − 1, yCtb + ( 1 << CtbLog2SizeY ) − 1 ), (735)
ySb + tempMv[1] )

– If subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 1, the following applies:
xColSb = Clip3( xCtb,

Min( SubPicRightBoundaryPos, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (736)
xSb + tempMv[0] )

– Otherwise ( subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 0), the following applies:
xColSb = Clip3( xCtb,

Min( pic_width_in_luma_samples − 1, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (737)
xSb + tempMv[0] )

– The variable currCb specifies the luma coding block covering the current coding subblock inside the current picture.
– The variable colCb specifies the luma coding block covering the modified location given by ( ( xColSb >> 3 ) << 3, ( yColSb >> 3 ) << 3 ) inside the ColPic.
– The luma location ( xColCb, yColCb ) is set equal to the top-left sample of the collocated luma coding block specified by colCb relative to the top-left luma sample of the collocated picture specified by ColPic.
…

8.5.5.4 Derivation process for subblock-based temporal merging base motion data
…
The location ( xColCb, yColCb ) of the collocated block inside ColPic is derived as follows.
– The following applies:
yColCb = Clip3( yCtb,

Min( pic_height_in_luma_samples − 1, yCtb + ( 1 << CtbLog2SizeY ) − 1 ), (742)
yColCtrCb + tempMv[1] )

– If subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 1, the following applies:
xColCb = Clip3( xCtb,

Min( SubPicRightBoundaryPos, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (743)
xColCtrCb + tempMv[0] )

– Otherwise ( subpic_treated_as_pic_flag[ SubPicIdx ] is equal to o, the following applies:
xColCb = Clip3( xCtb,

Min( pic_width_in_luma_samples − 1, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (744)
xColCtrCb + tempMv[0] )

…
– If colPredMode[xColCb][yColCb] is equal to MODE_INTER, the following applies:
– The variable currCb specifies the luma coding block covering ( xCtrCb ,yCtrCb ) inside the current picture.
– The variable colCb specifies the luma coding block covering the modified location given by ( ( xColCb >> 3 ) << 3, ( yColCb >> 3 ) << 3 ) inside the ColPic.
– The luma location ( xColCb, yColCb ) is set equal to the top-left sample of the collocated luma coding block specified by colCb relative to the top-left luma sample of the collocated picture specified by ColPic.

[jlchen]

Replying to zhou:

The item 1, 2, 3 and 4 make sense, will be integrated in the next release.

The item 5 is not necessary since the original vertical clipping operation restricts the colCb position within the same CTU row, which as well is in the same sub-picture.

The spec text is totally wrong in this case, as it requires writing/storing/loading of top-left CU locations of the col-located pictures. Suggested fix:

in Section 8.5.5.3

1) replace

1. The location ( xCtb, yCtb ) of the top-left sample of the luma coding tree block that contains the current coding block and the location ( xCtr, yCtr ) of the below-right center sample of the current luma coding block are derived as follows:

xCtb = ( xCb >> CtuLog2Size ) << CtuLog2Size (8 529)
yCtb = ( yCb >> CtuLog2Size ) << CtuLog2Size (8 530)
xCtr = xCb + ( cbWidth / 2 ) (8 531)
yCtr = yCb + ( cbHeight / 2 )

with

1. The location ( xCtb, yCtb ) of the top-left sample of the luma coding tree block that contains the current coding block and the location ( xCtrCb, yCtrCb ) of the below-right center sample of the current luma coding block are derived as follows:

xCtb = ( xCb >> CtuLog2Size ) << CtuLog2Size (8 529)
yCtb = ( yCb >> CtuLog2Size ) << CtuLog2Size (8 530)
xCtrCb = xCb + ( cbWidth / 2 ) (8 531)
yCtrCb = yCb + ( cbHeight / 2 )

2) delete the following clause

2. The luma location ( xColCtrCb, yColCtrCb ) is set equal to the top-left sample of the collocated luma coding block covering the location given by ( xCtr, yCtr ) inside ColPic relative to the top-left luma sample of the collocated picture specified by ColPic.

3) replace

3. The derivation process for subblock-based temporal merging base motion data as specified in clause 8.5.5.4 is invoked with the location ( xCtb, yCtb ), the location ( xColCtrCb, yColCtrCb ), ...

with

2. The derivation process for subblock-based temporal merging base motion data as specified in clause 8.5.5.4 is invoked with the location ( xCtb, yCtb ), the location ( xCtrCb, yCtrCb ), ...

in section 8.5.5.4

4) replace
– the location ( xColCtrCb, yColCtrCb ) of the top-left sample of the collocated luma coding block that covers the below-right center sample.

with

– the location ( xCtrCb, yCtrCb ) of the top-left sample of the luma coding block that covers the below-right center sample.

5) replace

The location ( xColCb, yColCb ) of the collocated block inside ColPic is derived as follows.
– The following applies:
yColCb = Clip3( yCtb,

Min( pic_height_in_luma_samples − 1, yCtb + ( 1 << CtbLog2SizeY ) − 1 ), (8 547)
yColCtrCb + tempMv[1] )

– If subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 1, the following applies:
xColCb = Clip3( xCtb,

Min( SubPicRightBoundaryPos, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 548)
xColCtrCb + tempMv[0] )

– Otherwise ( subpic_treated_as_pic_flag[ SubPicIdx ] is equal to o, the following applies:
xColCb = Clip3( xCtb,

Min( pic_width_in_luma_samples − 1, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 549)
xColCtrCb + tempMv[0] )

with

The location ( xColCb, yColCb ) of the collocated block inside ColPic is derived as follows.
– The following applies:
yColCb = Clip3( yCtb,

Min( pic_height_in_luma_samples − 1, yCtb + ( 1 << CtbLog2SizeY ) − 1 ), (8 547)
yCtrCb + tempMv[1] )

– If subpic_treated_as_pic_flag[ SubPicIdx ] is equal to 1, the following applies:
xColCb = Clip3( xCtb,

Min( SubPicRightBoundaryPos, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 548)
xCtrCb + tempMv[0] )

– Otherwise ( subpic_treated_as_pic_flag[ SubPicIdx ] is equal to o, the following applies:
xColCb = Clip3( xCtb,

Min( pic_width_in_luma_samples − 1, xCtb + ( 1 << CtbLog2SizeY ) + 3 ), (8 549)
xCtrCb + tempMv[0] )

Note that in the current spec text ( xCtrCb ,yCtrCb ) is undefined but is used in section 8.5.5.4.