Mismatch VTM/spec in DMVR process for bitdepth > 10

[forayr]

In VTM, before applying the bilinear interpolation filter for the DMVR, the mv are clipped to the picture size + 8 border pixel (InterPrediction::xinitMC()). Doing this, the fractional part of the MV become 0 when the mv is clipped.

In the specification (8.5.3.2.2), this clip is not present, only the position of the pixel is clipped to the picture when fetching its value. The fractional part of the MV is therefore not modified.

So we can have a case where no filtering is applied in a direction in the VTM, but this filtering is done in the spec.
The issue is only present when xFrac != 0 and yFrac != 0.

For example: if we consider a case where the mv target an area outside the picture and the margin horizontally, with xFrac = 8 and yFrac = 8, with the pixel value at the picture border:
75
56

In VTM, no filtering is applied horizontally, vertically we obtains

(75 * 8 + 56 * 8 + 1<<(Bitdepth - 7)) >> (Bidepth - 6) = 16 (if Bitdepth = 12)

In spec, the filtering is applied both horizontally and vertically:

horizontally:

(75 * 8 + 75 * 8 + 1<<(Bitdepth - 7)) >> (Bidepth - 6) = 19 (if Bitdepth = 12)
(56 * 8 + 56 * 8 + 1<<(Bitdepth - 7)) >> (Bidepth - 6) = 14 (if Bitdepth = 12)

vertically:

(19 * 8 + 14 * 8 + 8) >> 4 = 17

So we don't have the same result after interpolation. This issue is present only when Bitdepth > 10.

[Tomohiro Ikai]

Good catch.

75 75 75 75 75 75 75 75
56 56 56 56 56 56 56 56

So we assume 75 and 75's interporation is always 75 but in DMVR's reduced-precision intermediate case*1, it seems not.

19 << (BitDepth-10) = 76 (not 75)

Then we should*2 keep xFrac/yFrac part even if it's very large MVs and the MVs may indicate far from the border.

ipython check:

In [33]: for b in range(10,18,2): h0=(75*8+75*8+(1<<(b-7)))>>(b-6); h1=(56*8+56*8+(1<<(b-7)))>>(b-6); spec=(h0*8+h1*8+(b-1))>>4; vtm=(75*8+56*8+(1<<(b-7)))>>(b-6); print("bit:", b, "h0", h0, "h1", h1, "spec:", spec, "vtm:", vtm, "back:", spec<<(b-10), vtm<<(b-10), (56+75)/2)

bit: 10 h0 75 h1 56 spec: 66 vtm: 66 back: 66 66 65.5
bit: 12 h0 19 h1 14 spec: 17 vtm: 16 back: 68 64 65.5
bit: 14 h0 5 h1 4 spec: 5 vtm: 4 back: 80 64 65.5
bit: 16 h0 1 h1 1 spec: 1 vtm: 1 back: 64 64 65.5

In my opinion*3, VTM should be fixed. Am I right? Merge request is really appreciated.

*1: DMVR assumes kind of MSB part (or constant precision) should be sufficient for MV refinement and removes kind of LSB part in intermediate.
*2:because how interporate affects the value (fraction part matters)
*3: xinitMC's margin of 8 looks implement dependent then we might not assume this margin.

[forayr]

I agree that the VTM should be fixed. But I don't know how to do this with the current architecture.
It would be great if an expert in DMVR could take a look into it.

[fbossen]

Suggested fix:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2182