Mismatch between Spec and Software related to ISP

[k-unno]

There are mismatches between VTM-4.0 and VVC D4 on decision of intra interpolation filter for angular prediction in case of using ISP.

As written in bellow, the specification written in VVC D4 seems be not correct.

In VTM-4.0, if the width (or height which decided dependent on angle of prediction) of block is less than or equal to 8 pixels, DCTIF (reffered as Cubic in the software) will be applied.

#if JVET_M0102_INTRA_SUBPARTITIONS
          const bool                 useCubicFilter = useISP ? ( width <= 8 ) : ( !useFilteredPredSamples || multiRefIdx > 0 );
#else

On the other hand, in VVC D4, if the width (or height which decided dependent on angle of prediction) of block is greater than 8 pixels, DCTIF (reffered as fC) will be applied.

– If one or more of the following conditions is true, filterFlag is set equal to 0.

• predModeIntra is equal to INTRA_ANGULAR2, INTRA_ANGULAR34 or INTRA_ANGULAR66
• refIdx is not equal to 0
• IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT and cIdx is equal to 0 and predModeIntra is greater than or equal to INTRA_ANGULAR34 and nW is greater than 8
• IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT and cIdx is equal to 0 and predModeIntra is less than INTRA_ANGULAR34 and nH is greater than 8.

According to previous specification (e.g. JEM), specification in the VTM-4.0 seems be correct.

Additionally, nW and nH in VVC D4 means CU width and height in case of ISP applied, but width in VTM-4.0 means TU width or height.

[k-unno]

One possible modifications is as follows in the section 8.4.4.2.7 of VVC D4.

---

– If one or more of the following conditions is true, filterFlag is set equal to 0.

– predModeIntra is equal to INTRA_ANGULAR2, INTRA_ANGULAR34 or INTRA_ANGULAR66
– refIdx is not equal to 0
– IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT and cIdx is equal to 0 and predModeIntra is greater than or equal to INTRA_ANGULAR34 and nTbW is less than or equal to 8
– IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT and cIdx is equal to 0 and predModeIntra is less than INTRA_ANGULAR34 and nTbH is less than or equal to 8.

– Otherwise, the following applies:

– The variable minDistVerHor is set equal to Min( Abs( predModeIntra − 50 ), Abs( predModeIntra − 18 ) ).
– The variable intraHorVerDistThres[ nTbS ] is specified in Table 8-4.
– The variable filterFlag is derived as follows:

– If one or more of the following conditions is true, filterFlag is set equal to 1.

– minDistVerHor is greater than intraHorVerDistThres[ nTbS ]
– wideAngle is equal to 1
– IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT.

– Otherwise, filterFlag is set equal to 0.

---

[deluxan]

I agree with the first part of the changes regarding
"nTbW is less than or equal to 8"
and
"nTbH is less than or equal to 8".

However, could you explain why it is necessary to add the following?

" – If one or more of the following conditions is true, filterFlag is set equal to 1.

– minDistVerHor is greater than intraHorVerDistThres[ nTbS ]
– wideAngle is equal to 1
– IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT.

"

[k-unno]

Replying to deluxan:

Thank you for your confirmation and comment.

First of all, my proposal is to modify the spec according to the software.

In the software, DCTIF will be applied only when TU width (or height) is less than or equal to 8 in ISP case as follows.

#if JVET_M0102_INTRA_SUBPARTITIONS
          const bool                 useCubicFilter = useISP ? ( width <= 8 ) : ( !useFilteredPredSamples || multiRefIdx > 0 );
#else

On the other hand, if the second part of my modification is not appilied, when both of "minDistVerHor is greater than intraHorVerDistThres[ nTbS ]" and "wideAngle is eaual to 1" are FALSE, DCTIF will also be applied in ISP case.

To solve this mismatch between the spec and the software, I think the second part of my proposal is necessary.

I agree with the first part of the changes regarding
"nTbW is less than or equal to 8"
and
"nTbH is less than or equal to 8".

However, could you explain why it is necessary to add the following?

" – If one or more of the following conditions is true, filterFlag is set equal to 1.

– minDistVerHor is greater than intraHorVerDistThres[ nTbS ]
– wideAngle is equal to 1
– IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT.

"

[bbross]

Fixed in D5 v6