Mismatch in intra ref sample filtering between draft and VTM

[bbross]

Reported by Geert.
Change to align draft conditions with VTM:

8.2.4.2.4 Reference sample filtering process
Inputs to this process are:

• a variable refIdx specifying the intra prediction reference line index,
• a variable nTbW specifying the transform block width,
• a variable nTbH specifying the transform block height,
• a variable refW specifying the reference samples width,
• a variable refH specifying the reference samples height,
• the (unfiltered) neighbouring samples refUnfilt[ x ][ y ], with x = −1 − refIdx, y = −1 − refIdx..refH − 1 and x = −refIdx..refW − 1, y = −1 − refIdx,
• a variable cIdx specifying the colour component of the current block.

Outputs of this process are the reference samples p[ x ][ y ], with x = −1 − refIdx, y = −1 − refIdx..refH − 1 and x = −refIdx..refW − 1, y = −1 − refIdx.
The variable nTbS is set equal to ( Log2 ( nTbW ) + Log2 ( nTbH ) ) >> 1.
The variable filterFlag is derived as follows:

• If all of the following conditions are true, filterFlag is set equal to 1:
  • refIdx is equal to 0
  • nTbW * nTbH is greater than 32
  • cIdx is equal to 0
  • one or more of the following conditions is true:
    • predModeIntra is equal to INTRA_PLANAR and nTbW * nTbH is greater than 32
    • predModeIntra is equal to INTRA_ANGULAR34 and nTbS is greater than 2
    • predModeIntra is equal to INTRA_ANGULAR2 and nTbS is greater than 2 and nTbH is greater than or equal to nTbW
    • predModeIntra is equal to INTRA_ANGULAR66 and nTbS is greater than 2 and nTbW is greater than or equal to nTbH
• Otherwise, filterFlag is set equal to 0.