PROF with weighted prediction

[bheng]

There is a mismatch with the text when both weighted prediction and PROF are enabled for a CU.

The text would appear to apply both (PROF followed by weighted prediction).

The VTM software only applies weighted prediction. The PROF refinement code is not used. The functions xWeightedPredictionBi() and xWeightedPredictionUni() are used instead.

It is not clear what the intended behavior is supposed to be. Perhaps it is is intended to work like BDOF, where only BDOF is applied and weighted prediction is disabled?

[wchen1014]

Thank you for pointing out. When PROF is applied on uni-prediction, weighted prediction is used. But when PROF is applied on bi-prediction, weighted prediction is not used.

To fix the bug, the VTM code has been updated in this branch: ​https://vcgit.hhi.fraunhofer.de/wchen1014/VVCSoftware_VTM/tree/JVET-O0070.
Please let us know if the bug is really fixed. Thank you.

We will make SPEC changes accordingly later.

[hanhuang]

The SW should be modified to align with text. I don't see the reason to disable weighted prediction when PROF is applied on bi-prediction. The BCW can be applied for affine in bi-prediction.

[wchen1014]

Yes, weighted prediction should be enabled no matter PROF is applied in bi or uni.

What I mean in last comment was that weighted prediction is already enabled when PROF applied in uni, but not when PROF is applied in bi. This was the bug we identified last time.

In the code link we attached, the bug was fixed. Please review the code and let us know if any remaining issues. We will submit a merge request later.

[bheng]

Thanks for the fix, but as far as I can tell, I see no change in decoder behavior.

It looks like most of your changes are in xWeightedAverage. Unless I'm missing something, it seems that this code doesn't get executed when weighted prediction is enabled. I see the following.

Weighted Bi-Pred
xWeightedPredictionBi is called at this line:

if( !pu.cu->triangle && (!dmvrApplied) && (!bioApplied) && pps.getWPBiPred() && slice.getSliceType() == B_SLICE && pu.cu->GBiIdx==GBI_DEFAULT)
{

xWeightedPredictionBi( pu, srcPred0, srcPred1, pcYuvPred, m_maxCompIDToPred );

}

So, xWeightedAverage never gets called.

Weighted Uni-Pred
xWeightedPredictionUni is called at this line

else if( !pu.cu->triangle && pps.getUseWP() && slice.getSliceType() == P_SLICE )
{

xWeightedPredictionUni( pu, srcPred0, REF_PIC_LIST_0, pcYuvPred, -1, m_maxCompIDToPred );

}

So, xWeightedAverage never gets called.

Additionally, xPredInterUni gets called at this line with parameter "bi" set equal to "true"

if( ( (pps.getUseWP() && slice.getSliceType() == P_SLICE)

(pps.getWPBiPred() && slice.getSliceType() == B_SLICE) ) )

{

xPredInterUni ( pu, eRefPicList, pcMbBuf, true

, bioApplied
, true, true

);

}

xPredAffineBlk therefore gets called with "bi" set equal to "true", and ApplyProf never runs.

[wchen1014]

For bi-prediction, PROF has its own integration of weighted prediction for code efficiency purpose, which is inside applyWpBiPROFCore(). In other words, PROF does not use addWeightBi(). If you compare the two functions, you will notice that applyWpBiPROFCore() combines optical flow and weighted prediction.

For uni-prediction, PROF is applied in harmony without changing weighted prediction. For example, in motionCompensation(), xPredInterUni() -> xPredAffineBlk(), then xWeightedPredictionUni().

In summary, for both uni and bi, PROF is now enabled with weighted prediction.

[bheng]

For WP bi-prediction, you are expecting this to be run, yes?

xWeightedAverage() -> xApplyBiPROF() -> applyWpBiPROF()

Please take another look at the code below.

if( !pu.cu->triangle && (!dmvrApplied) && (!bioApplied) && pps.getWPBiPred() && slice.getSliceType() == B_SLICE && pu.cu->GBiIdx==GBI_DEFAULT)
{

xWeightedPredictionBi( )

}
else if( !pu.cu->triangle && pps.getUseWP() && slice.getSliceType() == P_SLICE )
{

xWeightedPredictionUni( )

}
else
{

xWeightedAverage( )

}

The first if-statement is going to pass when WP is enabled for PROF. Therefore your WpBiPROF code in the "else" will never be called.

So, PROF is currently disabled whenever WPBiPred is enabled in B-slices.

For WP uni-prediction, you are expecting this to be run, yes?

motionCompensation()-> xPredInterUni() -> xPredAffineBlk() -> applyPROF()

Please take another look at the code below.

if( ( (pps.getUseWP() && slice.getSliceType() == P_SLICE) | | (pps.getWPBiPred() && slice.getSliceType() == B_SLICE) ) )
{

xPredInterUni ( pu, eRefPicList, pcMbBuf, true <--- NOTE: bi set equal to true

, bioApplied
, true, true

);

}

And inside xPredAffineBlk():

if (!bi)
{

g_pelBufOP.applyPROF(dstY, dstBuf.stride, src, dstExtBuf.stride, blockWidth, blockHeight, gX, gY, gradXBuf.stride, dMvScaleHor, dMvScaleVer, blockWidth, shiftNum, offset, clpRng);

}

When WP is enabled "bi" is always true, even for uni-pred.
When "bi" is true, applyPROF() is not called.
So, PROF is currently disabled when WP is enabled in P-slices.

Therefore, I still believe PROF is currently disabled whenever WP or WPBi are enabled.

[zhou]

The fact that in applyPROFCore() the output is always clipped independent of the weighted prediction does not seem to be right, i.e.

dI = (src[w] + dI + offset) >> shiftNum;
dst[w] = (Pel)ClipPel(dI, clpRng);

Whether the refined prediction samples should be clipped or not should depend on whether the weighted prediction is on or not. If the weighted prediction is on, the output should be

dst[w] = src[w] + dI

[XiangLi]

Fixed in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1061