Bug in affine fallback mode trigger (8.5.5.9)

[chhuanb]

Affine fallback mode threshold is set depends on inter_pred_idc[ xCb ][ yCb ] as follow:

– If inter_pred_idc[ xCb ][ yCb ] is equal to PRED_BI and bxWX4 * bxHX4 is less than or equal to 225, fallbackModeTriggered is set equal to 0.
– Otherwise, if inter_pred_idc[ xCb ][ yCb ] is not equal to PRED_BI, both bxWXh * bxHXh is less than or equal to 165, and bxWXv * bxHXv is less than or equal to 165, fallbackModeTriggered is set equal to 0.

According to spec, inter_pred_idc[ xCb ][ yCb ] is inferred to PRED_L0 when not present. So for affine merge, this value cannnot be used to determine affine fallback mode.

[fbossen]

Indeed. predFlagLX[ 0 ][ 0 ] should probably be used instead of inter_pred_idc[ xCb ][ yCb ]. What is a bit unclear to me is whether predFlagL1[ 0 ][ 0 ] is defined when 8.5.5.9 is invoked for X=0.

In 8.5.5.1 we have
– Otherwise (merge_subblock_flag[ xCb ][ yCb ] is equal to 0), for X being replaced by either 0 or 1 in the variables predFlagLX, cpMvLX, MvdCpLX, and refIdxLX, in PRED_LX, and in the syntax element ref_idx_lX, the following ordered steps apply:

Not sure what this actually means. Do we apply each step for X=0 and X=1 before proceeding to the next step, or do we apply all steps for X=0 and then all steps for X=1?

[chhuanb]

Replying to fbossen:

Indeed. predFlagLX[ 0 ][ 0 ] should probably be used instead of inter_pred_idc[ xCb ][ yCb ]. What is a bit unclear to me is whether predFlagL1[ 0 ][ 0 ] is defined when 8.5.5.9 is invoked for X=0.

In 8.5.5.1 we have
– Otherwise (merge_subblock_flag[ xCb ][ yCb ] is equal to 0), for X being replaced by either 0 or 1 in the variables predFlagLX, cpMvLX, MvdCpLX, and refIdxLX, in PRED_LX, and in the syntax element ref_idx_lX, the following ordered steps apply:

Not sure what this actually means. Do we apply each step for X=0 and X=1 before proceeding to the next step, or do we apply all steps for X=0 and then all steps for X=1?

I think we apply all steps for X=0 and then all step2 for X=1.
One possible solution to fix this issue is to derived the pred-directoin in 8.5.5.1 and 8.5.5.2, and use it as input when invoked 8.5.5.9.

[jlchen]

Thanks to Huanbang and Frank for the discussion. Agree with the idea of Huanbang's solution to fix this. Basically go through the loop of X = 0..1 for parameter derivation, then invoke 8.5.5.9 with loop of x = 0..1.

[jlchen]

The related changes will be included in JVET-S2001-vE release and sent to related experts for reviewing.