JVET-P1026 - DST-7/DCT-8 TUs with coefficients outside 16x16 region

[bheng]

I assume it is still intended that MTS (DST-7/DCT-8) TUs are not allowed to have coefficients outside the lowest frequency 16x16 region. Is that correct?

And I believe the MtsZeroOutSigCoeffFlag was intended to prevent MTS when any luma coefficients are outside this region. Specifically, this condition.

( LastSignificantCoeffX > 15 | | LastSignificantCoeffY > 15 )

However, it fails to do so. There can be many coefficients after (15,15) in diagonal scan order that are outside the 16x16 region.

Each coefficient group would have to be checked individually for this to work right. Using LastSignificantCoeff alone is not enough.

[jani.lainema]

You are right. MTS TUs are not allowed to have coefficients outside of the 16x16 area. In the current design that is handled as a bitstream constraint and we should reflect that also in the specification. Alternatively we could change the design as you suggest and add checks in the coefficient group level. That might be helpful and I recommend bringing an input contribution to the next meeting proposing this aspect.

[bbross]

There are three cases to be distinguished here:

1. Explicit MTS

The variable MtsZeroOutSigCoeffFlag is used to gate the parsing of mts_idx, i.e. when MtsZeroOutSigCoeffFlag is equal to 0, mts_idx is not parsed and hence inferred to be equal to 0.
This is not a bitstream constraint and should be in VTM as well.
If the parsing of mts_idx is not conditioned that way in VTM, it is a VTM issue and should be fixed
-> However, I agree with Brian that the check based in last x / y is not enough and should be performed for each location in scan order. This, I will fix in my current draft.

2. Implicit MTS (intra/ISP)

Here, a transform other than DCT-2 (trType>0) is only set if the respective block size for that direction is >=4 and <= 16.
So here, no transform other than DCT-2 is selected for blocks >16x16 and thus no zero out is needed.

3. Implicit MTS (SBT)

-> Here there is no check and maybe this is the place where a bitstream restriction is needed.

Can someone please double check and confirm my findings also w.r.t VTM implementation?

[bheng]

This is my understanding at the moment:

1.) If this is supposed to be prevented by the syntax (my preference) rather than just a bitstream conformance constraint, both the spec text and the VTM need some changes. I believe VTM implements what it says in the current draft text, but as mentioned, checking last x/y is not enough.

2.) Correct.

3.) The syntax prevents anything for implicit SBT MTS outside 16x16 the same way it did prior to JVET-P1026. It uses conditions like the following to effectively clip the TU size.

if( ( sps_mts_enabled_flag && cu_sbt_flag &&

log2TbWidth < 6 && log2TbHeight < 6 ) )

&& cIdx = = 0 && log2TbHeight > 4 )
log2ZoTbHeight = 4

[bbross]

Thanks for the quick check Brian.

1.) Exactly, the condition in the draft and apparently in VTM as well only checks whether last x/y is inside the 16x16 block but not if there are coefficients outside this block along the whole scan path.
To me this seems odd but that is what we adopted as it is already in the original draft text of JVET-P0196 which is part of JVET-P1026.
@Jani, could you provide text for the missing bitstream constraint?

Just for my understanding, currently the VTM encoder only makes sure that there are no coefficients outside 16x16 for mst_idx > 0.
However for the DCT-2 case, there can be non-zero coefficients outside the 16x16 blocks if last pos is (15,15).
In this case, mts_idx is signalled but always 0.
If we would check every position in scan order instead just the last one, mts_idx would not be parsed and inferred to be equal to 0 instead.
To me this sounds more efficient but since it is a technical I guess it should be proposed at the next meeting.

3.) Yes, SBT should be OK although some unnecessary parenthesis can be removed in the conditions.

[jani.lainema]

Ben: That is aligned with my understanding. I will provide the text for the missing bitstream constraint.

[bbross]

Thanks and it will be fixed in JVET-P2001-vC by adding the following constraint:

It is a requirement of bitstream conformance that mts_idx shall be equal to 0 if in the current coding unit at least one coded_sub_block_flag[ xS ][ yS ] in the residual_coding( x0, y0, log2TbWidth, log2TbHeight, cIdx ) syntax structure is not equal to 0 for cIdx equal to 0 and xS or yS greater than 3.