Spec issues related to MTS

[jason]

1) Mismatch between draft 2 and VTM/BMS-2.0.1
According to the MTS scheme adopted in the last meeting and VTM/BMS-2.0.1, when intra predicted block has less than 3 non-zero coefficients and cu_mts_flag[ x0 ][ y0 ] is equal to 1, DST-VII is applied both horizontally and vertically. (mts_idx is not signalled)
However, in 7.4.5.8, mts_idx[ x0 ][ y0 ] is inferred to be equal to -1, which means DCT-II is applied both horizontally and vertically for that case.

Semantics and decoding process related to this issue should be revised.

Simple fix:
In 7.4.5.8,
When mts_idx[ x0 ][ y0 ] is not present, it is inferred as follows:

If cu_mts_flag[ x0 ][ y0 ] is equal to 0, mts_idx[ x0 ][ y0 ] is inferred to be equal to -1.
Otherwise (cu_mts_flag[ x0 ][ y0 ] is equal to 1), mts_idx[ x0 ][ y0 ] is inferred to be equal to 0.

In 8.3.4.1,
The variable trTypeHor specifying the horizontal transform kernel and the variable trTypeVer specifying the vertical transform kernel are derived as follows:
If cIdx is equl to 0, trTypeHor and trTypeVer are derived in Table 8-6 depending on mts_idx[ x ][ y ] and CuPredMode[ x ][ y ].
Otherwise (cIdx is equal to 1 or 2), trTypeHor and trTypeVer are equal to 0.

Table 8-6

mts_idx[ x ][ y ]	CuPredMode[x][y]==MODE_INTRA	CuPredMode[x][y]==MODE_INTER
	trTypeHor, trTypeVer	trTypeHor, trTypeVer
-1 (inferred)	0, 0	0, 0
0 (00 / inferred)	1, 1	2, 2
1 (01)	2, 1	1, 2
2 (10)	1, 2	2, 1
3 (11)	2, 2	1, 1

2)Redundant condition for mts_idx in 7.3.4.8
When cIdx is equal to 0 (luma TB) amd cu_mts_flag[ x0 ][ y0 ] is equal to 1, transform_skip_flag[ x0 ][ y0 ][ cIdx ] is inferred to be equal to 0 according to 7.4.5.8.
Thus, checking whether the transform skip flag is equal to 0 or not can be removed in the last of 7.3.4.8

Simple fix:
In 7.3.4.8
if ( cu_mts_flag[ x0 ][ y0 ] && ( cIdx == 0 ) &&

( ( CuPredMode[ x0 ][ y0 ] == MODE_INTRA && numSig > 2)

( CuPredMode[ x0 ][ y0 ] == MODE_INTER ) ) )

mts_idx[ x0 ][ y0 ]

[bbross]

fixed in VVC D2 v7