Missing the check of maximum transform size when CtbSizeY is 32

[chiaming]

In VVC D8, it mentioned:

When CtbSizeY is less than 64, the value of sps_max_luma_transform_size_64_flag shall be equal to 0.

It should add checks accordingly to avoid the maximum transform size is set to 64 when CtbSizeY is 32.
Suggest patch is also included.

[m.sarwer]

There is a contribution (JVET-R0097) related to this topic.

[chiaming]

Thanks to Mohammed's comment.

To replace the bitstream conformance constraint, another method is to move the sps_max_luma_transform_size_64_flag in front of the syntax of CTU size, and signal the size difference between CTU size and the maximum transform size instead. For example,

seq_parameter_set_rbsp( ) {

...
sps_max_luma_transform_size_64_flag
sps_log2_ctu_size_minus5_log2_max_transform_size
...
sps_max_luma_transform_size_64_flag
...

}

sps_max_luma_transform_size_64_flag equal to 1 specifies that the maximum transform size in luma samples is equal to 64. sps_max_luma_transform_size_64_flag equal to 0 specifies that the maximum transform size in luma samples is equal to 32.
When CtbSizeY is less than 64, the value of sps_max_luma_transform_size_64_flag shall be equal to 0.
The variables MinTbLog2SizeY, MaxTbLog2SizeY, MinTbSizeY, and MaxTbSizeY are derived as follows:

MinTbLog2SizeY = 2
MaxTbLog2SizeY = sps_max_luma_transform_size_64_flag ? 6 : 5
MinTbSizeY = 1 << MinTbLog2SizeY
MaxTbSizeY = 1 << MaxTbLog2SizeY

sps_log2_ctu_size_minus5_log2_max_transform_size plus 5 specifies the difference between the base 2 logarithm of luma coding tree block size of each CTU and the base 2 logarithm of maximum luma transform size. The value of sps_log2_ctu_size_minus5_log2_max_transform_size shall be in the range of 0 to (2 − sps_max_luma_transform_size_64_flag), inclusive. The value in the range of (3 − sps_max_luma_transform_size_64_flag) to 3 for sps_log2_ctu_size_minus5_log2_max_transform_size is are reserved for future use by ITU-T | ISO/IEC.
The variables CtbLog2SizeY and CtbSizeY are derived as follows:

CtbLog2SizeY = sps_log2_ctu_size_minus5_log2_max_transform_size + 5MaxTbLog2SizeY
CtbSizeY = 1 << CtbLog2SizeY