Wrong range check for sps_num_subpics_minus1

[hbteo]

Spec description:
sps_num_subpics_minus1 plus 1 specifies the number of subpictures in each picture in the CLVS. The value of sps_num_subpics_minus1 shall be in the range of 0 to Ceil( sps_pic_width_max_in_luma_samples ÷ CtbSizeY ) * Ceil( sps_pic_height_max_in_luma_samples ÷ CtbSizeY ) − 1, inclusive. When not present, the value of sps_num_subpics_minus1 is inferred to be equal to 0.

VTM implementation:
READ_UVLC(uiCode, "sps_num_subpics_minus1"); pcSPS->setNumSubPics(uiCode + 1);
CHECK(uiCode > (pcSPS->getMaxPicWidthInLumaSamples() / (1 << pcSPS->getCTUSize())) * (pcSPS->getMaxPicHeightInLumaSamples() / (1 << pcSPS->getCTUSize())) - 1, "Invalid sps_num_subpics_minus1 value");

Should be:
CHECK(uiCode > (pcSPS->getMaxPicWidthInLumaSamples() / pcSPS->getCTUSize()) * (pcSPS->getMaxPicHeightInLumaSamples() / pcSPS->getCTUSize()) - 1, "Invalid sps_num_subpics_minus1 value");

[ksuehring]

I think this would still be slightly incorrect, since this is an integer division, right?

[hbteo]

Replying to ksuehring:

I think this would still be slightly incorrect, since this is an integer division, right?

Yes, one possible change maybe:
CHECK(uiCode > ( (pcSPS->getMaxPicWidthInLumaSamples() + pcSPS->getCTUSize() - 1) / pcSPS->getCTUSize() ) * ( (pcSPS->getMaxPicHeightInLumaSamples() + pcSPS->getCTUSize() - 1) / pcSPS->getCTUSize() ) - 1, "Invalid sps_num_subpics_minus1 value");

[fbossen]

Suggested fix:
​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2187

[ksuehring]

The suggested fix was merged.