id,summary,reporter,owner,description,type,status,priority,milestone,component,version,resolution,keywords,cc
1431,Align VTM clipping after inverse transform with specification,adybrowne,,"
In section 8.7.4.1 (JVET-S2001vH), the equation (1163) specifies the clipping process for the inverse transform.
2. When nTbH and nTbW are both greater than 1, the intermediate sample values g[ x ][ y ] with x = 0..nonZeroW − 1, y = 0..nTbH − 1 are derived as follows:
g[ x ][ y ] = Clip3( CoeffMin, CoeffMax, ( e[ x ][ y ] + 64 ) >> 7 ) (1163)
Thus clipping is only specified at the intermediate stage for 2D transforms. However VTM (TrQuant::xIT) clips both the intermediate and final values to be between CoeffMin and CoeffMax for both 1D and 2D transforms.
This clipping leads to significant losses for 16 bit content when the ExtendedPrecision flag is not enabled.
It is suggested that VTM is brought into line with the spec.
This is most easily achieved by clipping the final value only to the numeric limits of the Pel data type (16 bits normally, 32 bits if the high bit depth macro is enabled). This is the solution adopted in HM.
",defect,closed,minor,,VTM,VTM-10.2,fixed,,ksuehring XiangLi fbossen jvet@…