id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
857	Mismatch between spec and VTM in derivation of coding sub-block boundaries	emora		"There seems to be a mismatch between the spec and VTM software regarding the derivation of max filtering length P/Q for coding sub-block boundaries. Section 8.8.3.4 states (for example, for vertical edges) that max filter length P/Q shall be equal to 2 (equations 1255 and 1256) if one or more of the following conditions are true:
–	xEdge is equal to 1,
–	xEdge is equal to ( nCbW / 8 ) − 1,
–	edgeTbFlags[ x − sbW ][ y ] is equal to 1,
–	edgeTbFlags[ x + sbW ][ y ] is equal to 1,

In VTM however, the first condition (xEdge == 1) is never checked. The code reads

""else if ( x > 0 && ( m_transformEdge[cIdx][ctuXOff+x-8][ctuYOff+y] | | ( (x + 8) >= areaPu.width ) | | m_transformEdge[cIdx][ctuXOff+x+8][ctuYOff+y] ) )""

I believe software is correct in this regards and so, first condition should be deleted. Same can be said for horizontal edges."	defect	closed	minor		spec		fixed		ksuehring bbross XiangLi fbossen jvet@…