Incorrect outputs for Subclause 8.5.8

[bheng]

The output from Subclause 8.5.8 is supposed to be resSamples:

"Output of this process is an (nTbW)x(nTbH) array resSamples."

However, when this subclause invokes itself in 3 places, the outputs are incorrect:

"the output is a modified reconstructed picture before in-loop filtering."

These 3 should be changed to output resSamples over the appropriate sub-range within x = 0..nTbW − 1, y = 0..nTbH − 1.

[jlchen]

Good catch! The same problem might also exist in section 8.4.5.1.

This might not be solved easily by setting the output as "resSamples over the appropriate sub-range" since the section 8.7.2 do not have stripe conception as implemented in the software.

[bbross]

Thanks, good but tricky catch! Recursion is beast here but I tried to fix it in JVET-S2001-vH by:

In 8.5.1 changing the three self-invocations from:
"...the output is a modified reconstructed picture before in-loop filtering."
to:

2. "...the output is assigned to the array resSamples[ x ][ y ] with x = 0..newTbW − 1, y = 0..newTbH − 1"
3. "If verSplitFirst is equal to TRUE,...the output is assigned to the array resSamples[ x ][ y ] with x = newTbW..nTbW − 1, y = 0..newTbH − 1"

"Otherwise (verSplitFirst is equal to FALSE),...the output is assigned to the array resSamples[ x ][ y ] with x = 0..newTbW − 1, y = newTbH..nTbH − 1"

That way the resSample array grows when coming back from the recursion branches. This might not be ideal but the most practical solution I could come up with within the given time.

As for the intra TB splitting process in 8.4.5.1, this is OK since the output of that process is a modified reconstructed picture before in-loop filtering and not the TB res sample array. In contrast to the inter process, intra performs prediction and inverse transform on TB basis