Extend border of picture correctly

[pierrick.bouvier]

There are two problems, that need to be fixed together.

First, margin must be increased to ensure we write the whole stride (line) in a picture. If not, we may access to uninitialized memory when using reconstructed picture as reference (left and right margin are not completely extended). This bug was found thanks to Memory sanitizer.

Doing this led me to another issue, where code was clearly incorrect.
In source/Lib/CommonLib/Picture.cpp and in source/Lib/CommonLib/Buffer.h, when extend image copying margin of first line. Alas, we are going one line too far:

  // pi is still (-marginX, height-1)
  p -= ( ( h - 1 ) * s );
  // pi is now (-marginX, 0)
  for( int y = 0; y < margin; y++ )
  {
    ::memcpy( p -( y+ 1 ) * s, p, sizeof( T ) * ( w + ( margin << 1 ) ) );
  }

Iteration should be from 1 to margin, and dest should be p -y*s.

In latest iteration of loop, we write line (margin), that is outside of picture. Luckily, it works most of the time because margin is (most of the time) incorrect (too small). Thus, it prevents writing out of picture memory.

Since margin is now correct, we started to trigger edge cases where memory after picture is touched, resulting in various crashes. This, was found using Address sanitizer.

Those two fixes result in a correct border extension.

​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/1233

[pierrick.bouvier]

Seems like fix concerning memcpy of margin is incorrect (I missed pointer initialization).

Still investigating original issue concerning uninitialized memory.

[pierrick.bouvier]

Closing the ticket.