Opened 4 years ago

Closed 4 years ago

#818 closed defect (invalid)

Extend border of picture correctly

Reported by: pierrick.bouvier Owned by:
Priority: major Milestone:
Component: VTM Version:
Keywords: Cc: vzakharc, yuwenhe, jvet@…


There are two problems, that need to be fixed together.

First, margin must be increased to ensure we write the whole stride (line) in a picture. If not, we may access to uninitialized memory when using reconstructed picture as reference (left and right margin are not completely extended). This bug was found thanks to Memory sanitizer.

Doing this led me to another issue, where code was clearly incorrect.
In source/Lib/CommonLib/Picture.cpp and in source/Lib/CommonLib/Buffer.h, when extend image copying margin of first line. Alas, we are going one line too far:

  // pi is still (-marginX, height-1)
  p -= ( ( h - 1 ) * s );
  // pi is now (-marginX, 0)
  for( int y = 0; y < margin; y++ )
    ::memcpy( p -( y+ 1 ) * s, p, sizeof( T ) * ( w + ( margin << 1 ) ) );

Iteration should be from 1 to margin, and dest should be p -y*s.

In latest iteration of loop, we write line (margin), that is outside of picture. Luckily, it works most of the time because margin is (most of the time) incorrect (too small). Thus, it prevents writing out of picture memory.

Since margin is now correct, we started to trigger edge cases where memory after picture is touched, resulting in various crashes. This, was found using Address sanitizer.

Those two fixes result in a correct border extension.

Change history (3)

comment:1 Changed 4 years ago by pierrick.bouvier

  • Component changed from 360Lib to VTM

comment:2 Changed 4 years ago by pierrick.bouvier

Seems like fix concerning memcpy of margin is incorrect (I missed pointer initialization).

Still investigating original issue concerning uninitialized memory.

comment:3 Changed 4 years ago by pierrick.bouvier

  • Resolution set to invalid
  • Status changed from new to closed

Closing the ticket.

Note: See TracTickets for help on using tickets.