VTM 6.0 crash in SCC TGM conditions

[ksuehring]

Crash was also observed in class TGM in the followings sequences/QPs:

[LB][ChineseEditing][QP27]
[LB][ChineseEditing][QP32]
[LB][ChineseEditing][QP37]
[LB][Desktop][QP22]
[LB][Desktop][QP27]
[LB][Desktop][QP32]
[LB][Desktop][QP37]

[LP][ChineseEditing][QP22]
[LP][ChineseEditing][QP27]
[LP][ChineseEditing][QP32]
[LP][ChineseEditing][QP37]
[LP][Desktop][QP22]
[LP][Desktop][QP27]
[LP][Desktop][QP32]
[LP][Desktop][QP37]

Unfortunately after 400 to 500 frames.

This may be related to #484, but the crash still occurs when JVET_O0050_LOCAL_DUAL_TREE is disabled.

I tried disabling JVET_O0119_BASE_PALETTE_444 because this seemed related, but even after fixing a number of macro encapsulations (context definitions, scaling lists), the encoder just crashed in the first frame.

[yhchao]

Hello Karsten,
Can you specify which color format (YUV4:2:0?) you are running.
Palette mode should be disabled for 4:2:0 in high level.

[tung.nguyen]

The configuration LB, ChineseEditing, QP37 crashes at POC 517.

When debugging on a Windows machine using VS2017, the encoder does not crash at POC 517, but already in POC 516, the check at the line 2800 in CABACWriter.cpp failed:

else if( cu.sbtInfo && partitioner.canSplit( PartSplit( cu.getSbtTuSplit() ), cs ) )
{

CHECK( !split, "transform split implied - sbt" );

}

[ksuehring]

Replying to yhchao:

Hello Karsten,
Can you specify which color format (YUV4:2:0?) you are running.
Palette mode should be disabled for 4:2:0 in high level.

I think that was unrelated. I was looking at code related to #484

[ksuehring]

It turns out that the issue was caused by a memory limit set on the compute cluster. VTM 6 seems to use more memory than previous versions. The peak usage must have exceeded 6.5GB RAM.