Decoding a stream with -d 8 -o out.yuv to get 8-bit output crashes when writing output frames. The problems seem to be in TVideoIOYuv::write. One when copying into an 8-bit picture, one when writing from it. Wrong strides are being used.

The attached patch shows a possible solution.

I run JEM1.0 code with valgrind and found the problem of uninitialized memory . The error is as below. It is generated with LCU16x16 All Intra setting.

==26659== Use of uninitialised value of size 8
==26659==    at 0x484FC7: ContextModel::getEntropyBits(short) (ContextModel.h:118)
==26659==    by 0x4AA525: TEncSbac::estCBFBit(estBitsSbacStruct*) (TEncSbac.cpp:2383)
==26659==    by 0x4AA43F: TEncSbac::estBit(estBitsSbacStruct*, int, int, ChannelType) (TEncSbac.cpp:2355)
==26659==    by 0x4A0F36: TEncEntropy::estimateBit(estBitsSbacStruct*, int, int, ChannelType) (TEncEntropy.cpp:944)
==26659==    by 0x4B045D: TEncSearch::xIntraCodingTUBlock(TComYuv*, TComYuv*, TComYuv*, short**, bool, unsigned int&, ComponentID, TComTU&, int, unsigned int*) (TEncSearch.cpp:1527)
==26659==    by 0x4B1237: TEncSearch::xRecurIntraCodingLumaQT_RSAF(TComYuv*, TComYuv*, TComYuv*, short**, unsigned int&, bool, double&, TComTU&) (TEncSearch.cpp:1924)
==26659==    by 0x4B1922: TEncSearch::xRecurIntraCodingLumaQT_RSAF(TComYuv*, TComYuv*, TComYuv*, short**, unsigned int&, bool, double&, TComTU&) (TEncSearch.cpp:2067)
==26659==    by 0x4B69CE: TEncSearch::estIntraPredLumaQT(TComDataCU*, TComYuv*, TComYuv*, TComYuv*, TComYuv*, short**) (TEncSearch.cpp:3669)
==26659==    by 0x49BDD1: TEncCu::xCheckRDCostIntra(TComDataCU*&, TComDataCU*&, double&, PartSize, int&) (TEncCu.cpp:2293)
==26659==    by 0x496969: TEncCu::xCompressCU(TComDataCU*&, TComDataCU*&, unsigned int, PartSize) (TEncCu.cpp:1077)
==26659==    by 0x4946A9: TEncCu::compressCtu(TComDataCU*) (TEncCu.cpp:398)
==26659==    by 0x45BE96: TEncSlice::compressSlice(TComPic*, bool, bool) (TEncSlice.cpp:855)

One bug I found is:

In TEnslice.cpp Ln 666,

  m_pcEntropyCoder->setEntropyCoder   ( m_pppcRDSbacCoder[0][CI_CURR_BEST] );
  m_pcEntropyCoder->resetEntropy      ( pcSlice );

here only the CI_CURR_BEST is initialized.

Later, in TEncSearch.cpp Line 2011 and Line 2591, in function xRecurIntraCodingLumaQT_RSAF() and xRecurIntraCodingLumaQT The sbac status of CI_QT_TRAFO_ROOT loaded is possibly uninitialized.

#if COM16_C806_EMT
        if( bSaveEmtResults && ( uiSingleCbfLuma || !bAllIntra || !m_pcEncCfg->getUseFastIntraEMT() ) )
        {
          m_pcRDGoOnSbacCoder->load ( m_pppcRDSbacCoder[ uiFullDepth ][ CI_QT_TRAFO_ROOT ] );
        }
#endif

The above problem caused windows program produce different output from linux. But fixing it did not fix all the memory problem. there are other memory bugs need to be fixed.

Jane

Work spaces for newer version of Visual Studio should be added.

It's probably easy to merge them from HM-dev.