In JEM4 software, the following code in TEncSearch.cpp causes different coding performance in some particular sequences/QP when D0127 is on/off.

#if JVET_D0127_REDUNDANCY_REMOVAL

updateCandList(uiMode, uiSad, numModesForFullRD+2, uiHadModeList, CandHadList);

We found that the intra mode candidate lists in the uiHadModeList array could be diffrent from the original order if the HAD cost of two intra modes are the same. In the situation, the coding performance may be different when the NSST index is equal to 3. The situation will cause the syntax change. Suggest to correct it.

The implementations of zero-out for large transforms are not consistent in JEM-5 and earlier versions. In one place, the non-zero-out region is checked to be the top-left quarter of a block,

#if JVET_C0046_ZO_ASSERT && JVET_C0046_ZO_ASSERT_CODED_SBK_FLAG

else if ( (uiLog2BlockWidth + uiLog2BlockHeight) > TH_LOG2TBAREASIZE &&

(!pcCU->getTransformSkip(compID) && !pcCU->getCUTransquantBypass(uiAbsPartIdx) ))

{

if ( iCGPosY >= (codingParameters.heightInGroups / 2)

iCGPosX >= (codingParameters.widthInGroups / 2) )

{

coded_sbk_flag(iCGX,iCGY) shall be equal to 0 assert(0 == uiSigCoeffGroupFlag[iCGBlkPos]);

}

}

#endif

While in another place, the non-zero-out region is checked to be no larger than 32x32 block, as below

assert((posLastX < JVET_C0024_ZERO_OUT_TH) && (posLastY < JVET_C0024_ZERO_OUT_TH));

Moreover, duplicated variables are declared for the thresholds, i.e., JVET_C0046_ZERO_OUT_TH and JVET_C0024_ZERO_OUT_TH.

It is proposed to make consistent implementation (always use threshold so that the max non-zero-out region is 32x32) to avoid confusion and clean redundant macros and code. A patch is attached for the proposed changes.

When encoding, nQP and QP is outputed in the console. nQP means the nearest QP to the step size (JCTVC-G382), and have different value with QP only when the option AdaptiveQpSelection(-aqps) is enabled. However, when LCU level rate control is enabled, nQP is different from QP for every frame, including I frames.

For current HM software, when LCU level rate control is enabled, the output value of nQP is the QP of last LCU in current frame. The QP of last LCU in current frame should not be assigned to TComSlice::m_iSliceQpBase.

The following code in the function body of TEncSlice::compressSlice should be removed:

#if ADAPTIVE_QP_SELECTION
      pCtu->getSlice()->setSliceQpBase( estQP );
#endif

After removing these code, all of output information in console remain the same except nQP value in RA and LD cases.