When you enable the QPA algorithm (Perceptually optimised QP adaptation, -qpa 1 --SliceChromaQPOffsetPeriodicity=1) is enabled and the internal bit-depth parameter is the default set in the main .cfg files (--internal-bitdepth=10), the result is as expected, a considerable rate saving and an improvement in perceptual quality.

However, if we force the internal bitdepth to be 8 (--internal-bitdepth=8), the result is a considerable increase in bitrate compared to not enabling the QPA algorithm.

As an example, I have encoded frames 1, 101, 201, 301, 401 and 501 of the BQTerrace sequence in All Intra mode, with base QPs 22, 27, 32, 37 and 42. Here are the average results:

InternalBitDepth=8, QPA off

base QP	Mbps	MS-SSIM
22	2.659	0.996
27	0.989	0.986
32	0.503	0.979
37	0.268	0.966
42	0.138	0.942

InternalBitDepth=10, QPA off

base QP	Mbps	MS-SSIM
22	2.724	0.996
27	1.021	0.987
32	0.517	0.979
37	0.275	0.967
42	0.142	0.944

InternalBitDepth=8, QPA on

base QP	Mbps	MS-SSIM
22	3.388	0.998
27	1.561	0.992
32	0.601	0.983
37	0.309	0.972
42	0.167	0.952

InternalBitDepth=10, QPA on

base QP	Mbps	MS-SSIM
22	2.454	0.996
27	0.820	0.986
32	0.408	0.977
37	0.210	0.963
42	0.115	0.935

As can be seen, for a base QP of 22, enabling the QPA algorithm increases the bit rate by almost 1 Mbps if we compare the bit depth level of 8 versus 10, when it would be expected to have a very similar value.

I attach the RD curves corresponding to the tables, where you can visually appreciate the mentioned rate increase.

I suspect that the problem, if any, may be in the calculation of picture or block energy (activity), or in the algorithm that modifies the Lagrangian during the rate control.

In VTM-16.0, function xRateDistOptQuant() (when depQuant off), remRegBins calculation looks not correct for 64x64 transform. The code in current master is as follows

int remRegBins = (uiWidth * uiHeight * ctxBinSampleRatio) >> 4;

When we have 64x64 transform, zeroout has to be applied. It looks not correct to always use uiWidth and uiHeight here.

In function CABACWriter::residual_coding, we are using cctx.regBinLimit = (tu.getTbAreaAfterCoefZeroOut(compID) * ctxBinSampleRatio) >> 4;

Using tu.getTbAreaAfterCoefZeroOut(compID) instead of "uiWidth * uiHeight" looks correct.

To summarize, the proposed change int remRegBins = (uiWidth * uiHeight * ctxBinSampleRatio) >> 4; => int remRegBins = (tu.getTbAreaAfterCoefZeroOut(compID) * ctxBinSampleRatio) >> 4;

When testing VTM-16.0rc1 decoder in debug mode and turn on "break on all C++ exceptions" in Visual Studio, the following exception is triggered at the end of the bitstream (as attached). The exception was triggered in function eofBeforeNBytes().

Command lines to reproduce this issue: EncoderApp -c cfg\encoder_randomaccess_vtm.cfg --InputFile=BQSquare_416x240_60.yuv --BitstreamFile=Bin_enc.bin --SourceWidth=416 --SourceHeight=240 --QP=37 --FrameSkip=0 --FramesToBeEncoded=17 --IntraPeriod=96 --FrameRate=60 --InputBitDepth=8 --ReconFile=recon.yuv --SEIDecodedPictureHash=1

DecoderApp -b Bin_enc.bin -o decoded.yuv --SEIDecodedPictureHash=1