Potential bug of Hash ME

[xinzhao]

In a recent version of VTM SW (SHA-1: 80c290c993593c0ed970319e23d08dfe84398283), it is noticed that for LDB configuration, current Hash ME implementation may cause big PSNR drop (up to 10dB drop) and severe visual quality degradation for certain frames of TGM sequence "Desktop". Some originally nonexistent content can show up on the decoded picture.

Below is an example: LDB configuration, TGM sequence "Desktop", QP 22, encoder log of frames 243~245 are shown below:

Test 1: VTM (SHA-1: 80c290c993593c0ed970319e23d08dfe84398283):

POC 243 TId: 0 ( B-SLICE, QP 27 ) 7384 bits [Y 52.9771 dB U 45.0448 dB V 45.5116 dB]
POC 244 TId: 0 ( B-SLICE, QP 23 ) 25392 bits [Y 42.6302 dB U 45.5357 dB V 45.9436 dB]
POC 245 TId: 0 ( B-SLICE, QP 27 ) 28464 bits [Y 52.9665 dB U 43.8098 dB V 44.0321 dB]

Test 2: VTM (SHA-1: 80c290c993593c0ed970319e23d08dfe84398283) with "--HashME=0":

POC 243 TId: 0 ( B-SLICE, QP 27 ) 9992 bits [Y 52.5057 dB U 46.0938 dB V 46.5207 dB]
POC 244 TId: 0 ( B-SLICE, QP 23 ) 18584 bits [Y 52.5740 dB U 46.2667 dB V 46.6286 dB]
POC 245 TId: 0 ( B-SLICE, QP 27 ) 27880 bits [Y 52.4223 dB U 44.7288 dB V 45.2189 dB]

It can be seen that frame 244 has a 10dB PSNR drop comparing to its neighboring frames as well as the same frame but using "-HashME=0" configuration.

The encoder log has been attached and the original and decoded frame 244 has been attached as well, it can be noticed that some nonexistent texts show up on the decoded picture, and some texts are heavily blurred (as highlighted in red box), which should be unexpected.

[xinzhao]

Original frame 244 of Desktop (Cropped due to file size limitation)

[xinzhao]

Decoded frame 244 of Desktop (Cropped due to file size limitation)

[xinzhao]

Encoder log using VTM (SHA-1: 80c290c993593c0ed970319e23d08dfe84398283)

[xinzhao]

Encoder log using VTM (SHA-1: 80c290c993593c0ed970319e23d08dfe84398283) with "--HashME=0"

[XiangLi]

A suggested fix has been provided by proponent in ​https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/merge_requests/536.

[XiangLi]

Fixed.