Opened 2 years ago
Closed 2 years ago
#1566 closed defect (fixed)
VTM encoder hits assert with --ISPFast=1 and --EncDbOpt =1
| Reported by: | Vadim | Owned by: | |
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | VTM | Version: | |
| Keywords: | Cc: | ksuehring, XiangLi, fbossen, jvet@… |
Description
VTM encoder hits assert with --ISPFast=1 and --EncDbOpt=1 in debug mode (for example in AI), it seems ISPFast implementation has issues in mode cost comparison.
If CHECKD is changed to CHECK, the encoder will crash in release mode as well.
1848: CHECKD( bestCostDct2NoIsp <= bestIspCost, "wrong cost!" );
ERROR: In function "EncCu::xCheckRDCostIntra" in H:\VVCSoftware_VTM\source\Lib\EncoderLib\EncCu.cpp:1848: wrong cost!
Is this CHECKD correct?
Change history (2)
comment:1 Changed 2 years ago by ksuehring
comment:2 Changed 2 years ago by ksuehring
- Resolution set to fixed
- Status changed from new to closed
The MR was merged
Note: See TracTickets for help on using tickets.
A fix is submitted as:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/merge_requests/2287
Wit EncDbOpt the code can be called when not in ISP mode, so an additional condition is needed in the check.
I ran full SD CTC (plus --ISPFast=1 and --EncDbOpt =1) with CHECK instead of CHECKD and did not see any crash. We would still rather like to keep the debug mode CHECKD so that there are no extra cycles needed in release mode.
The change should not have any performance change, but I can't compare to a regular CTC run since the parameter combination does not occur in CTC.