From b6c207f5358181f64e8cfadc929d61f0ed25266c Mon Sep 17 00:00:00 2001 From: Rostislav Pehlivanov Date: Sat, 7 May 2016 18:05:25 +0100 Subject: [PATCH] vc2enc_dwt: use 32 bit coefficients by default The problem is that with particularly complex images and especially at high bit depths and 5-level transforms the coefficients would overflow, causing huge artifacts to appear. This was discovered thanks to the fate tests, which will have to be redone as this fixes a multitude of problems and increases PSNR. There is a slight performance drop associated with this change, making the encoder slower by 1.15 times, however this is necessary in order to avoid undefined behavior and overflows. It would be worth to template the transforms to keep the performance for 8 bit images as 32 bit coefficients are unnecessary for that case, but the primary use of the encoder is to encode video at 10 bits. Reviewed-by: Christophe Gisquet Reviewed-by: Michael Niedermayer Signed-off-by: Rostislav Pehlivanov --- libavcodec/vc2enc_dwt.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavcodec/vc2enc_dwt.h b/libavcodec/vc2enc_dwt.h index af6fe3e2a2..7fbbfbe0ed 100644 --- a/libavcodec/vc2enc_dwt.h +++ b/libavcodec/vc2enc_dwt.h @@ -25,7 +25,7 @@ #include #include -typedef int16_t dwtcoef; +typedef int32_t dwtcoef; enum VC2TransformType { VC2_TRANSFORM_9_7 = 0, /* Deslauriers-Dubuc (9,7) */