This fixes the same overflow as in the RGB48/16-bit YUV scaling;
some filters can overflow both negatively and positively (e.g.
spline/lanczos), so we bias a signed integer so it's "half signed"
and "half unsigned", and can cover overflows in both directions
while maintaining full 31-bit depth.
Signed-off-by: Mans Rullgard <mans@mansr.com>
We're shifting individual components (8-bit, unsigned) left by 24,
so making them unsigned should give the same results without the
overflow.
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
For certain types of filters where the intermediate sum of coefficients
can go above the fixed-point equivalent of 1.0 in the middle of a filter,
the sum of a 31-bit calculation can overflow in both directions and can
thus not be represented in a 32-bit signed or unsigned integer. To work
around this, we subtract 0x40000000 from a signed integer base, so that
we're halfway signed/unsigned, which makes it fit even if it overflows.
After the filter finishes, we add the scaled bias back after a shift.
We use the same trick for 16-bit bpc YUV output routines.
Signed-off-by: Mans Rullgard <mans@mansr.com>
libswscale/swscale.c:2744:40: warning: to be safe all intermediate pointers in cast from ‘int16_t **’ to ‘const int16_t **’ must be ‘const’ qualified [-Wcast-qual]
libswscale/swscale.c:2745:41: warning: to be safe all intermediate pointers in cast from ‘int16_t **’ to ‘const int16_t **’ must be ‘const’ qualified [-Wcast-qual]
libswscale/swscale.c:2746:41: warning: to be safe all intermediate pointers in cast from ‘int16_t **’ to ‘const int16_t **’ must be ‘const’ qualified [-Wcast-qual]
libswscale/swscale.c:2747:78: warning: to be safe all intermediate pointers in cast from ‘int16_t **’ to ‘const int16_t **’ must be ‘const’ qualified [-Wcast-qual]
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
As old bits are shifted out of the accumulator, they cause signed
overflows when they reach the end. Making the variable unsigned fixes
this.
Signed-off-by: Mans Rullgard <mans@mansr.com>
This allows using more specific implementations for chroma/luma, e.g.
we can make assumptions on filterSize being constant, thus avoiding
that test at runtime.
We operated on 31-bits, but with e.g. lanczos scaling, values can
add up to beyond 0x80000000, thus leading to output of zeroes. Drop
one bit of precision fixes this.
We operated on 31-bits, but with e.g. lanczos scaling, values can
add up to beyond 0x80000000, thus leading to output of zeroes. Drop
one bit of precision fixes this.