omse goes from 0.03060703 (which fails for dct-test) to 0.01663750.
This also actually improve the error of decoding the sample generated
by fate-vsynth3-dnxhd1080i-10bit using simple_idct10 to FAANI, which
goes (when resampled to yuv422p) from:
stddev: 0.06 PSNR: 72.28 MAXDIFF: 1
to identical.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The current one, while correct, does not yield the best possible
results. The specificiations suggest another formula, which results
in quality gains in the decoded output from fate tests. This
justifies changing said formula.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Convert them to zigzag order, as the rest of them are.
When I was adding support for 10-bit DNxHD, I just copy-pasted the
missing quant matrices from the spec. Now it turns out the existing
matrices in dnxhddata.c were in zigzag order. This resulted in wrong
quantization for 10-bit DNxHD. The attached patch fixes the problem by
converting 10-bit quant matrices to zigzag order.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
The file is already present in git and by using it we can perform more tests
without the need of fate samples
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
The new reference.pnm is a freely licensed replacement. The photo has
been taken by Reinhard Tartler on August 28 2014, and is licensed under
the expat license as stated at http://www.jclark.com/xml/copying.txt
no changes in either standard deviation or PSNR is seen in any of the changed fate
cases
MSE changes from 0.05012422 to 0.04890000
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Convert them to zigzag order, as the rest of them are.
When I was adding support for 10-bit DNxHD, I just copy-pasted the
missing quant matrices from the spec. Now it turns out the existing
matrices in dnxhddata.c were in zigzag order. This resulted in wrong
quantization for 10-bit DNxHD. The attached patch fixes the problem by
converting 10-bit quant matrices to zigzag order.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Christophe Gisquet
Joseph Artsimovich
Michael Niedermayer
Reinhard Tartler
Janne Grunau
Mans Rullgard