This bug was reported by Danilo Ramos of Eideticom, Inc. It has
lain in wait 13 years before being found! The bug was introduced
in zlib 1.2.2.2, with the addition of the Z_FIXED option. That
option forces the use of fixed Huffman codes. For rare inputs with
a large number of distant matches, the pending buffer into which
the compressed data is written can overwrite the distance symbol
table which it overlays. That results in corrupted output due to
invalid distances, and can result in out-of-bound accesses,
crashing the application.
The fix here combines the distance buffer and literal/length
buffers into a single symbol buffer. Now three bytes of pending
buffer space are opened up for each literal or length/distance
pair consumed, instead of the previous two bytes. This assures
that the pending buffer cannot overwrite the symbol table, since
the maximum fixed code compressed length/distance is 31 bits, and
since there are four bytes of pending space for every three bytes
of symbol space.
This limits hash table inserts to the available data in the window
and to the sliding window size in deflate_stored(). The hash table
inserts are deferred until deflateParams() switches to a non-zero
compression level.
This commit allows a parameter change even if the input data has
not all been compressed and copied to the application output
buffer, so long as all of the input data has been compressed to
the internal pending output buffer. This also allows an immediate
deflateParams change so long as there have been no deflate calls
since initialization or reset.
This permits deflateParams to change the strategy and level right
after deflateInit, without having to wait until a header has been
written. The parameters can be changed immediately up until the
first deflate call that consumes any input data.
This avoids unnecessary filling of bytes in the sliding window
buffer when switching from level zero to a non-zero level. This
also provides a consistent indication of deflate having taken
input for a later commit ...
There have been many reports of bugs in the assembler codes
intended to speed up deflate and inflate. They are third-party
contributions in contrib, and so are not supported by the zlib
maintainers.
The previous code slid the window and the hash table and copied
every input byte three times in order to just write the data as
stored blocks with no compression. This commit minimizes sliding
and copying, especially for large input and output buffers.
Level 0 compression is now more than 20 times faster than before
the commit.
Most of the speedup is due to deferring hash table slides until
deflateParams() is called to change the compression level away
from 0. More speedup is due to copying directly from next_in to
next_out when the amounts of available input data and output space
permit it, avoiding the intermediate pending buffer. Additionally,
only the last 32K of the used input data is copied back to the
sliding window when large input buffers are provided.
This alters the specification in zlib.h, so that deflateParams()
will not change any parameters if there is not enough output space
in the event that a block is emitted in order to allow switching
the compression function.
Compression level 0 requests no compression, using only stored
blocks. When Z_HUFFMAN or Z_RLE was used with level 0 (granted,
an odd choice, but permitted), the resulting blocks were mostly
fixed or dynamic. The reason is that deflate_stored() was not
being called in that case. The compressed data was valid, but it
was not what the application requested. This commit assures that
only stored blocks are emitted for compression level 0, regardless
of the strategy selected.
This verifies that the state has been initialized, that it is the
expected type of state, deflate or inflate, and that at least the
first several bytes of the internal state have not been clobbered.
There is a bug in deflate for windowBits == 8 (256-byte window).
As a result, zlib silently changes a request for 8 to a request
for 9 (512-byte window), and sets the zlib header accordingly so
that the decompressor knows to use a 512-byte window. However if
deflateInit2() is used for raw deflate or gzip streams, then there
is no indication that the request was not honored, and the
application might assume that it can use a 256-byte window when
decompressing. This commit returns an error if the user requests
a 256-byte window when using raw deflate or gzip encoding.
While woolly mammoths still roamed the Earth and before Atlantis
sunk into the ocean, there were C compilers that could not handle
forward structure references, e.g. "struct name;". zlib dutifully
provided a work-around for such compilers. That work-around is no
longer needed, and, per the recommendation of a security audit of
the zlib code by Trail of Bits and TrustInSoft, in support of the
Mozilla Foundation, should be removed since what a compiler will
do with this is technically undefined. From the report: "there is
no telling what interactions the bug could have in the future with
link-time optimizations and type-based alias analyses, both
features that are present (but not default) in clang."
This updates the documentation to reflect the behavior of
deflateParams() when it is not able to compress all of the input
data provided so far due to insufficient output space. It also
assures that data provided is compressed before the parameter
changes, even if at the beginning of the stream.
If the compressed data was already at a block boundary, then
deflateParam() would report Z_BUF_ERROR, because there was nothing
to write. With this patch, Z_OK is returned in that case.
This patch allows zlib to compile cleanly with the -Wcast-qual gcc
warning enabled, but only if ZLIB_CONST is defined, which adds
const to next_in and msg in z_stream and in the in_func prototype.
A --const option is added to ./configure which adds -DZLIB_CONST
to the compile flags, and adds -Wcast-qual to the compile flags
when ZLIBGCCWARN is set in the environment.
This allows deflate to generate the same output when continuing after
a Z_SYNC_FLUSH vs. using deflateSetDictionary() after a Z_FULL_FLUSH
or a deflateReset(). It also slightly improves compression when
flushing by providing two more strings to possibly match at the start
of the new block.
Previously, the bit buffer would hold 1 to 16 bits after "all" of the
output is provided after a Z_BLOCK deflate() call. Now at most seven
bits remain in the output buffer after Z_BLOCK. flush_pending() now
flushes the bit buffer before copying out the byte buffer, in order
for it to really flush as much as possible.
Mark Adler
Mark Adler