The previous code slid the window and the hash table and copied
every input byte three times in order to just write the data as
stored blocks with no compression. This commit minimizes sliding
and copying, especially for large input and output buffers.
Level 0 compression is now more than 20 times faster than before
the commit.
Most of the speedup is due to deferring hash table slides until
deflateParams() is called to change the compression level away
from 0. More speedup is due to copying directly from next_in to
next_out when the amounts of available input data and output space
permit it, avoiding the intermediate pending buffer. Additionally,
only the last 32K of the used input data is copied back to the
sliding window when large input buffers are provided.
This alters the specification in zlib.h, so that deflateParams()
will not change any parameters if there is not enough output space
in the event that a block is emitted in order to allow switching
the compression function.
Compression level 0 requests no compression, using only stored
blocks. When Z_HUFFMAN or Z_RLE was used with level 0 (granted,
an odd choice, but permitted), the resulting blocks were mostly
fixed or dynamic. The reason is that deflate_stored() was not
being called in that case. The compressed data was valid, but it
was not what the application requested. This commit assures that
only stored blocks are emitted for compression level 0, regardless
of the strategy selected.
This verifies that the state has been initialized, that it is the
expected type of state, deflate or inflate, and that at least the
first several bytes of the internal state have not been clobbered.
There is a bug in deflate for windowBits == 8 (256-byte window).
As a result, zlib silently changes a request for 8 to a request
for 9 (512-byte window), and sets the zlib header accordingly so
that the decompressor knows to use a 512-byte window. However if
deflateInit2() is used for raw deflate or gzip streams, then there
is no indication that the request was not honored, and the
application might assume that it can use a 256-byte window when
decompressing. This commit returns an error if the user requests
a 256-byte window when using raw deflate or gzip encoding.
While woolly mammoths still roamed the Earth and before Atlantis
sunk into the ocean, there were C compilers that could not handle
forward structure references, e.g. "struct name;". zlib dutifully
provided a work-around for such compilers. That work-around is no
longer needed, and, per the recommendation of a security audit of
the zlib code by Trail of Bits and TrustInSoft, in support of the
Mozilla Foundation, should be removed since what a compiler will
do with this is technically undefined. From the report: "there is
no telling what interactions the bug could have in the future with
link-time optimizations and type-based alias analyses, both
features that are present (but not default) in clang."
This updates the documentation to reflect the behavior of
deflateParams() when it is not able to compress all of the input
data provided so far due to insufficient output space. It also
assures that data provided is compressed before the parameter
changes, even if at the beginning of the stream.
If the compressed data was already at a block boundary, then
deflateParam() would report Z_BUF_ERROR, because there was nothing
to write. With this patch, Z_OK is returned in that case.
This patch allows zlib to compile cleanly with the -Wcast-qual gcc
warning enabled, but only if ZLIB_CONST is defined, which adds
const to next_in and msg in z_stream and in the in_func prototype.
A --const option is added to ./configure which adds -DZLIB_CONST
to the compile flags, and adds -Wcast-qual to the compile flags
when ZLIBGCCWARN is set in the environment.
This allows deflate to generate the same output when continuing after
a Z_SYNC_FLUSH vs. using deflateSetDictionary() after a Z_FULL_FLUSH
or a deflateReset(). It also slightly improves compression when
flushing by providing two more strings to possibly match at the start
of the new block.
Previously, the bit buffer would hold 1 to 16 bits after "all" of the
output is provided after a Z_BLOCK deflate() call. Now at most seven
bits remain in the output buffer after Z_BLOCK. flush_pending() now
flushes the bit buffer before copying out the byte buffer, in order
for it to really flush as much as possible.
Previously when doing an empty flush, a extra static or stored block
could be emitted before the requested empty static or stored block.
This patch prevents the emission of empty blocks by the deflate_*
functions.
The incorporation of the Z_BLOCK flush did not update the rejection
of lower ranked flushes immediately after higher ranked flushes with
no more input data. This prevented an empty Z_SYNC_FLUSH right after
a Z_BLOCK flush, which would be desired to bring the deflate stream
to a byte boundary conditionally on whether or not it was already at
a byte boundary. This patch re-ranks Z_BLOCK above Z_NO_FLUSH but
below Z_PARTIAL_FLUSH, allowing stronger empty flushes to follow a
Z_BLOCK flush.
This patch adds the deflateResetKeep() function to retain the sliding
window for the next deflate operation, and fixes an inflateResetKeep()
problem that came from inflate() not updating the window when the
stream completed. This enables constructing and decompressing a series
of concatenated deflate streams where each can depend on the history of
uncompressed data that precedes it.
This generalizes deflateSetDictionary() and inflateSetDictionary() to
permit setting the dictionary in the middle of a stream for raw deflate
and inflate. This in combination with the Keep functions enables a
scheme for updating files block by block with the transmission of
compressed data, where blocks are sent with deflateResetKeep() to
retain history for better compression, and deflateSetDictionary() is
used for blocks already present at the receiver to skip compression but
insert that data in the history, again for better compression. The
corresponding inflate calls are done on the receiver side.
A common request has been the ability to compile zlib to require no
other libraries. This --solo option provides that ability. The price
is that the gz*, compress*, and uncompress functions are eliminated,
and that the user must provide memory allocation and free routines to
deflate and inflate when initializing.
Without this, Z_RLE could under some circumstances read one byte past
the end of the allocated sliding window. This would normally not be a
problem unless the window is right at the end of an allocated page, or
if a bounds checker is being used.
Also added "-motley" to ZLIB_VERSION in zlib.h, so that versions
in-between 1.2.5.1 and 1.2.5.2 that are pulled down from github
can be identified as such if bugs are reported on them.
Mark Adler