Chiebot-Mirror
/
zlib
mirror of https://github.com/madler/zlib.git
8
0
Fork 0
Code Issues Projects Releases Wiki Activity

Rewrite of zran in examples. See version history in zran.c.

Browse Source
pull/643/merge
Mark Adler 2 years ago
parent 12b345c430
commit eb0e038b29
2 changed files with 348 additions and 372 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 651
      examples/zran.c
  2. 69
      examples/zran.h

651
examples/zran.c
Unescape View File

@ -1,114 +1,101 @@
/* zran.c -- example of zlib/gzip stream indexing and random access /* zran.c -- example of deflate stream indexing and random access
* Copyright (C) 2005, 2012, 2018 Mark Adler * Copyright (C) 2005, 2012, 2018, 2023 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h * For conditions of distribution and use, see copyright notice in zlib.h
* Version 1.2 14 Oct 2018 Mark Adler */ * Version 1.3 18 Feb 2023 Mark Adler */
/* Version History: /* Version History:
1.0 29 May 2005 First version 1.0 29 May 2005 First version
1.1 29 Sep 2012 Fix memory reallocation error 1.1 29 Sep 2012 Fix memory reallocation error
1.2 14 Oct 2018 Handle gzip streams with multiple members 1.2 14 Oct 2018 Handle gzip streams with multiple members
Add a header file to facilitate usage in applications Add a header file to facilitate usage in applications
1.3 18 Feb 2023 Permit raw deflate streams as well as zlib and gzip
Permit crossing gzip member boundaries when extracting
Support a size_t size when extracting (was an int)
Do a binary search over the index for an access point
Expose the access point type to enable save and load
*/ */
/* Illustrate the use of Z_BLOCK, inflatePrime(), and inflateSetDictionary() // Illustrate the use of Z_BLOCK, inflatePrime(), and inflateSetDictionary()
for random access of a compressed file. A file containing a zlib or gzip // for random access of a compressed file. A file containing a raw deflate
stream is provided on the command line. The compressed stream is decoded in // stream is provided on the command line. The compressed stream is decoded in
its entirety, and an index built with access points about every SPAN bytes // its entirety, and an index built with access points about every SPAN bytes
in the uncompressed output. The compressed file is left open, and can then // in the uncompressed output. The compressed file is left open, and can then
be read randomly, having to decompress on the average SPAN/2 uncompressed // be read randomly, having to decompress on the average SPAN/2 uncompressed
bytes before getting to the desired block of data. // bytes before getting to the desired block of data.
//
An access point can be created at the start of any deflate block, by saving // An access point can be created at the start of any deflate block, by saving
the starting file offset and bit of that block, and the 32K bytes of // the starting file offset and bit of that block, and the 32K bytes of
uncompressed data that precede that block. Also the uncompressed offset of // uncompressed data that precede that block. Also the uncompressed offset of
that block is saved to provide a reference for locating a desired starting // that block is saved to provide a reference for locating a desired starting
point in the uncompressed stream. deflate_index_build() works by // point in the uncompressed stream. deflate_index_build() decompresses the
decompressing the input zlib or gzip stream a block at a time, and at the // input raw deflate stream a block at a time, and at the end of each block
end of each block deciding if enough uncompressed data has gone by to // decides if enough uncompressed data has gone by to justify the creation of a
justify the creation of a new access point. If so, that point is saved in a // new access point. If so, that point is saved in a data structure that grows
data structure that grows as needed to accommodate the points. // as needed to accommodate the points.
//
To use the index, an offset in the uncompressed data is provided, for which // To use the index, an offset in the uncompressed data is provided, for which
the latest access point at or preceding that offset is located in the index. // the latest access point at or preceding that offset is located in the index.
The input file is positioned to the specified location in the index, and if // The input file is positioned to the specified location in the index, and if
necessary the first few bits of the compressed data is read from the file. // necessary the first few bits of the compressed data is read from the file.
inflate is initialized with those bits and the 32K of uncompressed data, and // inflate is initialized with those bits and the 32K of uncompressed data, and
the decompression then proceeds until the desired offset in the file is // decompression then proceeds until the desired offset in the file is reached.
reached. Then the decompression continues to read the desired uncompressed // Then decompression continues to read the requested uncompressed data from
data from the file. // the file.
//
Another approach would be to generate the index on demand. In that case, // There is some fair bit of overhead to starting inflation for the random
requests for random access reads from the compressed data would try to use // access, mainly copying the 32K byte dictionary. If small pieces of the file
the index, but if a read far enough past the end of the index is required, // are being accessed, it would make sense to implement a cache to hold some
then further index entries would be generated and added. // lookahead to avoid many calls to deflate_index_extract() for small lengths.
//
There is some fair bit of overhead to starting inflation for the random // Another way to build an index would be to use inflateCopy(). That would not
access, mainly copying the 32K byte dictionary. So if small pieces of the // be constrained to have access points at block boundaries, but would require
file are being accessed, it would make sense to implement a cache to hold // more memory per access point, and could not be saved to a file due to the
some lookahead and avoid many calls to deflate_index_extract() for small // use of pointers in the state. The approach here allows for storage of the
lengths. // index in a file.
Another way to build an index would be to use inflateCopy(). That would
not be constrained to have access points at block boundaries, but requires
more memory per access point, and also cannot be saved to file due to the
use of pointers in the state. The approach here allows for storage of the
index in a file.
*/
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <limits.h>
#include "zlib.h" #include "zlib.h"
#include "zran.h" #include "zran.h"
#define WINSIZE 32768U /* sliding window size */ #define WINSIZE 32768U // sliding window size
#define CHUNK 16384 /* file input buffer size */ #define CHUNK 16384 // file input buffer size
/* Access point entry. */
struct point {
off_t out; /* corresponding offset in uncompressed data */
off_t in; /* offset in input file of first full byte */
int bits; /* number of bits (1-7) from byte at in-1, or 0 */
unsigned char window[WINSIZE]; /* preceding 32K of uncompressed data */
};
/* See comments in zran.h. */ // See comments in zran.h.
void deflate_index_free(struct deflate_index *index) void deflate_index_free(struct deflate_index *index) {
{
if (index != NULL) { if (index != NULL) {
free(index->list); free(index->list);
free(index); free(index);
} }
} }
/* Add an entry to the access point list. If out of memory, deallocate the // Add an access point to the list. If out of memory, deallocate the existing
existing list and return NULL. index->gzip is the allocated size of the // list and return NULL. index->mode is temporarily the allocated number of
index in point entries, until it is time for deflate_index_build() to // access points, until it is time for deflate_index_build() to return. Then
return, at which point gzip is set to indicate a gzip file or not. // index->mode is set to the mode of inflation.
*/ static struct deflate_index *add_point(struct deflate_index *index, int bits,
static struct deflate_index *addpoint(struct deflate_index *index, int bits, off_t in, off_t out, unsigned left,
off_t in, off_t out, unsigned left, unsigned char *window) {
unsigned char *window)
{
struct point *next;
/* if list is empty, create it (start with eight points) */
if (index == NULL) { if (index == NULL) {
// The list is empty. Create it, starting with eight access points.
index = malloc(sizeof(struct deflate_index)); index = malloc(sizeof(struct deflate_index));
if (index == NULL) return NULL; if (index == NULL)
index->list = malloc(sizeof(struct point) << 3); return NULL;
index->have = 0;
index->mode = 8;
index->list = malloc(sizeof(point_t) * index->mode);
if (index->list == NULL) { if (index->list == NULL) {
free(index); free(index);
return NULL; return NULL;
} }
index->gzip = 8;
index->have = 0;
} }
/* if list is full, make it bigger */ else if (index->have == index->mode) {
else if (index->have == index->gzip) { // The list is full. Make it bigger.
index->gzip <<= 1; index->mode <<= 1;
next = realloc(index->list, sizeof(struct point) * index->gzip); point_t *next = realloc(index->list, sizeof(point_t) * index->mode);
if (next == NULL) { if (next == NULL) {
deflate_index_free(index); deflate_index_free(index);
return NULL; return NULL;
@ -116,318 +103,291 @@ static struct deflate_index *addpoint(struct deflate_index *index, int bits,
index->list = next; index->list = next;
} }
/* fill in entry and increment how many we have */ // Fill in the access point and increment how many we have.
next = (struct point *)(index->list) + index->have; point_t *next = (point_t *)(index->list) + index->have++;
next->bits = bits; if (index->have < 0) {
next->in = in; // Overflowed the int!
deflate_index_free(index);
return NULL;
}
next->out = out; next->out = out;
next->in = in;
next->bits = bits;
if (left) if (left)
memcpy(next->window, window + WINSIZE - left, left); memcpy(next->window, window + WINSIZE - left, left);
if (left < WINSIZE) if (left < WINSIZE)
memcpy(next->window + left, window, WINSIZE - left); memcpy(next->window + left, window, WINSIZE - left);
index->have++;
/* return list, possibly reallocated */ // Return the index, which may have been newly allocated or destroyed.
return index; return index;
} }
/* See comments in zran.h. */ // Decompression modes. These are the inflateInit2() windowBits parameter.
int deflate_index_build(FILE *in, off_t span, struct deflate_index **built) #define RAW -15
{ #define ZLIB 15
int ret; #define GZIP 31
int gzip = 0; /* true if reading a gzip file */
off_t totin, totout; /* our own total counters to avoid 4GB limit */ // See comments in zran.h.
off_t last; /* totout value of last access point */ int deflate_index_build(FILE *in, off_t span, struct deflate_index **built) {
struct deflate_index *index; /* access points being generated */ // Set up inflation state.
z_stream strm; z_stream strm = {0}; // inflate engine (gets fired up later)
unsigned char input[CHUNK]; unsigned char buf[CHUNK]; // input buffer
unsigned char window[WINSIZE]; unsigned char win[WINSIZE] = {0}; // output sliding window
off_t totin = 0; // total bytes read from input
/* initialize inflate */ off_t totout = 0; // total bytes uncompressed
strm.zalloc = Z_NULL; int mode = 0; // mode: RAW, ZLIB, or GZIP (0 => not set yet)
strm.zfree = Z_NULL;
strm.opaque = Z_NULL; // Decompress from in, generating access points along the way.
strm.avail_in = 0; int ret; // the return value from zlib, or Z_ERRNO
strm.next_in = Z_NULL; off_t last; // last access point uncompressed offset
ret = inflateInit2(&strm, 47); /* automatic zlib or gzip decoding */ struct deflate_index *index = NULL; // list of access points
if (ret != Z_OK)
return ret;
/* inflate the input, maintain a sliding window, and build an index -- this
also validates the integrity of the compressed data using the check
information in the gzip or zlib stream */
totin = totout = last = 0;
index = NULL; /* will be allocated by first addpoint() */
strm.avail_out = 0;
do { do {
/* get some compressed data from input file */ // Assure available input, at least until reaching EOF.
strm.avail_in = fread(input, 1, CHUNK, in);
if (ferror(in)) {
ret = Z_ERRNO;
goto deflate_index_build_error;
}
if (strm.avail_in == 0) { if (strm.avail_in == 0) {
ret = Z_DATA_ERROR; strm.avail_in = fread(buf, 1, sizeof(buf), in);
goto deflate_index_build_error;
}
strm.next_in = input;
/* check for a gzip stream */
if (totin == 0 && strm.avail_in >= 3 &&
input[0] == 31 && input[1] == 139 && input[2] == 8)
gzip = 1;
/* process all of that, or until end of stream */
do {
/* reset sliding window if necessary */
if (strm.avail_out == 0) {
strm.avail_out = WINSIZE;
strm.next_out = window;
}
/* inflate until out of input, output, or at end of block --
update the total input and output counters */
totin += strm.avail_in; totin += strm.avail_in;
totout += strm.avail_out; strm.next_in = buf;
ret = inflate(&strm, Z_BLOCK); /* return at end of block */ if (strm.avail_in < sizeof(buf) && ferror(in)) {
totin -= strm.avail_in; ret = Z_ERRNO;
totout -= strm.avail_out;
if (ret == Z_NEED_DICT)
ret = Z_DATA_ERROR;
if (ret == Z_MEM_ERROR || ret == Z_DATA_ERROR)
goto deflate_index_build_error;
if (ret == Z_STREAM_END) {
if (gzip &&
(strm.avail_in || ungetc(getc(in), in) != EOF)) {
ret = inflateReset(&strm);
if (ret != Z_OK)
goto deflate_index_build_error;
continue;
}
break; break;
} }
/* if at end of block, consider adding an index entry (note that if if (mode == 0) {
data_type indicates an end-of-block, then all of the // At the start of the input -- determine the type. Assume raw
uncompressed data from that block has been delivered, and none // if it is neither zlib nor gzip. This could in theory result
of the compressed data after that block has been consumed, // in a false positive for zlib, but in practice the fill bits
except for up to seven bits) -- the totout == 0 provides an // after a stored block are always zeros, so a raw stream won't
entry point after the zlib or gzip header, and assures that the // start with an 8 in the low nybble.
index always has at least one access point; we avoid creating an mode = strm.avail_in == 0 ? RAW : // empty -- will fail
access point after the last block by checking bit 6 of data_type (strm.next_in[0] & 0xf) == 8 ? ZLIB :
*/ strm.next_in[0] == 0x1f ? GZIP :
if ((strm.data_type & 128) && !(strm.data_type & 64) && /* else */ RAW;
(totout == 0 || totout - last > span)) { ret = inflateInit2(&strm, mode);
index = addpoint(index, strm.data_type & 7, totin, if (ret != Z_OK)
totout, strm.avail_out, window); break;
if (index == NULL) { }
ret = Z_MEM_ERROR; }
goto deflate_index_build_error;
} // Assure available output. This rotates the output through, for use as
last = totout; // a sliding window on the uncompressed data.
if (strm.avail_out == 0) {
strm.avail_out = sizeof(win);
strm.next_out = win;
}
if (mode == RAW && index == NULL)
// We skip the inflate() call at the start of raw deflate data in
// order generate an access point there. Set data_type to imitate
// the end of a header.
strm.data_type = 0x80;
else {
// Inflate and update the number of uncompressed bytes.
unsigned before = strm.avail_out;
ret = inflate(&strm, Z_BLOCK);
totout += before - strm.avail_out;
}
if ((strm.data_type & 0xc0) == 0x80 &&
(index == NULL || totout - last >= span)) {
// We are at the end of a header or a non-last deflate block, so we
// can add an access point here. Furthermore, we are either at the
// very start for the first access point, or there has been span or
// more uncompressed bytes since the last access point, so we want
// to add an access point here.
index = add_point(index, strm.data_type & 7, totin - strm.avail_in,
totout, strm.avail_out, win);
if (index == NULL) {
ret = Z_MEM_ERROR;
break;
} }
} while (strm.avail_in != 0); last = totout;
} while (ret != Z_STREAM_END); }
/* clean up and return index (release unused entries in list) */ if (ret == Z_STREAM_END && mode == GZIP &&
(void)inflateEnd(&strm); (strm.avail_in || ungetc(getc(in), in) != EOF))
index->list = realloc(index->list, sizeof(struct point) * index->have); // There is more input after the end of a gzip member. Reset the
index->gzip = gzip; // inflate state to read another gzip member. On success, this will
// set ret to Z_OK to continue decompressing.
ret = inflateReset2(&strm, GZIP);
// Keep going until Z_STREAM_END or error. If the compressed data ends
// prematurely without a file read error, Z_BUF_ERROR is returned.
} while (ret == Z_OK);
inflateEnd(&strm);
if (ret != Z_STREAM_END) {
// An error was encountered. Discard the index and return a negative
// error code.
deflate_index_free(index);
return ret == Z_NEED_DICT ? Z_DATA_ERROR : ret;
}
// Shrink the index to only the occupied access points and return it.
index->mode = mode;
index->length = totout; index->length = totout;
point_t *list = realloc(index->list, sizeof(point_t) * index->have);
if (list == NULL) {
// Seems like a realloc() to make something smaller should always work,
// but just in case.
deflate_index_free(index);
return Z_MEM_ERROR;
}
index->list = list;
*built = index; *built = index;
return index->have; return index->have;
/* return error */
deflate_index_build_error:
(void)inflateEnd(&strm);
deflate_index_free(index);
return ret;
} }
/* See comments in zran.h. */ // See comments in zran.h.
int deflate_index_extract(FILE *in, struct deflate_index *index, off_t offset, ptrdiff_t deflate_index_extract(FILE *in, struct deflate_index *index,
unsigned char *buf, int len) off_t offset, unsigned char *buf, size_t len) {
{ // Do a quick sanity check on the index.
int ret, skip; if (index == NULL || index->have < 1 || index->list[0].out != 0)
z_stream strm; return Z_STREAM_ERROR;
struct point *here;
unsigned char input[CHUNK];
unsigned char discard[WINSIZE];
/* proceed only if something reasonable to do */ // If nothing to extract, return zero bytes extracted.
if (len < 0) if (len == 0 || offset < 0 || offset >= index->length)
return 0; return 0;
/* find where in stream to start */ // Find the access point closest to but not after offset.
here = index->list; int lo = -1, hi = index->have;
ret = index->have; point_t *point = index->list;
while (--ret && here[1].out <= offset) while (hi - lo > 1) {
here++; int mid = (lo + hi) >> 1;
if (offset < point[mid].out)
/* initialize file and inflate state to start there */ hi = mid;
strm.zalloc = Z_NULL; else
strm.zfree = Z_NULL; lo = mid;
strm.opaque = Z_NULL; }
strm.avail_in = 0; point += lo;
strm.next_in = Z_NULL;
ret = inflateInit2(&strm, -15); /* raw inflate */ // Initialize the input file and prime the inflate engine to start there.
int ret = fseeko(in, point->in - (point->bits ? 1 : 0), SEEK_SET);
if (ret == -1)
return Z_ERRNO;
int ch = 0;
if (point->bits && (ch = getc(in)) == EOF)
return ferror(in) ? Z_ERRNO : Z_BUF_ERROR;
z_stream strm = {0};
ret = inflateInit2(&strm, RAW);
if (ret != Z_OK) if (ret != Z_OK)
return ret; return ret;
ret = fseeko(in, here->in - (here->bits ? 1 : 0), SEEK_SET); if (point->bits)
if (ret == -1) inflatePrime(&strm, point->bits, ch >> (8 - point->bits));
goto deflate_index_extract_ret; inflateSetDictionary(&strm, point->window, WINSIZE);
if (here->bits) {
ret = getc(in);
if (ret == -1) {
ret = ferror(in) ? Z_ERRNO : Z_DATA_ERROR;
goto deflate_index_extract_ret;
}
(void)inflatePrime(&strm, here->bits, ret >> (8 - here->bits));
}
(void)inflateSetDictionary(&strm, here->window, WINSIZE);
/* skip uncompressed bytes until offset reached, then satisfy request */ // Skip uncompressed bytes until offset reached, then satisfy request.
offset -= here->out; unsigned char input[CHUNK];
strm.avail_in = 0; unsigned char discard[WINSIZE];
skip = 1; /* while skipping to offset */ offset -= point->out; // number of bytes to skip to get to offset
size_t left = len; // number of bytes left to read after offset
do { do {
/* define where to put uncompressed data, and how much */ if (offset) {
if (offset > WINSIZE) { /* skip WINSIZE bytes */ // Discard up to offset uncompressed bytes.
strm.avail_out = WINSIZE; strm.avail_out = offset < WINSIZE ? (unsigned)offset : WINSIZE;
strm.next_out = discard;
offset -= WINSIZE;
}
else if (offset > 0) { /* last skip */
strm.avail_out = (unsigned)offset;
strm.next_out = discard; strm.next_out = discard;
offset = 0;
} }
else if (skip) { /* at offset now */ else {
strm.avail_out = len; // Uncompress up to left bytes into buf.
strm.next_out = buf; strm.avail_out = left < UINT_MAX ? (unsigned)left : UINT_MAX;
skip = 0; /* only do this once */ strm.next_out = buf + len - left;
} }
/* uncompress until avail_out filled, or end of stream */ // Uncompress, setting got to the number of bytes uncompressed.
do { if (strm.avail_in == 0) {
if (strm.avail_in == 0) { // Assure available input.
strm.avail_in = fread(input, 1, CHUNK, in); strm.avail_in = fread(input, 1, CHUNK, in);
if (ferror(in)) { if (strm.avail_in < CHUNK && ferror(in)) {
ret = Z_ERRNO; ret = Z_ERRNO;
goto deflate_index_extract_ret; break;
} }
if (strm.avail_in == 0) { strm.next_in = input;
ret = Z_DATA_ERROR; }
goto deflate_index_extract_ret; unsigned got = strm.avail_out;
} ret = inflate(&strm, Z_NO_FLUSH);
strm.next_in = input; got -= strm.avail_out;
// Update the appropriate count.
if (offset)
offset -= got;
else
left -= got;
// If we're at the end of a gzip member and there's more to read,
// continue to the next gzip member.
if (ret == Z_STREAM_END && index->mode == GZIP) {
// Discard the gzip trailer.
unsigned drop = 8; // length of gzip trailer
if (strm.avail_in >= drop) {
strm.avail_in -= drop;
strm.next_in += drop;
}
else {
// Read and discard the remainder of the gzip trailer.
drop -= strm.avail_in;
strm.avail_in = 0;
do {
if (getc(in) == EOF)
// The input does not have a complete trailer.
return ferror(in) ? Z_ERRNO : Z_BUF_ERROR;
} while (--drop);
} }
ret = inflate(&strm, Z_NO_FLUSH); /* normal inflate */
if (ret == Z_NEED_DICT)
ret = Z_DATA_ERROR;
if (ret == Z_MEM_ERROR || ret == Z_DATA_ERROR)
goto deflate_index_extract_ret;
if (ret == Z_STREAM_END) {
/* the raw deflate stream has ended */
if (index->gzip == 0)
/* this is a zlib stream that has ended -- done */
break;
/* near the end of a gzip member, which might be followed by
another gzip member -- skip the gzip trailer and see if
there is more input after it */
if (strm.avail_in < 8) {
fseeko(in, 8 - strm.avail_in, SEEK_CUR);
strm.avail_in = 0;
}
else {
strm.avail_in -= 8;
strm.next_in += 8;
}
if (strm.avail_in == 0 && ungetc(getc(in), in) == EOF)
/* the input ended after the gzip trailer -- done */
break;
/* there is more input, so another gzip member should follow -- if (strm.avail_in || ungetc(getc(in), in) != EOF) {
validate and skip the gzip header */ // There's more after the gzip trailer. Use inflate to skip the
ret = inflateReset2(&strm, 31); // gzip header and resume the raw inflate there.
if (ret != Z_OK) inflateReset2(&strm, GZIP);
goto deflate_index_extract_ret;
do { do {
if (strm.avail_in == 0) { if (strm.avail_in == 0) {
strm.avail_in = fread(input, 1, CHUNK, in); strm.avail_in = fread(input, 1, CHUNK, in);
if (ferror(in)) { if (strm.avail_in < CHUNK && ferror(in)) {
ret = Z_ERRNO; ret = Z_ERRNO;
goto deflate_index_extract_ret; break;
}
if (strm.avail_in == 0) {
ret = Z_DATA_ERROR;
goto deflate_index_extract_ret;
} }
strm.next_in = input; strm.next_in = input;
} }
ret = inflate(&strm, Z_BLOCK); strm.avail_out = WINSIZE;
if (ret == Z_MEM_ERROR || ret == Z_DATA_ERROR) strm.next_out = discard;
goto deflate_index_extract_ret; ret = inflate(&strm, Z_BLOCK); // stop at end of header
} while ((strm.data_type & 128) == 0); } while (ret == Z_OK && (strm.data_type & 0x80) == 0);
/* set up to continue decompression of the raw deflate stream
that follows the gzip header */
ret = inflateReset2(&strm, -15);
if (ret != Z_OK) if (ret != Z_OK)
goto deflate_index_extract_ret; break;
inflateReset2(&strm, RAW);
} }
}
/* continue to process the available input before reading more */ // Continue until we have the requested data, the deflate data has
} while (strm.avail_out != 0); // ended, or an error is encountered.
} while (ret == Z_OK && left);
if (ret == Z_STREAM_END) inflateEnd(&strm);
/* reached the end of the compressed data -- return the data that
was available, possibly less than requested */
break;
/* do until offset reached and requested data read */
} while (skip);
/* compute the number of uncompressed bytes read after the offset */
ret = skip ? 0 : len - strm.avail_out;
/* clean up and return the bytes read, or the negative error */ // Return the number of uncompressed bytes read into buf, or the error.
deflate_index_extract_ret: return ret == Z_OK || ret == Z_STREAM_END ? len - left : ret;
(void)inflateEnd(&strm);
return ret;
} }
#ifdef TEST #ifdef TEST
#define SPAN 1048576L /* desired distance between access points */ #define SPAN 1048576L // desired distance between access points
#define LEN 16384 /* number of bytes to extract */ #define LEN 16384 // number of bytes to extract
/* Demonstrate the use of deflate_index_build() and deflate_index_extract() by
processing the file provided on the command line, and extracting LEN bytes
from 2/3rds of the way through the uncompressed output, writing that to
stdout. An offset can be provided as the second argument, in which case the
data is extracted from there instead. */
int main(int argc, char **argv)
{
int len;
off_t offset = -1;
FILE *in;
struct deflate_index *index = NULL;
unsigned char buf[LEN];
/* open input file */ // Demonstrate the use of deflate_index_build() and deflate_index_extract() by
// processing the file provided on the command line, and extracting LEN bytes
// from 2/3rds of the way through the uncompressed output, writing that to
// stdout. An offset can be provided as the second argument, in which case the
// data is extracted from there instead.
int main(int argc, char **argv) {
// Open the input file.
if (argc < 2 || argc > 3) { if (argc < 2 || argc > 3) {
fprintf(stderr, "usage: zran file.gz [offset]\n"); fprintf(stderr, "usage: zran file.raw [offset]\n");
return 1; return 1;
} }
in = fopen(argv[1], "rb"); FILE *in = fopen(argv[1], "rb");
if (in == NULL) { if (in == NULL) {
fprintf(stderr, "zran: could not open %s for reading\n", argv[1]); fprintf(stderr, "zran: could not open %s for reading\n", argv[1]);
return 1; return 1;
} }
/* get optional offset */ // Get optional offset.
off_t offset = -1;
if (argc == 3) { if (argc == 3) {
char *end; char *end;
offset = strtoll(argv[2], &end, 10); offset = strtoll(argv[2], &end, 10);
@ -437,14 +397,18 @@ int main(int argc, char **argv)
} }
} }
/* build index */ // Build index.
len = deflate_index_build(in, SPAN, &index); struct deflate_index *index = NULL;
int len = deflate_index_build(in, SPAN, &index);
if (len < 0) { if (len < 0) {
fclose(in); fclose(in);
switch (len) { switch (len) {
case Z_MEM_ERROR: case Z_MEM_ERROR:
fprintf(stderr, "zran: out of memory\n"); fprintf(stderr, "zran: out of memory\n");
break; break;
case Z_BUF_ERROR:
fprintf(stderr, "zran: %s ended prematurely\n", argv[1]);
break;
case Z_DATA_ERROR: case Z_DATA_ERROR:
fprintf(stderr, "zran: compressed data error in %s\n", argv[1]); fprintf(stderr, "zran: compressed data error in %s\n", argv[1]);
break; break;
@ -458,19 +422,20 @@ int main(int argc, char **argv)
} }
fprintf(stderr, "zran: built index with %d access points\n", len); fprintf(stderr, "zran: built index with %d access points\n", len);
/* use index by reading some bytes from an arbitrary offset */ // Use index by reading some bytes from an arbitrary offset.
unsigned char buf[LEN];
if (offset == -1) if (offset == -1)
offset = (index->length << 1) / 3; offset = ((index->length + 1) << 1) / 3;
len = deflate_index_extract(in, index, offset, buf, LEN); ptrdiff_t got = deflate_index_extract(in, index, offset, buf, LEN);
if (len < 0) if (got < 0)
fprintf(stderr, "zran: extraction failed: %s error\n", fprintf(stderr, "zran: extraction failed: %s error\n",
len == Z_MEM_ERROR ? "out of memory" : "input corrupted"); got == Z_MEM_ERROR ? "out of memory" : "input corrupted");
else { else {
fwrite(buf, 1, len, stdout); fwrite(buf, 1, got, stdout);
fprintf(stderr, "zran: extracted %d bytes at %llu\n", len, offset); fprintf(stderr, "zran: extracted %ld bytes at %lld\n", got, offset);
} }
/* clean up and exit */ // Clean up and exit.
deflate_index_free(index); deflate_index_free(index);
fclose(in); fclose(in);
return 0; return 0;

69
examples/zran.h
Unescape View File

@ -1,40 +1,51 @@
/* zran.h -- example of zlib/gzip stream indexing and random access /* zran.h -- example of deflated stream indexing and random access
* Copyright (C) 2005, 2012, 2018 Mark Adler * Copyright (C) 2005, 2012, 2018, 2023 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h * For conditions of distribution and use, see copyright notice in zlib.h
* Version 1.2 14 Oct 2018 Mark Adler */ * Version 1.3 18 Feb 2023 Mark Adler */
#include <stdio.h> #include <stdio.h>
#include "zlib.h" #include "zlib.h"
/* Access point list. */ // Access point.
typedef struct point {
off_t out; // offset in uncompressed data
off_t in; // offset in compressed file of first full byte
int bits; // 0, or number of bits (1-7) from byte at in-1
unsigned char window[32768]; // preceding 32K of uncompressed data
} point_t;
// Access point list.
struct deflate_index { struct deflate_index {
int have; /* number of list entries */ int have; // number of access points in list
int gzip; /* 1 if the index is of a gzip file, 0 if it is of a int mode; // -15 for raw, 15 for zlib, or 31 for gzip
zlib stream */ off_t length; // total length of uncompressed data
off_t length; /* total length of uncompressed data */ point_t *list; // allocated list of access points
void *list; /* allocated list of entries */
}; };
/* Make one entire pass through a zlib or gzip compressed stream and build an // Make one pass through a zlib, gzip, or raw deflate compressed stream and
index, with access points about every span bytes of uncompressed output. // build an index, with access points about every span bytes of uncompressed
gzip files with multiple members are indexed in their entirety. span should // output. gzip files with multiple members are fully indexed. span should be
be chosen to balance the speed of random access against the memory // chosen to balance the speed of random access against the memory requirements
requirements of the list, about 32K bytes per access point. The return value // of the list, which is about 32K bytes per access point. The return value is
is the number of access points on success (>= 1), Z_MEM_ERROR for out of // the number of access points on success (>= 1), Z_MEM_ERROR for out of
memory, Z_DATA_ERROR for an error in the input file, or Z_ERRNO for a file // memory, Z_BUF_ERROR for a premature end of input, Z_DATA_ERROR for a format
read error. On success, *built points to the resulting index. */ // or verification error in the input file, or Z_ERRNO for a file read error.
// On success, *built points to the resulting index.
int deflate_index_build(FILE *in, off_t span, struct deflate_index **built); int deflate_index_build(FILE *in, off_t span, struct deflate_index **built);
/* Deallocate an index built by deflate_index_build() */ // Use the index to read len bytes from offset into buf. Return the number of
void deflate_index_free(struct deflate_index *index); // bytes read or a negative error code. If data is requested past the end of
// the uncompressed data, then deflate_index_extract() will return a value less
// than len, indicating how much was actually read into buf. If given a valid
// index, this function should not return an error unless the file was modified
// somehow since the index was generated, given that deflate_index_build() had
// validated all of the input. If nevertheless there is a failure, Z_BUF_ERROR
// is returned if the compressed data ends prematurely, Z_DATA_ERROR if the
// deflate compressed data is not valid, Z_MEM_ERROR if out of memory,
// Z_STREAM_ERROR if the index is not valid, or Z_ERRNO if there is an error
// reading or seeking on the input file.
ptrdiff_t deflate_index_extract(FILE *in, struct deflate_index *index,
off_t offset, unsigned char *buf, size_t len);
/* Use the index to read len bytes from offset into buf. Return bytes read or // Deallocate an index built by deflate_index_build().
negative for error (Z_DATA_ERROR or Z_MEM_ERROR). If data is requested past void deflate_index_free(struct deflate_index *index);
the end of the uncompressed data, then deflate_index_extract() will return a
value less than len, indicating how much was actually read into buf. This
function should not return a data error unless the file was modified since
the index was generated, since deflate_index_build() validated all of the
input. deflate_index_extract() will return Z_ERRNO if there is an error on
reading or seeking the input file. */
int deflate_index_extract(FILE *in, struct deflate_index *index, off_t offset,
unsigned char *buf, int len);

Write Preview
Loading…
Cancel
Save