PZlib (Alternative to Precomp)

[RamiroCruzo]

I don't get all the hocus-pocus around the Zlib term...So, here's some database about Zlib, Reflate, Brute, Precomp & pZlib.

@ChronoCross You should read this as there are no Bad-Zlib Stream or Invalid Streams.

Originally, data is stored in streams called Deflate Stream. Those streams are also called Headerless Zlib stream. Once, you wrap these Deflate streams into Headers, they become Zlib stream or Gzip Stream.

Zlib Stream contains Header + CRC + Data. (Header + CRC) = 15 position upto the stream. So Deflate stream = ZlibStream -15.

Now, why Deflate Streams are difficult to search for: As the header contains so called Magic Bytes which help to identify them, but when you seek for Deflate streams, its a bit more Brute & Slow...

Magic Bytes:

Common:

78 01, 78 5e, 78 9c, 78 da

Rare:

08 1d, 08 5b, 08 99, 08 d7, 18 19, 18 57, 18 95, 18 d3,
28 15, 28 53, 28 91, 28 cf, 38 11, 38 4f, 38 8d, 38 cb,
48 0d, 48 4b, 48 89, 48 c7, 58 09, 58 47, 58 85, 58 c3,
68 05, 68 43, 68 81, 68 de

Very rare:

08 3c, 08 7a, 08 b8, 08 f6, 18 38, 18 76, 18 b4, 18 f2,
28 34, 28 72, 28 b0, 28 ee, 38 30, 38 6e, 38 ac, 38 ea,
48 2c, 48 6a, 48 a8, 48 e6, 58 28, 58 66, 58 a4, 58 e2,
68 24, 68 62, 68 bf, 68 fd, 78 3f, 78 7d, 78 bb, 78 f9

Now comes how Reflate works...
Rawdet reads all the offsets of the file & initiates Inflate from each of them. [Inflate = Decompression of Zlib Stream]. It usually gets error where the stream isn't present & where ever it successfully inflates, it creates buffer out of the data. Thats what .raw are, unprocessed Deflate streams. Raw2hif firstly Inflates the stream & then Deflates (Compresses) the streams with the level you have provided e.g. -c9 compresses the stream in Level9, thats why the higher the level, the smaller is the output.

Brute on the other hand, instead of creating buffers, it directly inflates them and creates the output.

Now, comes the Precomp part...Its a storehouse of processes. You can assume its slowness is due to it hit & trial method. It again & again reads the file to discover more & more stream & even to find info about stream it takes about 9 * 9 trials. Thats why its disk reading values are high e.g. for Brute of 1 GB file it approx. reads 98 GB data.

pZlib, it scans for the Magic Bytes & Inflates the streams, easy-peasy... Not in coding....