PZlib (Alternative to Precomp)

[felice2011]

Same thing, I tried one minute ago the idea of Gozarck but give 0 Streams on any file, even those they should work.

[RamiroCruzo]

I don't get all the hocus-pocus around the Zlib term...So, here's some database about Zlib, Reflate, Brute, Precomp & pZlib.

@ChronoCross You should read this as there are no Bad-Zlib Stream or Invalid Streams.

Originally, data is stored in streams called Deflate Stream. Those streams are also called Headerless Zlib stream. Once, you wrap these Deflate streams into Headers, they become Zlib stream or Gzip Stream.

Zlib Stream contains Header + CRC + Data. (Header + CRC) = 15 position upto the stream. So Deflate stream = ZlibStream -15.

Now, why Deflate Streams are difficult to search for: As the header contains so called Magic Bytes which help to identify them, but when you seek for Deflate streams, its a bit more Brute & Slow...

Magic Bytes:

Common:

78 01, 78 5e, 78 9c, 78 da

Rare:

08 1d, 08 5b, 08 99, 08 d7, 18 19, 18 57, 18 95, 18 d3,
28 15, 28 53, 28 91, 28 cf, 38 11, 38 4f, 38 8d, 38 cb,
48 0d, 48 4b, 48 89, 48 c7, 58 09, 58 47, 58 85, 58 c3,
68 05, 68 43, 68 81, 68 de

Very rare:

08 3c, 08 7a, 08 b8, 08 f6, 18 38, 18 76, 18 b4, 18 f2,
28 34, 28 72, 28 b0, 28 ee, 38 30, 38 6e, 38 ac, 38 ea,
48 2c, 48 6a, 48 a8, 48 e6, 58 28, 58 66, 58 a4, 58 e2,
68 24, 68 62, 68 bf, 68 fd, 78 3f, 78 7d, 78 bb, 78 f9

Now comes how Reflate works...
Rawdet reads all the offsets of the file & initiates Inflate from each of them. [Inflate = Decompression of Zlib Stream]. It usually gets error where the stream isn't present & where ever it successfully inflates, it creates buffer out of the data. Thats what .raw are, unprocessed Deflate streams. Raw2hif firstly Inflates the stream & then Deflates (Compresses) the streams with the level you have provided e.g. -c9 compresses the stream in Level9, thats why the higher the level, the smaller is the output.

Brute on the other hand, instead of creating buffers, it directly inflates them and creates the output.

Now, comes the Precomp part...Its a storehouse of processes. You can assume its slowness is due to it hit & trial method. It again & again reads the file to discover more & more stream & even to find info about stream it takes about 9 * 9 trials. Thats why its disk reading values are high e.g. for Brute of 1 GB file it approx. reads 98 GB data.

pZlib, it scans for the Magic Bytes & Inflates the streams, easy-peasy... Not in coding....

[coveiro]

The Witcher 3 Wild Hunt




Done at normal HD, I will post later on my SSD

[ChronoCross]

Gozarck's tool works fine for me. Copy rawdet and bat file in the same path of the folder to test and drag the folder. This bat works with folder and not with files directly.
@ramirocruzo thnx for the info. Check the bug info on reflate by author and this link about invalid stream or non valid streams about bug or issue.
Precomp source code page 2 thread 41
http://encode.ru/threads/2411-Precom...n-GitHub/page2

[RamiroCruzo]

Originally Posted by ChronoCross

Gozarck's tool works fine for me. Copy rawdet and bat file in the same path of the folder to test and drag the folder. This bat works with folder and not with files directly.
@ramirocruzo thnx for the info. Check the bug info on reflate by author and this link about invalid stream or non valid streams about bug or issue.
Precomp source code page 2 thread 41
http://encode.ru/threads/2411-Precom...n-GitHub/page2

Well, re-read his whole reply. With Non-Valid Streams. he meant streams which are of smaller size that the header size of Precomp. If you try to process them, the lossless restoration info will be greater that the original file. That means compression ruined. As, Reflate doesn't make headers, so yes...Reflate can process it...

[Razor12911]

Multi threading support comng

[Razor12911]

Update available

Changes

Added multi threading support

[Mini]

wait CLS thx!!!

[panker1992]

Mini you dont need CLS, use exe to stdout and magic is done

cls is only a lib for arc.exe its even slower than original exe itself and it is forced on smaller buffer.

[Mini]

Dark Souls 3
Data5.bdt - 11.4 gb

pzlib v1.0 (max level) + lzma = 9.95 gb
pzlib v1.1 + lzma = 11 gb (http://i.imgur.com/Jt9r2qn.png)

wtf

[ChronoCross]

Don't worry mini. Is beta test. Test pzlib only. I was tested a game 68gb with pzlib+lzma and my hdd now is full and the test never end. But the application not crash.

[Razor12911]

Originally Posted by Mini

Dark Souls 3
Data5.bdt - 11.4 gb

pzlib v1.0 (max level) + lzma = 9.95 gb
pzlib v1.1 + lzma = 11 gb (http://i.imgur.com/Jt9r2qn.png)

wtf

is restoration possible for version 1.0?

Originally Posted by ChronoCross

Don't worry mini. Is beta test. Test pzlib only. I was tested a game 68gb with pzlib+lzma and my hdd now is full and the test never end. But the application not crash.

I try to avoid errors.

Update available

Changes


Threads inversely proportional to speed for stability (new maximum for threads is 4 instead of 8 )
-Speed for multi threading improved
-ms conversion to readable time added as requested
-encode/decode can be shorted to e/d

[Mini]

is restoration possible for version 1.0?

dont know

reflate result - 9.96 gb

I want to receive the same result with the help pzlib

[felice2011]

Hi Razor i have a problem in the data decompression using "PZlib 1110" I get this error.

unpackcmd = pzlib d - -o - <stdin> <stdout>

[Razor12911]

that's weird. will check.