PZlib (Alternative to Precomp)

[Razor12911]

Pre-ZLIB (pzlib)

Alternative of precomp

This program detects and unpack zlib streams for a certain input
giving you an option to apply a stronger compression such as LZMA.

This only detects zlib streams compressed using default and maximum level
which have headers. (It doesn't detect headerless)

Usage:
pZLib encode/decode [options] inputfile -o outputfile
inputfile/outputfile can be specified as "-" for stdin/stdout

Options:
-t#: Number of threads to use
-t#p: (default = 50) Percentage of threads to use
-v: Verification so that restoration keeps CRC the same
-ai: Prints out additional information

Note:
Input and/or Output can be "-" to signal stdin and/or stdout usage.
Multi threading only works during restoration.

Example:
pzlib encode $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp

Created by Razor12911 and Panker1992

[Mini]

good job! waiting CLS-PZlib ^_^

[Razor12911]

CLS-PZlib is not really necessary. It will work with Inno no problem as long as arc.ini is there, it will show progress etc.

[Razor12911]

CLS uploaded

Took about 20 minutes to make CLS, its not properly tested but it works.

CLS works on its own, it doesn't need exe alongside it.

[panker1992]

This Method is actually a zlib killer :P

it can pack Dragon Age Inquisition, which precomp took 12 hours in about 1 our or so :D

i hope we see some results, this is the method i had once used to show reflate's speed packing dragon's dogma.

this is actually one of the 4 methods our fully fledged reflate has :D

enjoy !!

[panker1992]

Me and Razor decided to show you the progress we did so far,

this test is about how far we are going with reflate

input file was the entire game Dragon's Dogma

we only applied -mreflate and not fully repack for the sake of comparison

the game total input was 19.7 GB

the left CMD is HDD and right CMD is SSD

the process on HDD took 54 minutes, in which reflate took about 30 minutes to fully process the file

the process on SSD took 38 minutes, in which reflate took about 22 minutes to fully process the file


Edit : Extraction was Successful :D it took 28 minutes to decomp <stdin> <stdout> usage

Original Article here : http://fileforums.com/showthread.php?t=97672&highlight=reflate

[Razor12911]

The effect of "-l" option.

[Amsal]

Thank you panker and razor for the soft, I think your partnership is going well. I hope you come with more great works. Thank you again for this ;)

[Amsal]

Is this work with ASC?

[Razor12911]

Yea, I tested.

[Stained]

Can you explain this sentence, "348MB instead of 82.3MB" ? Does it mean that the original file size was 82.3MB and resultant file size came out to be 348MB?

[Razor12911]

It means that's the new file size from the input, from the 348MB, you then apply stronger compression to make the file size drop even further, for example this file. Originally, it is 82.3MB, after applying pZlib, it becomes 348MB, from that apply LZMA it becomes 9MB. (For this files, other files behave differently)

[Stained]

It means that's the new file size from the input, from the 348MB, you then apply stronger compression to make the file size drop even further, for example this file. Originally, it is 82.3MB, after applying pZlib, it becomes 348MB, from that apply LZMA it becomes 9MB. (For this files, other files behave differently)
9MB! That's amazing, man. And thank you, for explaining.😊

[ChronoCross]

Very nice tool works correctly.

[Mini]

1. what news about decompression FZR?
2. whether there will be a support multithreading Pzlib?

[Razor12911]

Currently busy with exams

[Andrey167]

Multithread work?

[panker1992]

this is faster than anything on market with minimum usage of RAW power.

its blazing speed will leave you almost speechless ....processing 20 GB game in less than half an hour. for 1 core.

The Multithreaded version of this is reserved for me and Zee for now :)


@lolaya this tool decompresses all zlib from an input thus you gain better end compression in the end.

[felice2011]

this is faster than anything on market with minimum usage of RAW power.

its blazing speed will leave you almost speechless ....processing 20 GB game in less than half an hour. for 1 core.

The Multithreaded version of this is reserved for me and Zee for now :)


@lolaya this tool decompresses all zlib from an input thus you gain better end compression in the end.

WOW.....:eek:

As I have a little of time to test with pleasure...;)

[Stained]

Guys, is this awesome work ready to be used in a conversion? I mean can I use it ?
Also, if yes, I will need to include the file in decompression also, right? Or just using it, then lmza and then conversion will wok?

[Razor12911]

yea, then just add cls then done.

[Stained]

Thank you man. Will try it out tomorrow ��

[felice2011]

Hey Razor tell me which file extensions you've tried with PZlib posted in your tests ...;)

[Razor12911]

Cpk for pes, sb for nfs tr, arc for re5, bndl for nfshp and nfsmw...

[felice2011]

I took some files sample of games currently installed on my computer ...

1) Game = BAK / File = *.upk

http://i65.tinypic.com/2111ooh.png

2) Game = BAK / File = *.usm

http://i65.tinypic.com/dq6353.png

3) Game = Dirt Rally / File= *.bk2

http://i65.tinypic.com/2j4qz9c.png

4) Game = Dirt Rally / File= *.pssg

http://i66.tinypic.com/2elftso.png

5) Game = MGS5 / File= *.dat

http://i63.tinypic.com/28i23rt.png

6) Game = MadMax / File= *.arc

http://i65.tinypic.com/23j50df.png

7) Game = ProjectCar / File= *.bff

http://i65.tinypic.com/97mlj8.png

You should be more precise, to specify what types of flows detecting in input, and what types of files, so use your new compressor with the desired result.
Except the positive result of the first test posted, with all the others I have not gotten the desired result.

************************************************** ************************************************** ****************
Edit: Test Only Upk Files

But here the situation is different, I wanted to test the only files *.upk, that in their results they have given a positive result to the new "PZlib"
Amazing speed in comparison to the use of the last precomp045, and better compression compared to reflate 1.4 ...

http://i63.tinypic.com/2lj6dxe.png

http://i67.tinypic.com/2lwr3wz.png

[Razor12911]

Haha, it aint about extensions sometimes mate, I scan files, I depend less on extension.

Strange, Dirt Rally does have zlibs and should have been detected by pZLib.

Game = Dirt Rally / File= *.bk2 (These are videos, no zlib can be located on that), same with usm.

Mad Max has headerless zlibs and those are not detected by pzlib.

[felice2011]

Haha, it aint about extensions sometimes mate, I scan files, I depend less on extension.

Strange, Dirt Rally does have zlibs and should have been detected by pZLib.

Game = Dirt Rally / File= *.bk2 (These are videos, no zlib can be located on that), same with usm.

Mad Max has headerless zlibs and those are not detected by pzlib.

:rolleyes:OH..OH..:eek:..and this was you aware only you and possibly Panker:D

Ok you're the maestro of compression, I'm not trying to belittle or criticize your work, I'm tested your new compressor, and better understand its potential, maintaining a high feedback of the discussion, on the contrary would not be interested in anyone, what do you do you think?
I will do additional testing on complete games with multiple files.
Summing up your work is that of Panker is a great project, I hope in more news and updates.;)

[gozarck]

Simple batch for drag an drop folder to generate a list of files ready to test with reflate or pzlib. Generate 2 files rsd1.txt is a list of files with streams and rsd0.txt list of files with 0 streams. Maybe panker can do a mask for files exclusive to use reflate or pzlib for save time.
Drag the game folder into a batch and this batch generate the list of files.
Edit. I was forget to remove "" in some part of the batch sorry now is fixed. Now you can use [email protected] in arc.exe

[ChronoCross]

Thnx gozarck this help a lot to select the test files for pzlib.

[Razor12911]

Simple batch for drag an drop folder to generate a list of files ready to test with reflate or pzlib. Generate 2 files rsd1.txt is a list of files with streams and rsd0.txt list of files with 0 streams. Maybe panker can do a mask for files exclusive to use reflate or pzlib for save time.
Drag the game folder into a batch and this batch generate the list of files.
Edit. I was forget to remove "" in some part of the batch sorry now is fixed. Now you can use [email protected] in arc.exe
Great but I somehow can't make it work.

Always says:
0 Streams Founded Available to apply Reflate :D

@ChronoCross and gozarck
I don't think that this is going to work. pZlib detects zlibs using their magic headers.
http://stackoverflow.com/questions/9050260/what-does-a-zlib-header-look-like
http://tools.ietf.org/html/rfc1950

For example, GTAV. Gozarck, your detector will work because rawdet has its methods of detecting that pZlib doesn't because pZlib itself has its own methods of detecting which I linked up there. Though I created Brute which works and the detector might work there, brute is still under development, restoring some streams is going to be hell, something big needs to be done first.

:rolleyes:OH..OH..:eek:..and this was you aware only you and possibly Panker:D

Ok you're the maestro of compression, I'm not trying to belittle or criticize your work, I'm tested your new compressor, and better understand its potential, maintaining a high feedback of the discussion, on the contrary would not be interested in anyone, what do you do you think?
I will do additional testing on complete games with multiple files.
Summing up your work is that of Panker is a great project, I hope in more news and updates.;)
I thought everybody knew, bk2 are bink 2 videos and usm is also a video format. It can't work on Mad Max because of the reason above I just explained.
We did try Dirt Rally though, recompressor needed 3.4GB from just 700mb worth of data and decompression failed, it needed 5GB somehow, Dirt Rally has huge zlib streams, we estimated that the game even when it is big already, originally with all the files updated, it is somewhere around 100GB just like GTAV.
Thank you for the test though.

It also works for Dragon Age BTW, Panker tested it a long time ago. Here are the results
http://fileforums.com/showpost.php?p=448875&postcount=6

[felice2011]

Same thing, I tried one minute ago the idea of Gozarck but give 0 Streams on any file, even those they should work.

[RamiroCruzo]

I don't get all the hocus-pocus around the Zlib term...So, here's some database about Zlib, Reflate, Brute, Precomp & pZlib.

@ChronoCross You should read this as there are no Bad-Zlib Stream or Invalid Streams.

Originally, data is stored in streams called Deflate Stream. Those streams are also called Headerless Zlib stream. Once, you wrap these Deflate streams into Headers, they become Zlib stream or Gzip Stream.

Zlib Stream contains Header + CRC + Data. (Header + CRC) = 15 position upto the stream. So Deflate stream = ZlibStream -15.

Now, why Deflate Streams are difficult to search for: As the header contains so called Magic Bytes which help to identify them, but when you seek for Deflate streams, its a bit more Brute & Slow...

Magic Bytes:

Common:

78 01, 78 5e, 78 9c, 78 da

Rare:

08 1d, 08 5b, 08 99, 08 d7, 18 19, 18 57, 18 95, 18 d3,
28 15, 28 53, 28 91, 28 cf, 38 11, 38 4f, 38 8d, 38 cb,
48 0d, 48 4b, 48 89, 48 c7, 58 09, 58 47, 58 85, 58 c3,
68 05, 68 43, 68 81, 68 de

Very rare:

08 3c, 08 7a, 08 b8, 08 f6, 18 38, 18 76, 18 b4, 18 f2,
28 34, 28 72, 28 b0, 28 ee, 38 30, 38 6e, 38 ac, 38 ea,
48 2c, 48 6a, 48 a8, 48 e6, 58 28, 58 66, 58 a4, 58 e2,
68 24, 68 62, 68 bf, 68 fd, 78 3f, 78 7d, 78 bb, 78 f9


Now comes how Reflate works...
Rawdet reads all the offsets of the file & initiates Inflate from each of them. [Inflate = Decompression of Zlib Stream]. It usually gets error where the stream isn't present & where ever it successfully inflates, it creates buffer out of the data. Thats what .raw are, unprocessed Deflate streams. Raw2hif firstly Inflates the stream & then Deflates (Compresses) the streams with the level you have provided e.g. -c9 compresses the stream in Level9, thats why the higher the level, the smaller is the output.

Brute on the other hand, instead of creating buffers, it directly inflates them and creates the output.

Now, comes the Precomp part...Its a storehouse of processes. You can assume its slowness is due to it hit & trial method. It again & again reads the file to discover more & more stream & even to find info about stream it takes about 9 * 9 trials. Thats why its disk reading values are high e.g. for Brute of 1 GB file it approx. reads 98 GB data.

pZlib, it scans for the Magic Bytes & Inflates the streams, easy-peasy... :p Not in coding....

[coveiro]

The Witcher 3 Wild Hunt


http://i.imgsafe.org/78f2bd7.png

Done at normal HD, I will post later on my SSD

[ChronoCross]

Gozarck's tool works fine for me. Copy rawdet and bat file in the same path of the folder to test and drag the folder. This bat works with folder and not with files directly.
@ramirocruzo thnx for the info. Check the bug info on reflate by author and this link about invalid stream or non valid streams about bug or issue.
Precomp source code page 2 thread 41
http://encode.ru/threads/2411-Precomp-source-code-on-GitHub/page2

[RamiroCruzo]

Gozarck's tool works fine for me. Copy rawdet and bat file in the same path of the folder to test and drag the folder. This bat works with folder and not with files directly.
@ramirocruzo thnx for the info. Check the bug info on reflate by author and this link about invalid stream or non valid streams about bug or issue.
Precomp source code page 2 thread 41
http://encode.ru/threads/2411-Precomp-source-code-on-GitHub/page2

Well, re-read his whole reply. With Non-Valid Streams. he meant streams which are of smaller size that the header size of Precomp. If you try to process them, the lossless restoration info will be greater that the original file. That means compression ruined. As, Reflate doesn't make headers, so yes...Reflate can process it...

[Razor12911]

Multi threading support comng

[Razor12911]

Update available

Changes

Added multi threading support

[Mini]

wait CLS ;) thx!!!

[panker1992]

Mini you dont need CLS, use exe to stdout and magic is done

cls is only a lib for arc.exe its even slower than original exe itself and it is forced on smaller buffer.

[Mini]

Dark Souls 3
Data5.bdt - 11.4 gb

pzlib v1.0 (max level) + lzma = 9.95 gb
pzlib v1.1 + lzma = 11 gb (http://i.imgur.com/Jt9r2qn.png)

wtf

[ChronoCross]

Don't worry mini. Is beta test. Test pzlib only. I was tested a game 68gb with pzlib+lzma and my hdd now is full and the test never end. But the application not crash. :)

[Razor12911]

Dark Souls 3
Data5.bdt - 11.4 gb

pzlib v1.0 (max level) + lzma = 9.95 gb
pzlib v1.1 + lzma = 11 gb (http://i.imgur.com/Jt9r2qn.png)

wtf
is restoration possible for version 1.0?

Don't worry mini. Is beta test. Test pzlib only. I was tested a game 68gb with pzlib+lzma and my hdd now is full and the test never end. But the application not crash. :)

I try to avoid errors.

Update available

Changes


Threads inversely proportional to speed for stability (new maximum for threads is 4 instead of 8 )
-Speed for multi threading improved
-ms conversion to readable time added as requested
-encode/decode can be shorted to e/d

[Mini]

is restoration possible for version 1.0?
dont know

reflate result - 9.96 gb

I want to receive the same result with the help pzlib :)

[felice2011]

Hi Razor i have a problem in the data decompression using "PZlib 1110" I get this error.

http://i63.tinypic.com/2s991dt.png

unpackcmd = pzlib d - -o - <stdin> <stdout>

:confused:

[Razor12911]

that's weird. will check.

[Prince4]

Can we integrate this with ISDone?

[felice2011]

Hi Razor, news of some stable update. I was willing to use it but I always problems on decompression.

[Razor12911]

Hi Razor, news of some stable update. I was willing to use it but I always problems on decompression.

What problems?

[Carldric Clement]

Hi Razor, news of some stable update. I was willing to use it but I always problems on decompression.

If you problem about your Compressor GUI, like my Mini Compressor, use this if helpful:

[External compressor:pzlib]
header = 0
unpackcmd = pzlib d $$arcpackedfile$$.tmp -o $$arcdatafile$$.tmp

[felice2011]

If you problem about your Compressor GUI, like my Mini Compressor, use this if helpful:

[External compressor:pzlib]
header = 0
unpackcmd = pzlib d $$arcpackedfile$$.tmp -o $$arcdatafile$$.tmp

I do not think that's what the problem was too obvious, already I tried to decompress with both lines:
unpackcmd = pzlib d $$ arcpackedfile $$. tmp -o $$ arcdatafile $$. tmp & unpackcmd = pzlib -d - -o - <stdin> <stdout>
In fact out of curiosity I wanted to try your utility, the same error.

http://i68.tinypic.com/sdzrqc.png
What problems?
The problem is always the same, already posted ...

http://www.fileforums.com/showthread.php?t=97888&page=3#45

Using the line "unpackcmd = pzlib -d - -o - <stdin> <stdout>"

&

http://i67.tinypic.com/9hszlz.png

Using the line "unpackcmd = pzlib d $$arcpackedfile$$.tmp -o $$arcdatafile$$.tmp"

Personally I think this is one of the best compression projects developed. On my part I have much interest in the excellent level of compression, with a high speed of execution of the packing, compared to the classic precompressor.
My intention was also to integrate the MSK in the latest version, but if errors are generated during decompression of the archive created, I cannot guarantee the complete functionality.
I examine of more to see if there any file or type of files that can generate the error during decompression.

[Razor12911]

add -v option during comprssion if you have CRC errors during decompression

[External compressor:pzlib]
header = 0
packcmd = pzlib e - -o - <stdin> <stdout>
unpackcmd = pzlib d - -o - <stdin> <stdout>

[felice2011]

OK I solved used PZlib v1100 (24/05/16), with this line everything works perfectly in Compr & Decomp..;)


[External compressor:pzlib]
header = 0
packcmd = pzlib e -v - -o - <stdin> <stdout>
unpackcmd = pzlib d - -o - <stdin> <stdout>


:p

[LuzhinE]

He tried in PES 2016, is good PZlib

[Razor12911]

OK I solved used PZlib v1100 (24/05/16), with this line everything works perfectly in Compr & Decomp..;)


[External compressor:pzlib]
header = 0
packcmd = pzlib e -v - -o - <stdin> <stdout>
unpackcmd = pzlib d - -o - <stdin> <stdout>


:p

I even mentioned beginning of topic, if you encounter CRC errors or just to make sure CRC stays the same, add -v

Options:
-t#: Number of threads to use (maximum = 8)
-t#p: (default = 50) Percentage of threads to use
-v: Verification so that restoration keeps CRC the same
-ai: Prints out additional information

[LuzhinE]

.arc not compress files

[felice2011]

.arc not compress files
:confused:

[felice2011]

Sorry Razor but two out of three times I get crash of the arc.exe at the beginning of compression ...

https://goo.gl/ylQXJL

I don't understand what the problem...:o

[Razor12911]

It happens, freearc bug.

[felice2011]

It happens, freearc bug. :eek:

Tests performed on 33 different compressors, combined with various methods and masks, never any crashes with arc.exe, the thing is quite strange.:confused:

EDIT:

Probably I solved the problem lies in the line of options, run other tests of safety and I post the right configuration.

Final Test PZlib V1100 recommended options if you are experiencing app-crash in arc.exe

Comp;


[External compressor:pzlib]
header = 0
packcmd = pzlib e -v $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp


Decomp;


[External compressor:pzlib]
header = 0
unpackcmd = pzlib d - -o - <stdin> <stdout>

[Razor12911]

It's now an ordinary bug on my side, compressors with stdin and stdout during compression, and if you're compressing a large number of files, this happens. Pretty much annoying.

[felice2011]

Pretty much annoying.:cool:

;)The important that the problem is solved, even with a large number of files...:p

[Razor12911]

:p but still, when everything is piped, overall speed is better.

[felice2011]

Razor I need of the command for the console help info of PZlib and Reflate, if you have implemented.
I tried it with the command "pzlib.exe -h" or "pzlib.exe -help" but it does not work.

[Razor12911]

well unfortunately, i haven't added that.

[felice2011]

well unfortunately, i haven't added that.
Ah..Ok.
I am entering a new source on the help info in MSK on 34 different compressors, when I will have finished to compile the ContextMenuStrip I send you a PM to complete the info on some compressors.
Thanks Bro..

[LuzhinE]

and error because?

http://i.imgur.com/gEl4I5o.png

[felice2011]

Check set..http://fileforums.com/showthread.php?t=97888&page=4#60

[Razor12911]

Update for PZLib available

Changes

Only added x64 support.

Comparison between x86 and x64

File = UI.sb
Size = 82.3MB > 348MB

Recompression times with verification off:
x86: 4.81 Seconds (17.93MB/s)
x64: 4.20 Seconds (20.57MB/s)

Recompression times with verification on:
x86: 38.27 Seconds (2.26MB/s)
x64: 26.65 Seconds (3.24MB/s)

Restoration times using 4 threads:
x86: 11.08 Seconds (7.79MB/s)
x64: 7.95 Seconds (10.87MB/s)

[Prince4]

Update for PZLib available

Changes

Only added x64 support.

Comparison between x86 and x64

File = UI.sb
Size = 82.3MB > 348MB

Recompression times with verification off:
x86: 4.81 Seconds (17.93MB/s)
x64: 4.20 Seconds (20.57MB/s)

Recompression times with verification on:
x86: 38.27 Seconds (2.26MB/s)
x64: 26.65 Seconds (3.24MB/s)

Restoration times using 4 threads:
x86: 11.08 Seconds (7.79MB/s)
x64: 7.95 Seconds (10.87MB/s)

Where's the download link? :P

[mausschieber]

Where's the download link? :P

look at first post

[LuzhinE]

It does not open how it works?

[Razor12911]

It does not open how it works?

-_-, this works with Freearc man...

[RamiroCruzo]

-_-, this works with Freearc man...

hahaha...

[Razor12911]

Update for PZLib available

Changes

Updated multi threading

[chum]

Hey bro, you said to use -v for CRC problems, but using -v just blocked any compressing gains, the compressed file just become out same size of input file. Without -v decompressing is just failing with CRC Always.

I like Pzlib for compresisng since it is much faster than Precomp and give nearly same final size but decompressing problem is just avoiding to it be a standard usage for me.

If you just remove the multithreading for decompressing dont it fix the problem since decompressing is known to work properly in single-thread..

[Razor12911]

Hey bro, you said to use -v for CRC problems, but using -v just blocked any compressing gains, the compressed file just become out same size of input file. Without -v decompressing is just failing with CRC Always.
That isn't the case most of the time.


I like Pzlib for compresisng since it is much faster than Precomp and give nearly same final size but decompressing problem is just avoiding to it be a standard usage for me.

If you just remove the multithreading for decompressing dont it fix the problem since decompressing is known to work properly in single-thread..

I don't understand this part here.

[aswadd]

I have a problem
CRC failed in "data\corecharacter.arch05". File is broken.
http://i.imgur.com/uwGpfaJ.png
it happened with any file type :\
I also turned off threads but still this error / tried -v option but it didn't even make an output file :(

[aswadd]

That isn't the case most of the time.


I don't understand this part here.

the problem goes with -v parameter , the way you add it in arc.ini :D
I finally got it work comp/decomp was success

Just add -v at the end of the line & not after pzlib ex:
packcmd = pzlib e - -o - <stdin> <stdout> -v

don't put it like this or even in the middle ... etc
packcmd = pzlib -v e - -o - <stdin> <stdout>

same problem happens with antiz Just add -r at the end :D

in the end :D thnx razor & Panker for this great work :)

[Razor12911]

Guys, I said this in the beginning of the thread, you must add -v if you're getting CRC errors.

-v is just verification, it just makes sure that the streams that were found match CRC when being restored, it permutes some streams which is why you get a better size without verification than when it's turned on.

[aswadd]

I got another problem Arc crash when trying pzlib on
Need for speed MW 2012 :( while other pre-compressors didn't make arc crash
http://i.imgur.com/bgaNOkF.png

[Gupta]

I got another problem Arc crash when trying pzlib on
Need for speed MW 2012 :( while other pre-compressors didn't make arc crash
http://i.imgur.com/bgaNOkF.png

this are the things which are already discussed in the forum

use

[External compressor:pzlib]
header = 0
packcmd = pzlib e -v $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp
unpackcmd = pzlib d - -o - <stdin> <stdout>

[chum]

Guys, I said this in the beginning of the thread, you must add -v if you're getting CRC errors.

-v is just verification, it just makes sure that the streams that were found match CRC when being restored, it permutes some streams which is why you get a better size without verification than when it's turned on.

The problem Razor is than using -v for Shadow of mordor cause the output data1.arc come same size of input folder files. Input folder files size=16.5Gb, data1.arc=16.5GB with -v added. Without -v data1.arc compress fine to 5GB but then unpack just give CRC error.

[Razor12911]

yea, it means the data could be recompressed but CRC could not be maintained.

[~MAK~]

yea, it means the data could be recompressed but CRC could not be maintained.

Then is it possible that you make 2 version of pzlib, first one is your this already released one, and second one will have this idea if possible while compression and decompression there should not be any CRC checks and later we could patch that file with xdelta to make it md5 perfect :)

[Razor12911]

Haha, that will not work. The difference is too much, it isn't just a few bytes that differ but the whole thing. Tried xdelta, it isn't going to help, instead it will make diff files almost the same size as original sometimes even more than which cannot be compressed further then end up with negative ratio, end up with ratio more than 100% even after compression.

[~MAK~]

Haha, that will not work. The difference is too much, it isn't just a few bytes that differ but the whole thing. Tried xdelta, it isn't going to help, instead it will make diff files almost the same size as original sometimes even more than which cannot be compressed further then end up with negative ratio, end up with ratio more than 100% even after compression.

hahaha, I already expected that :P . Anyways Nice to meet you

[Razor12911]

Nice to meet you too.

[ChronoCross]

Testing pzlib pipe downloaded from encode.ru.
7z tar pzlib e - - srep 7z lzma
compression good
decompression fail stuck in 39% i need to add -v option?

[Razor12911]

-v doesn't exist, it's a new program I'm creating, somehow it gets stuck. really frustrating.

[Razor12911]

Source uploaded

[Prince4]

Did anyone fixed the ERROR: Stream read error that felice showed no matter what version i use its the same :(

[Viper645]

The effect of "-l" option.

Then what is the use of l5 and l7 ? to make the compression slower? :p

[Razor12911]

Then what is the use of l5 and l7 ? to make the compression slower? :p

http://fileforums.com/showpost.php?p=448876&postcount=7

:p

[78372]

pZlib 1300

Removed. You don't need it, you won't get it

[Simorq]

pZlib with large files 10GB problem.

C:\Users\Administrator\Downloads\Compressed\UA Pro\UltraARC 2801 R3>arc a -ep1 -r -ed -dses -s; -di -i2 -wE:\FFOutput\New folder\ArcTmp -n@CfgTmp\List1.txt -mMMC E:\FFOutput\New folder\Data1.Bin -dpE:\Program Files\Hellblade Senuas Sacrifice\HellbladeGame\Content\Paks
FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: E:\FFOutput\New folder\Data1.Bin using pZLib1
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 17,325,579,914 => 26,739,971,247 bytes. Ratio 154.34%
Compression time: cpu 34.61 sec/real 3033.32 sec = 1%. Speed 5.71 mB/s
All OK

Extract
http://uupload.ir/files/suf6_bug.png

[78372]

May be because of stdio error?

[Simorq]

May be because of stdio error?

Well, I'll do another test with pZLib and report it

[Simorq]

where is the problem?
Direct use of pZLib is no problem.
But in FreeARC, when it decoded, it would give Disk Full.

http://uupload.ir/files/vvcr_hellbladegame-windowsnoeditor.pak.jpg

[78372]

Weird, will check tomorrow

[78372]

Try compressing with arcdatafile and arcoacked file, and decompress with stdio and ISDone. Then report back please.

[-NORO-]

I tried pzlib 1300 with 17.7 gb files , decompression stuck at 6 or 10 % :confused:
Used stdin/stdout
PS: precompression is really slow compared to pzlib v3 .

[Gupta]

i don't understand what is the whole point of having another variant, its just a waste of testing time
about speed, i will rather prefer stability

i mean you already have ztool,pzlibv3, reflate,precomp
posting another scoped variant just create headaches for novices, others will never bother download this even,
of course, in respect to razor's work, it just adds a unworking program to his name, headaches for novices

[felice2011]

The only way to avoid any problems is to use it under the mask, pzlib still suffers from severe bugs. Used with support FA during scanning and compression of the files, if some files do not contain streams, pzlib generates an error Inside not reported, but compression of the archive seems to be successful. At decompression we get the classic errors detected from pzlib.;)

[-NORO-]

i don't understand what is the whole point of having another variant, its just a waste of testing time
about speed, i will rather prefer stability

i mean you already have ztool,pzlibv3, reflate,precomp
posting another scoped variant just create headaches for novices, others will never bother download this even,
of course, in respect to razor's work, it just adds a unworking program to his name, headaches for novices

While you prefer stability others do prefer speed ,btw ztool doesnt give same ratio as pzlib do.
If he continues to work on it , it will be an improved version of pzlib.

[Chayan Manna]

btw ztool doesnt give same ratio as pzlib do.

Yeah, ZTool gives better ratio than pZLib.

[-NORO-]

Yeah, ZTool gives better ratio than pZLib.

I meant the contrary it has worst ratio :confused:
I tried few times with mutliple games pzlib and ztool
pzlib gave better ratio , weird

[Gupta]

Does pzlib is able to restore them back?

[Chayan Manna]

I meant the contrary it has worst ratio :confused:
I tried few times with mutliple games pzlib and ztool
pzlib gave better ratio , weird

No, ZTool gave better ratio at every game file I have tested, that's why Razor12911 made it private.
Unlike pZLib the parameter "-x" is not for the ultimate settings in ZTool. To gain the best ratio from ZTool the parameter is different. And ZTool is also Faster & more stable than pZLib.

[-NORO-]

No, ZTool gave better ratio at every game file I have tested, that's why Razor12911 made it private.
Unlike pZLib the parameter "-x" is not for the ultimate settings in ZTool. To gain the best ratio from ZTool the parameter is different. And ZTool is also Faster & more stable than pZLib.

Ohh i didnt test it with ultimate settings parameter , i know it is really faster and stable.

[78372]

Yeah, ZTool gives better ratio than pZLib.

Do ya know why ztool gives more ratio than pzlib?
Because ztool uses reflate libs even when detecting zlib streams(m1)

A week ago I tested with a 1GB file from jc2. pZlib 1350 gave 2.13GB and ztool gave 2.2GB.
Ofc pzlib 1350 was faster, while giving a smaller ratio.The 2.2GB file became 610mb after processing by srep, and 2.13GB by pzlib became 536mb. So it's proved that ztool uses reflate, which causes better ratio.

[-NORO-]

Do ya know why ztool gives more ratio than pzlib?
Because ztool uses reflate libs even when detecting zlib streams(m1)

A week ago I tested with a 1GB file from jc2. pZlib 1350 gave 2.13GB and ztool gave 2.2GB.
Ofc pzlib 1350 was faster, while giving a smaller ratio.The 2.2GB file became 610mb after processing by srep, and 2.13GB by pzlib became 536mb. So it's proved that ztool uses reflate, which causes better ratio.

Did you update pzlib 1350 ?
I had decompression errors with 17 gb files

[78372]

Yep, but you have pzlib v3, use that :)
ztool is faster than pzlib 1350 in most cases. Very few games are processed faster(about 2x) by pzlib 1350

[linggar17x]

i get error crc when decompression..
How to solve?

[78372]

Read the whole thread, you will find your answer

[GBT repack]

What is the best compression rate now?
Seeking to share
thinks

[MMBDON]

Can anybody help me, how to solve decompression problem with pzlib compressed data. I have already compressed some files of max payne 3 with pzlib v3. Compressed ratio became 44.9% from 6.78 GB - 3.05 GB. But when i'm decompressing this compressed file with WPI by including pzlib, some files were not extracting. I don't know why? I using compression method as, pzlib3+srep+delta+exe2+lzma2:ultra:a1:mfbt4:d518m: fb273:mc10000:lc8.
Please help me......

[shazzla]

Do not use stdio @compression !

[Kitsune1982]

hi guys. i am using razor12911's IC 3020 UA2008R2 modification, and i am having some problems with pzlib. the game i am trying to compress, syberia 1, has zlib streams inside the cmo folder in the root folder of the installed game, but when i try to install my repack after using pzlib on the folder with the zlib streams, the data/cmo folder, i keep getting a 'unarc returned an error error code -11 decompression fails - archive is corrupted' error and i know it has zlib streams because i scanned the folder with zlib checker. any ideas whats going on?

the cmo folder is only 95 mb but i want to get the hang of using pzlib.
please post back, i am frustrated and dont know why its not working.
ive tried using pzlib by itself which results in a larger file and pzlib with lzma both using UltraArc 2800 R2 and get the error code -11 with the second attempt.
any feedback will really help. thanks.

-Kitsune

[Andu21]

If you are using -f option remove it also try compressing from file to file instead of stdio.

[Kitsune1982]

hi guys, thanks for helping me with my pzlib question. i am still having issues so i thought i would post my arc.ini info here to further assist you all in helping me resolve my issue.

- arc.ini from F:\UA2800R2\Resources

My Method:
tor = srep+exe+exe2+delta+dict

Example (; = Disable)
;tor = /$File=srep+Lzma2/$Deflate=pzlib+srep+lzma:176mb:normal:bt4:273:mc10 00000000:lc8
;tor = precompmt+srep+lzma2
;tor = antiz+srep+nanozip

[External compressor:pzlib]
header = 0
packcmd = pZLib\pZLib e {options} -c128m -t75p - -o - $$arcdatafile$$.tmp -o - <stdout>
unpackcmd = pZLib\pZLib d {options} -t25p - -o - <stdin> <stdout>


[External compressor:pzlibv2]
header = 0
packcmd = pZLib v2\pZLib e -m3 -r6 {options} -c32m -t25p - -o - $$arcdatafile$$.tmp -o - <stdout>
unpackcmd = pZLib v2\pZLib d {options} -t100p - -o - <stdin> <stdout>

arc.ini from C:\Installer Creator 3.0\include

[External compressor:precomp]
header = 0
unpackcmd = precomp -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

[External compressor:srep,srep64]
header = 0
unpackcmd = srep {options} -d -s - - <stdin> <stdout>

[External compressor:pzlib]
header = 0
unpackcmd = pzlib d - -o - <stdin> <stdout>

[External compressor:lzma2,lz77,xz]
header = 0
unpackcmd = 7z x -txz -an -y -si -so <stdin> <stdout>

[External compressor:zstd]
unpackcmd = zstd -d -f <stdin> <stdout>

please take a look at my arc.ini files and tell me if i did anything wrong there. also i have raw2hif.dll hif2raw.dll pzlib.exe and cls-reflate.exe in my IC 3100 include folder. i know i could use CIU instead but i like the style of this app and its similar in design just a bit older. let me know if i need to change anything in either of the arc.ini files.. thanks!

-Kitsune

[Andu21]

[External compressor: pzlibv2]
header = 0
packcmd = pZLib v2\pZLib e -m3 -r6 {options} -c32m -t25p $$arcdatafile$$.tmp -o - <stdout>

Try the above, without these before the data file: - -o -

[Simorq]

hi guys, thanks for helping me with my pzlib question. i am still having issues so i thought i would post my arc.ini info here to further assist you all in helping me resolve my issue.

-Kitsune

IC 3020 UA2801R3.7z (http://www95.zippyshare.com/v/coebBc6S/file.html)

Use the UltraARC 2801 R3 Hotfix.

ZTool is better than pZLib 3.