FileForums - pZLib v3 (Final release)

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - pZLib v3 (Final release) (https://fileforums.com/showthread.php?t=98672)

pZLib v3 (Final release)

Here's a Christmas gift people, I wonder what vultures have for me though.

Anyways nothing more I can do to this project, it reached its limits in terms of optimisation and offering more speed while producing more or less the same output as precomp, reflate, antiz and etc.

Project was started in July 17, 2016, finished today December 21, 2016. No longer working on this anymore, no more potential updates because everything works, haven't tested on DOOM but it should work to roughly give you 25-27GB output as final size if you compressed it right while giving good speed. pZLib v2 bugs have been fixed as well, tested on PES2017, Mafia 3 and some other games that gave it troubles.

Deal still stands, don't use program if you aren't going to give credits, took 5 months and 4 days to make this.

You can place libraries near pzlib to use reflating functions to maximum precompression.

Use '-x' to get the best ratio, that's the ultra setting, use '-r#' to set level default value is 0 which is set by program for best output, 1-9 to set manually.

thank you
beautiful, as always

At Christmas it is all better people, and gifts will make even those who do not deserve it, great Razor ...:)

Quick Test ...

Code:

-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 08:16:12

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        pzlib_v3         981,1°MB           2,08°GB       212,01%               1°File     00:01:14:45   e -m2 -st50p -t8 -ct8 -c32m -b512k -v -o 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 08:20:00

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        pzlib_v2         981,1°MB           2,07°GB       210,99%               1°File     00:01:12:93   e -m2 -st50p -t8 -ct8 -c32m -b512k -v -o 

=============================================================================================================================================

Is perfect ...;)

can you do me a favour, do a comparison between precomp045, reflate by shelwien and antiz, use 1 thread. I want to see difference.

can u add this precomp too, compiled from latest commit
only uses 2 temp files per stream and lzma and xz compression in-built and also some other performance related tweaks

Code:

-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 10:51:41

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



         reflate         122,1°MB          240,4°MB       196,89%               1°File    00:01:58:992                  -process -c6 -t1  

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 10:52:40

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



           antiz         122,1°MB          241,0°MB       197,38%               1°File    00:00:18:920               --brute-window -i -o 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 10:55:59

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



      precomp045         122,1°MB          241,4°MB       197,71%               1°File    00:01:56:911                   -intense -cn -v  

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 10:57:51

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        pzlib_v3         122,1°MB          240,9°MB       197,30%               1°File    00:00:09:943   e -m2 -st50p -t1 -ct1 -c32m -b128k -v -o 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 11:00:04

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        pzlib_v3         122,1°MB          240,9°MB       197,30%               1°File     00:00:10:49   e -m2 -st50p -t1 -ct1 -c16m -b64k -v -o 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.6.9c ] -↓- Compressed Archive Completed At -↓- 21/12/2016 11:06:43

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        pzlib_v3         122,1°MB          240,9°MB       197,30%               1°File     00:00:10:54   e -m1 -st50p -t1 -ct1 -c16m -b64k -v -o 

=============================================================================================================================================

if you want to test with the FA support let me know ....

Ok first of all thanks Razor.
Now clear me some things... Where I wrong clear me.
First I used bf3 cas_10.cas file I used this with Freearc method used -m2 -f -t8 then I Begin
Input was 411mb output 714 mb took 47.53 second all done good
When I unpack it it's not even moving from zero℅ ok closed it
I go to arc.ini removed-f (I thought it may be an CRC error)
Again same output but extraction failure again(not actually failed it didn't moved to a bit after 7min)
Used in other game files too compression goes well decompression stucks.
Why?
Now reflate (used x64)
I put x64 version libraries (forgot to put -x in arc.ini)
Input 411 mb output 720 mb took 49 second ok good
But when I remembered and put -x in arc.ini it gives output less 411 mb input 714 mb output .
Why??

Quote:

Originally Posted by felice2011 (Post 454667)

At Christmas it is all better people, and gifts will make even those who do not deserve it, great Razor ...:)

A saying that I heard recently: heroes are people who died in vain for people who lived in vain. :)

you used -f with m2, you increased the chances of getting crc, -f breaks crcs sometimes but its mostly broken when you use m2 than m1, so remove -f

Thanks for reply
I completely removed -f and precompressed the file again and then extraction stucks again .
Only using -m2 -t8
I changed things many time. And then I used version x86 and I got an arc crash.
And about reflate why output changes if I put-x in arc.ini
If I do not put it than ratio is more good.???

i don't understand when you say, extration is stuck, try encoding and decoding without freearc.

I mean after precompression went successfull I decode/unpack it it stucks/hangs

I used without freearc same happens precompression went successfull decoding stucks/hangs
Status : restoration data
Data: 0.25MB > 0.00MB <0℅>
Speed: 0.00mb/s <read> 0.00mb/s <write>
Two minutes went nothing proceeds :(

that bug must go to hell, ain't got time for it right now.

Many thanks, mate, for this Christmas gift, you're awesome! Will continue to credit you in my future repacks, as I already do with PZlib2.

big thanks! I can't deal with unpacking, 0% and all hang :(

Quote:

Originally Posted by Joe Forster/STA (Post 454673)

A saying that I heard recently: heroes are people who died in vain for people who lived in vain. :)

The saying is interpreted by those people who throughout history have tried to be heroes but they were not able to be, the greatest heroes in the history still have a name in the most famous books, your thinking, like your sentence was written by those people who have never been able to distinguish themselves in the crowd, or to incise your name in the memories of mankind.;)

Many thanks my dear brother razor for this

how to unpack? hangs for 0%! help

Bro, How to Unpack Bin files

Yes :o ERROR: CRC failed .... File is broken with FA support :(

thnx razor but may i ask you why you release it by the version 3.7 why that specific number ? Why you always release your tools not according to the ordinary order !!

felice2011

In what does these parameters help, to understanding

-st50p -ct8 -c32m -b512k

Quote:

Originally Posted by LuzhinE (Post 454691)

felice2011

In what does these parameters help, to understanding

-st50p -ct8 -c32m -b512k

-st50p

Quote:

Originally Posted by Razor12911 (Post 454327)

I already see -t60p set, in order for that to be realistic, you must have a CPU with 25 threads, pzlib will use 15 threads which 60%. If you got 8 threads, pzlib will override your 60% order and opt for 50% because you literary can't use 4 and 8-tenth threads so 8 threads * 60% = 4.8, when this value is rounded off, it's 5. therefore 5/8 = 63%.

-st#: number of scanning threads

-ct8

-ct#: number of precompressing threads

-c32m

-c#: chunk size (default 16m)

-b512k

-b#: buffer size (default 256k)

If you'd used my "Cmd_Info" program you can have the detailed information directly in the info window.;)

Last thing, when it relates to the chunk size, happened several times that I have had requested in private on the issue of progress in block and not continues of the progress bar during installation through Inno-setup, with compressors that use the chunk size option, you should use a low memory block, or do not overdo in the order of 16m 32, 64 or 128m, so as to obtain during decompression a minor block for writing of the files.:)

Razor12911 thanx!!!

But help pleace guys, same as user Mini, not unpacked "hangs for 0%"

use
[External compressor:pzlib]
header = 0
packcmd = pZLib e -m2 -c256m -t8 - -o - <stdin> <stdout>
unpackcmd = pZLib d -t8 - -o - <stdin> <stdout>

before version working normal on this parameters

may be need another parameters?

-st50p -ct8

Uses 50% CPU to compress no?

This -t8 parameter uses the entire cpu to compress

or am I wrong?

Hotfix uploaded

Take note, no idea if the decompression bug is fixed because that's not what I experience from my end so can someone test this.

game files - 320 mb
after pzlib+srep+lzma - 258 mb

unpacking:
Extracting 1 file, 336,093,110 bytes. Processed 100%

game files - 1.5 gb also continues to increase, unpacking the file over and over again

hotfix didn't help, unfortunately

Quote:

Originally Posted by Razor12911 (Post 454697)

Hotfix uploaded

Take note, no idea if the decompression bug is fixed because that's not what I experience from my end so can someone test this.

OK ... successful decompression:D

Mini What game are you trying to accomplish

DOOM. mp_gameresources_003.patch

You are trying it with the fix?

Hey Guys don't forget the 4 days XD Man Many thnx , Hope you'll make a TUTs in the next few centuries

Try with the normal version

Now Battlefield 3 already works with the fix

Ok so it works and it doesn't work?

Quote:

Originally Posted by Razor12911 (Post 454709)

Ok so it works and it doesn't work?

For me initial V3 gave error (forever unpacking, actually), hotfix works. On tests sets, will test later on big files.

Thanks again and for hotfix ofc!

Cool

Here are some tips guys:

-The increment of -c# does give better output, but uses more memory, pzlib will reach 40GB ram usage if you set it up like an idiot, 16MB is the default value, there are very less inputs that will need a higher value like for example, Mad Max will need something like 256MB-320MB, GTAV with 128MB is fine, and DOOM needs something like 600-700MB to catch the most biggest stream I've ever seen.

-If you still want to get all the streams that a game has but you want to lower down the RAM usage as I explained with "-c#" then you have to adjust -st# scanning threads assigns the number of threads to scan data with, more threads = more speed = more ram usage, size is maintained whichever number you set here but speed and memory usage isn't.

-the fast verification option, this does break CRC sometimes, it quickly determines ways of precompressing data, it has a 95% accuracy, the other 5% is when it breaks CRC, you should use this when you use -m1 as method, with m2, accuracy goes down to 70%.

-decompression memory, well pzlib is actually not memory hungry, it all depends on what settings you used when encoding so, you might wonder if pzlib used 12GB memory when encoding, how much memory is it going to need for decompression. Well this is how you calculate, you must find out what ratio was the data inflated to, let's say GTAV for example, you used -c128mb and pzlib ended up using 8GB memory for encoding, pzlib encoded 47.6GB and made it become 95.7GB for example, the ratio is 201%. to calculate it's chunk size (-c128MB) multiplied by 2.01 which is 256MB, that's roughly the memory which will be used, things like IO buffer and processing memory will make it roughly 300-350MB as a result.

-this last version utilises raw2hif_dll.dll and hif2raw_dll.dll, these are reflate functions, the moment you place them near pzlib.exe, pzlib utilizes them and switches off internal functions, I ran many benchmarks, the usage of these libraries make pzlib slower than when it's using internal functions, however, with DOOM, you really don't have a choice but to use them because internal function cannot process the streams that game has, this is why precomp and pzlib v2 never works on that game, however I blended the internal functions with reflate to give the same size when reflate is used alone but more speed, meaning the best method to use when using pzlib is pzlib -m2 -x...., "-x" gives a better output than pzlib alone.

-for DOOM and some games that pzlib fails to process, remember I created ZlibChecker, you can first use this program to see what size should you really get and pzlib must give that output. for example, zlibchecker might report that 20GB became 37GB, but when you use pzlib, pzlib gives 21GB as inflated size, actually this is what happens when you use pzlib on DOOM, then you have to use "-s" for those special streams because clearly the game has special deflate streams. For DOOM, the best method you can use is -m2 -c640m -x -s -r9....

Thanks Razor Bro For hotfix ofc!
decompression works great bro

The normal version never decompresses.

Tested with x64 fix

packcmd = pZLib e -m2 -t8 - -o - <stdin> <stdout>

http://i.imgur.com/WxAu7Vq.jpg

packcmd = pZLib e -m1 -t8 - -o - <stdin> <stdout>

http://i.imgur.com/aFnNVlC.jpg

packcmd = pZLib e -m1 -t8 -f - -o - <stdin> <stdout>

http://i.imgur.com/CTSuJtX.jpg