FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   pZLib v3 (Final release) (https://fileforums.com/showthread.php?t=98672)

vollachr 24-02-2018 04:04

Quote:

Originally Posted by doofoo24 (Post 468336)
use Ztool...

Already tried, didn't give the desired compression ratio, so not an option in this specific case.

I've now changed my arc.ini a little and trying pZLib3 again

doofoo24 24-02-2018 04:27

did you add hif2raw_dll.dll & raw2hif_dll.dll to ztool ?
game like crysis 1 need hif2raw_dll.dll & raw2hif_dll.dll, setting r9

KaktoR 24-02-2018 06:13

Have you changed decompress arc.ini too?

vollachr 25-02-2018 00:31

Quote:

Originally Posted by doofoo24 (Post 468340)
did you add hif2raw_dll.dll & raw2hif_dll.dll to ztool ?
game like crysis 1 need hif2raw_dll.dll & raw2hif_dll.dll, setting r9

I'm compressing Assetto Corsa, Not Crysis, but yes, these dll files are in the same folder as ztool

Quote:

Originally Posted by KaktoR (Post 468344)
Have you changed decompress arc.ini too?

My CIUv2 arc.ini for pZLib3:

Code:

[External compressor:pzlib3,pZLib3,pZLib32,pzlib32,pZlib32,pzLib32,pZLib64,pzlib64,pZlib64,pzLib64]
header = 0
unpackcmd = pZLib d {options} -t100p - -o - <stdin> <stdout>

and my current compression arc.ini for pZLib3:

Code:

[External compressor:pZLib3]
header = 0
packcmd = pZLib e {options} -m2 -t25p -c128m - -o - $$arcdatafile$$.tmp -o - $$arcpackedfile$$.tmp
;packcmd = pZLib e {options} -t100p - -o - $$arcdatafile$$.tmp -o - <stdout>
unpackcmd = pZLib d {options} -t75p - -o - <stdin> <stdout>

UPDATE:

OK, as for the time of the original compression that gave me the CRC error the compression arc.ini was this:

Code:

[External compressor:pZLib3]
header = 0
packcmd = pZLib e {options} -m2 -t25p -c128m - -o - $$arcdatafile$$.tmp -o - <stdout>
;packcmd = pZLib e {options} -t100p - -o - $$arcdatafile$$.tmp -o - <stdout>
unpackcmd = pZLib d {options} -t75p - -o - <stdin> <stdout>

I have now changed the pzlib3 arc.ini entry for CIUv205g to be this:

Code:

[External compressor:pzlib3,pZLib3,pZLib32,pzlib32,pZlib32,pzLib32,pZLib64,pzlib64,pZlib64,pzLib64]
header = 0
;unpackcmd = pZLib d {options} -t100p - -o - <stdin> <stdout>
unpackcmd = pZLib d {options} -t25p - -o - $$arcdatafile$$.tmp <stdout>

So far, 4 minutes into setup & no CRC Checksum error, it was practically immediate before.

The only issue now is that the elapsed time keeps progressing while the time remaining stays 0 and the progress bar stays at 0.0% .

But I'll give it some time, could be because of the extreme compression ratio (40GB game to 3.80GB archive)

UPDATE 2:

Around 10 minutes in, I got the following ISDone error:

Quote:

An error occurred when unpacking!
Unarc.dll returned an error code: -1
ERROR: archive data corrupted (decompression fails)

doofoo24 25-02-2018 01:34

wow extreme compression ratio (40GB game to 3.80GB archive) ???
what setting ???

vollachr 25-02-2018 01:50

Quote:

Originally Posted by doofoo24 (Post 468385)
wow extreme compression ratio (40GB game to 3.80GB archive) ???
what setting ???

These:

Code:

pZLib3:st1+srep64+lzma64:a1:mfbt4:d256m:fb128:mc1000:lc8
With the following arc.ini values:

Code:

[External compressor:pZLib3]
header = 0
packcmd = pZLib e {options} -m2 -t25p -c128m - -o - $$arcdatafile$$.tmp -o - <stdout>
;packcmd = pZLib e {options} -t100p - -o - $$arcdatafile$$.tmp -o - <stdout>
unpackcmd = pZLib d {options} -t75p - -o - <stdin> <stdout>

[External compressor:srep64]
header = 0
packcmd  = srep64 {options} -v0 -m3f -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep64 {options} -d -s - - <stdin> <stdout>

[External compressor:lzma64]
header = 0
packcmd  = lzma64 e lzma{:option} <stdin> <stdout>
unpackcmd = lzma64 d lzma{:option} <stdin> <stdout>

But as I mentioned, so far I fail to unpack it with CIUv205g.

I'm trying to perform a test on the archive directly with FreeArc UI program to see if the archive is actually corrupt or not, this will at least give me an indication where my problem is.

doofoo24 25-02-2018 01:53

ok, i am tring to compress the game now with ztool with pzlib:m3:r9:c256m i will report back...

vollachr 25-02-2018 01:59

Quote:

Originally Posted by doofoo24 (Post 468388)
ok, i am tring to compress the game now with ztool with pzlib:m3:r9:c256m i will report back...

I did a lot of attempts with ztool pzlib, never got below 8.99GB final size (2xDVD5+1x300MB+ Disc) for Assetto Corsa.

I even used the ultra preset (pzlib:ultra), further more, because of the game size, it requires a lot of free space or it just end with errors during compression (that's true for both ztool & pZLib3)

doofoo24 25-02-2018 02:08

after ztool i get 42618mb not alot of zlib detect...
as for pzlib i didn't use it for long time last time i tried to use it i got a lot of crc error's.
try to disable superfetch in windows or use safe mode...

vollachr 25-02-2018 02:15

Quote:

Originally Posted by doofoo24 (Post 468392)
after ztool i get 42618mb not alot of zlib detect...
as for pzlib i didn't use it for long time last time i tried to use it i got a lot of crc error's.
try to disable superfetch in windows or use safe mode...

OK...

Anyway, so far, I keep getting when performing test on the archive with FreeArc GUI an error unknown method or parameters in pZLib3:st1

And I did edit the arc.ini in the FreeArc installation folder and added the relevant exe & dll files, trying one more time with a slight change in arc.ini but at the same time trying to re-do my compression without the st1 parameter for pzlib3 and with lzma instead of lzma64.

Hopefully it'll give the same archive result with less decompression problems.

doofoo24 25-02-2018 02:24

what size you get after pzlib before srep ?

vollachr 25-02-2018 02:28

Quote:

Originally Posted by doofoo24 (Post 468395)
what size you get after pzlib before srep ?

I'm not sure I remember correctly, but I think it was around 24GB with pzlib3, really don't remember about ztool's pzlib, why?

doofoo24 25-02-2018 02:32

not pzlib that in ztool i meant pzlib v3 seem that ztool does not detected zlib, now i am tring to compress with pzlib v3 instead of ztool and i will try ztool without hif2raw_dll.dll & raw2hif_dll.dll...

vollachr 25-02-2018 02:39

Quote:

Originally Posted by doofoo24 (Post 468397)
not pzlib that in ztool i meant pzlib v3 seem that ztool does not detected zlib, now i am tring to compress with pzlib v3 instead of ztool and i will try ztool without hif2raw_dll.dll & raw2hif_dll.dll...

OK, Now, I remember that at one point during compression the tempfile was around 11GB too, just not sure if it was before or after srep.

Anyway, let me know what you get.

Also, I'm currently trying the setup again after replacing the pzlib.exe from CIUv205g with the one I used for compression, it's been almost 16 minutes without errors so far.

UPDATE: Nope, same problems, just took longer to happen

KaktoR 25-02-2018 02:51

There are some games/files, where pzlib3 or ztool have errors from the beginning in compressing. A good and new example is Kingdom Come Deliverance, where some files have some kind of false detection with pzlib, result is either crc error or archive data corrupted in extracting.

Same was a single file for NBA2k18 i guess, where the ratio was over 2000% inflation.

I don't know if it is a generall problem with pzlib or just switches.

vollachr 25-02-2018 02:58

Quote:

Originally Posted by KaktoR (Post 468402)
There are some games/files, where pzlib3 or ztool have errors from the beginning in compressing. A good and new example is Kingdom Come Deliverance, where some files have some kind of false detection with pzlib, result is either crc error or archive data corrupted in extracting.

Same was a single file for NBA2k18 i guess, where the ratio was over 2000% inflation.

Hopefully, that's not the issue here, because that will be really annoying, I'm trying something now, hopefully it will work, if not, well, I guess it'll back to the drawing board.

To be honest, I didn't try to compress it to 1xDVD5 size, that was a shock when I've seen the result, I'll be happy with getting it to work with a size of 2xDVD5 without the extra 300MB+ 3rd disc.

Problem is, so far, only pzlib3 managed to compress the game below 8.99GB size (which result in a 3rd, nearly empty disc)

doofoo24 25-02-2018 03:33

pzlibv3 and ztool didn't work for me (i mean it did not detect zlib) not decompression error...
i tried to scan different files type (kn5 / ai ) with gfs from razor zlib with levels(slow) no zlib file with dynamic streams option it detect zlib...

doofoo24 25-02-2018 03:53

@vollachr by any chance did you used msc ?
there almost 9.7gb of dds files...

vollachr 25-02-2018 03:58

Quote:

Originally Posted by Simorq (Post 468406)
pZLib2 / pZLib3.0.1 / pZLib3.2.0 / ZTool

I only encountered some problems with pZLib3.2.0.
They all work great.

I'll try and see what works and what not, currently using pZLib3.2.0, will try 3.0.1 maybe.

vollachr 25-02-2018 04:01

Quote:

Originally Posted by doofoo24 (Post 468407)
@vollachr by any chance did you used msc ?
there almost 9.7gb of dds files...

No, I mean, I tried but I got crashes from MSC, so I gave up on it, other than srep/srep64 & lzma/lzma64 I only tried ztool, pzlib3.2.0 and precomp0.4.3 (which I abandoned mid-way due to way too long times)

doofoo24 25-02-2018 04:32

i tried ztool and both pzlib 3.2 and 3.0.1 i get the same ratio 105.36%...
only applied pzlib 42,935,540,198 => 45,238,050,951 bytes.
what ratio you get ???
now i am tring precompmt...

vollachr 25-02-2018 04:43

Quote:

Originally Posted by doofoo24 (Post 468410)
i tried ztool and both pzlib 3.2 and 3.0.1 i get the same ratio 105.36%...
only applied pzlib 42,935,540,198 => 45,238,050,951 bytes.
what ratio you get ???

That makes no sense...

When I used pzlib 3.2 my ratio at end of pzlib3 part of compression was under 50%, don't remember the exact number but it was in the 40-50% area.

I'm currently testing compression with pzlib3.0.1+srep64+lzma, will report back results when I have them, gonna take a few hours.

doofoo24 25-02-2018 05:13

precompmt.dll used with precomp 0.46
i get 42,935,540,198 => 43,687,319,971 bytes. Ratio 101.75%
worst than ztool /pzlibv3

doofoo24 25-02-2018 05:18

[QUOTE=vollachr;468409]No, I mean, I tried but I got crashes from MSC, so I gave up on it...
during compression or deccompression it crashes ?

vollachr 25-02-2018 05:26

Quote:

Originally Posted by doofoo24 (Post 468415)
Quote:

Originally Posted by vollachr (Post 468409)
No, I mean, I tried but I got crashes from MSC, so I gave up on it...

during compression or deccompression it crashes ?

During compression, each time it got to the MSC part, MSC crashed, could never figure out why.

doofoo24 25-02-2018 05:36

it seems that only psd have zlib...
all psd 2.72gb

vollachr 25-02-2018 06:24

Quote:

Originally Posted by doofoo24 (Post 468417)
it seems that only psd have zlib...
all psd 2.72gb

Really? OK...

Anyway, here's the weird thing though, pZLib 3.0.1, unlike pZLib 3.2, so far at least (still far from finished), give me a much worse ratio (97-100%), that said, it's not even half way through the process, so I'll wait and see.

Truth be told, I only need it to compress additional 400MB or so compared to srep64+lzma without pZLib3 and It'll be enough.

Maybe I'll re-try MSC later on.

UPDATE: Okay, pZLib 3.0.1 part of compression finished, final ratio: 106%, I'll wait and see what the final result will be.

doofoo24 25-02-2018 07:10

i'm tring msc on dds file so far no crash...

vollachr 25-02-2018 07:16

Quote:

Originally Posted by doofoo24 (Post 468421)
i'm tring msc on dds file so far no crash...

Could be something with my arc.ini entry for MSC, or it's placement within the method, when I tried it it was pzlib(ztool)+msc+srep64+lzma

my ini for msc:

Code:

[External compressor:msc]
header = 0
packcmd = msc c -v {options} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp


doofoo24 25-02-2018 07:18

[External compressor:msc]
;Alt: -f = Forced on bigger data //-frog=9 use this instead of -tak=9 = yields error on decomp.
header = 0
packcmd = msc.exe c -v -wav=1 -raw=1 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=1 -bmf=9s -tak=9 -dxt=1 -lzma=hc4,lc8,lp2,pb2,fb273,mc1000 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

usually i compress filles separately zlib / media / etc...
on small game i use msc before pzlib
msc+pzlib+srep+lzma

vollachr 25-02-2018 07:23

Quote:

Originally Posted by doofoo24 (Post 468423)
[External compressor:msc]
;Alt: -f = Forced on bigger data //-frog=9 use this instead of -tak=9 = yields error on decomp.
header = 0
packcmd = msc.exe c -v -wav=1 -raw=1 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=1 -bmf=9s -tak=9 -dxt=1 -lzma=hc4,lc8,lp2,pb2,fb273,mc1000 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

usually i compress filles separately zlib / media / etc...
on small game i use msc before pzlib
msc+pzlib+srep+lzma

Well, I'm using CLS-diskspan to create the disc size split so separately is not an option but I'll try your MSC arc.ini entry, maybe it will work.

Thanks.

doofoo24 25-02-2018 07:47

ok msc done on dds
Compressed 18,571 files, 11,173,425,105 => 8,818,080,553 bytes. Ratio 78.92%
test decompression all file crc ok...
well all that left is to use msc+pzlib+srep+lolz to get the best result when it done i will update...
so the final size msc+pzlib+srep+lolz it worst without msc ???

CANNIBAL 25-02-2018 09:36

How About U Add "-v" To Pzlib3

vollachr 26-02-2018 06:14

Quote:

Originally Posted by doofoo24 (Post 468426)
ok msc done on dds
Compressed 18,571 files, 11,173,425,105 => 8,818,080,553 bytes. Ratio 78.92%
test decompression all file crc ok...
well all that left is to use msc+pzlib+srep+lolz to get the best result when it done i will update...
so the final size msc+pzlib+srep+lolz it worst without msc ???

Any news?

From some reason, whenever I try msc+pzlib+srep64+lzma64 the compression just ends after MSC, without any errors but without completing the compression and writing the data file so I end up with a final disc folder that only have the Setup.exe, Setup.ico & Autorun.inf and is less than 60MB in size (which is the Setup.exe size)

I tried it with different pzlib versions and settings, it keeps happening.

doofoo24 26-02-2018 06:56

the ratio seem to be worst with MSC+pzlib+srep+lolz. i get 9.8gb ???
other pople who repack Assetto Corsa get 6.8gb...
i don't know what setting.
i am out of idea right now ???

vollachr 26-02-2018 07:05

Quote:

Originally Posted by doofoo24 (Post 468455)
the ratio seem to be worst with MSC+pzlib+srep+lolz. i get 9.8gb ???
other pople who repack Assetto Corsa get 6.8gb...
i don't know what setting.
i am out of idea right now ???

6.8GB could be with not fully updated game and/or not all DLCs, I honestly haven't seen a repack for v1.16 and all DLCs that is less than 9.0GB.

The 6.8GB repacks seems to be for v1.14.

I'm gonna do another 1-2 tries, if nothing works I'll just settle for either 2xDVD5+1xCD or maybe 1xDVD9+1xDVD5 (or both) using Srep64+Lzma64 only,

Because so far, the differences between all attempts were minor and insignificant, and the Srep64+Lzma64 was the quickest one.

Simorq 26-02-2018 10:20

Quote:

Originally Posted by doofoo24 (Post 468455)
the ratio seem to be worst with MSC+pzlib+srep+lolz. i get 9.8gb ???
other pople who repack Assetto Corsa get 6.8gb...
i don't know what setting.
i am out of idea right now ???

MSC=LOLZ:D
pzlib+srep+lolz

doofoo24 26-02-2018 21:55

Compressed 33,999 files, 42,886,060,604 => 8,475,378,350 bytes. Ratio 19.76%

vollachr 27-02-2018 00:45

Quote:

Originally Posted by doofoo24 (Post 468472)
Compressed 33,999 files, 42,886,060,604 => 8,475,378,350 bytes. Ratio 19.76%
pzlib:max:c128m+srep:m5f:a2+delta+lolz:mc1023:fba4 096:bc8:blr8:bm8:oh14:pc4:mtt1:mt2:d256m

Nice, I'll try, although LOLZ is currently giving me some on-screen text output problems due to >nul at end of the compression command, I'm working on a workaround but I'm not sure it will work yet.

Could there be an alternative to LOLZ?, what about RZM or ZST? have you tried LZMA or LZMA2? instead of lolz? if so, what was the result?

doofoo24 27-02-2018 01:08

no i didn't try lzma or zst, i believe the ratio would be worst maybe you will get 8.10g to 8.15g...
from past experience the best setting for lzma i recommend lzma:a1:mfbt4:d512m:fb273:mc10000:lc8 or if you have alot of ram you could increase d512m to 756m or 1024mb still better than lzma2 or lzma:ultra that used by panker...
just be sure that you using fazip 64 for lzma to overcome memory limit in freearc.
the max lzma is d182mb in freearc alone without fazip...

[External compressor:lzma,delta,exe,mm,rep,tor,4x4]
header = 0
packcmd = fazip -i0 compress:{compressor}{:option} <stdin> <stdout>

* srep need less tan 1gb decompresion memory for Assetto...
i will try to use pzlib:max:c64m+srep:m5f:a2+delta+lzma:a1:mfbt4:d51 2m:fb273:mc10000:lc8
no need for more than 64m for pzlib...


All times are GMT -7. The time now is 03:38.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com