Best Compression Methods for 'Specific' Games. Q&A

[ZakirAhmad]

Originally Posted by Simorq

Kingdom Come Deliverance

Original Size: 28.9 GB
Compressed Size: 22.7 GB 
Compression Time: 7 - 8 Hours (2 Cores + 12GB RAM + HDD) (DiskSpan)
Decompression Time: 24 Minutes (2 Cores + HDD)
Method: pzlib:max:t100p+srep:m3f+4x4:b100mb:lzma:a1:mfbt4:d200m:fb273:mc10000:lc8

Nioh Complete Edition

Original Size: 73.8 GB
Compressed Size: 22 GB 
Compression Time: 2 - 3 Hours (2 Cores + 12GB RAM + HDD) (DiskSpan)
Decompression Time: 15 Minutes (2 Cores + HDD)
Method: srep:m3f+4x4:b100mb:lzma:a1:mfbt4:d200m:fb273:mc10000:lc8

Watch Dogs 2 Gold Edition

Original Size: 25.1 GB
Compressed Size: 20.2 GB 
Compression Time: 70 Minutes (2 Cores + 12GB RAM + HDD) (DiskSpan)
Decompression Time: 8 Minutes (2 Cores + HDD)
Method: srep:m3f:l128+lzma2

Need for Speed Payback

Original Size: 27.0 GB
Compressed Size: 15.6 GB 
Compression Time: 102 Minutes (2 Cores + 12GB RAM + HDD) (DiskSpan)
Decompression Time: 6 Minutes (2 Cores + HDD)
Method: srep:m3f:l128+lzma2

lzma2 is best choice than lzma, u can use single thread while compressing (which increases ratio ) but u will get same decompression speed (around 50mbps on my system) which u get in multithreading.

[KaktoR]

Originally Posted by Simorq

Kingdom Come Deliverance

Original Size: 28.9 GB
Compressed Size: 22.7 GB 
Compression Time: 7 - 8 Hours (2 Cores + 12GB RAM + HDD) (DiskSpan)
Decompression Time: 24 Minutes (2 Cores + HDD)
Method: pzlib:max:t100p+srep:m3f+4x4:b100mb:lzma:a1:mfbt4:d200m:fb273:mc10000:lc8

These 3 files give me CRC error with this method (i have tested a few other options with pzlib and pzlib3, all the same)

+ the same with all files in "\Data\_fastload" and "\Engine"

[Simorq]

Originally Posted by KaktoR

These 3 files give me CRC error with this method (i have tested a few other options with pzlib and pzlib3, all the same)

+ the same with all files in "\Data\_fastload" and "\Engine"

GOG Version?

[KaktoR]

^
I can do it for *.blp files, but not for entire game (takes ages).

What switches do you recommend?

Will do it tomorrow and report back then.

[doofoo24]

using srep with setting srep:m2f:l128:a1 reduce alot of memory decompression with minimum compression ratio hit as long used with lolz, the setting that i used is lolz:mc1023:fba4096:bc8:blr8:bm8:oh14:pc4:mtt1:d12 8m:mt8 (change mt depend on cpu core)...
i test mad max pzlib:max:c256m+srep:m2f:l128:a1+lolz:mc1023:fba40 96:bc8:blr8:bm8:oh14:pc4:mtt1:mt8:d128m.
i get 34,354,331,688 => 4,266,484,514 bytes. Ratio 12.42%

as in the picture lolz need 261mb per thread no need for more than 2 thread to use with pzlib becase pzlib is the bottleneck...

*srep with m3f need 21g of ram for decompression with m2f:l128:a1 it reduce to 2696mb...
the difference ratio in the end between m3f and m2f 250mb but a lot less memory usage in decompresion and much faster...
total installation time 6 min

[ZakirAhmad]

Originally Posted by doofoo24

using srep with setting srep:m2f:l128:a1 reduce alot of memory decompression with minimum compression ratio hit as long used with lolz, the setting that i used is lolz:mc1023:fba4096:bc8:blr8:bm8h14c4:mtt1:d12 8m:mt8 (change mt depend on cpu core)...
i test mad max pzlib:max:c256m+srep:m2f:l128:a1+lolz:mc1023:fba40 96:bc8:blr8:bm8h14c4:mtt1:mt8:d128m.
i get 34,354,331,688 => 4,266,484,514 bytes. Ratio 12.42%

*srep with m3f need 21g of ram for decompression with m2f:l128:a1 it reduce to 2696mb...
the difference ratio in the end between m3f and m2f 250mb but a lot less memory usage in decompresion and much faster...
total installation time 6 min

can u please post ur srep which u used for compression.
thanks.

[doofoo24]

srep 3.93a beta srep:m2f:l128:a1 but you need about 18600mb of ram to compress if you have 16gb use l256

[ZakirAhmad]

Originally Posted by doofoo24

srep 3.93a beta srep:m2f:l128:a1 but you need about 18600mb of ram to compress if you have 16gb use l256

Thanks .
can u tell me how to launch compressors from setup. like u posted in above screen shot of setup in task manager.

[doofoo24]

is like m3f:l512 try m2f:l512:a1 it shoud be less than 8gb but the ratio you may need to increase lolz to d256m but make sure to be mt2 if you don't have enough ram i used lolz wit d128:mt8 it fast but need around 12gb or ram in compression...

[ZakirAhmad]

Originally Posted by doofoo24

using srep with setting srep:m2f:l128:a1 reduce alot of memory decompression with minimum compression ratio hit as long used with lolz, the setting that i used is lolz:mc1023:fba4096:bc8:blr8:bm8:oh14:pc4:mtt1:d12 8m:mt8 (change mt depend on cpu core)...
i test mad max pzlib:max:c256m+srep:m2f:l128:a1+lolz:mc1023:fba40 96:bc8:blr8:bm8:oh14:pc4:mtt1:mt8:d128m.
i get 34,354,331,688 => 4,266,484,514 bytes. Ratio 12.42%

as in the picture lolz need 261mb per thread no need for more than 2 thread to use with pzlib becase pzlib is the bottleneck...

*srep with m3f need 21g of ram for decompression with m2f:l128:a1 it reduce to 2696mb...
the difference ratio in the end between m3f and m2f 250mb but a lot less memory usage in decompresion and much faster...
total installation time 6 min

how did u make compressors like ztool lolz group with setup in above taskmanager screenshot.

[ZakirAhmad]

Compressed Mad Max Fury Road with ztool srep lolz to 3.90 from 32gb.<br>
The srep dictionary size is only 2200mb. I used settings srep:m2f:t1:l512:c512.<br>
srep compressed it to 9.9gb then lolz to 3.90gb. <br>
The only problem with doofoo settings is that he used multithreaded srep because srep with m1 and m2 uses multithreading hence reduction in co.pression ration. He also used l32 and c 32 whre as i used l512 and c 512 because my ram is only 4gb otherwise i might have gained may be better than this. <br>
The whole point is to limit creation of temp file which interupts fast installation.<br>
<br>
How can we achieve fast installation?<br>
The things i recommend is <br>
1. If useing prexompressor with multithreaded support, do not set processor useage to 100% because their r other tools like lolz srep etc in compression chain which also use cpu. If u set precompressor(1st tool in chain) usage to 100%,, it will restrict other compressors like srep and lolz to use cpu in parallel and hence the compressors will run one after another not parallel.<br>
#2. If a game is heavely compressed with srep like in mad max use m2f setting instead of m3f or m5f. It will in most cases restrict decompression dectionary to managable size.<br>
#3 useing multithreading for lolz when using precompressor is of no use in most cases i guess because if precompressor cant process much, it will automatically restrict lolz data processing or lazma etc.<br>
<br>
#4 use latest srep by profrager. And set memory usage in cls as 100%- 1250m. Why 1250m?, i observed if ur win 10 pc is idle, it uses only (1250mb ) of ram (at least in my case), what if we use 100% memory usage or 100% - 256m? <br>
I would not recommand doing this. U known in win 10 their r two ways to free RAM when it senses RAM is full, one is to use page file and another is to use compression and then page file. So when u set ram usage to 100% and if srep needs ur 100% of ram, it will first fill ur 100% ram and then will create virtual temp file file. But in prder to iuse 100% ram windows has to move own components to page file and or compress, this will increse instillation time exponentially and will make using pc hell impossible!<br>
However I have read that srep uses memory from free pool but i havent foumd doing it that. It just uses full ram if I do not use it that way.<br>
Also m5f -ia- swotches of srep promises to ise less ram without the cost of speed, so if any one has got fast computer please compress madmax with m5f -ia- switch and report decompression memory isage by srep back here. Use srep 3.93 because it provides decompression ram usage after processing the file.<br>
One more thing, please do press thanks, if u find this helpfull.<br>
<br>
<br>

[doofoo24]

Originally Posted by ZakirAhmad

Compressed Mad Max Fury Road with ztool srep lolz to 3.90 from 32gb.<br>
The srep dictionary size is only 2200mb. I used settings srep:m2f:t1:l512:c512.<br>
srep compressed it to 9.9gb then lolz to 3.90gb. <br>
The only problem with doofoo settings is that he used multithreaded srep because srep with m1 and m2 uses multithreading hence reduction in co.pression ration. He also used l32 and c 32 whre as i used l512 and c 512 because my ram is only 4gb otherwise i might have gained may be better than this. <br>
The whole point is to limit creation of temp file which interupts fast installation.<br>
<br>
How can we achieve fast installation?<br>
The things i recommend is <br>
1. If useing prexompressor with multithreaded support, do not set processor useage to 100% because their r other tools like lolz srep etc in compression chain which also use cpu. If u set precompressor(1st tool in chain) usage to 100%,, it will restrict other compressors like srep and lolz to use cpu in parallel and hence the compressors will run one after another not parallel.<br>
#2. If a game is heavely compressed with srep like in mad max use m2f setting instead of m3f or m5f. It will in most cases restrict decompression dectionary to managable size.<br>
#3 useing multithreading for lolz when using precompressor is of no use in most cases i guess because if precompressor cant process much, it will automatically restrict lolz data processing or lazma etc.<br>
<br>
#4 use latest srep by profrager. And set memory usage in cls as 100%- 1250m. Why 1250m?, i observed if ur win 10 pc is idle, it uses only (1250mb ) of ram (at least in my case), what if we use 100% memory usage or 100% - 256m? <br>
I would not recommand doing this. U known in win 10 their r two ways to free RAM when it senses RAM is full, one is to use page file and another is to use compression and then page file. So when u set ram usage to 100% and if srep needs ur 100% of ram, it will first fill ur 100% ram and then will create virtual temp file file. But in prder to iuse 100% ram windows has to move own components to page file and or compress, this will increse instillation time exponentially and will make using pc hell impossible!<br>
However I have read that srep uses memory from free pool but i havent foumd doing it that. It just uses full ram if I do not use it that way.<br>
Also m5f -ia- swotches of srep promises to ise less ram without the cost of speed, so if any one has got fast computer please compress madmax with m5f -ia- switch and report decompression memory isage by srep back here. Use srep 3.93 because it provides decompression ram usage after processing the file.<br>
One more thing, please do press thanks, if u find this helpfull.<br>
<br>
<br>

*srep 3.93 when you use m3f:l512:c256 always fail in decompression...
also i found that using m3f is the best for final results after applying lzma/lolz better than m5f...
*you will get better results with m5f alone but seems to me that lzma or lolz mange to compress better srep:m3f over m5f...

[Simorq]

Mad Max
pzlib+srep:m3f:l1024+lolz:d1024:al1:mc1023:tt16 (3.95 GB)

[L33THAK0R]

Originally Posted by Simorq

Mad Max
pzlib+srep:m3f:l1024+lolz:d1024:al1:mc1023:tt16 (3.95 GB)

Did you have to decrypt or unpack the archives first? Try as I might, I can't seem to get the blasted thing under 21.8gb.

[Masquerade]

Originally Posted by L33THAK0R

Did you have to decrypt or unpack the archives first? Try as I might, I can't seem to get the blasted thing under 21.8gb.

There's no decryption involved, or unpacking, 100% lossless. Try using a different xtool version (e.g. xtool 2020 with -mzlib+reflate in xtool packcmd)