FileForums - Best Compression Methods for 'Specific' Games. Q&A

Do not use XTool chunk size of greater than 128mb otherwise it will get stuck during compression. (tested on my 64GB, 16 thread PC). XTool v0.12 used. Thanks to Razor12911 for the tools.

Final Size: 37.9GB

Code:

 13:03:18 - Selected ARC/DS method for Data1a-01.bin was: oo2recm8+xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 13:03:18 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

----------------------------------------------------------------------------------------------------------------------------------------

 19:01:31 - Overall input size: 51.51 GB

 19:01:31 - Overall output size: 44.30 GB (Ratio 86.01%)

 19:01:31 - Data files creation time: 05:31:39

Just 200mb for lolz, bad.

Even worse that with xtool applied the size is bigger then my last repack with just oo2recm https://fileforums.com/showpost.php?...postcount=2136

KaktoR

12-11-2021 11:32

They are Billions
v1.1.4

Code:

 19:48:52 - Selected ARC/SPLIT method for Data1a-01.bin was: srep_new+lolz:d64:mtt1:mt12:mc1023/$storepack=FA-MIN/$libpack=srep_new+razor_mtx/$oggretheyarebillions=oggre/$deflatetheyarebillions=xt_zlib+srep_new+lolz:d64:mtt1:mt12:mc1023

----------------------------------------------------------------------------------------------------------------------------------------

 20:30:49 - Overall input size: 5.16 GB

 20:30:49 - Overall output size: 1.84 GB (Ratio 35.70%)

 20:30:49 - Overall conversion time: 00:41:53

KaktoR

12-11-2021 11:59

Battlefield 2042 test

Code:

srep+4x4:lzma

Overall input size: 246.54 MB

Overall output size: 55.90 MB (Ratio 22.67%)



oo2reck+srep+4x4:lzma (oo2reck takes very long)

Overall input size: 246.54 MB

Overall output size: 54.91 MB (Ratio 22.27%)



xtool:kraken+srep+4x4:lzma

Overall input size: 246.54 MB

Overall output size: 54.94 MB (Ratio 22.28%)

KaktoR

12-11-2021 15:39

Skyrim Anniversary Edition test

Code:

Compressed 15 files, 9,955,981,643 => 16,269,399,549 bytes. Ratio 163.41%

Compression time: cpu 11.56 sec/real 320.70 sec = 4%. Speed 31.05 mB/s

All OK



Extracted 15 files, 16,269,399,549 => 9,955,981,643 bytes. Ratio 163.41%

Extraction time: cpu 17.92 sec/real 271.19 sec = 7%. Speed 36.71 mB/s

All OK



Data\Skyrim - Animations.bsa

Data\Skyrim - Interface.bsa

Data\Skyrim - Meshes0.bsa

Data\Skyrim - Meshes1.bsa

Data\Skyrim - Misc.bsa

Data\Skyrim - Shaders.bsa

Data\Skyrim - Textures0.bsa

Data\Skyrim - Textures1.bsa

Data\Skyrim - Textures2.bsa

Data\Skyrim - Textures3.bsa

Data\Skyrim - Textures4.bsa

Data\Skyrim - Textures5.bsa

Data\Skyrim - Textures6.bsa

Data\Skyrim - Textures7.bsa

Data\Skyrim - Textures8.bsa

Working bsa plugin https://fileforums.com/showpost.php?...&postcount=223
Use lz4 v1.8.3 library

PS: t100p stuck for me, so use a lower thread usage for xtool

kj911

14-11-2021 01:13

Microsoft Train Simulator *.ACE files:

older:

Quote:

xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt32:fba4096:mc1023

newest:

Quote:

xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt1:oh14:gm01:lm4:fba4096:mc1023

Green its optional switches, probably testing any game. Defaults, its 00, 11 or 21 worse results than 00.
Red the magic switch! Nice compression gain! And overall usage all physical RAM! (8GB and 87% usage vs. 97-99%)
oh14 its minimally better compression. Defaults: oh12
tt1 its very fast mode, max usage: tt16 or tt32!! Higher values not better and extremely slow compression.

Difference? Old version 2.93GB! New, with tt1 (compress again with tt16 and d160m switches in last time)* 2.80GB!! ~5% compression rate gain.

UPDATE: Aaarrgh! The test package don't decompresseable! CLS-ini error or any tips?

*tt32 its probably very best results. Calculated compression time avg. ~2days! Overkill.

Cesar82

16-11-2021 06:19

Question about LOLZ

Could someone familiar with LOLZ parameters help me with some questions?

How is actual memory usage calculated using for multi thread used An X value for dictionary (-d)?

Considering that the supported values for dictionary are from 0 to 2047 "-d[0..2047]): 64m". If you enter a value above 2047 (such as 2050) will the default value be set to the dictionary (64m) or will the maximum value (2047m) be used?

I need this information because I created presets for LOLZ in DiskSpan GUI.
The memory and thread information will be obtained when starting the DiskSpan GUI and then starting the compression will be calculated based on the threads and RAM of the system.
It will also be informed the number of system theads replaced <thread> by the number of virtual cores (threads) of the current CPU during the conversion.
So one more question arises...
In the lolz documentation it states that the supported values are 1 to 16 thread: "mt threads(-mt[1..16]): 1"
If using a Ryzen processor 9, 5900x is 24 threads, so if you use -mt24, lolz will set it to 1 (default) or to 16 (Maximum supported).

Below is the example where it will be in LOLZ presets in DiskSpan GUI.

Code:

LOLZ_HIGH = lolz:dtb1:d<expr:0.65*memory/(thread+1)/1048576>m:mtt1:mt<thread>:mc1023



Expression example: <expr:0.65*memory/(thread+1)/1048576>



0.65    = percentage of calcule (65%)

memory  = full system memory (RAM)

thread  = number of logical cores

core    = number of Physical cores

1048576 = convert result in bytes to mb

If someone can help me with these questions, I would appreciate it!

kj911

16-11-2021 08:47

Its calculated result correct?? See the linked picture from my friend laptop specifications. --> https://i.kek.sh/zUjUwL6MVZE.jpg

Example: Intel Core I3-1005G1 CPU, Win10 x64 with 8GB memory. (Total cores: 2, Total Threads: 4)

Available physical RAM 8GB --> 8589934592 byte

PreCalculate: 0.65*8589934592 = 5583457484,8
Cores --> 4+1 = 5
Final calculate: 5583457484,8/5/1048576 = 1064.96 MB?? Try it added extra /8 values? 132MB values its really.

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

Cesar82

16-11-2021 15:35

1 Attachment(s)

Quote:

Originally Posted by kj911 (Post 494938)

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

For your system it would result as if using from the command line:

Code:

LOLZ_HIGH = lolz:dtb1:d1065m:mtt1:mt4:mc1023



So lolz would use: 4 * 1065 = 4260 MB of 8192 MB.

I believe the memory usage of lolz is calculated as: theads*dictionary

Example: If using a Ryzen 5900x (24 theads) and 32 GB of RAM we would have:

Code:

LOLZ_HIGH = lolz:dtb1:d852m:mtt1:mt24:mc1023



So lolz would use: 24 * 852 = 20448 MB of 32768 MB.

It may be recommended to do the calculation using available memory rather than total system memory.

Maybe it's not just simply multiplying the dictionary by the number of threads, that's my question.
Can not understand. See the attached video.

its_gnar_again

16-11-2021 18:22

Has anyone here had any luck getting Final Fantasy X/X-2 HD to decompress without any issues? The method on here is basically xtool(xzlib)+srep+lolz, but that ends up with a corrupted archive error during decompression not even half-way. I have tried the following:

Tried different configs for LOLZ
Tried different configs for SREP
Tried outright replacing LOLZ with LZMA
Tried various versions of XTool and ZTool, and different configs for them as well.

I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

KaktoR

17-11-2021 09:39

Hellblade: Senua's Sacrifice Enhanced
Video documentary ripped

Code:

 17:51:27 - Selected ARC/DS method for Data1a-01.bin was: srep_new+LZMA-MT

 17:51:27 - Selected ARC/DS method for Data1b-01.bin was: bpk

----------------------------------------------------------------------------------------------------------------------------------------

 18:38:43 - Overall input size: 17.25 GB

 18:38:43 - Overall output size: 8.25 GB (Ratio 47.84%)

 18:38:43 - Data files creation time: 00:38:17

KaktoR

17-11-2021 11:48

Quote:

Originally Posted by its_gnar_again (Post 494944)

Tried different configs for LOLZ
Tried different configs for SREP
Tried outright replacing LOLZ with LZMA
Tried various versions of XTool and ZTool, and different configs for them as well.

I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

xtool:reflate works just fine for me.

Code:

[External compressor:xtool]

header    = 0

default   = -c64mb -t75p

packcmd   = "XTool.exe" precomp { -option} - - <stdin> <stdout>

unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

its_gnar_again

17-11-2021 17:06

Quote:

Originally Posted by KaktoR (Post 494957)

xtool:reflate works just fine for me.

Code:

[External compressor:xtool]

header    = 0

default   = -c64mb -t75p

packcmd   = "XTool.exe" precomp { -option} - - <stdin> <stdout>

unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

My xtool setup is pretty much the same from when I last tested it.

Quote:

[External compressor:xtool]
header = 0
packcmd = XT\xtool.exe precomp -mzlib -c64mb -t75p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

I haven't tested JUST xtool only to see if that works, but I will get back to you here from when I do. I don't understand what the problem is on my part really.

Grumpy

17-11-2021 22:22

@its_gnar_again

Quote:

forgot the pass & email I used on my old account lmao

Surely you remember one of them, you've had 4 or 5 accounts since 2020 :eek:

its_gnar_again

18-11-2021 19:05

Quote:

Originally Posted by its_gnar_again (Post 494959)

I haven't tested JUST xtool only to see if that works, but I will get back to you.

UPDATE: I tried specifically using reflate on xtool+srep+lolz, and still get a decompression error. However, I did try using xtool only and it decompressed just fine...what could honestly be the issue? I would like to get this figured out. :confused:

KaktoR

19-11-2021 00:32

Maybe check your CLS.ini settings for lolz and srep.

And make sure you're using the correct cls files for lolz and srep

All times are GMT -7. The time now is 09:43.

Page 177 of 312

177