FileForums - Best Compression Methods for 'Specific' Games. Q&A

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

The Hopebringer

Code:

 23:48:33 - Selected ARC/DS method for Data1a-01.bin was: xtool:mmermaid:mhopebringer:dd3+4x4:lzma

 23:48:33 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+4x4:lzma

-----------------------------------------------------------------------------------------

 00:20:16 - Overall input size: 9,73 GB

 00:20:16 - Overall output size: 7,78 GB (Ratio 79.95%)

 00:20:16 - Overall conversion time: 00:31:41

Some Diskspan GUI settings for The Sinking City - The Merciful Madness and Starfield
Feel free to alter the LOLZ_ST lines to whatever you feel comfortable with if you don't like to wait for unpacks.

Code:

The Sinking City - The Merciful Madness 47.5 GB -> 27.1 GB



[Game1Settings]

Name=The Sinking City

Exec=

MethodState=1

Data1.Name=Data1.bin

Data1.ExclText=

Data1.InclText=*.pak

Data1.Method1=xtool:c128mb:d1:mue5,k0xAAF9AF1F40225A1241C120533277804B09A56CE78CBF100C131423788DB0E34F:mkraken:core_2.9.5+xtool:dd3+LOLZ_ST|0

Data1.SplitMode=2

Data2.Name=Data2.bin

Data2.Method1=xtool:dd3+LOLZ_ST|0

Data2.InclText=

Data2.SplitMode=2

Data2.ExclText=*.pak



---- DSG_arc.ini addition for LOLZ_ST

LOLZ_ST  = lolz:dtb1:d1536:mtt0:mtb64:mc1023

----could just sub for other LOLZ methods for less compression and faster decompression



Also Starfieldv1888 eng only 118 GB -> 62.3 GB



[Game1Settings]

Name=Starfield

Exec=

MethodState=1

Data1.Name=Data1.bin

Data1.ExclText=*WwiseSounds*|*Voices*

Data1.InclText=*.ba2|*.esm

EGPath=C:\Program Files\Bethesda Softworks\Starfield

Data1.Method1=xtool:mstarfield+xtool:mzlib:mreflate:dd3+LOLZ_ST|0

Data1.SplitMode=2

Data2.Name=Data2.bin

Data2.Method1=msc+xtool:dd3+LOLZ_ST|0

Data2.InclText=*WwiseSounds*|*Voices*

Data2.SplitMode=2

Data3.Name=Data3.bin

Data3.Method1=xtool:c512mb:mbk2:df20p+LOLZ_ST|0

Data3.InclText=*.bk2

Data3.SplitMode=2

Data4.Name=Data4.bin

Data4.Method1=xtool:dd3+LOLZ_ST/$mp3pack=7zdll-pm:m0=mp3det:m1=packmp3c|0

Data4.ExclText=*.ba2|*.bk2|*.esm

Data4.SplitMode=1

Quote:

Originally Posted by KaktoR (Post 503389)

I tested a bit.

Do not use msc for any of the files. msc_frog will work for encoding, but decoding will give you errors. So use xtool with flac plugin and l8 setting for now for all the WAV files.

For the OGG files you have to extract them first (with BAE tool), erase them out of all the files (with xtool erase command), compress them with oggre. After installation, you have to put all the OGG files back inside the original bsa files (with xtool decode command + XTL file).

BIK file use bpk.

Edit: You can also use bethesda.bms with quickbms instead of BAE if you want.

Edit 2:

Code:

 17:55:35 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib:dd3+lolz:d81:mtt1:mt12:mc1023

 17:55:35 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8

 17:55:35 - Selected ARC/DS method for Data1c-01.bin was: oggre

 17:55:35 - Selected ARC/DS method for Data1d-01.bin was: 7zdll-pm:m0=mp3det:m1=packmp3c

 17:55:35 - Selected ARC/DS method for Data1e-01.bin was: bpk+xtool:dd3+lolz:d81:mtt1:mt12:mc1023



 18:53:04 - Overall input size: 11,92 GB

 18:53:04 - Overall output size: 5,41 GB (Ratio 45.41%)

 18:53:04 - Overall conversion time: 00:57:28

Install time + decoding BSA files 10 minutes.

Edit 3: Keep in mind that the input size is a bit inaccurate due to extracted files, as the BSA files are still the original size but with zero'd data inside.

Good evening and thanks for your help! These are my results:

18:37:14 - Selected ARC/DS method for BPK.bin was: bpk
18:37:14 - Selected ARC/DS method for BSA.bin was: xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
18:37:14 - Selected ARC/DS method for ESM.bin was: xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
18:37:14 - Selected ARC/DS method for MP3.bin was: msc_tak:mp3=1:wav=1:raw=1:ddsraw=1:bmf=9:TAK=9:bmp =1:lzma,bt4,fb128,lc8+srep_old:392+lolz:dtb1:d512m :mtb96:ldl5:mc1023
18:37:14 - Selected ARC/DS method for OGG.bin was: oggre+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc10 23
18:37:14 - Selected ARC/DS method for WAV.bin was: xtool:mflac,l8+srep_old:392+lolz:dtb1:d512m:mtb96: ldl5:mc1023
18:37:14 - Selected ARC/DS method for OTHERS.bin was: srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc1023

23:55:30 - Overall input size: 12,22 GB
23:55:30 - Overall output size: 5,27 GB (Ratio 43.18%)
23:55:30 - Overall conversion time: 05:18:12

One cons is decompression time: about 19 minutes! How can I reduce this time? How did you obtain 10 minutes of decompression time?

^^
MSC's MP3 codec is quite slow. I'd recommend try mpzmt by BlackFire69 with srep after.

Don't use srep+lolz after oggre.

There is a 7zip MP3 plugin which is super good, even better then mpz, and it will not be detected as virus unlike mpz (false positive)

Can it process stream ,or only for file ?

Both.

Code:

7zdll

Compressed 1 file, 1,135,385,600 => 918,525,479 bytes. Ratio 80.90%

Compression time: cpu 0.73 sec/real 78.90 sec = 1%. Speed 14.39 mB/s

All OK



mpzmt

Compressed 1 file, 1,135,385,600 => 922,505,962 bytes. Ratio 81.25%

Compression time: cpu 0.44 sec/real 75.45 sec = 1%. Speed 15.05 mB/s

All OK

Quote:

Originally Posted by KaktoR (Post 503419)

There is a 7zip MP3 plugin which is super good, even better then mpz, and it will not be detected as virus unlike mpz (false positive)

Thanks so much for your help!!! Not only compression ratio is slightly better, but i've also saved at least 5 minutes of decompression time! Now, the bottleneck is .ogg file decompression, it still takes too long. Is there any compression that can allow me faster decompression time?
I'll try to extract only .mp3 and .wav files from .bsa files to understand if it worth it in terms of decompression time!

^^
You cannot speed up oggre.

Quote:

Originally Posted by Dragonis40 (Post 503424)

Good evening! Concerning Fallout New Vegas, if you extract only .mp3 and .wav files, the final size is compressed videogame about 140 MB bigger than the version with .mp3, .ogg and .wav, but compression time is about half time!!!
So it can worth it!

If some kind soul can reveal method used for Monster Hunter Rise .pak files? Can't figure this one out...only two .pak files should be encrypted and with zlib and zstd streams but there is no AES key for this anywhere and using xtool on this will produce only small gain...but I saw few repackers ratio was *much* better. Tried creating database also but without luck.
Thanks in advance, regards.

^
zstd.

Use xtool 0.8.4 + libzstd 1.1.4

https://www.patreon.com/posts/97128362

Code:

Streams: 77311 / 77311

Time: 00:05:06 (CPU 00:01:06)



Size: 34.5 GB >> 48.2 GB



Done!!!

Actually there are even some zlib streams.

Code:

Streams: 20756 / 20756

Time: 00:03:52 (CPU 00:14:47)



Size: 34.5 GB >> 36.0 GB



Done!!!

^you're right KaktoR,:) I was using most recent version of libzstd without any success but using 1.1.4 from 2017 worked like a charm! If I may ask, how can one pinpoint exact version of libzstd to use? I tried everything, yes I even tried different versions of libzstd but it never occurred to me that I should be using 7 years old dll version :eek:
Thanks again,
best regards

xtool library checker :)

Never seen a ReEngine game use anything else but libzstd.dll v1.1.4 when zstd is used by that engine.