Best Compression Methods for 'Specific' Games. Q&A

[Dragonis40]

Originally Posted by kj911

Dragonis40: Try it use very slooow settings:

xZLib(old Xtool 0.12)+srep:m3f:a0(srep 3.92!)+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:d512m:fba4096:mc1023:tt16:mt1:ldmf1:ldc0:ldl5

Questions: The *.bsa archives contain goodly compressibble raw/wav audio files? MP3's or any? Videos?

Alternative1: xZLib+srep+delta+lolz
Alternative2: xZLib+msc+srep+delta(used or not)+lolz

I've made some test:

xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
21 files: 8,588,002,315 byte => 4,776,969,655 bytes. Ratio 55.62% (17127.25 sec)

xt_zlib+srep_old:392+lolz:dt:dtb1:dtw1:dtm1:dto1:d td1:d512m:fba4096:mc1023:tt16:mt1:ldmf1:ldc0:ldl5
21 files: 8,588,002,315 => 4,764,178,217 bytes. Ratio 55.47% (35516.07 sec)

The last combination is NOT worth it!

Several .bsa files are plenty of .ogg file (hundreds), so I should extract these .ogg files, compress and before the end of installation make a backup of original .bsa file. I think it takes very long to install the videogame!!!

[KaktoR]

Tales of Arise
v12.12.23

 17:25:12 - Selected ARC/DS method for Data1a-01.bin was: xtool:c128mb:mzlib+xtool:dd3+4x4:lzma
 17:25:12 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+bcm
 17:25:12 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:lzma
------------------------------------------------------------------------------------------
 19:54:25 - Overall input size: 52,19 GB
 19:54:25 - Overall output size: 34,37 GB (Ratio 65.86%)
 19:54:25 - Overall conversion time: 02:29:11

Japanese audio was ripped out and wem's are encoded. Gives you another 100mb size reduction (only 100kb+ files), but honestly not sure if the game is working without crashes with ripped files as this was just meant to be as a test with fixed xtool version (like I was doing since days for now) and I am not planning to play this game actually. So if you want to test this for yourself use "unreal_tournament_4_0.4.25e.bms" script with quickbms, then erase with xtool.

Key: 0x61FC71A1376A29598393E2BFB976B275705F460E29399CFD1F9C060C8CD6CE40

Just keep in mind that any patch for the game will fail, as long as "pakchunk0-WindowsNoEditor.pak" is modified.

[KaktoR]

I tested a bit.

Do not use msc for any of the files. msc_frog will work for encoding, but decoding will give you errors. So use xtool with flac plugin and l8 setting for now for all the WAV files.

For the OGG files you have to extract them first (with BAE tool), erase them out of all the files (with xtool erase command), compress them with oggre. After installation, you have to put all the OGG files back inside the original bsa files (with xtool decode command + XTL file).

BIK file use bpk.

Edit: You can also use bethesda.bms with quickbms instead of BAE if you want.

Edit 2:

 17:55:35 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib:dd3+lolz:d81:mtt1:mt12:mc1023
 17:55:35 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8
 17:55:35 - Selected ARC/DS method for Data1c-01.bin was: oggre
 17:55:35 - Selected ARC/DS method for Data1d-01.bin was: 7zdll-pm:m0=mp3det:m1=packmp3c
 17:55:35 - Selected ARC/DS method for Data1e-01.bin was: bpk+xtool:dd3+lolz:d81:mtt1:mt12:mc1023

 18:53:04 - Overall input size: 11,92 GB
 18:53:04 - Overall output size: 5,41 GB (Ratio 45.41%)
 18:53:04 - Overall conversion time: 00:57:28

Install time + decoding BSA files 10 minutes.

Edit 3: Keep in mind that the input size is a bit inaccurate due to extracted files, as the BSA files are still the original size but with zero'd data inside.

[Dragonis40]

Originally Posted by KaktoR

I tested a bit.

Do not use msc for any of the files. msc_frog will work for encoding, but decoding will give you errors. So use xtool with flac plugin and l8 setting for now for all the WAV files.

For the OGG files you have to extract them first (with BAE tool), erase them out of all the files (with xtool erase command), compress them with oggre. After installation, you have to put all the OGG files back inside the original bsa files (with xtool decode command + XTL file).

BIK file use bpk.

Edit: You can also use bethesda.bms with quickbms instead of BAE if you want.

Edit 2:

 17:55:35 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib:dd3+lolz:d81:mtt1:mt12:mc1023
 17:55:35 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8
 17:55:35 - Selected ARC/DS method for Data1c-01.bin was: oggre
 17:55:35 - Selected ARC/DS method for Data1d-01.bin was: 7zdll-pm:m0=mp3det:m1=packmp3c
 17:55:35 - Selected ARC/DS method for Data1e-01.bin was: bpk+xtool:dd3+lolz:d81:mtt1:mt12:mc1023

 18:53:04 - Overall input size: 11,92 GB
 18:53:04 - Overall output size: 5,41 GB (Ratio 45.41%)
 18:53:04 - Overall conversion time: 00:57:28

Install time + decoding BSA files 10 minutes.

Edit 3: Keep in mind that the input size is a bit inaccurate due to extracted files, as the BSA files are still the original size but with zero'd data inside.

Good evening and thanks for your help! These are my results:

18:37:14 - Selected ARC/DS method for BPK.bin was: bpk
18:37:14 - Selected ARC/DS method for BSA.bin was: xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
18:37:14 - Selected ARC/DS method for ESM.bin was: xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
18:37:14 - Selected ARC/DS method for MP3.bin was: msc_tak:mp3=1:wav=1:raw=1:ddsraw=1:bmf=9:TAK=9:bmp =1:lzma,bt4,fb128,lc8+srep_old:392+lolz:dtb1:d512m :mtb96:ldl5:mc1023
18:37:14 - Selected ARC/DS method for OGG.bin was: oggre+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc10 23
18:37:14 - Selected ARC/DS method for WAV.bin was: xtool:mflac,l8+srep_old:392+lolz:dtb1:d512m:mtb96: ldl5:mc1023
18:37:14 - Selected ARC/DS method for OTHERS.bin was: srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc1023

23:55:30 - Overall input size: 12,22 GB
23:55:30 - Overall output size: 5,27 GB (Ratio 43.18%)
23:55:30 - Overall conversion time: 05:18:12

One cons is decompression time: about 19 minutes! How can I reduce this time? How did you obtain 10 minutes of decompression time?

[Masquerade]

^^
MSC's MP3 codec is quite slow. I'd recommend try mpzmt by BlackFire69 with srep after.

Don't use srep+lolz after oggre.

[KaktoR]

Fallout 3 GOTY

 20:13:34 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib:dd3+lolz:d84:mtt1:mt12:mc1023
 20:13:34 - Selected ARC/DS method for Data1b-01.bin was: msc_tak
 20:13:34 - Selected ARC/DS method for Data1c-01.bin was: oggre
 20:13:34 - Selected ARC/DS method for Data1d-01.bin was: 7zdll-pm:m0=mp3det:m1=packmp3c
 20:13:34 - Selected ARC/DS method for Data1e-01.bin was: bpk+xtool:dd3+lolz:d84:mtt1:mt12:mc1023
-------------------------------------------------------------------------------------------------------------------------------------------
 21:22:39 - Overall input size: 11,25 GB
 21:22:39 - Overall output size: 5,08 GB (Ratio 45.13%)
 21:22:39 - Overall conversion time: 01:00:31

Used quickbms + bethesda.bms to extract WAV, OGG and MP3 files out of all the BSA sound archives.

Example

for %%f in (*.bsa) do quickbms_4gb_files -d -o -Y -f "{}.wav,{}.ogg,{}.mp3" bethesda.bms "%%f"

Used xtool with erase command to erase all WAV, OGG and MP3 files out of the sound archives (zero data).

Example

"xtool" erase "Fallout - Sound.bsa_extract" "Fallout - Sound.bsa" "fallout3.xtl"
...
...

Data1a -> All BSA files
Data1b -> WAV
Data1c -> OGG
Data1d -> MP3
Data1e -> Rest of the files

After installation, use xtool again with decode command to copy all the extracted files back into their original places inside the BSA archives.

Example

"xtool" decode "fallout3.xtl" "Fallout - Sound.bsa_extract" "Fallout - Sound.bsa"
...
...

Remove extracted files from game dir.

PS: Original input size is about 8.70 GB, but due to extracted data this is a bit bigger. However all the sound BSA files will be compressable something like near 1%, so only the install step will require some additional space, but once xtool decode is finished, the size is the same as original but also the crc of all the files.

Edit: Install time + xtool decode about 10mins

[Masquerade]

A Way Out

xtool_zlib_dd5+lolz

20.6GB ---> 14.4GB

[KaktoR]

The Hopebringer

 23:48:33 - Selected ARC/DS method for Data1a-01.bin was: xtool:mmermaid:mhopebringer:dd3+4x4:lzma
 23:48:33 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+4x4:lzma
-----------------------------------------------------------------------------------------
 00:20:16 - Overall input size: 9,73 GB
 00:20:16 - Overall output size: 7,78 GB (Ratio 79.95%)
 00:20:16 - Overall conversion time: 00:31:41

[murphy78]

Some Diskspan GUI settings for The Sinking City - The Merciful Madness and Starfield
Feel free to alter the LOLZ_ST lines to whatever you feel comfortable with if you don't like to wait for unpacks.

The Sinking City - The Merciful Madness 47.5 GB -> 27.1 GB

[Game1Settings]
Name=The Sinking City
Exec=
MethodState=1
Data1.Name=Data1.bin
Data1.ExclText=
Data1.InclText=*.pak
Data1.Method1=xtool:c128mb:d1:mue5,k0xAAF9AF1F40225A1241C120533277804B09A56CE78CBF100C131423788DB0E34F:mkraken:core_2.9.5+xtool:dd3+LOLZ_ST|0
Data1.SplitMode=2
Data2.Name=Data2.bin
Data2.Method1=xtool:dd3+LOLZ_ST|0
Data2.InclText=
Data2.SplitMode=2
Data2.ExclText=*.pak

---- DSG_arc.ini addition for LOLZ_ST
LOLZ_ST  = lolz:dtb1:d1536:mtt0:mtb64:mc1023
----could just sub for other LOLZ methods for less compression and faster decompression

Also Starfieldv1888 eng only 118 GB -> 62.3 GB

[Game1Settings]
Name=Starfield
Exec=
MethodState=1
Data1.Name=Data1.bin
Data1.ExclText=*WwiseSounds*|*Voices*
Data1.InclText=*.ba2|*.esm
EGPath=C:\Program Files\Bethesda Softworks\Starfield
Data1.Method1=xtool:mstarfield+xtool:mzlib:mreflate:dd3+LOLZ_ST|0
Data1.SplitMode=2
Data2.Name=Data2.bin
Data2.Method1=msc+xtool:dd3+LOLZ_ST|0
Data2.InclText=*WwiseSounds*|*Voices*
Data2.SplitMode=2
Data3.Name=Data3.bin
Data3.Method1=xtool:c512mb:mbk2:df20p+LOLZ_ST|0
Data3.InclText=*.bk2
Data3.SplitMode=2
Data4.Name=Data4.bin
Data4.Method1=xtool:dd3+LOLZ_ST/$mp3pack=7zdll-pm:m0=mp3det:m1=packmp3c|0
Data4.ExclText=*.ba2|*.bk2|*.esm
Data4.SplitMode=1

[KaktoR]

There is a 7zip MP3 plugin which is super good, even better then mpz, and it will not be detected as virus unlike mpz (false positive)

[Dragonis40]

Originally Posted by KaktoR

There is a 7zip MP3 plugin which is super good, even better then mpz, and it will not be detected as virus unlike mpz (false positive)

Thanks so much for your help!!! Not only compression ratio is slightly better, but i've also saved at least 5 minutes of decompression time! Now, the bottleneck is .ogg file decompression, it still takes too long. Is there any compression that can allow me faster decompression time?
I'll try to extract only .mp3 and .wav files from .bsa files to understand if it worth it in terms of decompression time!

[Masquerade]

^^
You cannot speed up oggre.

[Dragonis40]

Originally Posted by Dragonis40

Thanks so much for your help!!! Not only compression ratio is slightly better, but i've also saved at least 5 minutes of decompression time! Now, the bottleneck is .ogg file decompression, it still takes too long. Is there any compression that can allow me faster decompression time?
I'll try to extract only .mp3 and .wav files from .bsa files to understand if it worth it in terms of decompression time!

Good evening! Concerning Fallout New Vegas, if you extract only .mp3 and .wav files, the final size is compressed videogame about 140 MB bigger than the version with .mp3, .ogg and .wav, but compression time is about half time!!!
So it can worth it!

[shazzla]

Can it process stream ,or only for file ?

[KaktoR]

Both.

7zdll
Compressed 1 file, 1,135,385,600 => 918,525,479 bytes. Ratio 80.90%
Compression time: cpu 0.73 sec/real 78.90 sec = 1%. Speed 14.39 mB/s
All OK

mpzmt
Compressed 1 file, 1,135,385,600 => 922,505,962 bytes. Ratio 81.25%
Compression time: cpu 0.44 sec/real 75.45 sec = 1%. Speed 15.05 mB/s
All OK