Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Does anyone know how BMP detection under MSC works? Got a few titles I'm working on that mainly use BMP images for texture assets, however MSC seems to skip over the majority of files. I had a brief trawl through some posts here but searching with shorter search terms is a massive pain in the arse and didnt yield a lot of information.

[Masquerade]

Maybe the thread on Krinkels.org may help you?

[KaktoR]

Saints Row (2022)
v1.1.6.4392638

 21:38:42 - Selected ARC/DS method for Data1a-01.bin was: xtool:mlz4f,l9:lz4_192+srep+4x4:lzma
 21:38:42 - Selected ARC/DS method for Data1b-01.bin was: srep+4x4:lzma
-------------------------------------------------------------------------------------------------------------------------------------------
 23:25:52 - Overall input size: 44.08 GB
 23:25:52 - Overall output size: 25.51 GB (Ratio 57.88%)
 23:25:52 - Overall conversion time: 01:47:08

[KaktoR]

Wolfenstein II The New Colossus

 15:56:42 - Selected ARC/DS method for Data1a-01.bin was: xtool:mkraken:5+srep+4x4:lzma
 15:56:42 - Selected ARC/DS method for Data1b-01.bin was: bpk
 15:56:42 - Selected ARC/DS method for Data1c-01.bin was: srep+4x4:lzma
-------------------------------------------------------------------------------------------------------------------------------------------
 21:50:55 - Overall input size: 58.80 GB
 21:50:55 - Overall output size: 43.14 GB (Ratio 73.37%)
 21:50:55 - Overall conversion time: 05:53:20

Notes:
- Majority of *.resources files using level 4, majority of *.texdb and *.cache using level 5. I was too lazy to sort them out properly so I didn't use the l# option for xtool.
- On many files I didn't got all kraken streams, I checked with every library I have and even used different libraries with --oodle= option like with Elden Ring but to no avail.
- *.pages files using lz4hc but most likely weak compression, but couldn't get all streams (overall about half of streams), so I decided to just compress them with srep+lzma.

[dixen]

Somerville (Steam version)

*.wem, *.bnk - srep+razorMT
6.3 gb> 1.6 gb

Rest files - srep+lolz
3.7 gb > 430 mb

Install time - 1 min.

[Masquerade]

Unlike it's original variant, paks in Crysis 3 Remastered are not encrypted. They are just standard zip files and can be precompressed with xtool:

Compressed Animations.pak, 144,174,121 => 229,174,364 bytes. Ratio 158.96%

[KaktoR]

Is anyone here working with xtool's --dedup option?

I made many tests since yesterday and I think this will replace srep command entirely, at least for me.

xtool+srep+4x4:lzma
Compressed 27 files, 4,594,892,800 => 2,346,970,567 bytes. Ratio 51.08%
Compression time: cpu 3.69 sec/real 921.49 sec = 0%. Speed 4.99 mB/s
All OK

15min 23sec

xtool+4x4:lzma
Compressed 27 files, 4,594,892,800 => 2,346,861,976 bytes. Ratio 51.08%
Compression time: cpu 2.69 sec/real 660.86 sec = 0%. Speed 6.95 mB/s
All OK

11min 02sec

--dedup=(1..5)
srep.exe must be present next to xtool.exe for compression and decompression (no cls-srep required)

[KaktoR]

Here is another test. You can use xtool without any precompression too.

xtool+4x4:lzma
Compressed 7,383 files, 31,661,584,708 => 8,601,282,118 bytes. Ratio 27.17%
Compression time: cpu 16.39 sec/real 1506.53 sec = 1%. Speed 21.02 mB/s
All OK

28min

srep+4x4:lzma
Compressed 7,383 files, 31,661,584,708 => 8,601,527,510 bytes. Ratio 27.17%
Compression time: cpu 24.27 sec/real 2312.41 sec = 1%. Speed 13.69 mB/s
All OK

38min

[External compressor:xtool]
header    = 0
default   = -c64mb -t100p
packcmd   = "{compressor}.exe" precomp { -moption} --dbase --dedup=3 - - <stdin> <stdout>
unpackcmd = "{compressor}.exe" decode -t100p - - <stdin> <stdout>

[L33THAK0R]

Originally Posted by Gehrman

Is there a way to control memory usage during decompression in SREP.exe?

LAST MINUTE EDIT: Just realized you're talking about using xtool's srep functionality, so not sure if this is useful at all

I use the following settings in my "CLS.ini" file, I do use the CLS version of Srep for decompression so not sure if that would have any impact if you're using a non-CLS package.

[Srep]
Bufsize=24m
transfer_ReadBufSize=512k
transfer_WriteBufSize=512k
Memory=60%-2048m
TempPath={app}

[Srep_new]
Bufsize=24m
transfer_ReadBufSize=512k
transfer_WriteBufSize=512k
Memory=60%-2048m
TempPath={app}

[KaktoR]

Crysis 2 Remastered

 14:52:34 - Selected ARC/DS method for Data1a-01.bin was: oggre
 14:52:34 - Selected ARC/DS method for Data1b-01.bin was: xtool+4x4:lzma
 14:52:34 - Selected ARC/DS method for Data1c-01.bin was: xtool:mcrysis2remastered+4x4:lzma
 14:52:34 - Selected ARC/DS method for Data1d-01.bin was: xtool+4x4:lzma
-------------------------------------------------------------------------------------------------------------------------------------------
 16:53:20 - Overall input size: 53.76 GB
 16:53:20 - Overall output size: 34.79 GB (Ratio 64.72%)
 16:53:20 - Overall conversion time: 02:00:44

Data1a
GameCrysis2\music.pak
GameCrysis2\music_chk0.pak

Data1b
GameCrysis2\Video*.pak

Data1c
*.pak (excl 1a+1b)

Data1d
rest of the files

Thanks to Masquerade for database file

[Masquerade]

For Spiderman Miles Morales, use XTool Spidey plugin with liblz4 v1.9.4 to get all streams.

[fabrieunko]

Originally Posted by Masquerade

For Spiderman Miles Morales, use XTool Spidey plugin with liblz4 v1.9.4 to get all streams.

thank you very much but where to find?

[Masquerade]

Originally Posted by fabrieunko

thank you very much but where to find?

https://github.com/lz4/lz4/releases/tag/v1.9.4

C'mon man, just use Google

[KaktoR]

Edit: Have mixed up with another title

[fabrieunko]

^^ I had used google and found the same link but I was not sure and I did not understand how to replace the dll...