Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Originally Posted by Masquerade

xtool+srep+lolz

On *.webm and *.feral

srep+lzma2

*feral is ogg audio archives, so you could use oggre but no idea of the benefit (or any issue in decompression)

50.1GB ----> 17.3GB

You can rip languages from webm videos with ffmpeg (thanks to devs streams are labeled).

Only 1 language: 1.78 GB -> 1.68 GB

You can use MSC on the following files

Contents\Resources\Data\data\sounds\sfx.dat.feral

Contents\Resources\Data\data\sounds\localised\de\voice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\de\voice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\en\voice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\en\voice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\it\voice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\it\voice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\es\voice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\es\voice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\ru\voice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\ru\voice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\fr\voice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\fr\voice_strat.dat.feral

[KaktoR]

Total War Rome Remastered
v2.0.2, 1 language, other languages ripped, no hd pack

 10:53:41 - Selected ARC/SPLIT method for Data1a-01.bin was: 
srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
$deflatepack=xtool:mreflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
$mscromeremastered=msc_tak:mp3=1:wav=1:raw=1:ddsraw=1:bmf=9:TAK=9:bmp=1:lzma,bt4,fb128,lc8
$textpack=srep_new+lzma:a1:mfbt4:256m:fb273:mc1000000000:lc8
$precomppack=precomp_mtx+srep_new+lolz:mtt1:mt<thread>:tt1:mc2
$libpack=srep_new+razor_mtx
----------------------------------------------------------------------------------------------------------------------------------------
 13:29:13 - Overall input size: 43.22 GB
 13:29:13 - Overall output size: 19.46 GB (Ratio 45.02%)
 13:29:13 - Overall conversion time: 02:27:30

$deflatepack: *.png
$msc: see post above
$textpack: *.txt, *.xml, *.json, *.strings
$precomppack: *.gif
$libpack: *.dll, *.exe

video files for ffmpeg: Contents\Resources\Data\data\fmv\intros (1 language: 1.78 GB -> 1.68 GB)

[KaktoR]

Far Cry 5
v1.011, 2 languages, incl HD pack

 21:41:34 - Selected ARC/DS method for Data1a-01.bin was: xtool:mreflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
 02:20:26 - Overall input size: 91.60 GB
 02:20:26 - Overall output size: 39.95 GB (Ratio 43.61%)
 02:20:26 - Overall conversion time: 04:38:50

Used DELZORec_x64 because xtool with fc5 plugin does not work properly atm.

DELZORec times:
8min ssd
25min hdd

PS: DELZORec does not work on any hd game files (*_hd.*)

DELZORec usage from batch (common.dat example):
DELZORec_x64.exe u "data_final\pc\common.fat" "data_final\pc\common.dat" "data_final\pc\common.unp"
DELZORec_x64.exe r "data_final\pc\common.unp" "data_final\pc\common.dat" "data_final\pc\common.fat"

[hwang4]

What's Best Method for compression?

[Masquerade]

Originally Posted by hwang4

What's Best Method for compression?

This game's PKG archives are compressed using LZ4, so first you need to find a way to decompress files inside the archives. Since the header of the files appear to be encrypted, unlike previous games, there's no BMS script to unpack the PKG.

[hwang4]

Masquerade , can you show me your method plz !

I tried (xtool+srep+lolz), i got 25,5gb (english+french)!
Your method is better (21gb)
thx!

[Masquerade]

Originally Posted by hwang4

Masquerade , can you show me your method plz !

I tried (xtool+srep+lolz), i got 25,5gb (english+french)!
Your method is better (21gb)
thx!

I used VGM toolbox to "extract" the PKGs, since every lz4 stream is very easy to detect with the 0x04224D18 header with PKGB before the steam and PKGE after the stream. When each file is extracted, you can use lz4 exe to decompress the extracted lz4 streams, then compress with xtool:zlib+srep+lolz.

Then, to remake the pkgs, you just need to compress each file again individually with the lz4 exe using -9 --no-frame-crc arguments. Then, concatenate the files together (I use bincat for this since it's fast and simple, not sure of native cmd option) and xdelta.

Make your xdeltas before compressing your archive, since you will need to decompress all of the extracted files from the pkgs, compress them, then concatenate them to make a temp file. You can compare the temp file with the original pkg to make a delta.

For the sound files, I used razor.

[Hexagon123]

I found out that Yoku's Island Express use ZLIB?

[dixen]

Life Is Strange: Before The Storm (GOG)

*.bytes - xtool (v0.12 + lz4 codec) + srep + lolz
15 gb > 33,6 gb > 15 gb > 4.6 gb

*.usm, *.awb, *.acb - Srep + zstd (cls)
6 gb > 4.6 gb

Rest files - srep + lolz
105 mb > 31 mb

RePack size - 9.3 gb
Install time - 6 min. (R5 3600, 16 gb RAM, HDD>HDD)

NOTE. Why ZSTD? Easy)))

Tested 290 files, 5,035,921,557 => 6,416,827,196 bytes. Ratio 78.48%
Testing time: cpu 17.61 sec/real 17.95 sec = 98%. Speed 357.46 mB/s
All OK

[Walker96082]

Using FreeArc in Ultra Profile
from 25.7 GB to 7.98 GB

[Jarmen_Kell]

Guys, any idea for "Tales of Arise"?

[macut18]

NBA 2K22
xtool_zlib+srep+4x4
96.3 GB > 70.2 GB

[L33THAK0R]

Originally Posted by dixen

Pathfinder: Kingmaker Enhancend + Imperial Edition 2.0.8 (35465)

*.bnk, *.wem - srep+razor
*.resS, *.assets - srep+lolz
Bundle folder = xtool(v0.11 zlib+lz4)+srep+lolz

45 gb > 15.1 gb

Compression time - 11 hours
Decompression time - 35 mins

Has anyone been able to get the lz4 xtool plugin working with this title? No matter the version or combination of methods I always seem to get decompression errors when packing with lz4. Tried a handful of other libraries but doesnt seem to help.

[KaktoR]

@L33THAK0R
I don't get it to work either, but for me the end is right in the begining of compression no matter what library I use
"ERROR: general (de)compression error in xprecomp"

Edit: Now I used xtool 0.3.16H + unity plugin + lz4 v180 library and it worked fine. Will test some more.

I don't know if the unity plugin is compatible with latest xtool version?

Edit 2: Ok unity plugin works with latest xtool version 0.3.21

newdunswardoutdoor_light_night_bundle

Compressed 1 file, 124,807,756 => 219,835,242 bytes. Ratio 176.14%
Compression time: cpu 0.11 sec/real 3.73 sec = 3%. Speed 33.42 mB/s
All OK

Extracted 1 file, 219,835,242 => 124,807,756 bytes. Ratio 176.14%
Extraction time: cpu 0.03 sec/real 3.16 sec = 1%. Speed 39.54 mB/s
All OK

[L33THAK0R]

Originally Posted by KaktoR

@L33THAK0R
I don't get it to work either, but for me the end is right in the begining of compression no matter what library I use
"ERROR: general (de)compression error in xprecomp"

Edit: Now I used xtool 0.3.16H + unity plugin + lz4 v180 library and it worked fine. Will test some more.

I don't know if the unity plugin is compatible with latest xtool version?

Edit 2: Ok unity plugin works with latest xtool version 0.3.21

newdunswardoutdoor_light_night_bundle

Compressed 1 file, 124,807,756 => 219,835,242 bytes. Ratio 176.14%
Compression time: cpu 0.11 sec/real 3.73 sec = 3%. Speed 33.42 mB/s
All OK

Extracted 1 file, 219,835,242 => 124,807,756 bytes. Ratio 176.14%
Extraction time: cpu 0.03 sec/real 3.16 sec = 1%. Speed 39.54 mB/s
All OK

This is quite the relief that I wasn't the only one having issues! Would it be possible for you to pretty please upload the v180 lz4 library? I can't seem to find it online (my collection of lz4 libraries is only the different ones included in Razor's xtool builds) unfortunately.