Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Does anyone know roughly what the average variance is for the final ratio of a given input, that's run through LOLZ?

[KaktoR]

The Last of Us Part 1
v1.1.0.0

 19:08:20 - Selected ARC/DS method for Data1a-01.bin was: xtool:c192mb:mpsarc:core_2.9.9+xtool:mpreflate:dd3+4x4:lzma
 19:08:20 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+4x4:lzma
-----------------------------------------------------------------------------------------
 21:16:38 - Overall input size: 70.29 GB
 21:16:38 - Overall output size: 48.29 GB (Ratio 68.70%)
 21:16:38 - Overall conversion time: 02:08:17

You will get better ratio in increasing chunk size for xtool.

[kj911]

L33THAK0R: Very input data dependent! Older, very older gamez, betterly compressible than previous 10-15 years ago to last years games. Average few percent to upto 10-25% betterly final compression ratio than LZMA. After, many data precompression (ex: xtool), prefiltering (ex: exe/delta) and such more...

Simply, this cannot be said in advance! It is necessary to experiment, it is possible to derive approximate information about the possible degree of compression from existing repacked games online.

Small examples: This very old modded MSTS games *.ACE files with xzlib+srep+lolz combo avg. 25-40% final ratio than uncompressed sizes.
*.RAW files in rare case upto 1.5-2* betterly compressed via delta+(srep)+lolz combo than simple LZMA only.
Sound files final sizes difference its minimal few percentage. Achieve better rates use sound specific compression.

KaktoR: What "mpsarc"??

[KaktoR]

Originally Posted by kj911

KaktoR: What "mpsarc"??

https://fileforums.com/showpost.php?...&postcount=352

[L33THAK0R]

Originally Posted by kj911

L33THAK0R: Very input data dependent! Older, very older gamez, betterly compressible than previous 10-15 years ago to last years games. Average few percent to upto 10-25% betterly final compression ratio than LZMA. After, many data precompression (ex: xtool), prefiltering (ex: exe/delta) and such more...

Simply, this cannot be said in advance! It is necessary to experiment, it is possible to derive approximate information about the possible degree of compression from existing repacked games online.

Small examples: This very old modded MSTS games *.ACE files with xzlib+srep+lolz combo avg. 25-40% final ratio than uncompressed sizes.
*.RAW files in rare case upto 1.5-2* betterly compressed via delta+(srep)+lolz combo than simple LZMA only.
Sound files final sizes difference its minimal few percentage. Achieve better rates use sound specific compression.

KaktoR: What "mpsarc"??

@@kj911 ah thank you for the explanation, what I meant was more how much could be chalked up to variance due to LOLZ being a non-deterministic compression algorithm (or series of, I'm not too sure on its inner-workings). I like to compare my results to my peers, for some titles, take the 2020 title "Manifold Garden" I got a result of 797 MB, however one of our peers got a result of 792 MB. Now I was working with the latest version of the title and I also included the weight of the installer, with mine being slightly larger (due to some un-optimised storing of the "VCL styles" library that I've yet to figure out how to compress and decompress during run-time), and I'm wondering whether I can chalk up this difference to variance or whether I've missed some data that can be packed more optimally.

[kj911]

L33THAK0R: From what I have experienced with LOLZ so far, its operation is a bit more unusual compared to the standard LZMA. A given result cannot be reproduced, exactly byte, as with more commercial compressors. (Given compressed file size, like enwik8-10 results.) Certain switches of "Arc.exe" also affect the degree of final compression, I'm thinking here of the "-ds" switch, which is the default in principle or even if you (not) define it the final result may be different, say with "-dses/dsns" than otherwise.. LOLZ can sometimes be better without -dt/dtb, in some cases, than if we set everything to the max.

This small image shows the results of data files precompressed with xZLib+srep:m3f:a0 mode, and separately compressed to 7z, with the same compression settings. The difference between them is whether or not "-ds" or "-dses" switch was used during precompression. We would think that the end result would be the same if LZMA/LOLZ final compression were used for that archive.

[Wanterlude]

Hi all, can anyone look at the files of the game Bramble The Mountain King, there UE4 and encrypted files inside + kraken.
I tried the latest version of the unreal plugin, but it didn't work. (xtool last ver.). With ucas database the same thing.
Can someone please see if it's the plugin not working or am I crooked?
Key:

0x0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3

//edit. woks fine with latest xtool

[Masquerade]

Originally Posted by Wanterlude

Hi all, can anyone look at the files of the game Bramble The Mountain King, there UE4 and encrypted files inside + kraken.
I tried the latest version of the unreal plugin, but it didn't work. (xtool last ver.). With ucas database the same thing.
Can someone please see if it's the plugin not working or am I crooked?
Key:

0x0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3

Apologies if you tried already, but please may you try XTool v0.7.1 with Unreal Plugin and Oodle core v2.9.5 with this packcmd? I cpuldn't help but notice you mentioned UE4 whereas the UTOC/UCAS structure is a fearure of UE5 that got added backwards.

xtool precomp -c128mb -t50p -mue5:0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3+kraken

[Wanterlude]

Originally Posted by Masquerade

Apologies if you tried already, but please may you try XTool v0.7.1 with Unreal Plugin and Oodle core v2.9.5 with this packcmd? I cpuldn't help but notice you mentioned UE4 whereas the UTOC/UCAS structure is a fearure of UE5 that got added backwards.

xtool precomp -c128mb -t50p -mue5:0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3+kraken

XTool v0.7.1 with Unreal Plugin and Oodle core v2.9.5

xtool precomp -c128mb -t50p -d1 -mue5:0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3+kraken
Streams: 8910 / 8910
Time: 00:00:18 (CPU 00:01:30)

Size: 6.70 GB >> 6.71 GB

xtool precomp -c128mb -t50p -d1 -mue4:0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3+kraken
Streams: 984 / 984
Time: 00:00:14 (CPU 00:01:01)

Size: 6.70 GB >> 6.72 GB

*xtool 0.7.2h same results

[dixen]

Originally Posted by Wanterlude

Hi all, can anyone look at the files of the game Bramble The Mountain King, there UE4 and encrypted files inside + kraken.
I tried the latest version of the unreal plugin, but it didn't work. (xtool last ver.). With ucas database the same thing.
Can someone please see if it's the plugin not working or am I crooked?
Key:

0x0D29D2B91A3564686C106FA7A96BE98D3AA13A55C80D4E3200EE608DA2C105E3

My test with xtool v 0.77 + oodle 2.9.5

Compressing Bramble_TMK-WindowsNoEditor.ucas 99.2%
Compressing 7,111,026,672 bytes with xtool.exe precomp -mue5:k0x0D29D2B91A3564686C106FA7A96BE98D3AA13A55C8 0D4E3200EE608DA2C105E3:n128+kraken -d1 -c128mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
100.0%XTool is created by Razor12911

Streams: 221925 / 222121
Time: 00:06:00 (CPU 00:37:28)

Size: 6.62 GB >> 12.5 GB

100.0%
Errorlevel=0

Compressed 1 file, 7,111,026,672 => 13,462,584,251 bytes. Ratio 189.32%
Compression time: cpu 0.95 sec/real 592.84 sec = 0%. Speed 11.99 mB/s
All OK

[Wanterlude]

Originally Posted by dixen

My test with xtool v 0.77 + oodle 2.9.5

Sorry I forgot to edit that...
After Razor updated xtool (or plugin), everything works fine

[KaktoR]

Just read a topic about unpacking PCK files. Nothing new, honestly, but an idea came in my mind about not just unpacking them, but also encode/decode the WEMs inside the PCK files. So far, nothing new. Just wanted to know, if there is a "elegant" way instead of my idea floating in my head. Maybe somthing more "automated".

I already know that this method works and it's CRC perfect after the last step.

1. Unpack PCK files with quickbms + wwise.bms
2. Copy unpacked files to somewhere else, so you have 2 copys of them
3. Use WEMTool on the first copy in game folder
4. Null all WEM files on the second copy outside of game folder and reimport them back with quickbms to PCK files in game folder. This way you can compress the PCK files* with ~0-1% ratio
5. Compress the game with nulled PCK files* and encoded WEM files

Installing the repack
1. After installing, you have to decode all processed WEM files back to their original with WEMTool again
2. Reimport decoded WEM files and BNK files back to PCK files with quickbms import feature

[Masquerade]

This will absolutely work however you can replace The QuickBMS reimport with XTool erase/decode since the contents of a PCK aren't compressed so the wems will appear in plain bytes.

I've wanted in the past to find some way of adding a wem plugin to XTool so wems don't need to be extracted to disk (which is a long time when there are often thousands of tiny wems).

[KaktoR]

Only bad thing left is that quickbms isn't able to create extracted folders in the same (sub)folder where the input files are. In AC games you have several dlc folders with pck files.
In those cases I just use a FOR loop for it, works great. However you have to use "cd /D" command everytime to switch the folder if you are on root level as working directory, otherwise quickbms will create the extracted folders inside root folder without _extracted prefix and you have to move them manually afterwards.

[kj911]

Any idea, methods, tricks recompressing Trainz 2010 (and PTP2*) game *.CDP files/complete games???

Example file the following CDP format: http://lat-trainz-addons.1w.lv/DR1A-...20Pack.cdp.zip

XTool 0.12/0.71 with zlib or zlib+(p)reflate methods or many precomp 0.4.x versions, not handle inflating the following file. GFS found ~19k ZLIB streams packed in level:9 mode.

*Pro Train Perfect 2. Public copy not available. (The game based on TS2010.)