Best Compression Methods for 'Specific' Games. Q&A

[Masquerade]

The Riftbreaker [v27746]

Audio .zips:

msc+oggre+srep+lzma2

Everything else:

xzlib+srep+lolz

6.4GB ---> 4.4GB

[Carldric Clement]

Need for Speed Carbon

Original Size: 4.91 GB
Compress Size: 2.41 GB

Method:

*  any jpg files = packjpg
*  rest of files = xprecomp+srep:m3f:a2+exe2+7z:d128m:mt2

XTool Used Plugins = ZLib, Reflate, LZ4HC

[Carldric Clement]

Broforce v1131

Original Size: 457 MB
Compress Size: 289 MB

Method:

* jpg = packjpg
* rest of files = xprecomp+srep:m3f:a2+exe2+lolz:d128:mt4:mtt1:lmdf1
* asset bundle unity = xtool:zlib:unity+srep:m3f:a2+exe2+lolz:d128m:mt4:mtt1:lmdf1

Thanks to L33THAK0R for suggesting reduced size.
Update on 1 May 2022.

[L33THAK0R]

Originally Posted by Carldric Clement

Broforce v1131

Original Size: 457 MB
Compress Size: 337 MB

Method:

* jpg = packjpg
* rest of files = xprecomp+srep:m3f:a2+exe2+lolz:d128:mt4:mtt1:lmdf1

you can reduce this to ~290mb (Incl. installer) if you decompress the Unity assets.

[KaktoR]

Age of Empires II Definitive Edition
Update 61591, 1 Language, all DLC

 17:20:52 - Selected ARC/DS method for Data1a-01.bin was: precomp_mtx+srep_new+4x4:lzma
 17:20:52 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:lzma
-----------------------------------------------------------------------------------------
 21:51:31 - Overall input size: 44.71 GB
 21:51:31 - Overall output size: 24.40 GB (Ratio 54.57%)
 21:51:31 - Data files creation time: 04:09:05

Data1a = *.aoe2scenario|*.aoe2campaign|*.scx|*.png|*.aoe2spgame|*.pdf

Why precomp? Xtool is ~200mb bigger

WemTool on wem files gives some mb, not worth it

[kj911]

Quick noob decompression questions: This varying methods betterly safer from extraction process??

A: xZLib+delta+srep+lolz
B: xZlib+srep+delta+lolz

My test, its compression percentage differnece less than 0.1%. (B its smaller.)

[Masquerade]

Project CARS 3 [v1.0.0.0.0705]

• *.bff
  

  xPCARS3+srep+lolz

• *.bk2
  

  bpk (hash fail on decompression so use bpk exes and hdiff patch them)

• *.bank
  

  msc+srep+lzma2

• Everything else
  

  precomp:d0+srep+lolz

49.4GB ----> 19.1GB

[L33THAK0R]

Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript ), however I can't seem to figure out which library version to use, the best I've gotten so far is ~850MB. I think if I exhaust all my options I'll just look into re-importing the decompressed assets to a modified archive, since some Russian modders were able to do so, in order to apply a total conversion translation mod to the titular title in question.

[Masquerade]

Originally Posted by L33THAK0R

Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript )

I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work

I checked this game out and it uses lz4hc, not lz4. By using lz4 precompressor, you will end up getting 21/5539 streams. By switching the codec to lz4hc alone you will instantly get 4215/5539 [liblz4 v1.7.4].

This alone would seem pretty cool, until you see that the only streams that were inflated were the small ones, so you get very little size increase. In addition to this, you still have over 1000 compressed LZ4 streams still in the file.

By digging a bit deeper into the info XTool is giving me (using verbose output), my next intention was to lock precompression at a certain level of lz4hc. I chose lz4hc level 9. This is because most of the streams previously were compressed at level 9. After making a new database and testing, we get these results:

• 5524/5539 streams [liblz4 v1.7.3, v1.7.4, v1.8.0].
• 5488/5539 streams [liblz4 v1.8.1].
• 5424/5539 streams [liblz4 v1.8.2, v1.8.3, v1.9.3].

This clearly shows that using anything newer than liblz4 v1.8.0 is going to yield extremely poor results. With the oldest 3 libraries, you will only miss 15 streams. In my opinion, this is completely acceptable. At this point, I stop testing and publish my findings in the XTool plugins thread.

We all have a lot to thank Razor12911 for.

[L33THAK0R]

Originally Posted by Masquerade

I checked this game out and it uses lz4hc, not lz4.

Ah that'd definitely explain things, I didn't even think to see if I was using the wrong codec

Originally Posted by Masquerade

I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work

Yeah I was using bms2xtl, just thought rebuilding an archive would be the next step if I couldn't figure out how to pick up more streams.

Cheers for the help and testing mate

[L33THAK0R]

Over the last couple days I've been testing "BMS2XTOOL" on ".pxp" archives to try and see if through this method it's possible to both decrypt + expand the compressed streams present within a given ".pxp" archive, to make re-compressing said archives a viable prospect. However I've hit a roadblock unfortunately, as I can get xTool (v0.5.0) to detect some streams, but can't seem to figure out what codec to use, to actually process the majority of said streams.

Currently I'm associating the compressor type, "ZENPXP" with the "zlib" codec for use with "BMS2XTOOL", and combining this with the "reflate" codec in my arc.ini.

For anyone interested, attached below is a sample testing suite, as well as just a copy of BMS2XTOOL, with the appropriate resources to generate your own database for said sample, should anyone be interested in looking at this. I've also tried to see if it's been documented anywhere within QuickBMS the compression algorithm used for a given "comtype" in a QuickBMS script but alas this doesn't seem to be the case

[Spotless]

VALORANT: Episode 4 Act 3

srep+lolz

23.5 GB> 7.2GB

[iamsmeg]

sorry I forgot to say this is for Watch dogs 1 using the database method posted here

[pslove]

Hey guys, what's the best method for The Walking Dead Definitive Series? I'm using DiskSpan GUI btw. Thank you.

[Masquerade]

Originally Posted by pslove

Hey guys, what's the best method for The Walking Dead Definitive Series? I'm using DiskSpan GUI btw. Thank you.

Telltale game, so all of the ttarch2 files that have headers beginning with ECTT are encrypted using Blowfish encryption.

In this case, DiskspanGUI has nothing to handle those files. I would use cls-ttgd by Shegorat, but that isn't a public tool.

Use this to decrypt the files: https://www.fileforums.com/showthread.php?p=483980

Then, compress with xtool(zlib+reflate)+srep+lolz. You can re-encrypt your decrypted files from earlier using the same tool to get MD5 perfectiom upon installation.