Best Compression Methods for 'Specific' Games. Q&A

[Masquerade]

Originally Posted by L33THAK0R

Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript )

I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work

I checked this game out and it uses lz4hc, not lz4. By using lz4 precompressor, you will end up getting 21/5539 streams. By switching the codec to lz4hc alone you will instantly get 4215/5539 [liblz4 v1.7.4].

This alone would seem pretty cool, until you see that the only streams that were inflated were the small ones, so you get very little size increase. In addition to this, you still have over 1000 compressed LZ4 streams still in the file.

By digging a bit deeper into the info XTool is giving me (using verbose output), my next intention was to lock precompression at a certain level of lz4hc. I chose lz4hc level 9. This is because most of the streams previously were compressed at level 9. After making a new database and testing, we get these results:

• 5524/5539 streams [liblz4 v1.7.3, v1.7.4, v1.8.0].
• 5488/5539 streams [liblz4 v1.8.1].
• 5424/5539 streams [liblz4 v1.8.2, v1.8.3, v1.9.3].

This clearly shows that using anything newer than liblz4 v1.8.0 is going to yield extremely poor results. With the oldest 3 libraries, you will only miss 15 streams. In my opinion, this is completely acceptable. At this point, I stop testing and publish my findings in the XTool plugins thread.

We all have a lot to thank Razor12911 for.

[L33THAK0R]

Originally Posted by Masquerade

I checked this game out and it uses lz4hc, not lz4.

Ah that'd definitely explain things, I didn't even think to see if I was using the wrong codec

Originally Posted by Masquerade

I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work

Yeah I was using bms2xtl, just thought rebuilding an archive would be the next step if I couldn't figure out how to pick up more streams.

Cheers for the help and testing mate