LZ4 MOD for *certain* game's data archives | repacks

[elit]

This is a modified LZ4 compressor for repacking game files before their injection back into their original(or zeroed) data archives. This will very likely only work on certain small selection of games, specifically Raiders of the Broken Planes - Wardog Fury, for which it was tested, and probably other Mercury Steam games that use LZ4 data archives(not all of them use lz4), like other Raiders games and perhaps 2 Castlevania games as well. This will compress files originally extracted(and unpacked) with castlevania.bms using quickbms back into their exact crc perfect compressed dump. For re-injecting, the script need to be modified to raw dump using log command(everything included below). Data packs will/must be crc exact as original. Compressed files will consist of only compressed blocks, with block size and crc preceding it, and without LZ4 header and tail. Also frame crc must be disabled. Command line is therefore "lz4 -9 -B5 --no-frame-crc FILE". -9 and -B5 were for this game. Do not use -B262144 as it is not same for some reason. Also read further for more details. It may need to be re-moded again for different game/version.

The story:
LZ4 is a whore. I meant it, I almost lost my mind. It took me weeks to figure out everything. But I had to do it, this game was perfect opportunity for me to learn, to get closer to that "FitGirl" status. Bms script was simple but at the same time challenges that arose made me learn so much more... I am glad.

First I had to figure out parameters. (Actually, first I had to learn basic bms scripting but I will skip that.) This wasnt so hard as from the bms script I already knew game compress with 256k blocks(0x040000). LZ4 specification state that -B5 == 256kb blocks. I compressed random extracted file with every option and compared to original raw(not decompressed) file dump.
<<<(For raw dump to extract, you just comment out "comtype" lines as well as chunk extract function and use classic log command - or same moded bms file as for reinjecting(see in attached files). Also this method make quickbms pretty much same as Razor's Injector Maker but quicker and no file size/memory limit.). >>>
I found compression -9 was closest to match most bytes in between files, it was clear this was correct setting. But files were not same(not taking head/tail and block's crc position into account).

I ended up downloading *every single* LZ4 version from github. I learned hard way that each few versions compression changed(different output), this was hell. I even found that each compression level is better or worse for different files, sometimes highest one gave worse results, sometimes level 10 gave worse than level 9 but level 11 again better, then on next file scenario switched etc. Then version differences in compression on top of that. I found that files had to be crc exact because game crashed, probably because it kept frame crc in other place of archive, but funny it crashed if 2 smaller game packs were moded but other 2 biggest ones not. And so on, my god...

Finally I found that v128-v131 were exact match(after removing header/tail and... oh yes, block crc - I will explain later). Yet, from say 30000+ files all were exact but ~300 were still wrong. This was killing me, eventually I found that there is certain "switch" in compression function that I had to force one way only. Then it compressed correct all, but some files made app crash. After another headache I made a *very* dirty workaround that even to my own surprise worked and I got 90000+ files all crc perfect and no crash. I still may redone it though.

So long story short, with LZ4 you not only need to match/guess correct parameters, you also need right version AND you still may need to make certain changes in code to match same compression behavior. With that said, now it should be easy for everyone to figure and re-mod for other games as needed - once you compare src diffs for changes - as they are very small.

Oh and that block crc... game archive files use "block size -> block crc -> cmp data" block pattern(you learn that from bms script), lz4 specification use crc after block end not beginning! So that had to be taken care of as well. Now whole dump is match including block's crc btw.

Ok done with story:
I include both original and moded bms script as well as both original and modded lz4 src, so you can learn. There is also compiled moded binary as well. This was tested for Raiders of the Broken Planet Wardog Fury, but could/should work for all series as well as Castlevanias from Mercury Steam. Other games I dont know, it depend on how their files lz4 blocks are structured, how trigger happy are they with crc's and so on. You can learn that from their bms scripts. If files are same pattern as above then its only matter of matching compression options and/or lz4 version. For this game you use "-9 -B5 --no-frame-crc" parameters and nothing else. Compressor was not tested with other parameters(but should work)!

Hopefully this will help encourage people into advanced repacking, join and become *Russian Hacker* NOW .

lz4-r131_orig.zip

lz4-r131_mod.zip

castlevania.zip

castlevania_mod.zip

NEW:
lz4-r131_mod2.zip