LZ4 MOD for *certain* game's data archives | repacks

[elit]

This is a modified LZ4 compressor for repacking game files before their injection back into their original(or zeroed) data archives. This will very likely only work on certain small selection of games, specifically Raiders of the Broken Planes - Wardog Fury, for which it was tested, and probably other Mercury Steam games that use LZ4 data archives(not all of them use lz4), like other Raiders games and perhaps 2 Castlevania games as well. This will compress files originally extracted(and unpacked) with castlevania.bms using quickbms back into their exact crc perfect compressed dump. For re-injecting, the script need to be modified to raw dump using log command(everything included below). Data packs will/must be crc exact as original. Compressed files will consist of only compressed blocks, with block size and crc preceding it, and without LZ4 header and tail. Also frame crc must be disabled. Command line is therefore "lz4 -9 -B5 --no-frame-crc FILE". -9 and -B5 were for this game. Do not use -B262144 as it is not same for some reason. Also read further for more details. It may need to be re-moded again for different game/version.

The story:
LZ4 is a whore. I meant it, I almost lost my mind. It took me weeks to figure out everything. But I had to do it, this game was perfect opportunity for me to learn, to get closer to that "FitGirl" status. Bms script was simple but at the same time challenges that arose made me learn so much more... I am glad.

First I had to figure out parameters. (Actually, first I had to learn basic bms scripting but I will skip that.) This wasnt so hard as from the bms script I already knew game compress with 256k blocks(0x040000). LZ4 specification state that -B5 == 256kb blocks. I compressed random extracted file with every option and compared to original raw(not decompressed) file dump.
<<<(For raw dump to extract, you just comment out "comtype" lines as well as chunk extract function and use classic log command - or same moded bms file as for reinjecting(see in attached files). Also this method make quickbms pretty much same as Razor's Injector Maker but quicker and no file size/memory limit.). >>>
I found compression -9 was closest to match most bytes in between files, it was clear this was correct setting. But files were not same(not taking head/tail and block's crc position into account).

I ended up downloading *every single* LZ4 version from github. I learned hard way that each few versions compression changed(different output), this was hell. I even found that each compression level is better or worse for different files, sometimes highest one gave worse results, sometimes level 10 gave worse than level 9 but level 11 again better, then on next file scenario switched etc. Then version differences in compression on top of that. I found that files had to be crc exact because game crashed, probably because it kept frame crc in other place of archive, but funny it crashed if 2 smaller game packs were moded but other 2 biggest ones not. And so on, my god...

Finally I found that v128-v131 were exact match(after removing header/tail and... oh yes, block crc - I will explain later). Yet, from say 30000+ files all were exact but ~300 were still wrong. This was killing me, eventually I found that there is certain "switch" in compression function that I had to force one way only. Then it compressed correct all, but some files made app crash. After another headache I made a *very* dirty workaround that even to my own surprise worked and I got 90000+ files all crc perfect and no crash. I still may redone it though.

So long story short, with LZ4 you not only need to match/guess correct parameters, you also need right version AND you still may need to make certain changes in code to match same compression behavior. With that said, now it should be easy for everyone to figure and re-mod for other games as needed - once you compare src diffs for changes - as they are very small.

Oh and that block crc... game archive files use "block size -> block crc -> cmp data" block pattern(you learn that from bms script), lz4 specification use crc after block end not beginning! So that had to be taken care of as well. Now whole dump is match including block's crc btw.

Ok done with story:
I include both original and moded bms script as well as both original and modded lz4 src, so you can learn. There is also compiled moded binary as well. This was tested for Raiders of the Broken Planet Wardog Fury, but could/should work for all series as well as Castlevanias from Mercury Steam. Other games I dont know, it depend on how their files lz4 blocks are structured, how trigger happy are they with crc's and so on. You can learn that from their bms scripts. If files are same pattern as above then its only matter of matching compression options and/or lz4 version. For this game you use "-9 -B5 --no-frame-crc" parameters and nothing else. Compressor was not tested with other parameters(but should work)!

Hopefully this will help encourage people into advanced repacking, join and become *Russian Hacker* NOW .

lz4-r131_orig.zip

lz4-r131_mod.zip

castlevania.zip

castlevania_mod.zip

NEW:
lz4-r131_mod2.zip

[elit]

Mod2 upped, small code cleanup and final changes. It should be final for these games.

"-9, --no-frame-crc and -B5" flags are now made default, also shown in help cmd options. No need to invoke with explicit params anymore(unless you need different), just lz4.exe FILE.

[Razor12911]

Originally Posted by elit

LZ4 is a whore.

I couldn't have said it better myself

[elit]

Originally Posted by Razor12911

I couldn't have said it better myself

I am curious, do lzo, zlib and others(zstd,...) have similar problems(of different output per different version) or are they more stable in that regard?

[Razor12911]

zstd has this problem as well. lzo, not so sure but it shouldn't have this problem since all variants of lzo come with binaries/libraries, lzo1x, lzo2a and so on.
zlib doesn't have this.

[Gupta]

you can build zstd with legacy support. Then it can decompress any version

[elit]

Originally Posted by PrinceGupta2000

you can build zstd with legacy support. Then it can decompress any version

I am more interested in compression variability though, you need to match crc in most game archives.

[78372]

Probably zlib is the one which has almost no problems

[Gupta]

/*! ZSTD_isFrame() :
 *  Tells if the content of `buffer` starts with a valid Frame Identifier.
 *  Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
 *  Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
 *  Note 3 : Skippable Frame Identifiers are considered valid. */
ZSTDLIB_API unsigned ZSTD_isFrame(const void* buffer, size_t size);

[Razor12911]

PrinceGupta there is a reason ztool comes with external libraries, it’s for user to change them until he finds one that was used in a particular game. Decompression is never a problem, but recompression is. When it comes lz4 and zstd, everything must be precise else you will get crc mismatch every time. Perhaps a bit with ztool since it comes with internal diff patching functions when user used a dll that was not used for compression in first place

[elit]

Originally Posted by Razor12911

zstd has this problem as well. lzo, not so sure but it shouldn't have this problem since all variants of lzo come with binaries/libraries, lzo1x, lzo2a and so on.
zlib doesn't have this.

Now I see the pattern, both lz4 and zstd are developed by same author. Well, at least his code is nicely organized and easy to learn.

Originally Posted by Razor12911

PrinceGupta there is a reason ztool comes with external libraries, it’s for user to change them until he finds one that was used in a particular game.

OMG I kill you man, thats great! How come I did not know.. Time to do my games all over again .
EDIT: Well actually, thinking of it it may not be enough with lz4/zstd. I dont know how you coded it but in this case for example, default library would read block structure wrong even if compression match because of mentioned crc position which is part of the block, unless your tool is skipping it and can focus only on game data.
Still, it gave me an idea that one perhaps doesnt need to repack whole game archives with quickbms anymore, because depending on how your ztool works, maybe all I need is to modify library as I did here and ztool could function again. That would be great and would made ztool universal, then all you do is compile all lz4 versions and recycle them forever. But I would like to know more in detail how your tool work/read data exactly to understand, though its probably your secret so I am not going to ask.

[Razor12911]

Originally Posted by elit

OMG I kill you man, thats great! How come I did not know.. Time to do my games all over again .
EDIT: Well actually, thinking of it it may not be enough with lz4/zstd. I dont know how you coded it but in this case for example, default library would read block structure wrong even if compression match because of mentioned crc position which is part of the block, unless your tool is skipping it and can focus only on game data.
Still, it gave me an idea that one perhaps doesnt need to repack whole game archives with quickbms anymore, because depending on how your ztool works, maybe all I need is to modify library as I did here and ztool could function again. That would be great and would made ztool universal, then all you do is compile all lz4 versions and recycle them forever. But I would like to know more in detail how your tool work/read data exactly to understand, though its probably your secret so I am not going to ask.

Yes, tool focuses on game data, error codes are returned all the time, like you said, library reading block structure incorrectly, it happens a lot, but it still returns decompressed data.
I can tell you how it works if you want, it's no secret.

[elit]

Originally Posted by Razor12911

...I can tell you how it works if you want, it's no secret.

That would be helpful, you can PM me or post it here whatever you prefer.

I actually tried to use lib from this version that I know works with the game and could not get it unpack anything with ztool(output same size). But I am not sure if I did it right, you see your lz4.dll have ~60kb while my compiled had around ~200+kb and I made .dll by manually changing function in makefile code - in function "add_library" where I added "SHARED" flag, something like this:

add_library(liblz4 ${LZ4_SRCS_LIB}) >> add_library(liblz4 SHARED ${LZ4_SRCS_LIB})

Now the thing is if I dont do that and just check to make lib in cmake gui, it does output library around same size as yours(~60kb), but its a .lib not windows dll. When I tried that big dll ztool did not complain with errors but output same size, if I rename lib to dll it output error(obviously not same). Maybe someone can hint me here what I did wrong, you can also try yourself if you can compile more properly.

Additionally, I found on zenhax forums that zstd may not be so bad after all, it does keep track of different version in header:
"zstd
It starts with a little endian 32bit magic number, when seen with a hex editor only the first byte (the low 8bit) is different because it depends by the version of the algorithm:

1e b5 2f fd    v0.1
22 b5 2f fd    v0.2
23 b5 2f fd    v0.3
24 b5 2f fd    v0.4
25 b5 2f fd    v0.5
26 b5 2f fd    v0.6
27 b5 2f fd    v0.7
28 b5 2f fd    v0.8, current version"

Unfortunately lz4 dont do it, but its the only one at least. Of course, there still have to be a header present in game resource to know.

[Razor12911]

Well first of all, do you know ztool's plz4 and pzstd wasn't set to be universal?

[elit]

Originally Posted by Razor12911

Well first of all, do you know ztool's plz4 and pzstd wasn't set to be universal?

Yeah I guess you hardcoded it in a specific way, but feel free to elaborate on it in more detail if you can, any additional info is more than welcome. I also do know that some game data packs utilize various kind of obfuscation and encryption. But since you mentioned that external libs were made intentionally so that user can replace them with different version for different games, I assumed it was designed to be more universal at least for basic archive formats.

Aside of that, I just realized why even my modified version may not work here with ztool. I modified compression function, but not decompression - which still likely read block's incorrectly. If ztool use it to verify data, no wonder it wont work. Of course, as I said any additional info regarding this issue that may help me understand your tool better is more than welcome.