Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #1  
Old 08-01-2018, 14:04
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
LZ4 MOD for *certain* game's data archives | repacks

This is a modified LZ4 compressor for repacking game files before their injection back into their original(or zeroed) data archives. This will very likely only work on certain small selection of games, specifically Raiders of the Broken Planes - Wardog Fury, for which it was tested, and probably other Mercury Steam games that use LZ4 data archives(not all of them use lz4), like other Raiders games and perhaps 2 Castlevania games as well. This will compress files originally extracted(and unpacked) with castlevania.bms using quickbms back into their exact crc perfect compressed dump. For re-injecting, the script need to be modified to raw dump using log command(everything included below). Data packs will/must be crc exact as original. Compressed files will consist of only compressed blocks, with block size and crc preceding it, and without LZ4 header and tail. Also frame crc must be disabled. Command line is therefore "lz4 -9 -B5 --no-frame-crc FILE". -9 and -B5 were for this game. Do not use -B262144 as it is not same for some reason. Also read further for more details. It may need to be re-moded again for different game/version.

The story:
LZ4 is a whore. I meant it, I almost lost my mind. It took me weeks to figure out everything. But I had to do it, this game was perfect opportunity for me to learn, to get closer to that "FitGirl" status. Bms script was simple but at the same time challenges that arose made me learn so much more... I am glad.

First I had to figure out parameters. (Actually, first I had to learn basic bms scripting but I will skip that.) This wasnt so hard as from the bms script I already knew game compress with 256k blocks(0x040000). LZ4 specification state that -B5 == 256kb blocks. I compressed random extracted file with every option and compared to original raw(not decompressed) file dump.
<<<(For raw dump to extract, you just comment out "comtype" lines as well as chunk extract function and use classic log command - or same moded bms file as for reinjecting(see in attached files). Also this method make quickbms pretty much same as Razor's Injector Maker but quicker and no file size/memory limit.). >>>
I found compression -9 was closest to match most bytes in between files, it was clear this was correct setting. But files were not same(not taking head/tail and block's crc position into account).

I ended up downloading *every single* LZ4 version from github. I learned hard way that each few versions compression changed(different output), this was hell. I even found that each compression level is better or worse for different files, sometimes highest one gave worse results, sometimes level 10 gave worse than level 9 but level 11 again better, then on next file scenario switched etc. Then version differences in compression on top of that. I found that files had to be crc exact because game crashed, probably because it kept frame crc in other place of archive, but funny it crashed if 2 smaller game packs were moded but other 2 biggest ones not. And so on, my god...

Finally I found that v128-v131 were exact match(after removing header/tail and... oh yes, block crc - I will explain later). Yet, from say 30000+ files all were exact but ~300 were still wrong. This was killing me, eventually I found that there is certain "switch" in compression function that I had to force one way only. Then it compressed correct all, but some files made app crash. After another headache I made a *very* dirty workaround that even to my own surprise worked and I got 90000+ files all crc perfect and no crash. I still may redone it though.

So long story short, with LZ4 you not only need to match/guess correct parameters, you also need right version AND you still may need to make certain changes in code to match same compression behavior. With that said, now it should be easy for everyone to figure and re-mod for other games as needed - once you compare src diffs for changes - as they are very small.

Oh and that block crc... game archive files use "block size -> block crc -> cmp data" block pattern(you learn that from bms script), lz4 specification use crc after block end not beginning! So that had to be taken care of as well. Now whole dump is match including block's crc btw.

Ok done with story:
I include both original and moded bms script as well as both original and modded lz4 src, so you can learn. There is also compiled moded binary as well. This was tested for Raiders of the Broken Planet Wardog Fury, but could/should work for all series as well as Castlevanias from Mercury Steam. Other games I dont know, it depend on how their files lz4 blocks are structured, how trigger happy are they with crc's and so on. You can learn that from their bms scripts. If files are same pattern as above then its only matter of matching compression options and/or lz4 version. For this game you use "-9 -B5 --no-frame-crc" parameters and nothing else. Compressor was not tested with other parameters(but should work)!

Hopefully this will help encourage people into advanced repacking, join and become *Russian Hacker* NOW .

lz4-r131_orig.zip

lz4-r131_mod.zip

castlevania.zip

castlevania_mod.zip

NEW:
lz4-r131_mod2.zip

Last edited by elit; 08-01-2018 at 19:33.
Reply With Quote
The Following 6 Users Say Thank You to elit For This Useful Post:
COPyCAT (24-01-2018), doofoo24 (10-01-2018), EzzEldin16 (10-01-2018), felice2011 (08-01-2018), oltjon (08-01-2018), Razor12911 (08-01-2018)
Sponsored Links
  #2  
Old 08-01-2018, 19:23
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
Mod2 upped, small code cleanup and final changes. It should be final for these games.

"-9, --no-frame-crc and -B5" flags are now made default, also shown in help cmd options. No need to invoke with explicit params anymore(unless you need different), just lz4.exe FILE.
Reply With Quote
  #3  
Old 08-01-2018, 19:46
Razor12911's Avatar
Razor12911 Razor12911 is offline
Coder
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,154
Thanks: 1,671
Thanked 7,478 Times in 1,739 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by elit View Post
LZ4 is a whore.
I couldn't have said it better myself
__________________
Looking for something?
Visit the
Tutorial Index

Last edited by Razor12911; 08-01-2018 at 19:53.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
elit (17-04-2018)
  #4  
Old 09-01-2018, 05:50
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
I couldn't have said it better myself
I am curious, do lzo, zlib and others(zstd,...) have similar problems(of different output per different version) or are they more stable in that regard?
Reply With Quote
  #5  
Old 09-01-2018, 20:21
Razor12911's Avatar
Razor12911 Razor12911 is offline
Coder
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,154
Thanks: 1,671
Thanked 7,478 Times in 1,739 Posts
Razor12911 is on a distinguished road
zstd has this problem as well. lzo, not so sure but it shouldn't have this problem since all variants of lzo come with binaries/libraries, lzo1x, lzo2a and so on.
zlib doesn't have this.
__________________
Looking for something?
Visit the
Tutorial Index
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
elit (10-01-2018), EzzEldin16 (10-01-2018)
  #6  
Old 09-01-2018, 23:31
Gupta Gupta is offline
Registered User
 
Join Date: Aug 2016
Location: India
Posts: 350
Thanks: 123
Thanked 533 Times in 197 Posts
Gupta is on a distinguished road
you can build zstd with legacy support. Then it can decompress any version
__________________
XD
Reply With Quote
  #7  
Old 10-01-2018, 04:14
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
Quote:
Originally Posted by PrinceGupta2000 View Post
you can build zstd with legacy support. Then it can decompress any version
I am more interested in compression variability though, you need to match crc in most game archives.
Reply With Quote
  #8  
Old 10-01-2018, 04:40
78372's Avatar
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 532
Thanks: 594
Thanked 608 Times in 232 Posts
78372 is on a distinguished road
Probably zlib is the one which has almost no problems

Last edited by 78372; 10-01-2018 at 04:59.
Reply With Quote
The Following User Says Thank You to 78372 For This Useful Post:
elit (10-01-2018)
  #9  
Old 10-01-2018, 05:08
Gupta Gupta is offline
Registered User
 
Join Date: Aug 2016
Location: India
Posts: 350
Thanks: 123
Thanked 533 Times in 197 Posts
Gupta is on a distinguished road
Code:
/*! ZSTD_isFrame() :
 *  Tells if the content of `buffer` starts with a valid Frame Identifier.
 *  Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
 *  Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
 *  Note 3 : Skippable Frame Identifiers are considered valid. */
ZSTDLIB_API unsigned ZSTD_isFrame(const void* buffer, size_t size);
__________________
XD
Reply With Quote
The Following User Says Thank You to Gupta For This Useful Post:
EzzEldin16 (10-01-2018)
  #10  
Old 10-01-2018, 05:28
Razor12911's Avatar
Razor12911 Razor12911 is offline
Coder
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,154
Thanks: 1,671
Thanked 7,478 Times in 1,739 Posts
Razor12911 is on a distinguished road
PrinceGupta there is a reason ztool comes with external libraries, it’s for user to change them until he finds one that was used in a particular game. Decompression is never a problem, but recompression is. When it comes lz4 and zstd, everything must be precise else you will get crc mismatch every time. Perhaps a bit with ztool since it comes with internal diff patching functions when user used a dll that was not used for compression in first place
__________________
Looking for something?
Visit the
Tutorial Index
Reply With Quote
The Following 3 Users Say Thank You to Razor12911 For This Useful Post:
78372 (10-01-2018), elit (10-01-2018), EzzEldin16 (10-01-2018)
  #11  
Old 10-01-2018, 12:00
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
zstd has this problem as well. lzo, not so sure but it shouldn't have this problem since all variants of lzo come with binaries/libraries, lzo1x, lzo2a and so on.
zlib doesn't have this.
Now I see the pattern, both lz4 and zstd are developed by same author. Well, at least his code is nicely organized and easy to learn.

Quote:
Originally Posted by Razor12911 View Post
PrinceGupta there is a reason ztool comes with external libraries, it’s for user to change them until he finds one that was used in a particular game.
OMG I kill you man, thats great! How come I did not know.. Time to do my games all over again .
EDIT: Well actually, thinking of it it may not be enough with lz4/zstd. I dont know how you coded it but in this case for example, default library would read block structure wrong even if compression match because of mentioned crc position which is part of the block, unless your tool is skipping it and can focus only on game data.
Still, it gave me an idea that one perhaps doesnt need to repack whole game archives with quickbms anymore, because depending on how your ztool works, maybe all I need is to modify library as I did here and ztool could function again. That would be great and would made ztool universal, then all you do is compile all lz4 versions and recycle them forever. But I would like to know more in detail how your tool work/read data exactly to understand, though its probably your secret so I am not going to ask.

Last edited by elit; 10-01-2018 at 12:32.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
EzzEldin16 (11-01-2018)
  #12  
Old 11-01-2018, 19:14
Razor12911's Avatar
Razor12911 Razor12911 is offline
Coder
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,154
Thanks: 1,671
Thanked 7,478 Times in 1,739 Posts
Razor12911 is on a distinguished road
Quote:
Originally Posted by elit View Post
OMG I kill you man, thats great! How come I did not know.. Time to do my games all over again .
EDIT: Well actually, thinking of it it may not be enough with lz4/zstd. I dont know how you coded it but in this case for example, default library would read block structure wrong even if compression match because of mentioned crc position which is part of the block, unless your tool is skipping it and can focus only on game data.
Still, it gave me an idea that one perhaps doesnt need to repack whole game archives with quickbms anymore, because depending on how your ztool works, maybe all I need is to modify library as I did here and ztool could function again. That would be great and would made ztool universal, then all you do is compile all lz4 versions and recycle them forever. But I would like to know more in detail how your tool work/read data exactly to understand, though its probably your secret so I am not going to ask.
Yes, tool focuses on game data, error codes are returned all the time, like you said, library reading block structure incorrectly, it happens a lot, but it still returns decompressed data.
I can tell you how it works if you want, it's no secret.
__________________
Looking for something?
Visit the
Tutorial Index
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
elit (12-01-2018)
  #13  
Old 12-01-2018, 12:34
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
...I can tell you how it works if you want, it's no secret.
That would be helpful, you can PM me or post it here whatever you prefer.

I actually tried to use lib from this version that I know works with the game and could not get it unpack anything with ztool(output same size). But I am not sure if I did it right, you see your lz4.dll have ~60kb while my compiled had around ~200+kb and I made .dll by manually changing function in makefile code - in function "add_library" where I added "SHARED" flag, something like this:
Code:
add_library(liblz4 ${LZ4_SRCS_LIB}) >> add_library(liblz4 SHARED ${LZ4_SRCS_LIB})
Now the thing is if I dont do that and just check to make lib in cmake gui, it does output library around same size as yours(~60kb), but its a .lib not windows dll. When I tried that big dll ztool did not complain with errors but output same size, if I rename lib to dll it output error(obviously not same). Maybe someone can hint me here what I did wrong, you can also try yourself if you can compile more properly.

Additionally, I found on zenhax forums that zstd may not be so bad after all, it does keep track of different version in header:
"zstd
It starts with a little endian 32bit magic number, when seen with a hex editor only the first byte (the low 8bit) is different because it depends by the version of the algorithm:
Code:
1e b5 2f fd    v0.1
22 b5 2f fd    v0.2
23 b5 2f fd    v0.3
24 b5 2f fd    v0.4
25 b5 2f fd    v0.5
26 b5 2f fd    v0.6
27 b5 2f fd    v0.7
28 b5 2f fd    v0.8, current version"
Unfortunately lz4 dont do it, but its the only one at least. Of course, there still have to be a header present in game resource to know.

Last edited by elit; 12-01-2018 at 12:38.
Reply With Quote
  #14  
Old 12-01-2018, 20:51
Razor12911's Avatar
Razor12911 Razor12911 is offline
Coder
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,154
Thanks: 1,671
Thanked 7,478 Times in 1,739 Posts
Razor12911 is on a distinguished road
Well first of all, do you know ztool's plz4 and pzstd wasn't set to be universal?
__________________
Looking for something?
Visit the
Tutorial Index
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
Jiva newstone (13-01-2018)
  #15  
Old 13-01-2018, 07:48
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 156
Thanks: 107
Thanked 223 Times in 79 Posts
elit is on a distinguished road
Quote:
Originally Posted by Razor12911 View Post
Well first of all, do you know ztool's plz4 and pzstd wasn't set to be universal?
Yeah I guess you hardcoded it in a specific way, but feel free to elaborate on it in more detail if you can, any additional info is more than welcome. I also do know that some game data packs utilize various kind of obfuscation and encryption. But since you mentioned that external libs were made intentionally so that user can replace them with different version for different games, I assumed it was designed to be more universal at least for basic archive formats.

Aside of that, I just realized why even my modified version may not work here with ztool. I modified compression function, but not decompression - which still likely read block's incorrectly. If ztool use it to verify data, no wonder it wont work. Of course, as I said any additional info regarding this issue that may help me understand your tool better is more than welcome.

Last edited by elit; 13-01-2018 at 07:52.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Ultimate Conversion Compressor (UCC) vollachr Conversion Tutorials 38 13-06-2018 00:42
PSX Modchip FAQ Ne0 PSX Games 1 07-08-2009 08:18
QUERY: editing CloneCD images & best for game data & music CDs andwan0 CD/DVD Software & Utilities 2 18-05-2009 18:13
CD/DVD Data Recovery 123walter CD/DVD Software & Utilities 1 05-08-2005 21:53
99min CD-R's, my great experience w/Audio & Data abbadon CD/DVD Recordable Media 10 15-05-2002 23:16



All times are GMT -7. The time now is 15:59.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.
Copyright 2000-2018, FileForums @ https://fileforums.com