QuickBMS: File Extractor and Reimporter

[Gupta]

Files extractor and reimporter, archives and file formats parser, an advanced tool for reverse engineers and power users, and much more.

Features:
open source and multiplatform, tested on Windows (even Win98) and GNU/Linux (x86 and PPC)
works from both command-line and GUI (Windows only)
reimport feature for reinjecting the modified files back to the archives (if smaller/equal than the original files)
support for tons of encryption algorithms, even some proprietary ones
support for tons of compression algorithms (over 500), even some proprietary ones
support for tons of hashing algorithms
support for other types of algorithms (like base64) and/or obfuscations (xor, rot and so on)
support for calling DLLs and raw dumped functions with almost any known calling convention
support for bits operation and switchable little/big endian
simple and dynamic language that allows to make many operations reducing the percentage of archives and formats that can't be supported easily
verbose option (-V) that displays all the needed details during reversing and testing of file formats
HTML output (-H) that automatically applies colors and names to the fields parsed during the usage of the scripts: example for zip.bms on q3infoboom.zip
possibility to use the tool as a blind scanner of compression, encryption and crc/checksum algorithms
support for network sockets and SSL
support for other alternative input/output interfaces like processes (included debugging and automatic breakpoint restoring), audio, video and Windows messages
support for the visualization and creation of various types of data like x86 assembly, IP addresses, time_t, FILETIME, ClassID and more
support for C-like structures and basic syntax for easy handling of file formats
support for the WCX packer plugins of Total Commander
support for any command-line decompressor/decrypter/anything_else via the EXECUTE method (Comtype and Encryption commands)
read and write operations
quickbms_4gb_files.exe is a native 32bit program with all the variables set as 64bit useful in some situations
planned future development that will cover more advanced features for reverse engineers and power users
customizable CRC engine
tons of scripts supporting hundreds of file formats and thousands of games
experimental games localization (strings editing)
creation of ISO images and ZIP archives as alternative outputs

Source Code / Linux:
:quickbms_src.zip
the full source code of the tool, it's separated from the executables to reduce the size of the download and avoiding confusion to the users.
on Linux it's enough to go in the src folder and launch make.

Download Executable:
:quickbms.zip

Scripts for Specific Games And HomePage itself can Be Found Here
:http://aluigi.altervista.org/quickbms.htm

YOUTUBE TUTORIAL

[elit]

Its good program but we need something to reinject/reimport back from scratch without keeping existing pack files. I believe FitGirl use modified version of quickbms that have this ability but its her private tool.

[Gupta]

use quickbms -r (see reimport.bat)
or use razor's file injector

PS: you can replace the files to extracted in the original file with 0s to achieve the better compression
PPS: some scripts also transforms raw data to other formats like DDS etc which can't be injected as they are you have to preserve the rawdata

[elit]

I tried quickbms not long ago to recreate pack file from scratch(read reinject data files into *new* pack file) with "-r", but it gave me error that I need original pack file for output. For that reason I believe you cannot use it to extract/uncompress data files, delete original pack file and then recreate it from scratch. I think you meant that instead of deleting pack file, you "zero" it inside with that option to be able to compress it to almost nothing and ship it along for later reinjection? So just to keep its structure hierarchy with empty data?

As for Injector Maker I know it can create from scratch and I saw video how to do it with .forge files, but I thought its not universal for other formats and especially in conjunction with quickbms extract? Because quickbms also decompress individual data inside. I am not sure how would Injector Maker handle this. As you say you have to preserve raw data but I havent found an option in quickbms to not decompress. If somebody can step in here and give a few tips/examples I would really appreciate it, this is the only thing limiting me after spending lot of time understanding SREP/ARC/LZMA.

Also one more thing, I had problems re-importing using quickbms way too often even when it unpacked data without error, thats why I never felt very reliable about it. Errors and/or warnings like, I think even some files skipped. That would be problem if you "zero" on unpack and then it fail or skip to re-inject some files randomly?

[elit]

OMG I just did it and it work! I was able to repack in FitGirl-style game(Raiders of the Broken Planet) by repacking its resource files. But! Not only simple quickbms command with default script like everybody else can do, I had to modify it for reimporting because memory mapped import doesnt work on that game. So I ended up learning about BMS scripting language, headers/flags, compression etc. You know for a n00b like me this was quite achievement, so glad... and finally can actually have a sleep.

Big thanks PrinceGupta2000 for hinting on zeroing data pack capability of quickbms, that was the trigger .

PS(For those curious how I was able to put " compressed blocksized" pack back together, simple, I used normal "clog" without fragmenting through memory mapped "append" command. I expected it not to work and was ready to modify header flags but alas, it wasnt necessary. Game use lz4 btw(ztool wont work), original script for extract is castlevania.bms)

[elit]

I am not sure if I should not had opened new topic, but since it is related to my previous posts here I will reply here for now.
So regarding my "reimporting journey", that I spent pretty much these holidays on, I found that most pack files worked fine with above method except 2: Data02.packed and Data00.packed. I dont like half-backed job even though these 2 represent no more than 25% of game. So I started experiencing custom repacking with lz4 command line compressor and almost came to same data.

The thing is, original dump files have different position of the block crc - in beginning, but latest command line compressor is putting it at the end of the block.
Example:

- original: 22 16 00 00 33 9D BE 62 FF 08...(data)
^First 4 bytes are (compressed size - 8 bytes), next 4 bytes is block crc, then goes typical token and data.

- repacked(omitting first 4 bytes header): 70 40 AD 14 16 00 00 FF 08...(data)
^here size is right before FF 08, or 14 16 00 00. Block crc in this case would be located at the end of the data block not between "size" and the "token/signature"

Also every original file dump regardless of size have this exact 8 byte(4byte "size" followed by 4byte crc) structure.
Can someone tell me which version of lz4 had this format? I am sure its not a matter of different command options.
Thanks a lot.

[Gupta]

In a nutshell:
Most probably streaming compression,
If i can recall it correctly there is option '--no-crc'
In commandline version if not play around with the api ur self, if don't wanna do it either use diff bt remember if diff are too big it doesn't make sense to use it, try to reproduce dumps as near as possible
I myself will play around with streaming api,,, they are well documented and very easy

If u r sure lz4 is giving you original dump just not with the crc , log them and append them to dumps yourself and then inject them

[elit]

It is chunk streaming indeed, of 256kb's.

Crc is needed, its just that command line is putting it in wrong place(after block not before it). I am considering recompiling lz4.exe with changes but if my basic changes give compile/runtime error I dont know if I have a stamina to go through all code.

I was considering xdelta but found that compressed data are still not exact, they are almost but very few variances still, but any other settings combination went only further away. Thats why i need new crc not just dump original data with xdelta changes. I believe it should work after that.

[Gupta]

or maybe you can only replace only compressed data with 0 and leave checksum
and after compression(of already extracted lz4-unpacked dumps), copy compressed(restored) dumps leaving last 4 bytes of checksum

[elit]

Finally it seems to be working! I modded lz4 cmd tool to include crc before data and then manually removed header and tail. Tested single file that previously crashed the game worked! I am confident it should hopefully work for the rest.
But I need to modify lz4 further to remove frame header and tail as I wont do it manually for thousand files. Once done, I will create new topic and upload it here. With such lz4 users will be able to re-import certain previously non-doable games with simple quickbms+custom lz4 - ala FitGirl.

Btw Prince, that wouldnt work, crc is block crc and since compression is not exact it would crash the game. Also block crc is in each few kb's within the compressed file not only beginning and you would have to be able to recognize it among other random bytes. But I did had that in mind at beginning.

[Gupta]

I asked u if compression is exact or not and assumed it was exact for the replies

Find the exact level or use diff

[elit]

I told you its not exact:
"..but found that compressed data are still not exact, they are almost but very few variances still.."

I was not able to get same data with *any* settings combination, I believe they used completely different version. Also original lz4 author told me he never made any version where block crc is at the beginning.
And diff is no solution because of crc as mentioned already.

But, things should work now with my modded lz4 tool, wait and see...