XTool 2020 (Main Project)

[Razor12911]

Update available

Changes

- added IO functions (find, extract, patch)
- generate database feature and IO functions now can search for streams larger than chunk size

Notes

More IO functions introduced. Find simply helps you track positions of extracted files from a given input, while extract uses a generated decode file data input to extract streams in case if you wanted another person to extract their very own streams using your own findings and patch, well patch compares two inputs and generates a diff file which can allow you to patch an input to make it similar to the new data.

Find

xtool find [parameters] extracted_streams original_data [decode_data]

Find can be used to search for streams to then produce decode data which you can upload here on the forum in cases where you wanted to inform people what files should be extracted if they wanted for example to separate audio files per language from a game.

An example where I wanted to make a guide of how to separate english, german, italian, russian and spanish languages from the game would be something like:

xtool find "extracted\en\*" "game\*" en.xtl
xtool find "extracted\de\*" "game\*" de.xtl
xtool find "extracted\it\*" "game\*" it.xtl
xtool find "extracted\es\*" "game\*" es.xtl
xtool find "extracted\ru\*" "game\*" ru.xtl

These files are then uploaded for people if they wanted to extract the same data by themselves by using xtool via the "extract" function or for later use by yourself.

Extract

xtool extract decode_data original_data extracted_streams

Extract works mostly hand in hand with with the find function as its job is to use the decode_data to extract the data that was used for its generation.

A use case for it would be similar to the find example but from another user's perspective, so if someone uploaded decode data to use to extract my very own streams from their investigations, rather than them giving me position ranges for languages or the tools to use, we can just use their uploaded decode data to do it ourselves

xtool extract en.xtl "game\*" "extracted\en\*"
xtool extract de.xtl "game\*" "extracted\de\*"
xtool extract it.xtl "game\*" "extracted\it\*"
xtool extract es.xtl "game\*" "extracted\es\*"
xtool extract ru.xtl "game\*" "extracted\ru\*"

Then from here, we use these extracted files, to even use the erase function introduced in the version prior to blank out the sectors to prepare our own repack

xtool erase "extracted\*" "game\*" languages.xtl

which we can then restore after installation using

xtool decode languages.xtl "extracted\*" "game\*"

Patch

xtool patch [parameters] old_data new_data patch_data
xtool decode patch_data old_data

Patch's use case can be the generation of update patches in cases where you already made a repack but for whatever reason refuse to re-repack the game so instead choose to generate a patch file which can be used to update the old repack with newer files

xtool patch "Sims4\*" "Sims4_updated\*" sims4_wedding_stories.patch

Then after the installation of the original repack, you can add an additional archive that has sims4_wedding_stories.patch which will be used to update the game

xtool decode sims4_wedding_stories.patch "Sims4\*"

I noticed after compiling xtool.exe that xdelta is set to compress these patch files which means that they cannot be precompressed nor compressed with something better, however this compression in the next version will be removed.

[Razor12911]

Update available

Changes

- added IO functions (archive, execute)
- fixed issue in patch io function
- removed compression on patch diff files

Notes

More IO functions added. Archive behaves like -m0 but allows you to the archive to stdout and read from stdin when decoding (if you ever need that). Then there is execute which allows you execute several instances of another executable in parallel mode while all their inputs are fed from one source and all their output is fed to one destination.

Archive

xtool archive files1 files2... archive

There is nothing to add here as I have personal uses for this but the example would be

xtool archive game\* mygame.xtl
xtool decode mygame.xtl extracted\*

Execute

xtool execute [parameters] input output [exec_syntax]

Too lazy to write description but, here's an example

xtool.exe execute -c64mb -t8 UI.sb UI.bin bcm.exe -b64 [filein] [fileout]
xtool.exe decode -t100p UI.bin UI_dec.sb bcm.exe -d [filein] [fileout]

The left side of the syntax is to command xtool and after specifying input and output files, the right side begins and here you can write the command line that should be used to perform execution.

[filein], [fileout], [stdin], [stdout] can be used and denote what IO the program being executed uses.
[] is used to avoid conflicting with Freearc's <>

Freearc example would look like this

[External compressor:xbcm]
header    = 0
packcmd   = xtool.exe execute { -option} - - <stdin> <stdout> bcm.exe -b64 [filein] [fileout]
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout> bcm.exe -d [filein] [fileout]

and the method would be -mxbcm:c64mb:t75p

[Razor12911]

Update available

Changes

- added png stream preprocessor
- removed grittibanzli codec (since nobody uses it)

Notes

Xtool is able to process deflate/zlib streams and png images do contain these streams however, they are split up into several blocks which at times does prevent the program from being able to process them. The png stream preprocessor's job is to concatenate (rejoin all these blocks into a single stream) which can then be processed by xtool, so if you know your input contains these images then it's best to include -mpng into the method chain and use -d1 to first preprocess the streams then process them using zlib, reflate or preflate (preflate is the preferred method to use).

Results

without png preprocessor:

Compressed 1 file, 7,232,549 => 8,291,632 bytes. Ratio 114.64% -mreflate
Compressed 1 file, 7,232,549 => 7,232,655 bytes. Ratio 100.00% -mpreflate

with png preprocessor:

Compressed 1 file, 7,232,549 => 26,075,119 bytes. Ratio 360.52% -mpng+reflate -d1
Compressed 1 file, 7,232,549 => 25,289,529 bytes. Ratio 349.66% -mpng+preflate -d1

[Cesar82]

@Razor12911, can i use the parameters in that order in freearc commandline?

xtool:mpreflate:mpng:d1

[Razor12911]

yes but you'll have to change arc.ini from { -moption} to { -option}

[Cesar82]

Originally Posted by Razor12911

yes but you'll have to change arc.ini from { -moption} to { -option}

Thanks.
Yes, I use { -option} in Arc.ini of the DSG.
I tested precompressing pngs with just "xtool:mpreflate" or with "xtool:mpreflate:mpng:d1" and got similar (near)I figured I was using the command order wrong.

[yasitha]

sorry, for asking this question here, i don't know where to ask.

someone please tell me and give me,
best .Wav (.pck) compressor
with arc.ini info and required files.
Thanks 😊

hint:
i try Pck files, on GFC.
Game File Scanner showing WAV streams. so i guess i can use Wav compressor.

[KaktoR]

msc

And I don't know what this has to do with xtool.

[yasitha]

Originally Posted by KaktoR

msc

And I don't know what this has to do with xtool.

yes, my bad, this is not related to xtool
i try msc, but wont work. can you help me.

[kj911]

yasitha: Try it use testing tta methods from FreeArc and totally ignored file extension from during compression.

This compression oks, use final packing the following switches:

tta+srep+lzma or lolz

[yasitha]

Originally Posted by kj911

yasitha: Try it use testing tta methods from FreeArc and totally ignored file extension from during compression.

This compression oks, use final packing the following switches:

tta+srep+lzma or lolz

thanks bro can you please share tta+ config required files as (.zip)
so i can try and use

thank you!!

[shazzla]

Afaik freearc's tta works on per file basis,while msc works on stream(/file)?!

[yasitha]

Originally Posted by shazzla

Afaik freearc's tta works on per file basis,while msc works on stream(/file)?!

thank you

[KaktoR]

Posting it here aswell

xtool:png+xtool:preflate+srep+lzma2
Compressed 8,178 files, 2,202,105,870 => 1,112,132,188 bytes. Ratio 50.50%
Compression time: cpu 17.70 sec/real 1593.72 sec = 1%. Speed 1.38 mB/s
All OK

precomp_mtx+srep+lzma2
Compressed 8,178 files, 2,202,105,870 => 1,120,270,164 bytes. Ratio 50.87%
Compression time: cpu 16.72 sec/real 2599.17 sec = 1%. Speed 0.85 mB/s
All OK

[yasitha]

Originally Posted by KaktoR

Posting it here aswell

xtool:png+xtool:preflate+srep+lzma2
Compressed 8,178 files, 2,202,105,870 => 1,112,132,188 bytes. Ratio 50.50%
Compression time: cpu 17.70 sec/real 1593.72 sec = 1%. Speed 1.38 mB/s
All OK

precomp_mtx+srep+lzma2
Compressed 8,178 files, 2,202,105,870 => 1,120,270,164 bytes. Ratio 50.87%
Compression time: cpu 16.72 sec/real 2599.17 sec = 1%. Speed 0.85 mB/s
All OK

what is this ?