XTool 2020 (Main Project)

[Razor12911]

Originally Posted by FitGirl

I suspect it depends on number of duplicated streams

you answered yourself

but as I said, I have a few ideas to reduce the memory requirements like getting rid of streams from memory that have restored the duplicated streams. I am not sure if srep does this already but if it doesn't then it should be a bonus. GPU memory usage is another thing that will be added but in future. I need start adding support for other codecs, especially the external ones. What I plan is for people to write their own plugins regardless whether they can write code or not. If you can figure out a header structure of a game, just write and ini file and give it to xtool and it will do the rest. This is mostly for lz4 games, encrypted ones or ones that require something special. I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

Here's an example of Saints Row IV Remastered, instead of making another side project, you just write this and give it to xtool.

[stream1]
Codec=lz4hc:l10 // lz4hc compression used, level 10
Header=00BADBEE0FEEDBEE // this is the magic bytes
Structure={Header(8)} {CSize(4)} {DSize(4)} {Stream} // this is the header structure
Condition1=DSize > CSize // the conditions for xtool to accept the stream
Condition2=CSize > 64
Condition3=CSize < 16 x 1024 x 1024

Maybe it's possible for xtool to report it after initial decompression is done?

this I can do


@everyone

here are some benchmarks, I finally written the decompression code, though it needs some tweaking before I can post update

normal xtool

Compressed 1 file, 320,782,304 => 108,018,914 bytes. Ratio 33.67%
Compression time: cpu 0.47 sec/real 22.64 sec = 2%. Speed 14.17 mB/s

Tested 1 file, 108,018,914 => 320,782,304 bytes. Ratio 33.67%
Testing time: cpu 0.38 sec/real 14.91 sec = 3%. Speed 21.51 mB/s

+ memory database
+ deduplication

Compressed 1 file, 320,782,304 => 107,858,828 bytes. Ratio 33.62%
Compression time: cpu 0.44 sec/real 15.98 sec = 3%. Speed 20.08 mB/s

Tested 1 file, 107,858,828 => 320,782,304 bytes. Ratio 33.62%
Testing time: cpu 0.44 sec/real 9.32 sec = 5%. Speed 34.41 mB/s

thus far, the output is slightly better for more speed