XTool 2020 (Main Project)

[KaktoR]

Maybe files are crypted?

[Masquerade]

Originally Posted by KaktoR

Maybe files are crypted?

The files are definitely encrypted. Sorry for the problems...

500mb sample ---> 450mb when 7z compressed.

[shazzla]

If there r oodle libs then those streams r oodle streams imho.
Doesnt matter what scanners says its zlib.
Try with different oodle libs.
Again : imho. But maybe im very wrong...

@Kaktor : +1

[Razor12911]

Next update will have two new features, database and stream deduplication and here are some benchmarks as to what these features can do.

Input is Need for Speed Hot Pursuit\SEACREST

pzlib v2
Compressed 225 files, 3,253,110,016 => 859,932,872 bytes. Ratio 26.43%
Compression time: cpu 5.73 sec/real 300.50 sec = 2%. Speed 10.83 mB/s

pzlib v3
fails, no idea why

xtool2018 (v012)
Compressed 225 files, 3,253,110,016 => 722,889,785 bytes. Ratio 22.22%
Compression time: cpu 5.06 sec/real 188.09 sec = 3%. Speed 17.30 mB/s

xtool2019 (1908_R6)
Compressed 225 files, 3,253,110,016 => 722,934,572 bytes. Ratio 22.22%
Compression time: cpu 4.89 sec/real 189.14 sec = 3%. Speed 17.20 mB/s

xtool2020
Compressed 225 files, 3,253,110,016 => 722,927,890 bytes. Ratio 22.22%
Compression time: cpu 4.86 sec/real 165.26 sec = 3%. Speed 19.68 mB/s

xtool2020 with deduplication
Compressed 225 files, 3,253,110,016 => 719,194,176 bytes. Ratio 22.11%
Compression time: cpu 4.08 sec/real 154.45 sec = 3%. Speed 21.06 mB/s

xtool2020 with deduplication + memory database
Compressed 225 files, 3,253,110,016 => 719,195,766 bytes. Ratio 22.11%
Compression time: cpu 4.20 sec/real 75.00 sec = 6%. Speed 43.38 mB/s

xtool2020 with deduplication + imported database
Compressed 225 files, 3,253,110,016 => 719,194,905 bytes. Ratio 22.11%
Compression time: cpu 4.50 sec/real 61.77 sec = 7%. Speed 52.66 mB/s

precomp
Compressed 225 files, 3,253,110,016 => 754,538,183 bytes. Ratio 23.19%
Compression time: cpu 6.69 sec/real 547.05 sec = 1%. Speed 5.95 mB/s

In precomp's defence, it uses disk instead of memory to work and stream type was never narrowed to only focus on deflate streams.

All tests ran using 4 threads with 32mb chunk size

NOTE: This input has repeated streams, some games have similar characteristics therefore xtool stores a database in memory to remember the configuration and then removes these repeated streams to give more speed.
It's free speed, there is no penalty in performance if you use the new options at all times however, some tests need to be run due to the possibility of collisions in the dictionary used...

This project has come a long way

[FitGirl]

Cool update, waiting impatiently
Any predictions on RAM requirements for both imported/memory DBs? I suspect it depends on number of duplicated streams, but it would be good beforehand to know what additional RAM will be required upon recompression. Maybe it's possible for xtool to report it after initial decompression is done?

[Razor12911]

Originally Posted by FitGirl

I suspect it depends on number of duplicated streams

you answered yourself

but as I said, I have a few ideas to reduce the memory requirements like getting rid of streams from memory that have restored the duplicated streams. I am not sure if srep does this already but if it doesn't then it should be a bonus. GPU memory usage is another thing that will be added but in future. I need start adding support for other codecs, especially the external ones. What I plan is for people to write their own plugins regardless whether they can write code or not. If you can figure out a header structure of a game, just write and ini file and give it to xtool and it will do the rest. This is mostly for lz4 games, encrypted ones or ones that require something special. I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

Here's an example of Saints Row IV Remastered, instead of making another side project, you just write this and give it to xtool.

[stream1]
Codec=lz4hc:l10 // lz4hc compression used, level 10
Header=00BADBEE0FEEDBEE // this is the magic bytes
Structure={Header(8)} {CSize(4)} {DSize(4)} {Stream} // this is the header structure
Condition1=DSize > CSize // the conditions for xtool to accept the stream
Condition2=CSize > 64
Condition3=CSize < 16 x 1024 x 1024

Maybe it's possible for xtool to report it after initial decompression is done?

this I can do


@everyone

here are some benchmarks, I finally written the decompression code, though it needs some tweaking before I can post update

normal xtool

Compressed 1 file, 320,782,304 => 108,018,914 bytes. Ratio 33.67%
Compression time: cpu 0.47 sec/real 22.64 sec = 2%. Speed 14.17 mB/s

Tested 1 file, 108,018,914 => 320,782,304 bytes. Ratio 33.67%
Testing time: cpu 0.38 sec/real 14.91 sec = 3%. Speed 21.51 mB/s

+ memory database
+ deduplication

Compressed 1 file, 320,782,304 => 107,858,828 bytes. Ratio 33.62%
Compression time: cpu 0.44 sec/real 15.98 sec = 3%. Speed 20.08 mB/s

Tested 1 file, 107,858,828 => 320,782,304 bytes. Ratio 33.62%
Testing time: cpu 0.44 sec/real 9.32 sec = 5%. Speed 34.41 mB/s

thus far, the output is slightly better for more speed

[dixen]

Originally Posted by Razor12911

I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

For LEGO series games this would be fine)

[FitGirl]

Originally Posted by dixen

For LEGO series games this would be fine)

It will never work with LEGO series, cause there are no public compressors for algos used in those games, and the decompressors used by Luigi are in binary (ripped from DLL) format, so they can only be extracted, not packed back.

[Razor12911]

Originally Posted by FitGirl

Any predictions on RAM requirements for both imported/memory DBs?

back on the SEACREST input from Need For Speed Hot Pursuit

I ran a memory usage test as you asked and these were the results.

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

Decompression memory is 1485 mb. 147,537 matches = 2,360,592 bytes = 0.18% of file

= total 1485 MB (without deduplication)

NOTE: Xtool uses memory of its own and this was not factored, about 190 MB
so if you do the math, that's 350+190 = about 540 MB total memory while if xtool doesn't remove deduplicates and lets srep all the work that's 1485+190 = 1675 MB

540 MB vs 1675 MB memory usage, which is better?

when 540 MB ram was used, xtool was decoding at 88.91 MB/s
and when 1675 MB ram was used, xtool decoded at 27.45 MB/s

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine )

[Masquerade]

Originally Posted by Razor12911

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?

[FitGirl]

Originally Posted by Masquerade

I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?

The idea of new xtool is not replace srep. It's purpose is not to recompress duplicate chunks, but just copy them, which makes less CPU load and faster overall decompression. You still need to run srep after xtool, cause it will deal with unpacked duplicated chunks.

[Masquerade]

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB



So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

[Harsh ojha]

Originally Posted by Razor12911

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine )

Same here potato pc 😅

FitGirl Repack take more HHD space for installing game ex. Far Cry 3 duology
Need 17 GB HHD space free
But the game Size was 10 GB ( language english.
Without FC3 mapEditor )

*Sorry for my bad English as English is not my mother tongue 😅*

[FitGirl]

Configuration feature is great! It would be so much easier to repack some obscure formats.

[Razor12911]

and the final size of 2020?