XTool 2020 (Main Project)

[Razor12911]

There's a feature I'd like to point out which was briefly added in 0.6.8 but wasn't finished until this current release and that is on the fly compression, basically if you wanted to quickly find out the theoretical compressed size of a game without consuming disk space.

Reasons you may want to use this feature:
+ If you just wanted to find out the compressed size
+ If a game uses kraken and wanted to see if it was worth it to compress it at all
+ If the game is big and you don't have space for compression
+ If you don't want to reduce lifespan on SSD

XTool is created by Razor12911

Streams: 1415335 / 1415335
Time: 00:01:21 (CPU 00:04:03)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB

Done!!!

13GB is the size that srep is expecting, so you change duplication preset to srep, then set input size to at least 15GB then enable compression, set desired level and dictionary.

Streams: 1415335 / 1415335
Time: 00:04:36 (CPU 00:34:05)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 751 MB [5.15 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.03 GB >> 4.65 GB

Done!!!

So then you know that 15.3 GB becomes 4.65 GB in less than 5 mins but without creating any temp files as the whole compression takes place in memory. You also know srep decompression memory too.

_________________________________

The other thing, as you know how much I tried to make xtool to perform as fast as possible, the thing that always bothered me was that lzma was slow and that's why I added fast-lzma2. I noticed something interesting about this compression library in the sense that it's able to increase the number of threads used for compression without increasing memory usage.

Here's a few results

4x4:lzma (via fazip) (b128mb:t6), ultra, 64mb dictionary, 12 threads used 2781 MB ram

Compressed 288 files, 16,391,993,193 => 4,869,896,897 bytes. Ratio 29.71%
Compression time: cpu 13.50 sec/real 662.85 sec = 2%. Speed 24.73 mB/s

fast lzma2, level 10, 64mb dictionary, 12 threads used 403 MB ram

Compressed 288 files, 16,391,993,193 => 4,929,451,310 bytes. Ratio 30.07%
Compression time: cpu 11.84 sec/real 425.39 sec = 3%. Speed 38.53 mB/s

Not only is memory used by fast lzma2 low, but it's much faster while producing similar results and what is impressive is, if you had like 32 thread cpu, you can increase its speed even further without increasing memory usage by too much but if you use the 4x4 method, the memory usage will increase even more.

[infovs]

Thank you Razor12911 for this fantastic tool...have one question, if it is not too much trouble, can you please consider naming x64 srep.exe (since xtool is hardcoded using this exact name) like Bulat original named it as srep64.exe?
I'm using original naming for many years, I need 32bit srep.exe for some compatibility reasons of my own...and your renaming 64bit srep64.exe as srep.exe is causing problems.
Maybe at least consider making xtool using srep64.exe name first, if found, then if not found use srep.exe or whatever srep name you consider acceptable.

Thanks in advance,
best regards