XTool 2020 (Main Project)

[Razor12911]

Update available

Changes

- memory usage optimizations

Notes

Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1

XTool is created by Razor12911

Streams: 1415315 / 1415335
Time: 00:07:20 (CPU 00:44:37)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

0.7.2

XTool is created by Razor12911

NVIDIA GeForce GTX 1060 6GB (4.50 GB loaded)
Streams: 1415315 / 1415335
Time: 00:05:31 (CPU 00:46:09)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

[Masquerade]

Nothing short of incredible, always love seeing XTool updates. Thanks again Razor!