XTool 2020 (Main Project)

[shazzla]

It would be nice if someone document this parameters....

[Masquerade]

Originally Posted by shazzla

It would be nice if someone document this parameters....

CMD> xtool precomp

XTool is created by Razor12911

precomp - data precompressor

Usage:
  xtool precomp [parameters] input output


Parameters:
  -m#  - codecs to use for precompression (separate with "+" if more than one)
  -c#  - scanning range of precompressor [16mb]
  -t#  - number of working threads [50p]
  -d#  - scan depth [0]
  -dd  - use stream deduplication
  -l#  - compress data using fast lzma2 (separate params with ":")
               d# - dictionary size
  -lm  - low memory mode
  -s   - skip stream verification
  -v   - enables verbose
  -df# - set xdelta threshold to accept streams [5p]
  -x#  - extract streams to directory path
  -dm# - deduplication memory usage limit (#=size) [75p]
  -sm# - srep memory usage limit (#=size) [75p]

[Razor12911]

Executing xtool with the command as the only parameter will display information of how the command is used alongside the available parameters. The reason certain parameters remain undocumented is simply because they were added for personal usage or as a gimmick and not a lot of testing has been done and as such, they may not work as intended and may corrupt your data in certain instances.

Any of the listed parameter is fine to be used by you, the end user however anything not listed here is limited to advanced users who may have gone through the source code over on github to even know of these parameters existence.

[shazzla]

Thanks,but i miss exactly those undocumented parameters!

Somehow i missed github. lol
Going there now,btw thanks!

[Razor12911]

Update available

Changes

- memory usage optimizations

Notes

Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1

XTool is created by Razor12911

Streams: 1415315 / 1415335
Time: 00:07:20 (CPU 00:44:37)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

0.7.2

XTool is created by Razor12911

NVIDIA GeForce GTX 1060 6GB (4.50 GB loaded)
Streams: 1415315 / 1415335
Time: 00:05:31 (CPU 00:46:09)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

[Masquerade]

Nothing short of incredible, always love seeing XTool updates. Thanks again Razor!

[infovs]

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

[Razor12911]

Originally Posted by infovs

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

Hi, optimise was added but not documented simply because the conditions needed for the benefits to even be noticeable are obscure, essentially the game Returnal was the reason this feature was added. The game used leviathan which takes a long time to process, to put it into perspective, if a stream was compressed using level 9 and let's say it takes 20 seconds to process, what optimise does is try lesser compression settings that give the same output, like level 8 which takes half the time on average compared to level 9 so that's about 10 seconds, 50% gain in speed, it does this again and again until the compression setting no longer gives the same output. As a result, some level 9 compressed streams of Returnal could be compressed with level 6, some level 7.

With this in mind, the installation should be faster right? Well yes it is but I wasn't impressed by the improvements so I left it undocumented.

Benchmark on pakchunk2_s8-WindowsNoEditor.pak of Returnal

without -o

XTool is created by Razor12911

Time: 00:22:17 (CPU 02:12:41)

with -o

XTool is created by Razor12911

Time: 00:18:50 (CPU 01:52:02)

We got about 15% speed improvement. You might see this as a win, but precompression took 32 mins with -o than 23 mins without -o so 40% more time for precompression.

As for srep/srep64, just let 32-bit programs die.

[Razor12911]

Update available

Changes

- Minor bug fixes

[infovs]

Originally Posted by Razor12911

Update available
Changes
- Minor bug fixes

Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?
And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only..
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards

[Razor12911]

Originally Posted by infovs

And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only..
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards

Initially only the x86 libraries existed and after the developer of reflate returned years after, he provided x64 versions however they function the same. (Also the libraries included in 0.3.21 are actually x64, that xtool release was also x64 )

Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?

I've actually added Denuvo to limit installation of xtool compressed games to five per day.

[infovs]

Originally Posted by Razor12911

I've actually added Denuvo to limit installation of xtool compressed games to five per day.

Glad to see you added older Denuvo version otherwise xtool.exe size would be like 1GB ..

Joking aside, regarding x86 hif2raw_dll.dll/raw2hif_dll.dll..you're right - 0.3.21 version did not contained x86, 0.3.19 did but only x86 xtool.exe.
x86 hif2raw_dll.dll/raw2hif_dll.dll libs versions I'm mentioning are contained with xtool_1908_R6, they seems newer than ones distributed with 0.7.3, hence the confusion

[Razor12911]

Update available

Changes

- user can specify srep parameters to use via -sp#
- updated zlib codec

Notes

the -sm# parameter is removed, now if you want to control the memory srep is bound to use, you use refer to the list of srep's parameters and use it as such -spmem75p or whatever it is you prefer, -sp is also available when encoding in order to specify additional srep parameters like the user Gehrman has requested. -spa0:l512.

The executable got bigger because I've been busy trying to add some features to the program that have not come into fruition but if you're a nosy person, you can check the source code to see what caused the exe to get big.

[KaktoR]

What is wrong here? I don't get it.

Streams: 12050 / 12446
Time: 00:00:03 (CPU 00:00:18)

Size: 78.9 MB >> 211 MB

Compressed 2 files, 84,074,653 => 84,074,928 bytes. Ratio 100.00%
Compression time: cpu 0.06 sec/real 2.99 sec = 2%. Speed 28.16 mB/s
All OK

[External compressor:xtool_test]
header = 0
packcmd   = xtool.exe precomp -mscarletnexus+zlib -c128mb -t100p -d1 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

[Gehrman]

Thank you Razor12911 for the new version


XTool 0.7.4 test
Forza Horizon 5

134 GB > 198 GB > 78.8 GB

Creating archive: FH5PE - 1.594.508 - 001.Bin using xtool:c32mb:mzlib:dd3:spa0:hash-+4x4:lzma:fb273:lc8:mfbt4:64m+diskspan:1395864372:1557135360
Compressed 17,216 files, 143,891,581,840 => 1,395,864,412 bytes. Ratio 0.97%
Compression time: cpu 477.42 sec/real 28649.30 sec = 2%. Speed 5.02 mB/s
All OK
CT: 07:57:29

Tested 17,216 files, 1,395,864,412 => 143,891,581,840 bytes. Ratio 0.97%   
Directory 171,831 => 873,633 bytes. Ratio 19.67%
Testing time: cpu 181.02 sec/real 6340.69 sec = 3%. Speed 22.69 mB/s
All OK

DT: 01:45:40 (In another compressed version that used XTool 2020, the extraction time was 1 hour. It seems that this version has decreased in decompression speed.)

a problem
Heavy CPU consumption in Normal CPU Priority mode, which causes Mouse Lag.
This problem does not exist in "Below Normal" CPU Priority.