Forgive me if I have missed something obvious, but the first post in this thread mentions documentation, but I'm not able to locate any documentation for xtool. Specifically, I'm trying to figure out the command line. I can get the results I want from the GUI plus a plugin, but ultimately I'd like to pair it with a batch file that does the same thing.

xtool.chm is what you are looking for.

Oh wow, it really was right there all along. Thanks!

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

I think the parameter for srep memory usage in XTool 0.70 is "-sm#" and not anymore "--srepmem=#".

It would be nice if someone document this parameters.... :)

:confused::confused:

Executing xtool with the command as the only parameter will display information of how the command is used alongside the available parameters. The reason certain parameters remain undocumented is simply because they were added for personal usage or as a gimmick and not a lot of testing has been done and as such, they may not work as intended and may corrupt your data in certain instances.

Any of the listed parameter is fine to be used by you, the end user however anything not listed here is limited to advanced users who may have gone through the source code over on github to even know of these parameters existence.

Thanks,but i miss exactly those undocumented parameters! :D

Somehow i missed github. lol
Going there now,btw thanks!

Update available

Changes

- memory usage optimizations

Notes

Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1
0.7.2
There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

Nothing short of incredible, always love seeing XTool updates. Thanks again Razor!

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

Hi, optimise was added but not documented simply because the conditions needed for the benefits to even be noticeable are obscure, essentially the game Returnal was the reason this feature was added. The game used leviathan which takes a long time to process, to put it into perspective, if a stream was compressed using level 9 and let's say it takes 20 seconds to process, what optimise does is try lesser compression settings that give the same output, like level 8 which takes half the time on average compared to level 9 so that's about 10 seconds, 50% gain in speed, it does this again and again until the compression setting no longer gives the same output. As a result, some level 9 compressed streams of Returnal could be compressed with level 6, some level 7.

With this in mind, the installation should be faster right? Well yes it is but I wasn't impressed by the improvements so I left it undocumented.

Benchmark on pakchunk2_s8-WindowsNoEditor.pak of Returnal

without -o
with -o
We got about 15% speed improvement. You might see this as a win, but precompression took 32 mins with -o than 23 mins without -o so 40% more time for precompression.

As for srep/srep64, just let 32-bit programs die. :rolleyes:

external ogg codec/compressor
 

Hello everyone,
first of all thank you Razor12911 and eveyone else who made this possible.
Im currently experimenting with xtool and noticed no dedicated handling of ogg files. Im trying to compress some of my minecraft backups where mods with ogg sound files contribute a significant share.
I have already extraced some and compressed them with standalone oggre. Now i want to include them the same way panker1992 (i recreated his wavpack config and it worked ;D) did but im not really a hex guru. Did anyone already have a working ogg config ?
So far ive found the Signature but im not sure what to fill in for the rest of the ini values:

^^
I'd advise against using XTool with oggre because oggre has built in deduplication that will not be working when each ogg is compressed individually with XTool.