Hi, the older versions of xtool and the current version have different code infrastructure and porting the deduplication code would be time consuming, also a standalone deduplication program wouldn't make sense simply because the xtool version is stream based meaning, it relies on whatever streams are detected, however if you just wanted deduplication in stdio mode, you can just use srep in -m0 mode rather than -m#f, not sure if that answers your request :)

Update available

Changes

- added ability to redirect base directory for plugins and libraries
- added restrictions to avoid errors with experimental codecs
- added optimize option to speed up the decoding process for zstd and oodle codecs
- added dictionary parameter for fast-lzma2
- added memory caching when decoding to alleviate speed bottleneck
- fixed bug with download feature for inputs in URL format
- fixed issues with exporting precompression database
- fixed issues with executable plugin support
- fixed issues advanced configuration based plugin support
- fixed potential decoding issue upon using plugin support functions
- fixed issues with deduplication feature
- fixed issues with jojpeg codec
- replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature
- replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios
- improved user interface
- improved oodle codec performance for 2.6.0+ libraries
- improved encoding speed when using internal codecs
- improved processing speed when depth is used
- removed fast lzma2 multi threaded decompression due to excessive memory requirements
- removed debugging information when using the patch function
- removed ability to toggle database feature and ability to export database files (now enabled by default)
- updated deduplication virtual memory allocation
- updated reflate codec to verify streams prone to data corruption

Notes
Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools.
Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added.

Speed when using -d# parameter has been improved as show below on the game Star Ocean The Divine Force's chunk_2.cpk file.

0.6.9
0.7.0

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

Something is wrong with Xtool 07 hotfix.

It doesnt catch up any stream on Snowrunner's EDITOR.PAK ,while Xtool 069 does ! (069 fails at the end on this file with reflate ,zlib works fine !)
See below :

......

Edit #2 :
Ohh ,maaan! :D ALL DLL's must be in the libraries folder ! Not only the external ones ,like unreal ,unity,etc... :D

I'm not sure what the question is, but the information shows up when you use the program w/o FA.

@shazzla

be mindful of base directory or -bd#, this allows xtool to relocate libraries and plugins (including xtl and ini files) from another directory

I added this because there were just too many files near xtool.exe, also if you want to compress games but rather than moving and deleting oo2core..., you can just change the base directory.

This "base directory or -bd#..." is not so clear...
How to use/set -bd# ?

Let's say you want want to move all libraries and plugins to a folder called "libs" which is near xtool.exe, you specify base directory as -bdlibs or -bd.\libs. or you can just leave it empty to make xtool work the same as before.

Sorry for dumb question, but how get this info in xtool?

With these settings
https://i.imgur.com/KrVjKWN.png

The information is always visible and verbose option isn't necessary because that will force xtool to use 1 thread, whether you use GUI or CLI. It's always there.

There's a feature I'd like to point out which was briefly added in 0.6.8 but wasn't finished until this current release and that is on the fly compression, basically if you wanted to quickly find out the theoretical compressed size of a game without consuming disk space.

Reasons you may want to use this feature:
+ If you just wanted to find out the compressed size
+ If a game uses kraken and wanted to see if it was worth it to compress it at all
+ If the game is big and you don't have space for compression
+ If you don't want to reduce lifespan on SSD

https://i.ibb.co/1n8BL2G/otfcom1.png
13GB is the size that srep is expecting, so you change duplication preset to srep, then set input size to at least 15GB then enable compression, set desired level and dictionary.
https://i.ibb.co/L0Kdzrs/otfcom2.png
So then you know that 15.3 GB becomes 4.65 GB in less than 5 mins but without creating any temp files as the whole compression takes place in memory. You also know srep decompression memory too.

_________________________________

The other thing, as you know how much I tried to make xtool to perform as fast as possible, the thing that always bothered me was that lzma was slow and that's why I added fast-lzma2. I noticed something interesting about this compression library in the sense that it's able to increase the number of threads used for compression without increasing memory usage.

Here's a few results

4x4:lzma (via fazip) (b128mb:t6), ultra, 64mb dictionary, 12 threads used 2781 MB ram
fast lzma2, level 10, 64mb dictionary, 12 threads used 403 MB ram
Not only is memory used by fast lzma2 low, but it's much faster while producing similar results and what is impressive is, if you had like 32 thread cpu, you can increase its speed even further without increasing memory usage by too much but if you use the 4x4 method, the memory usage will increase even more.

Thank you Razor12911 for this fantastic tool...have one question, if it is not too much trouble, can you please consider naming x64 srep.exe (since xtool is hardcoded using this exact name) like Bulat original named it as srep64.exe?
I'm using original naming for many years, I need 32bit srep.exe for some compatibility reasons of my own...and your renaming 64bit srep64.exe as srep.exe is causing problems.
Maybe at least consider making xtool using srep64.exe name first, if found, then if not found use srep.exe or whatever srep name you consider acceptable.

Thanks in advance,
best regards

Final compressed size without srep?? Better or worser?

FLZMA2: This really? Small memory footprint + utilize extra more cores? :eek: (This feature available from native 7-Zip releases?)
Likelys the feature from LOLZ. More 4-12C/4-24T CPU's from in visible this ONE virtually core utiiized during in compression. (Faster speed in LDMF1 mode.)

Guys, does anyone know how to fix this?
Game: A Plague Tale Requiem
Files: *.DPC
Xtool: last ver 0.7.0 hotfix
Problem: after unpacking 1 file (28.9gb) xtool goes beyond its allocated memory. (x4)
The picture shows the information from windows resource monitoring. (I have 16 GB RAM)
https://i.ibb.co/tBFnGqR/111759.png
In 1 case it happens as shown in the picture. In 2, the memory increase can be seen even in the windows task manager.

arc.ini:
-----------------------
upd. (14 may): preflate has the same problem | xtool 0.6.9 - same
-----------------------
upd. (15 may): fixed by 2nd xtool and -dm20p -sm20p parameters.
I think the problem was that I used 1 xtool for both zlib and dedup
-----------------------

Update available

Changes

- fixed issues with fast-lzma2 being unable to set correct compression level
- updated deflate stream scanner

Forgive me if I have missed something obvious, but the first post in this thread mentions documentation, but I'm not able to locate any documentation for xtool. Specifically, I'm trying to figure out the command line. I can get the results I want from the GUI plus a plugin, but ultimately I'd like to pair it with a batch file that does the same thing.

xtool.chm is what you are looking for.

Oh wow, it really was right there all along. Thanks!

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

I think the parameter for srep memory usage in XTool 0.70 is "-sm#" and not anymore "--srepmem=#".

It would be nice if someone document this parameters.... :)

:confused::confused:

Executing xtool with the command as the only parameter will display information of how the command is used alongside the available parameters. The reason certain parameters remain undocumented is simply because they were added for personal usage or as a gimmick and not a lot of testing has been done and as such, they may not work as intended and may corrupt your data in certain instances.

Any of the listed parameter is fine to be used by you, the end user however anything not listed here is limited to advanced users who may have gone through the source code over on github to even know of these parameters existence.

Thanks,but i miss exactly those undocumented parameters! :D

Somehow i missed github. lol
Going there now,btw thanks!

Update available

Changes

- memory usage optimizations

Notes

Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1
0.7.2
There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

Nothing short of incredible, always love seeing XTool updates. Thanks again Razor!

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

Hi, optimise was added but not documented simply because the conditions needed for the benefits to even be noticeable are obscure, essentially the game Returnal was the reason this feature was added. The game used leviathan which takes a long time to process, to put it into perspective, if a stream was compressed using level 9 and let's say it takes 20 seconds to process, what optimise does is try lesser compression settings that give the same output, like level 8 which takes half the time on average compared to level 9 so that's about 10 seconds, 50% gain in speed, it does this again and again until the compression setting no longer gives the same output. As a result, some level 9 compressed streams of Returnal could be compressed with level 6, some level 7.

With this in mind, the installation should be faster right? Well yes it is but I wasn't impressed by the improvements so I left it undocumented.

Benchmark on pakchunk2_s8-WindowsNoEditor.pak of Returnal

without -o
with -o
We got about 15% speed improvement. You might see this as a win, but precompression took 32 mins with -o than 23 mins without -o so 40% more time for precompression.

As for srep/srep64, just let 32-bit programs die. :rolleyes:

external ogg codec/compressor
 

Hello everyone,
first of all thank you Razor12911 and eveyone else who made this possible.
Im currently experimenting with xtool and noticed no dedicated handling of ogg files. Im trying to compress some of my minecraft backups where mods with ogg sound files contribute a significant share.
I have already extraced some and compressed them with standalone oggre. Now i want to include them the same way panker1992 (i recreated his wavpack config and it worked ;D) did but im not really a hex guru. Did anyone already have a working ogg config ?
So far ive found the Signature but im not sure what to fill in for the rest of the ini values:

^^
I'd advise against using XTool with oggre because oggre has built in deduplication that will not be working when each ogg is compressed individually with XTool.

Do you know any other ogg compressors ? Either way running it on single stream still resulted in a noticable reduction and i would see it as an example of how to correctly fill out these ini files. Can you help me with that ? From the xtool chm im not quite able to do that yet.

Update available

Changes

- Minor bug fixes

Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?
And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only.. :)
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards

Initially only the x86 libraries existed and after the developer of reflate returned years after, he provided x64 versions however they function the same. (Also the libraries included in 0.3.21 are actually x64, that xtool release was also x64 :confused:)

I've actually added Denuvo to limit installation of xtool compressed games to five per day. :rolleyes:

Glad to see you added older Denuvo version otherwise xtool.exe size would be like 1GB .. :p :rolleyes:

Joking aside, regarding x86 hif2raw_dll.dll/raw2hif_dll.dll..you're right - 0.3.21 version did not contained x86, 0.3.19 did but only x86 xtool.exe.
x86 hif2raw_dll.dll/raw2hif_dll.dll libs versions I'm mentioning are contained with xtool_1908_R6, they seems newer than ones distributed with 0.7.3, hence the confusion :confused:

Used -a0 in many times ago! :D

Try it -m5f mode, its minimally smaller memory eat than -m3f. And combined in -a0. This m5f:a0 smaller memory footprint from decompression than m3f:a0??

In last newer ~20kB sized hif/raw dlls, in have different results than older ~80kB sizd versions. 50% case bigger inflated/compressed results.
The one problematic parts, from x86 build packages, the "preflate_dll.dll" file, not compatible from WinXP and not working from any precompressors usage. (ex: ZREC) Still exist working one in have preflate library available in many precomp 0.4.8 and older versions. (x86 releases.) ;)

Update available

Changes

- user can specify srep parameters to use via -sp#
- updated zlib codec

Notes

the -sm# parameter is removed, now if you want to control the memory srep is bound to use, you use refer to the list of srep's parameters and use it as such -spmem75p or whatever it is you prefer, -sp is also available when encoding in order to specify additional srep parameters like the user Gehrman has requested. -spa0:l512.

The executable got bigger because I've been busy trying to add some features to the program that have not come into fruition but if you're a nosy person, you can check the source code to see what caused the exe to get big.

What is wrong here? I don't get it.

Thank you Razor12911 for the new version


XTool 0.7.4 test
Forza Horizon 5

134 GB > 198 GB > 78.8 GB

DT: 01:45:40 (In another compressed version that used XTool 2020, the extraction time was 1 hour. It seems that this version has decreased in decompression speed.)

a problem
Heavy CPU consumption in Normal CPU Priority mode, which causes Mouse Lag.
This problem does not exist in "Below Normal" CPU Priority.

The log information via xtool when repacking FH5:
Something you have to keep in mind when repacking Forza Horizon 5 is that this game has a lot of duplicates because its also a big game. If you use srep, it will request 20 GB for decompression memory or (5 GB if you used -dd while xtool itself will request 8.5 GB, about 13 GB overall, ~7 GB memory savings) and since you made a post that you wanted to control srep's a# parameter, I can only assume that when you ran decompression before, you had more system memory available than when you ran it this time.

The speed reductions seems to be related to you not having enough memory to install the game without srep or xtool making virtual temp files which incur speed penalty.

Games like these are the reason I added GPU memory feature in xtool because it would offload some memory onto the GPU which typically isn't used when installing games (if you have 8GB GPU, xtool would require 1GB instead of 8.5 GB, while 7.5GB will be on the GPU then installation would need 6GB instead of 20 GB which srep initially required), though not ready because it's not thoroughly tested on a lot of systems to see if it works without issues. :(

I ran a test between 0.7.4 against older releases, they all produce the same results and have the same speeds when decoding (at least within margin of error) and 0.7.0+ encodes faster. Decoding speed in 0.7.4 is slightly slower but reasonable because of memory usage optimisations that were made in 0.7.2, all versions used 220 MB+ for decoding while 0.7.4 used 73 MB.

ForzaHorizon5\DLC\MicrosoftExpansion2FH5\media\Tra cks\Falcon\bin.zip