XTool 2020 (Main Project)

[shazzla]

Something is wrong with Xtool 07 hotfix.

It doesnt catch up any stream on Snowrunner's EDITOR.PAK ,while Xtool 069 does ! (069 fails at the end on this file with reflate ,zlib works fine !)
See below :

......

Edit #2 :
Ohh ,maaan! ALL DLL's must be in the libraries folder ! Not only the external ones ,like unreal ,unity,etc...

[shazzla]

This "base directory or -bd#..." is not so clear...
How to use/set -bd# ?

[Razor12911]

Let's say you want want to move all libraries and plugins to a folder called "libs" which is near xtool.exe, you specify base directory as -bdlibs or -bd.\libs. or you can just leave it empty to make xtool work the same as before.

[dixen]

Sorry for dumb question, but how get this info in xtool?

Streams: 58080 / 58080
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

[KaktoR]

Originally Posted by dixen

Sorry for dumb question, but how get this info in xtool?

With these settings

[Razor12911]

The information is always visible and verbose option isn't necessary because that will force xtool to use 1 thread, whether you use GUI or CLI. It's always there.

[Razor12911]

There's a feature I'd like to point out which was briefly added in 0.6.8 but wasn't finished until this current release and that is on the fly compression, basically if you wanted to quickly find out the theoretical compressed size of a game without consuming disk space.

Reasons you may want to use this feature:
+ If you just wanted to find out the compressed size
+ If a game uses kraken and wanted to see if it was worth it to compress it at all
+ If the game is big and you don't have space for compression
+ If you don't want to reduce lifespan on SSD

XTool is created by Razor12911

Streams: 1415335 / 1415335
Time: 00:01:21 (CPU 00:04:03)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB

Done!!!

13GB is the size that srep is expecting, so you change duplication preset to srep, then set input size to at least 15GB then enable compression, set desired level and dictionary.

Streams: 1415335 / 1415335
Time: 00:04:36 (CPU 00:34:05)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 751 MB [5.15 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.03 GB >> 4.65 GB

Done!!!

So then you know that 15.3 GB becomes 4.65 GB in less than 5 mins but without creating any temp files as the whole compression takes place in memory. You also know srep decompression memory too.

_________________________________

The other thing, as you know how much I tried to make xtool to perform as fast as possible, the thing that always bothered me was that lzma was slow and that's why I added fast-lzma2. I noticed something interesting about this compression library in the sense that it's able to increase the number of threads used for compression without increasing memory usage.

Here's a few results

4x4:lzma (via fazip) (b128mb:t6), ultra, 64mb dictionary, 12 threads used 2781 MB ram

Compressed 288 files, 16,391,993,193 => 4,869,896,897 bytes. Ratio 29.71%
Compression time: cpu 13.50 sec/real 662.85 sec = 2%. Speed 24.73 mB/s

fast lzma2, level 10, 64mb dictionary, 12 threads used 403 MB ram

Compressed 288 files, 16,391,993,193 => 4,929,451,310 bytes. Ratio 30.07%
Compression time: cpu 11.84 sec/real 425.39 sec = 3%. Speed 38.53 mB/s

Not only is memory used by fast lzma2 low, but it's much faster while producing similar results and what is impressive is, if you had like 32 thread cpu, you can increase its speed even further without increasing memory usage by too much but if you use the 4x4 method, the memory usage will increase even more.

[infovs]

Thank you Razor12911 for this fantastic tool...have one question, if it is not too much trouble, can you please consider naming x64 srep.exe (since xtool is hardcoded using this exact name) like Bulat original named it as srep64.exe?
I'm using original naming for many years, I need 32bit srep.exe for some compatibility reasons of my own...and your renaming 64bit srep64.exe as srep.exe is causing problems.
Maybe at least consider making xtool using srep64.exe name first, if found, then if not found use srep.exe or whatever srep name you consider acceptable.

Thanks in advance,
best regards

[Wanterlude]

Guys, does anyone know how to fix this?
Game: A Plague Tale Requiem
Files: *.DPC
Xtool: last ver 0.7.0 hotfix
Problem: after unpacking 1 file (28.9gb) xtool goes beyond its allocated memory. (x4)
The picture shows the information from windows resource monitoring. (I have 16 GB RAM)

[External compressor:xtool]
header = 0
unpackcmd = "xtool.exe" decode -t100p -dm20p -sm20p - - <stdin> <stdout>

[External compressor:xtool]
header = 0
unpackcmd = "xtool.exe" decode -t100p -sm20p --mem=20p - - <stdin> <stdout>

In 1 case it happens as shown in the picture. In 2, the memory increase can be seen even in the windows task manager.

arc.ini:

[External compressor:xtool]
header    = 0
default   = -c128mb -t100p
packcmd   = xtool.exe precomp -mzlib -d1 -db -dd5 - - <stdin> <stdout>

-----------------------
upd. (14 may): preflate has the same problem | xtool 0.6.9 - same
-----------------------
upd. (15 may): fixed by 2nd xtool and -dm20p -sm20p parameters.
I think the problem was that I used 1 xtool for both zlib and dedup
-----------------------

[Razor12911]

Update available

Changes

- fixed issues with fast-lzma2 being unable to set correct compression level
- updated deflate stream scanner

[huluwizard]

Forgive me if I have missed something obvious, but the first post in this thread mentions documentation, but I'm not able to locate any documentation for xtool. Specifically, I'm trying to figure out the command line. I can get the results I want from the GUI plus a plugin, but ultimately I'd like to pair it with a batch file that does the same thing.

[KaktoR]

xtool.chm is what you are looking for.

[huluwizard]

Originally Posted by KaktoR

xtool.chm is what you are looking for.

Oh wow, it really was right there all along. Thanks!

[dixen]

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

[Cesar82]

Originally Posted by dixen

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

I think the parameter for srep memory usage in XTool 0.70 is "-sm#" and not anymore "--srepmem=#".