[Dev]XTool

[panker1992]

xcompress if pure windows 10 compression, and i think it supports it by default.

[Razor12911]

Can you guys test this. I added preflate as an alternative of reflate in case if you are having issues with crc errors when you use reflate or prolonged precompression times such as this:

Originally Posted by dixen

Dishonored: Death of the Outsider

*.resources - 17 gb > 25.1 gb for 23 minutes on 4 threads

For example. ZTool or Xtool (v0.12) - 30% for 4 hours

Thanks BLACKFIRE69

test unpack

I'm no expert when it comes to C++ and I did my best to compile a library for xtool to use so it may have bugs.

I ran a benchmark and here are the results

benchmark results on "game1.resources" from "Dishonored Death of the Outsiders"
xtool_2019:

8.30 GB >> ?? >> ?? >> ??
precompression time using 4 threads = it's better to watch paint dry (takes hours)

xtool_2020 (WIP):

8.30 GB >> 11.3 GB >> 8.74 GB >> 6.24 GB
precompression time using 2 threads = 8 minutes, 48 seconds
precompression time using 4 threads = 5 minutes, 7 seconds

PrecompX:

8.30 GB >> 11.1 GB >> 8.79 GB >> 6.28 GB
precompression time using 4 threads = 8 minutes, 29 seconds

[dixen]

Dishonored Death of the Outsiders

game1.resources

Compressed 1 file, 8,920,763,754 => 11,774,785,195 bytes. Ratio 131.99%
Compression time: cpu 11.31 sec/real 385.94 sec = 3%. Speed 23.11 mB/s
All OK

When unpack function?)

[Razor12911]

Originally Posted by dixen

Dishonored Death of the Outsiders

game1.resources



When unpack function?)

Soon, maybe this week. Still busy writing the main code while looking for more ways to speed up processing

[lolaya]

what is ztool and xtool and lolz?

[Grumpy]

Originally Posted by lolaya

what is ztool and xtool and lolz?

That information is already available on these forums, search and you will find, you can not expect for others to always hold your hand and spoon feed you all the time.

[lolaya]

Grumpy@

more posts here I can't find it easy

[BLACKFIRE69]

Originally Posted by lolaya

Grumpy@

more posts here I can't find it easy

that's the why studying hard....

[doofoo24]

can't wait to test the new xtool

[Razor12911]

Here's an early working version of xtool. I'm still busy adding preflate, reflate and other things so currently only the zlib function works which means this will not work on Dishonored 2 or DOOM but it will work on most titles.

Drag and drop game files that are zlib compressed on the bat file

Things to test:

* Stability
* Can you cancel installation when xtool was used in setup
* If you have 16 threads or more, make xtool use 100% of your cpu to see if it can handle it
* if you have 32GB+ ram, set high chunk size, something like -c1gb and see if xtool was able to allocate the necessary memory
* If you have potato pc, don't worry I got a job for you check if there are no problems if you are using 1 thread both in encoding and decoding

Whatever you do, don't place preflate_dll.dll near the exe, not yet. else it will fail, the preflate code hasn't been added yet

Notes:

The file check must report
FC: no differences encountered
if the test went successfully, if it failed kindly upload the file it failed on.

This xtool uses slightly more memory than the 2019 version, I opted for stability than less memory usage

xtool now accepts maths expressions in command line, if you wanted it to use all cores minus 1, you can just write -t100p-1 or -t100-2, totally up to you

Next release/test focuses on:

+ preflate
+ reflate (reflate actually isn't replaced by preflate, it sometimes gives better output than preflate so I'll keep it just in case you are after best results)
+ depth setting

[Masquerade]

Hello Razor, here is my test (I did make a post before but I had to delete it since I used a reflate files by mistake):

My spec: Ryzen 7 2700 (8c/16t) + 16GB RAM.

Bat file editied for 128mb chunk size.

Here's is XTool using all 16 of my threads:



Testing on pak file from Astroneer:

Start size: 2.13GB

End Size: 3.75GB

No differences encountered

Not entirely sure how I'd go about testing it in a setup...

[doofoo24]

tested on just cause2 arc files...
default setting nothing change "ProcProfile64.exe xtool.exe precomp -mzlib -c16mb -t100p-1 %1 %1.out"
system ryzen 3900x with 32gb
*is there arc.ini setting ?

[dixen]

Rayman Orgins
bundle_pc.pak (1.9 gb)

Process ID : 8952
Thread ID : 4228
Process Exit Code: 0
Thread Exit Code : 0

User Time : 149.796s
Kernel Time : 4.375s
Process Time : 154.171s
Clock Time : 74.923s

Working Set : 372312 KB
Paged Pool : 115 KB
Nonpaged Pool : 12 KB
Pagefile : 462572 KB
Page Fault Count : 147577

IO Read : 2046448 KB (in 31978 reads )
IO Write : 3038303 KB (in 47474 writes)
IO Other : 6 KB (in 297 others)

PS. tested on my 2nd pc (fx-4100, 2 cores, 4 threads).

[Razor12911]

Originally Posted by doofoo24

*is there arc.ini setting ?

[External compressor:xzlib]
header    = 0
packcmd   = xtool.exe precomp -c16mb -t100p-1 -mzlib - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

here's preflate test, test on all zlib compressed games + dishonored and doom

[doofoo24]

tested on Dishonored2 with c16m and 512m and set -t100p-1...
work fine on ryzen 3900x and 32gb use all core and ram...

42.1gb to 57.2gb with c16mb
42.1gb to 59.8gb with c512mb

*used on all game file...

tested unpacking with inno
i get ERROR: archive data corrupted...