[Dev]XTool

[Razor12911]

..

[JustFun]

Hmm, a hint? Hopefully! :D

[doofoo24]

I can't wait for april fools day

[DedSec]

I can't wait for april fools day

And I will be ready again to fool someone:p:p:p

[Jiva newstone]

i think its xcore or may be it will be a april fool

[KaktoR]

And I will be ready again to fool someone:p:p:p

Now you have destroyed the whole anticipation :D

[Razor12911]

..
XD, People be thanking two dots ".."

[oltjon]

XD, People be thanking two dots ".."


they thank you because you have a reputation. no the two dots ....

[Razor12911]

Good to know, I need these tested.

Test #1

* I want to know whether xtool performs give better output. (better output size is irrelevant, compress the output files individually and check which one was compressed better)
*For speed comparison, run tests on ramdisk or SSD

Games/Files to test on:
Video files (they may freeze xtool which is what I want)
DOOM 2016
Dishonored 2
And all other zlib games where pzlib failed to process

Post results like (Include times if you ran on ramdisk):
Xtool: 825GB > 1.69GB (Gave better output) took 12.592s
Ztool197: 825GB > 1.24GB (Gave better output) took 12.592s
Ztool198: 825GB > 1.24GB (Gave better output) took 12.592s

Don't change the parameters of ztool, they are made to match xtool

Test #2

* For this test, I want to see if the memory improvements worked, so just simply post the entire cmd output of the test you have run
* This test must be run on ramdisk or ssd

Games\Files to test on:
Anything that does not have deflate/zlib streams (e.g. Video files)

[ZakirAhmad]

No Ram disk, but if u want i can perform on HDD.

[ZakirAhmad]

I cant open test1.7z, winrar and 7zip says test 1 is corrupted. i downloaded twice

[-NORO-]

I cant open test1.7z, winrar and 7zip says test 1 is corrupted. i downloaded twice

use latest version of winrar

[Mini]

Dishonored_DeathOfTheOutsider game2_001.resources 36.1 MB
TEST1



Process ID : 126624
Thread ID : 126816
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.640s
Kernel Time : 0.046s
Process Time : 0.686s
Clock Time : 0.583s

Working Set : 187852 KB
Paged Pool : 99 KB
Nonpaged Pool : 12 KB
Pagefile : 256260 KB
Page Fault Count : 88384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 2 KB (in 1505 others)


Process ID : 130628
Thread ID : 125304
Process Exit Code: 0
Thread Exit Code : 0

User Time : 2.000s
Kernel Time : 0.062s
Process Time : 2.062s
Clock Time : 1.175s

Working Set : 78864 KB
Paged Pool : 96 KB
Nonpaged Pool : 9 KB
Pagefile : 85704 KB
Page Fault Count : 32166

IO Read : 37035 KB (in 581 reads )
IO Write : 37035 KB (in 580 writes)
IO Other : 2 KB (in 983 others)


Process ID : 129012
Thread ID : 132776
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.609s
Kernel Time : 0.187s
Process Time : 0.796s
Clock Time : 0.527s

Working Set : 205708 KB
Paged Pool : 98 KB
Nonpaged Pool : 8 KB
Pagefile : 229604 KB
Page Fault Count : 106539

IO Read : 37035 KB (in 581 reads )
IO Write : 99441 KB (in 1555 writes)
IO Other : 2 KB (in 955 others)


game2_001.resources.out1 97.2 МБ (101 969 487)
game2_001.resources.out2 36.1 MB (37 924 455)
game2_001.resources.out3 97.1 MB (101 827 433)

TEST2 - M.2 SSD
Process ID : 127384
Thread ID : 133524
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.078s
Process Time : 0.765s
Clock Time : 0.575s

Working Set : 164796 KB
Paged Pool : 101 KB
Nonpaged Pool : 12 KB
Pagefile : 221256 KB
Page Fault Count : 83384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1724 others)


Process ID : 128596
Thread ID : 129300
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.062s
Process Time : 0.749s
Clock Time : 0.550s

Working Set : 175800 KB
Paged Pool : 101 KB
Nonpaged Pool : 11 KB
Pagefile : 249272 KB
Page Fault Count : 72029

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1688 others)

[Razor12911]

Dishonored_DeathOfTheOutsider game2_001.resources 36.1 MB
TEST1



game2_001.resources.out1 97.2 МБ (101 969 487)
game2_001.resources.out2 36.1 MB (37 924 455)
game2_001.resources.out3 97.1 MB (101 827 433)

TEST2 - M.2 SSD
Process ID : 127384
Thread ID : 133524
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.078s
Process Time : 0.765s
Clock Time : 0.575s

Working Set : 164796 KB
Paged Pool : 101 KB
Nonpaged Pool : 12 KB
Pagefile : 221256 KB
Page Fault Count : 83384

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1724 others)


Process ID : 128596
Thread ID : 129300
Process Exit Code: 0
Thread Exit Code : 0

User Time : 0.687s
Kernel Time : 0.062s
Process Time : 0.749s
Clock Time : 0.550s

Working Set : 175800 KB
Paged Pool : 101 KB
Nonpaged Pool : 11 KB
Pagefile : 249272 KB
Page Fault Count : 72029

IO Read : 37035 KB (in 580 reads )
IO Write : 99579 KB (in 1556 writes)
IO Other : 6 KB (in 1688 others)

Thanks, so. The issue of not being able to detect DH2 streams under normal scan is fixed.
And data loader uses less memory.

[Razor12911]

Test 3

What is required here is running test on data with deflate/zlib streams by replacing zlibwapi.dll with different other zlib dlls that you can find, submit the dlls that do not work.

zlibwapi.dll
zlib1.dll
zlib.dll

are the dlls to look for to see whether xtool can handle all and at least give inflation.
N.B. exe is x86, so get x86 dll and not x64

[Andu21]

Anything below version 1.2.11.0 doesn't work on my test file, it gives 0 bytes. It only happens if i change original name to "zlibwapi", strange enough xtool works without zlibwapi.dll.

[Razor12911]

Anything below version 1.2.11.0 doesn't work on my test file, it gives 0 bytes. It only happens if i change original name to "zlibwapi", strange enough xtool works without zlibwapi.dll.

Thanks for the test, xtool can work without zlibwapi, but only if zlib1.dll or zlib.dll is detected elsewhere and since Freearc registers dlls when it installs, probably it used the one in FA's directory.

:)

[Webster17]

used it on Darksiders 2 file, test 1 recompressed it to 1gb but apperror occured, test 3 successfully worked

[Webster17]

Preocmp gave better output, 1.12gb whereas 1.06 with xtool on above mentioned file.

[Razor12911]

Hmm, interesting. Did you compress the file afterwards, both outputs of xtool and precomp to see which one did a good precompression job?

[Webster17]

working on it, afai tested previously, precomp is the lead winner against ztool and pzlibv3, its slow but efficient

[Razor12911]

Same here, ran many tests and somehow precomp seems to find streams that I have no idea as to where it finds them but compression after the precompressed data tells the true story.

[PsYcHo_RaGE]

i am still thinking about it!:confused:

[ZakirAhmad]

@Razor In my case precomp and test3 produced same results. i tried files from halo 2, tom raider 2013 etc.

[Webster17]

XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.

[Razor12911]

XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.

I want that 16mb, can you upload that file so I can see what's really going on.

[masen485]

Dev] XTool I want to use or try this application but I do not know anything, can you help?

[Andu21]

XTool+srep+lzma = 459mb = 1.06gb = 257mb
Precomp+srep+lzma = 459mb = 1.12gb = 239mb
Even after this I am damn impressed with XTool, its 10x faster compared to precomp, but the difference is about 16mb and it will be more if used on bigger files. Still ill ran some more tests. Also about Ztool it had the same issue while recompressing a zip file.
Second to this Razor. Also, pdfs aren't properly detected either.

[Webster17]

I want that 16mb, can you upload that file so I can see what's really going on.

Very sorry bro, i wish i could but my internet speed sucks, it will take me 24 hours or more to upload it. This might help, I used offzip on it and the output size extracted was about 1.12gb, same as precomp output.

[Razor12911]

Cool, I think I know what the issue is then. Thanks for participating in tests.
Will roll out another test where perhaps this issue will be fixed then move onto lz4 codec.

[KaktoR]

I had a lot of work to do last days (f.e. compressing 2 games at the same time is a really pain). On weekend i will participate into test some usefull stuff with xtool.

Cheers and good work, as always ;)

[Webster17]

Cool, I think I know what the issue is then. Thanks for participating in tests.
Will roll out another test where perhaps this issue will be fixed then move onto lz4 codec.

Welcome and I look forward to it

[Webster17]

Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

[Razor12911]

Another test result here used on characters.vpp_pc from Saints Row Gat out of Hell Game. Ztool(pzlib) and precomp doesn't detect and recompress this file so the Competitors are Xtool, Pzlib3, reflate and Pzlib3+reflate compressed with srep+lzma. Here are the results

Thanks, I'm now thinking of pushing forward the first working codec in xtool.

Since the new rewritten scanner seems improved.

A few ideas have popped up since the last time I posted the tests so I may post a few more tests, regarding the processing part.

[Webster17]

Welcome and bring it on, it's a pleasure to help out.

[Razor12911]

Test 4

The rewritten new xtool project is really surprising lol, I didn't know I can draw even more speed from just rewriting the code... Unless if I made a mess in the older versions, I wish I had the old sources just to see what the hell I wrote in the code editor.

Here are my benchmarks:
21587

What I need from this test is you guys to post the cmd output.

if possible compress the outputs made my all the test programs and post results.

[KaktoR]

Eh, i guess just drag&drop file to batch? Because it doesn't work "The command ProcProfile64.exe is wrong" or something.

[Andu21]

^Yes.

FF-C03C_MAIN.ZIP from Freedom Fighters:

Process ID : 2604
Thread ID : 1524
Process Exit Code: 0
Thread Exit Code : 0

User Time : 2.137s
Kernel Time : 0.078s
Process Time : 2.215s
Clock Time : 1.310s

Working Set : 80056 KB
Paged Pool : 84 KB
Nonpaged Pool : 8 KB
Pagefile : 167364 KB
Page Fault Count : 34069

IO Read : 16161 KB (in 254 reads )
IO Write : 35065 KB (in 548 writes)
IO Other : 1 KB (in 188 others)


Process ID : 2904
Thread ID : 1372
Process Exit Code: 0
Thread Exit Code : 0

User Time : 1.794s
Kernel Time : 0.062s
Process Time : 1.856s
Clock Time : 1.718s

Working Set : 61408 KB
Paged Pool : 84 KB
Nonpaged Pool : 9 KB
Pagefile : 66436 KB
Page Fault Count : 26514

IO Read : 16161 KB (in 254 reads )
IO Write : 35065 KB (in 548 writes)
IO Other : 1 KB (in 137 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 15.8 MB > 34.1 MB (216%)
Speed: 7.61 MB/s (read) 16.5 MB/s (write)
Duration: 2075 ms

Process ID : 2312
Thread ID : 1180
Process Exit Code: 0
Thread Exit Code : 0

User Time : 4.368s
Kernel Time : 0.031s
Process Time : 4.399s
Clock Time : 2.584s

Working Set : 61504 KB
Paged Pool : 84 KB
Nonpaged Pool : 9 KB
Pagefile : 67600 KB
Page Fault Count : 26718

IO Read : 16161 KB (in 68 reads )
IO Write : 35065 KB (in 147 writes)
IO Other : 4 KB (in 97 others)

PACK5.000.tiger:

Process ID : 2624
Thread ID : 1328
Process Exit Code: 0
Thread Exit Code : 0

User Time : 31.028s
Kernel Time : 0.514s
Process Time : 31.542s
Clock Time : 14.948s

Working Set : 247680 KB
Paged Pool : 84 KB
Nonpaged Pool : 11 KB
Pagefile : 281144 KB
Page Fault Count : 248746

IO Read : 288344 KB (in 4507 reads )
IO Write : 472134 KB (in 7378 writes)
IO Other : 2 KB (in 379 others)


Process ID : 2784
Thread ID : 792
Process Exit Code: 0
Thread Exit Code : 0

User Time : 149.963s
Kernel Time : 3.572s
Process Time : 153.535s
Clock Time : 42.476s

Working Set : 296384 KB
Paged Pool : 84 KB
Nonpaged Pool : 11 KB
Pagefile : 333916 KB
Page Fault Count : 2684676

IO Read : 288342 KB (in 4507 reads )
IO Write : 472199 KB (in 7379 writes)
IO Other : 2 KB (in 316 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 282 MB > 461 MB (164%)
Speed: 4.80 MB/s (read) 7.87 MB/s (write)
Duration: 58609 ms

Process ID : 2264
Thread ID : 2292
Process Exit Code: 0
Thread Exit Code : 0

User Time : 218.744s
Kernel Time : 0.514s
Process Time : 219.258s
Clock Time : 58.854s

Working Set : 280776 KB
Paged Pool : 84 KB
Nonpaged Pool : 11 KB
Pagefile : 322656 KB
Page Fault Count : 181916

IO Read : 288341 KB (in 1132 reads )
IO Write : 472199 KB (in 2145 writes)
IO Other : 35 KB (in 268 others)

All files compressed using lolz.
FF-C03C_MAIN.ZIP.out1 34.2 MB (35,907,044 bytes) > 10.3 MB (10,840,373 bytes)
FF-C03C_MAIN.ZIP.out2 34.2 MB (35,907,118 bytes) > 10.3 MB (10,837,830 bytes)
FF-C03C_MAIN.ZIP.out3 34.2 MB (35,907,121 bytes) > 10.3 MB (10,840,977 bytes)
FF-C03C_MAIN.pcf(precomp047:intense:brute) 35.2 MB (36,985,675 bytes) > 10.1 MB (10,675,558 bytes)

PACK5.000.tiger.out1 461 MB (483,465,220 bytes) > 188 MB (197,647,249 bytes)
PACK5.000.tiger.out2 461 MB (483,532,609 bytes) > 189 MB (199,192,457 bytes)
PACK5.000.tiger.out3 461 MB (483,532,612 bytes) > 190 MB (199,349,867 bytes)
PACK5.000.pcf(precomp047:intense:brute) 461 MB (483,591,682 bytes) > 189 MB (198,587,760 bytes)

[Gupta]

x64w.rpf



Process ID : 9128
Thread ID : 7836
Process Exit Code: 0
Thread Exit Code : 0

User Time : 278.125s
Kernel Time : 1.375s
Process Time : 279.500s
Clock Time : 87.822s

Working Set : 384048 KB
Paged Pool : 94 KB
Nonpaged Pool : 12 KB
Pagefile : 428284 KB
Page Fault Count : 406712

IO Read : 915372 KB (in 14304 reads )
IO Write : 2078038 KB (in 32470 writes)
IO Other : 4 KB (in 396 others)


Process ID : 4284
Thread ID : 3984
Process Exit Code: 0
Thread Exit Code : 0

User Time : 281.593s
Kernel Time : 1.500s
Process Time : 283.093s
Clock Time : 81.404s

Working Set : 397436 KB
Paged Pool : 93 KB
Nonpaged Pool : 12 KB
Pagefile : 477376 KB
Page Fault Count : 436067

IO Read : 915372 KB (in 14305 reads )
IO Write : 2073414 KB (in 32399 writes)
IO Other : 4 KB (in 255 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 894 MB > 1.98 GB (226%)
Speed: 6.82 MB/s (read) 15.5 MB/s (write)
Duration: 131016 ms

Process ID : 11244
Thread ID : 5772
Process Exit Code: 0
Thread Exit Code : 0

User Time : 33.581s
Kernel Time : 1.203s
Process Time : 34.784s
Clock Time : 131.038s

Working Set : 397756 KB
Paged Pool : 93 KB
Nonpaged Pool : 13 KB
Pagefile : 439408 KB
Page Fault Count : 252851

IO Read : 915372 KB (in 3579 reads )
IO Write : 2073358 KB (in 9056 writes)
IO Other : 183 KB (in 918 others)


x64w.rpf.out1.lzma 594185541
x64w.rpf.out2.lzma 597874786
x64w.rpf.out3.lzma 597841685

[Mini]

arcrowd01.pac (size 95 207 424) from WWE2K18


Process ID : 309132
Thread ID : 309128
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.750s
Kernel Time : 0.203s
Process Time : 16.953s
Clock Time : 8.460s

Working Set : 226684 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 251260 KB
Page Fault Count : 112870

IO Read : 92976 KB (in 1454 reads )
IO Write : 166352 KB (in 2600 writes)
IO Other : 2 KB (in 1299 others)


Process ID : 309272
Thread ID : 309256
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.156s
Kernel Time : 0.328s
Process Time : 16.484s
Clock Time : 8.180s

Working Set : 294064 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 336720 KB
Page Fault Count : 206861

IO Read : 92976 KB (in 1455 reads )
IO Write : 166354 KB (in 2601 writes)
IO Other : 2 KB (in 886 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 90.8 MB > 162 MB (179%)
Speed: 8.05 MB/s (read) 14.4 MB/s (write)
Duration: 11281 ms

Process ID : 309356
Thread ID : 309352
Process Exit Code: 0
Thread Exit Code : 0

User Time : 22.375s
Kernel Time : 0.250s
Process Time : 22.625s
Clock Time : 11.308s

Working Set : 272072 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 308976 KB
Page Fault Count : 169115

IO Read : 92976 KB (in 369 reads )
IO Write : 166354 KB (in 696 writes)
IO Other : 20 KB (in 657 others)

[KaktoR]

GameData.pak from KCD
(268.562.432 Bytes)

Process ID : 6964
Thread ID : 8532
Process Exit Code: 0
Thread Exit Code : 0

User Time : 31.703s
Kernel Time : 0.484s
Process Time : 32.187s
Clock Time : 10.829s

Working Set : 350452 KB
Paged Pool : 98 KB
Nonpaged Pool : 12 KB
Pagefile : 380008 KB
Page Fault Count : 258439

IO Read : 262266 KB (in 4099 reads )
IO Write : 500723 KB (in 7824 writes)
IO Other : 2 KB (in 304 others)


Process ID : 8396
Thread ID : 7768
Process Exit Code: -1
Thread Exit Code : -1

User Time : 78.218s
Kernel Time : 0.734s
Process Time : 78.952s
Clock Time : 25.580s

Working Set : 381624 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 433332 KB
Page Fault Count : 402705

IO Read : 262266 KB (in 4100 reads )
IO Write : 408768 KB (in 6388 writes)
IO Other : 2 KB (in 153 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 256 MB > 488 MB (191%)
Speed: 5.43 MB/s (read) 10.4 MB/s (write)
Duration: 47172 ms

Process ID : 9104
Thread ID : 6936
Process Exit Code: 0
Thread Exit Code : 0

User Time : 145.437s
Kernel Time : 0.562s
Process Time : 145.999s
Clock Time : 47.723s

Working Set : 372160 KB
Paged Pool : 96 KB
Nonpaged Pool : 13 KB
Pagefile : 424788 KB
Page Fault Count : 224668

IO Read : 262266 KB (in 1027 reads )
IO Write : 500082 KB (in 2123 writes)
IO Other : 49 KB (in 381 others)

[Panky]

animations-part1.pak (KCD)
Process ID : 4208
Thread ID : 12228
Process Exit Code: 0
Thread Exit Code : 0

User Time : 49.296s
Kernel Time : 0.890s
Process Time : 50.186s
Clock Time : 28.901s

Working Set : 516576 KB
Paged Pool : 98 KB
Nonpaged Pool : 13 KB
Pagefile : 590036 KB
Page Fault Count : 401659

IO Read : 342891 KB (in 5359 reads )
IO Write : 552344 KB (in 8631 writes)
IO Other : 2 KB (in 303 others)


Process ID : 5184
Thread ID : 12152
Process Exit Code: -1
Thread Exit Code : -1

User Time : 16.406s
Kernel Time : 1.218s
Process Time : 17.624s
Clock Time : 5.726s

Working Set : 797276 KB
Paged Pool : 97 KB
Nonpaged Pool : 15 KB
Pagefile : 913920 KB
Page Fault Count : 605176

IO Read : 147520 KB (in 2306 reads )
IO Write : 61952 KB (in 969 writes)
IO Other : 2 KB (in 236 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 335 MB > 541 MB (161%)
Speed: 11.0 MB/s (read) 17.7 MB/s (write)
Duration: 30578 ms

Process ID : 1532
Thread ID : 5836
Process Exit Code: 0
Thread Exit Code : 0

User Time : 53.265s
Kernel Time : 0.812s
Process Time : 54.077s
Clock Time : 30.574s

Working Set : 536468 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 591260 KB
Page Fault Count : 325641

IO Read : 342891 KB (in 1342 reads )
IO Write : 553956 KB (in 2315 writes)
IO Other : 53 KB (in 425 others)

[Razor12911]

arcrowd01.pac (size 95 207 424) from WWE2K18


Process ID : 309132
Thread ID : 309128
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.750s
Kernel Time : 0.203s
Process Time : 16.953s
Clock Time : 8.460s

Working Set : 226684 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 251260 KB
Page Fault Count : 112870

IO Read : 92976 KB (in 1454 reads )
IO Write : 166352 KB (in 2600 writes)
IO Other : 2 KB (in 1299 others)


Process ID : 309272
Thread ID : 309256
Process Exit Code: 0
Thread Exit Code : 0

User Time : 16.156s
Kernel Time : 0.328s
Process Time : 16.484s
Clock Time : 8.180s

Working Set : 294064 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 336720 KB
Page Fault Count : 206861

IO Read : 92976 KB (in 1455 reads )
IO Write : 166354 KB (in 2601 writes)
IO Other : 2 KB (in 886 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 90.8 MB > 162 MB (179%)
Speed: 8.05 MB/s (read) 14.4 MB/s (write)
Duration: 11281 ms

Process ID : 309356
Thread ID : 309352
Process Exit Code: 0
Thread Exit Code : 0

User Time : 22.375s
Kernel Time : 0.250s
Process Time : 22.625s
Clock Time : 11.308s

Working Set : 272072 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 308976 KB
Page Fault Count : 169115

IO Read : 92976 KB (in 369 reads )
IO Write : 166354 KB (in 696 writes)
IO Other : 20 KB (in 657 others)

0.3 seconds slower :mad:

I guess I have to work harder on this :)

[KaktoR]

FFXII_TZA.vbf
(31.188.934.656 Bytes)

Process ID : 4180
Thread ID : 6696
Process Exit Code: 0
Thread Exit Code : 0

User Time : 341.998s
Kernel Time : 43.109s
Process Time : 385.107s
Clock Time : 2455.929s

Working Set : 1156248 KB
Paged Pool : 98 KB
Nonpaged Pool : 11 KB
Pagefile : 1376796 KB
Page Fault Count : 2663686

IO Read : 30457942 KB (in 475907 reads )
IO Write : 57503890 KB (in 898499 writes)
IO Other : 2 KB (in 338 others)


Process ID : 744
Thread ID : 6400
Process Exit Code: -1
Thread Exit Code : -1

User Time : 48.187s
Kernel Time : 0.859s
Process Time : 49.046s
Clock Time : 31.565s

Working Set : 168704 KB
Paged Pool : 96 KB
Nonpaged Pool : 11 KB
Pagefile : 191520 KB
Page Fault Count : 102259

IO Read : 671808 KB (in 10498 reads )
IO Write : 618240 KB (in 9661 writes)
IO Other : 2 KB (in 248 others)

pZLib version 3 is created by Razor12911

Status: Done!!!
Data: 29.0 GB > 54.8 GB (189%)
Speed: 9.10 MB/s (read) 17.2 MB/s (write)
Duration: 3269438 ms

Process ID : 8420
Thread ID : 5912
Process Exit Code: 0
Thread Exit Code : 0

User Time : 9.981s
Kernel Time : 44.671s
Process Time : 54.652s
Clock Time : 3269.772s

Working Set : 1370460 KB
Paged Pool : 96 KB
Nonpaged Pool : 13 KB
Pagefile : 1543420 KB
Page Fault Count : 876193

IO Read : 30457942 KB (in 118980 reads )
IO Write : 57500176 KB (in 250895 writes)
IO Other : 4959 KB (in 19455 others)

As you can see, ztool just gave up on this, at least given options :D

[Razor12911]

Thanks for the tests guys. Here's another one, I added reflate lib support on this test, the inputs I'm interested in is data where reflate is required otherwise it will be just the same results as test number 4.

Png images, pdfs, linux isos and zip files contain streams that can be processed by reflate, games however that's DOOM and Dishonored 2 afaik.

So just like test 4, post cmd output.

I just like to point out, I'm not interested in the size that much but memory used and the time it took to perform the test.

These are the results of the tests I have performed:
21616

[Gupta]

x64w.rpf(checked twice)



Process ID : 4960
Thread ID : 2752
Process Exit Code: 0
Thread Exit Code : 0

User Time : 299.953s
Kernel Time : 1.515s
Process Time : 301.468s
Clock Time : 128.594s

Working Set : 344464 KB
Paged Pool : 93 KB
Nonpaged Pool : 13 KB
Pagefile : 373868 KB
Page Fault Count : 358433

IO Read : 915372 KB (in 14304 reads )
IO Write : 2078066 KB (in 32470 writes)
IO Other : 1 KB (in 138 others)


Process ID : 7556
Thread ID : 7144
Process Exit Code: 0
Thread Exit Code : 0

User Time : 315.703s
Kernel Time : 2.078s
Process Time : 317.781s
Clock Time : 110.843s

Working Set : 443228 KB
Paged Pool : 92 KB
Nonpaged Pool : 13 KB
Pagefile : 497848 KB
Page Fault Count : 496542

IO Read : 915372 KB (in 14305 reads )
IO Write : 2073414 KB (in 32399 writes)
IO Other : 1 KB (in 120 others)



x64w.rpf.out1.lzma: 2,127,939,711 -> 595,208,759: 27.97%
x64w.rpf.out2.lzma: 2,123,176,204 -> 598,727,541: 28.20%




School.tar(~30 pdfs, 200kB to 150mB)


Process ID : 9736
Thread ID : 896
Process Exit Code: 0
Thread Exit Code : 0

User Time : 300.541s
Kernel Time : 103.625s
Process Time : 404.166s
Clock Time : 431.191s

Working Set : 1053160 KB
Paged Pool : 93 KB
Nonpaged Pool : 13 KB
Pagefile : 1171552 KB
Page Fault Count : 86155222

IO Read : 581265 KB (in 9084 reads )
IO Write : 2164138 KB (in 33815 writes)
IO Other : 1 KB (in 210 others)


Process ID : 2492
Thread ID : 5196
Process Exit Code: 0
Thread Exit Code : 0

User Time : 365.791s
Kernel Time : 125.078s
Process Time : 490.869s
Clock Time : 486.174s

Working Set : 1157188 KB
Paged Pool : 92 KB
Nonpaged Pool : 42 KB
Pagefile : 1281496 KB
Page Fault Count : 98349470

IO Read : 581265 KB (in 9085 reads )
IO Write : 2173087 KB (in 33956 writes)
IO Other : 1 KB (in 148 others)


@KaktoR

User Time : 48.187s
Kernel Time : 0.859s
Process Time : 49.046s
Clock Time : 31.565s


why ztool so fast here or it crashed in testing?

[KaktoR]

@Gupta: It crashed :D

[KaktoR]

Thanks for the tests guys. Here's another one, I added reflate lib support on this test, the inputs I'm interested in is data where reflate is required otherwise it will be just the same results as test number 4.

Png images, pdfs, linux isos and zip files contain streams that can be processed by reflate, games however that's DOOM and Dishonored 2 afaik.

So just like test 4, post cmd output.

I just like to point out, I'm not interested in the size that much but memory used and the time it took to perform the test.

These are the results of the tests I have performed:
21616

I can't tell you how much RAM it used (where i can look? TM is quiet impractical).

*.png files (~8k) all i could find on HDD (except 20k+ from sysdrive). I packed them in a stored m0 zip file (however streams are the same) just because your batch can't process a folder, only a single file.
366 MB (384.172.032 Bytes)

Process ID : 9160
Thread ID : 4300
Process Exit Code: 0
Thread Exit Code : 0

User Time : 384.144s
Kernel Time : 33.265s
Process Time : 417.409s
Clock Time : 856.020s

Working Set : 1435080 KB
Paged Pool : 100 KB
Nonpaged Pool : 12 KB
Pagefile : 1590936 KB
Page Fault Count : 24544849

IO Read : 364298 KB (in 5694 reads )
IO Write : 3283916 KB (in 51312 writes)
IO Other : 5 KB (in 461 others)


Process ID : 7896
Thread ID : 1392
Process Exit Code: 0
Thread Exit Code : 0

User Time : 338.692s
Kernel Time : 84.125s
Process Time : 422.817s
Clock Time : 1614.952s

Working Set : 1589080 KB
Paged Pool : 98 KB
Nonpaged Pool : 87 KB
Pagefile : 1844716 KB
Page Fault Count : 64115442

IO Read : 364298 KB (in 5695 reads )
IO Write : 3296645 KB (in 51512 writes)
IO Other : 5 KB (in 418 others)

*.pdf files (16 files). Same as png, packed with m0 in zip file
256 MB (269.295.616 Bytes)

Process ID : 3904
Thread ID : 5204
Process Exit Code: 0
Thread Exit Code : 0

User Time : 300.359s
Kernel Time : 23.937s
Process Time : 324.296s
Clock Time : 97.433s

Working Set : 473056 KB
Paged Pool : 98 KB
Nonpaged Pool : 13 KB
Pagefile : 526072 KB
Page Fault Count : 17154634

IO Read : 262959 KB (in 4111 reads )
IO Write : 564426 KB (in 8820 writes)
IO Other : 2 KB (in 300 others)


Process ID : 6356
Thread ID : 6364
Process Exit Code: 0
Thread Exit Code : 0

User Time : 399.656s
Kernel Time : 28.109s
Process Time : 427.765s
Clock Time : 128.859s

Working Set : 574620 KB
Paged Pool : 97 KB
Nonpaged Pool : 26 KB
Pagefile : 644664 KB
Page Fault Count : 20138944

IO Read : 262959 KB (in 4112 reads )
IO Write : 566978 KB (in 8861 writes)
IO Other : 2 KB (in 148 others)

------------------------------------------------------------------------------

Single PNG file
15 MB (15.794.176 Bytes)

Process ID : 5764
Thread ID : 2916
Process Exit Code: 0
Thread Exit Code : 0

User Time : 23.031s
Kernel Time : 0.125s
Process Time : 23.156s
Clock Time : 23.248s

Working Set : 74904 KB
Paged Pool : 98 KB
Nonpaged Pool : 10 KB
Pagefile : 161384 KB
Page Fault Count : 38067

IO Read : 15422 KB (in 242 reads )
IO Write : 34757 KB (in 544 writes)
IO Other : 2 KB (in 170 others)


Process ID : 5784
Thread ID : 10224
Process Exit Code: 0
Thread Exit Code : 0

User Time : 115.343s
Kernel Time : 0.687s
Process Time : 116.030s
Clock Time : 116.374s

Working Set : 180072 KB
Paged Pool : 97 KB
Nonpaged Pool : 12 KB
Pagefile : 200228 KB
Page Fault Count : 600207

IO Read : 15422 KB (in 243 reads )
IO Write : 36515 KB (in 572 writes)
IO Other : 2 KB (in 136 others)

Single PDF file (The_Art_Of_SGWarlords.pdf)
45.8 MB (48.070.656 Bytes)

Process ID : 5056
Thread ID : 10140
Process Exit Code: 0
Thread Exit Code : 0

User Time : 83.156s
Kernel Time : 1.140s
Process Time : 84.296s
Clock Time : 23.200s

Working Set : 332048 KB
Paged Pool : 98 KB
Nonpaged Pool : 13 KB
Pagefile : 424816 KB
Page Fault Count : 1070466

IO Read : 46941 KB (in 735 reads )
IO Write : 263631 KB (in 4120 writes)
IO Other : 2 KB (in 170 others)


Process ID : 3472
Thread ID : 484
Process Exit Code: 0
Thread Exit Code : 0

User Time : 250.109s
Kernel Time : 4.390s
Process Time : 254.499s
Clock Time : 73.490s

Working Set : 469956 KB
Paged Pool : 97 KB
Nonpaged Pool : 28 KB
Pagefile : 536544 KB
Page Fault Count : 5065503

IO Read : 46941 KB (in 736 reads )
IO Write : 265579 KB (in 4151 writes)
IO Other : 2 KB (in 138 others)

[Simorq]

DMCL-0.nbz (http://www.mediafire.com/file/7klm63vs4oo7koc/DMCL-0.nbz) (165 MB):eek:
Process ID : 6860
Thread ID : 5652
Process Exit Code: 0
Thread Exit Code : 0

User Time : 72.450s
Kernel Time : 9.078s
Process Time : 81.528s
Clock Time : 577.212s

Working Set : 369332 KB
Paged Pool : 97 KB
Nonpaged Pool : 12 KB
Pagefile : 442580 KB
Page Fault Count : 3711470

IO Read : 168982 KB (in 2642 reads )
IO Write : 548729 KB (in 8574 writes)
IO Other : 5 KB (in 293 others)


Process ID : 6420
Thread ID : 2940
Process Exit Code: 0
Thread Exit Code : 0

User Time : 188.023s
Kernel Time : 30.140s
Process Time : 218.163s
Clock Time : 1565.767s

Working Set : 440868 KB
Paged Pool : 96 KB
Nonpaged Pool : 32 KB
Pagefile : 501228 KB
Page Fault Count : 11514042

IO Read : 168982 KB (in 2643 reads )
IO Write : 573538 KB (in 8963 writes)
IO Other : 5 KB (in 304 others)


Dmc2-0.nbz (http://www.mediafire.com/file/31z5i3fg37f8jy9/dmc2-0.nbz) (409 MB):confused:
http://s8.picofile.com/file/8323070650/Xtool_Bug.jpg

[Andu21]

TombRaider_Beginning.pdf - 904 MB (948,140,632 bytes)
Process ID : 3144
Thread ID : 2536
Process Exit Code: 0
Thread Exit Code : 0

User Time : 129.174s
Kernel Time : 3.619s
Process Time : 132.793s
Clock Time : 409.011s

Working Set : 914324 KB
Paged Pool : 85 KB
Nonpaged Pool : 11 KB
Pagefile : 1008020 KB
Page Fault Count : 1268744

IO Read : 925918 KB (in 14469 reads )
IO Write : 4782205 KB (in 74722 writes)
IO Other : 1 KB (in 172 others)


Process ID : 3116
Thread ID : 2036
Process Exit Code: 0
Thread Exit Code : 0

User Time : 357.273s
Kernel Time : 99.856s
Process Time : 457.129s
Clock Time : 6069.699s

Working Set : 1422572 KB
Paged Pool : 85 KB
Nonpaged Pool : 14 KB
Pagefile : 1632072 KB
Page Fault Count : 92402768

IO Read : 925918 KB (in 14469 reads )
IO Write : 4899729 KB (in 76559 writes)
IO Other : 1 KB (in 121 others)

TombRaider_Beginning.pdf.out1 - 4.56 GB (4,896,978,011 bytes)
TombRaider_Beginning.pdf.out2 - 4.67 GB (5,017,323,321 bytes)

[Razor12911]

@PrinceGupta
Turns out I left dynamic entropy calculator in the scanner, as a result it slowed down the scanner but re-ran the test without the entropy function and this came:

x64w.rpf

Process ID : 9004
Thread ID : 8760
Process Exit Code: 0
Thread Exit Code : 0

User Time : 277.531s
Kernel Time : 2.890s
Process Time : 280.421s
Clock Time : 85.264s

Working Set : 398088 KB
Paged Pool : 98 KB
Nonpaged Pool : 12 KB
Pagefile : 446216 KB
Page Fault Count : 377597

IO Read : 915376 KB (in 14305 reads )
IO Write : 2077976 KB (in 32469 writes)
IO Other : 2 KB (in 200 others)


Process ID : 20692
Thread ID : 17040
Process Exit Code: 0
Thread Exit Code : 0

User Time : 306.671s
Kernel Time : 3.093s
Process Time : 309.764s
Clock Time : 93.089s

Working Set : 453512 KB
Paged Pool : 97 KB
Nonpaged Pool : 13 KB
Pagefile : 509212 KB
Page Fault Count : 422732

IO Read : 915376 KB (in 14306 reads )
IO Write : 2073414 KB (in 32399 writes)
IO Other : 2 KB (in 155 others)

@Simorq

Downloaded the file and seems to have worked fine here:

dmc2-0.nbz

Process ID : 18168
Thread ID : 12628
Process Exit Code: 0
Thread Exit Code : 0

User Time : 58.462s
Kernel Time : 30.437s
Process Time : 88.899s
Clock Time : 388.581s

Working Set : 481856 KB
Paged Pool : 98 KB
Nonpaged Pool : 13 KB
Pagefile : 533624 KB
Page Fault Count : 12609675

IO Read : 419569 KB (in 6558 reads )
IO Write : 1482029 KB (in 23157 writes)
IO Other : 4 KB (in 393 others)


Process ID : 11436
Thread ID : 3180
Process Exit Code: 0
Thread Exit Code : 0

User Time : 35.757s
Kernel Time : 70.593s
Process Time : 106.350s
Clock Time : 1520.643s

Working Set : 544004 KB
Paged Pool : 97 KB
Nonpaged Pool : 32 KB
Pagefile : 611236 KB
Page Fault Count : 28470168

IO Read : 419579 KB (in 6559 reads )
IO Write : 1511624 KB (in 23621 writes)
IO Other : 3 KB (in 304 others)

Press any key to continue . . .

[Webster17]

Thanks for the tests guys. Here's another one, I added reflate lib support on this test, the inputs I'm interested in is data where reflate is required otherwise it will be just the same results as test number 4.

Png images, pdfs, linux isos and zip files contain streams that can be processed by reflate, games however that's DOOM and Dishonored 2 afaik.

So just like test 4, post cmd output.

I just like to point out, I'm not interested in the size that much but memory used and the time it took to perform the test.

These are the results of the tests I have performed:
21616

Test 3 took seconds to recompress Darksiders 2 file anim_streams.upak but test 5 took more like precomp and the output file is smaller than test 3 output, BTW if you are "not interested in the size that much" why build it in the first place?

[Webster17]

oh and sorry i couldn't participate in the tests, i was sick just recovered today

[Razor12911]

DMCL-0.nbz (http://www.mediafire.com/file/7klm63vs4oo7koc/DMCL-0.nbz) (165 MB):eek:
Process ID : 6860
Thread ID : 5652
Process Exit Code: 0
Thread Exit Code : 0

User Time : 72.450s
Kernel Time : 9.078s
Process Time : 81.528s
Clock Time : 577.212s

Working Set : 369332 KB
Paged Pool : 97 KB
Nonpaged Pool : 12 KB
Pagefile : 442580 KB
Page Fault Count : 3711470

IO Read : 168982 KB (in 2642 reads )
IO Write : 548729 KB (in 8574 writes)
IO Other : 5 KB (in 293 others)


Process ID : 6420
Thread ID : 2940
Process Exit Code: 0
Thread Exit Code : 0

User Time : 188.023s
Kernel Time : 30.140s
Process Time : 218.163s
Clock Time : 1565.767s

Working Set : 440868 KB
Paged Pool : 96 KB
Nonpaged Pool : 32 KB
Pagefile : 501228 KB
Page Fault Count : 11514042

IO Read : 168982 KB (in 2643 reads )
IO Write : 573538 KB (in 8963 writes)
IO Other : 5 KB (in 304 others)


Dmc2-0.nbz (http://www.mediafire.com/file/31z5i3fg37f8jy9/dmc2-0.nbz) (409 MB):confused:
http://s8.picofile.com/file/8323070650/Xtool_Bug.jpg

from what game are these files coming from?

[Simorq]

Razor12911

Devil May Cry HD Collection

[Razor12911]

Razor12911

Devil May Cry HD Collection

Well it seems like the scanner is improved.

dmc2-0.nbz:
xtool: 409 MB > 1.48 GB > 162 MB
ztool: 409 MB > 1.44 GB > 181 MB

DMCL-0.nbz:
xtool: 165 MB > 617 MB > 130 MB
ztool: 165 MB > 560 MB > 148 MB

I have to run more tests though to confirm this.

[Simorq]

animations-part1.pak (334 MB) (Kingdom Come)
Process ID : 4524
Thread ID : 7792
Process Exit Code: 0
Thread Exit Code : 0

User Time : 75.093s
Kernel Time : 0.796s
Process Time : 75.889s
Clock Time : 51.127s

Working Set : 501300 KB
Paged Pool : 97 KB
Nonpaged Pool : 13 KB
Pagefile : 554228 KB
Page Fault Count : 336186

IO Read : 342891 KB (in 5359 reads )
IO Write : 561879 KB (in 8780 writes)
IO Other : 4 KB (in 293 others)


Process ID : 7052
Thread ID : 2176
Process Exit Code: 0
Thread Exit Code : 0

User Time : 301.703s
Kernel Time : 6.234s
Process Time : 307.937s
Clock Time : 284.318s

Working Set : 617036 KB
Paged Pool : 96 KB
Nonpaged Pool : 14 KB
Pagefile : 674440 KB
Page Fault Count : 3790034

IO Read : 342891 KB (in 5360 reads )
IO Write : 561876 KB (in 8781 writes)
IO Other : 4 KB (in 268 others)

animations-part1.pak
XTool: 334 MB > 548 MB > 301 MB
ZTool: 334 MB > 548 MB > 301 MB

[Simorq]

Asset2.bra (475 MB) (Tokyo Xanadu eX)
Process ID : 5792
Thread ID : 4656
Process Exit Code: 0
Thread Exit Code : 0

User Time : 251.675s
Kernel Time : 9.171s
Process Time : 260.846s
Clock Time : 202.494s

Working Set : 247496 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 268176 KB
Page Fault Count : 3418788

IO Read : 487304 KB (in 7616 reads )
IO Write : 839258 KB (in 13114 writes)
IO Other : 1 KB (in 165 others)


Process ID : 5656
Thread ID : 2980
Process Exit Code: 0
Thread Exit Code : 0

User Time : 154.153s
Kernel Time : 31.609s
Process Time : 185.762s
Clock Time : 571.207s

Working Set : 335744 KB
Paged Pool : 95 KB
Nonpaged Pool : 20 KB
Pagefile : 383708 KB
Page Fault Count : 12619767

IO Read : 487304 KB (in 7617 reads )
IO Write : 838611 KB (in 13105 writes)
IO Other : 1 KB (in 148 others)

Asset2.bra
XTool: 475 MB > 819 MB > 315 MB (330,378,763 Bytes)
ZTool: 475 MB > 818 MB > 315 MB (330,628,132 Bytes)

[Razor12911]

Nice, about 5 times faster :) while using a bit less memory

[Simorq]

EAC.tat (229 MB) (226 EAC file of Final Fantasy XV)
Process ID : 7912
Thread ID : 5432
Process Exit Code: 0
Thread Exit Code : 0

User Time : 19.859s
Kernel Time : 0.562s
Process Time : 20.421s
Clock Time : 9.266s

Working Set : 164912 KB
Paged Pool : 96 KB
Nonpaged Pool : 11 KB
Pagefile : 189484 KB
Page Fault Count : 241612

IO Read : 235237 KB (in 3677 reads )
IO Write : 257963 KB (in 4031 writes)
IO Other : 1 KB (in 165 others)


Process ID : 8056
Thread ID : 5940
Process Exit Code: 0
Thread Exit Code : 0

User Time : 30.250s
Kernel Time : 0.671s
Process Time : 30.921s
Clock Time : 9.985s

Working Set : 221512 KB
Paged Pool : 95 KB
Nonpaged Pool : 13 KB
Pagefile : 253772 KB
Page Fault Count : 318112

IO Read : 235237 KB (in 3678 reads )
IO Write : 257915 KB (in 4031 writes)
IO Other : 1 KB (in 131 others)

EAC.tat
XTool: 229 MB > 251 MB > 218 MB (229,372,585 Bytes)
ZTool: 229 MB > 251 MB > 218 MB (229,571,558 Bytes)

[Simorq]

sa.tar (524 MB) (119 .arc file of Devil May Cry 4 Special Edition)
Process ID : 6136
Thread ID : 4772
Process Exit Code: 0
Thread Exit Code : 0

User Time : 188.984s
Kernel Time : 2.265s
Process Time : 191.249s
Clock Time : 110.771s

Working Set : 772124 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 835900 KB
Page Fault Count : 439800

IO Read : 537106 KB (in 8394 reads )
IO Write : 3485762 KB (in 54466 writes)
IO Other : 1 KB (in 165 others)


Process ID : 3996
Thread ID : 4672
Process Exit Code: 0
Thread Exit Code : 0

User Time : 183.000s
Kernel Time : 2.718s
Process Time : 185.718s
Clock Time : 115.975s

Working Set : 840024 KB
Paged Pool : 95 KB
Nonpaged Pool : 13 KB
Pagefile : 948848 KB
Page Fault Count : 573475

IO Read : 537106 KB (in 8395 reads )
IO Write : 3485763 KB (in 54467 writes)
IO Other : 1 KB (in 135 others)

sa.tar
XTool: 524 MB > 3.32 GB > 19.1 MB (20,114,206 Bytes)
ZTool: 524 MB > 3.32 GB > 19.3 MB (20,242,361 Bytes)

[Simorq]

770000620.zip (76.7 MB) (41 dat file of Act Of Aggression Reboot Edition)
Process ID : 6512
Thread ID : 6772
Process Exit Code: 0
Thread Exit Code : 0

User Time : 211.828s
Kernel Time : 15.640s
Process Time : 227.468s
Clock Time : 64.519s

Working Set : 138124 KB
Paged Pool : 96 KB
Nonpaged Pool : 11 KB
Pagefile : 159664 KB
Page Fault Count : 7294826

IO Read : 78592 KB (in 1230 reads )
IO Write : 109977 KB (in 1719 writes)
IO Other : 1 KB (in 165 others)


Process ID : 4372
Thread ID : 7616
Process Exit Code: 0
Thread Exit Code : 0

User Time : 13.875s
Kernel Time : 0.359s
Process Time : 14.234s
Clock Time : 10.203s

Working Set : 163216 KB
Paged Pool : 95 KB
Nonpaged Pool : 12 KB
Pagefile : 183656 KB
Page Fault Count : 217615

IO Read : 78592 KB (in 1231 reads )
IO Write : 84815 KB (in 1327 writes)
IO Other : 1 KB (in 130 others)

770000620.zip
XTool: 76.7 MB > 107 MB > 23.7 MB
ZTool: 76.7 MB > 82.8 MB> 26.1 MB

[Simorq]

LINKDATA_ENG_LANG.BIN (83.1 MB) (Samurai Warriors Spirit Of Sanada)
Process ID : 6448
Thread ID : 4776
Process Exit Code: 0
Thread Exit Code : 0

User Time : 43.359s
Kernel Time : 0.937s
Process Time : 44.296s
Clock Time : 12.656s

Working Set : 514516 KB
Paged Pool : 96 KB
Nonpaged Pool : 13 KB
Pagefile : 600588 KB
Page Fault Count : 429073

IO Read : 85148 KB (in 1332 reads )
IO Write : 444360 KB (in 6944 writes)
IO Other : 1 KB (in 165 others)


Process ID : 7076
Thread ID : 5964
Process Exit Code: 0
Thread Exit Code : 0

User Time : 50.796s
Kernel Time : 1.406s
Process Time : 52.202s
Clock Time : 17.095s

Working Set : 509488 KB
Paged Pool : 95 KB
Nonpaged Pool : 13 KB
Pagefile : 575032 KB
Page Fault Count : 492995

IO Read : 85148 KB (in 1333 reads )
IO Write : 444253 KB (in 6943 writes)
IO Other : 1 KB (in 131 others)

LINKDATA_ENG_LANG.BIN
XTool: 83.1 MB > 433 MB > 34.3 MB
ZTool: 83.1 MB > 433 MB > 35.6 MB

[Simorq]

ReflateTEST.tar (21.5 MB) (txt/dat/png file)
Process ID : 1840
Thread ID : 5204
Process Exit Code: 0
Thread Exit Code : 0

User Time : 45.593s
Kernel Time : 0.453s
Process Time : 46.046s
Clock Time : 15.235s

Working Set : 336704 KB
Paged Pool : 96 KB
Nonpaged Pool : 11 KB
Pagefile : 448428 KB
Page Fault Count : 318504

IO Read : 22106 KB (in 347 reads )
IO Write : 204363 KB (in 3194 writes)
IO Other : 1 KB (in 165 others)


Process ID : 1812
Thread ID : 7996
Process Exit Code: 0
Thread Exit Code : 0

User Time : 294.703s
Kernel Time : 7.796s
Process Time : 302.499s
Clock Time : 93.083s

Working Set : 436028 KB
Paged Pool : 95 KB
Nonpaged Pool : 18 KB
Pagefile : 473932 KB
Page Fault Count : 3814774

IO Read : 22106 KB (in 348 reads )
IO Write : 206464 KB (in 3228 writes)
IO Other : 1 KB (in 134 others)

ReflateTEST.tar
XTool: 21.5 MB > 199 MB > 13.5 MB
ZTool: 21.5 MB > 201 MB > 15.6 MB

[Simorq]

cp_mi_sing_biodomes2.ff (333 MB) (Call of Duty Black Ops III)
Process ID : 1856
Thread ID : 2412
Process Exit Code: 0
Thread Exit Code : 0

User Time : 396.034s
Kernel Time : 19.156s
Process Time : 415.190s
Clock Time : 230.527s

Working Set : 287844 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 325872 KB
Page Fault Count : 9175977

IO Read : 341624 KB (in 5339 reads )
IO Write : 596941 KB (in 9328 writes)
IO Other : 1 KB (in 165 others)


Process ID : 2296
Thread ID : 508
Process Exit Code: 0
Thread Exit Code : 0

User Time : 88.509s
Kernel Time : 20.906s
Process Time : 109.415s
Clock Time : 378.870s

Working Set : 355288 KB
Paged Pool : 95 KB
Nonpaged Pool : 18 KB
Pagefile : 425748 KB
Page Fault Count : 9814034

IO Read : 341624 KB (in 5340 reads )
IO Write : 607518 KB (in 9494 writes)
IO Other : 1 KB (in 143 others)

cp_mi_sing_biodomes2.ff
XTool: 333 MB > 582 MB > 238 MB
ZTool: 333 MB > 593 MB > 254 MB

[Razor12911]

Final Zlib/Deflate test

Here's the final test guys, after this if no issues are found, I'll be posting the first working alpha/beta release.

What I'm looking for in this test is the tool working well, without any errors. upload files and pc specs if errors occured.

CRC MUST MATCH

A message in cmd output must show:
FC: no differences encountered

[Razor12911]

The next codec is Crilayla, I decided to postpone lz4.

So if you have games that use this compression algorithm, like Nier Automata and Sonic Forces I think... do run this test.

NB: I've set this to use 1 thread, it will be slow af so do run it on small files or big if you want, up to you.

What I'm looking for in this is any sort of precompression without errors.

If possible, could you also upload a few samples that I can work on for this codec.

[Simorq]

Crilayla Test0
NARUTO - Ultimate Ninja Storm (http://www.mediafire.com/file/tb1lp1nd4y8m9j4/NUNS.tar) >> Crilayla Stream but has XTool not identify.
NARUTO SHIPPUDEN - Ultimate Ninja STORM 2 (http://www.mediafire.com/file/tcm3k730nbmip6q/NSUNS2.tar) >> Crilayla Stream but has XTool not identify.

__________________________________________________ ___________________
Crilayla Test1
Naruto Shippuden Ultimate Ninja Storm Revolution
stage.cpk 1.40 GB >> 2.42 GB

Process ID : 3728
Thread ID : 844
Process Exit Code: 0
Thread Exit Code : 0

User Time : 300.425s
Kernel Time : 4.531s
Process Time : 304.956s
Clock Time : 786.084s

Working Set : 111264 KB
Paged Pool : 96 KB
Nonpaged Pool : 9 KB
Pagefile : 117988 KB
Page Fault Count : 39052

IO Read : 1471542 KB (in 22994 reads )
IO Write : 2544481 KB (in 39758 writes)
IO Other : 1 KB (in 189 others)
__________________________________________________ ___________________
Crilayla Test2
NARUTO SHIPPUDEN - Ultimate Ninja STORM 3 Full Burst HD (Crilayla Stream)
dlc.cpk 39.4 MB >> 180 MB

Process ID : 7860
Thread ID : 7924
Process Exit Code: 0
Thread Exit Code : 0

User Time : 113.250s
Kernel Time : 0.421s
Process Time : 113.671s
Clock Time : 118.511s

Working Set : 158236 KB
Paged Pool : 97 KB
Nonpaged Pool : 9 KB
Pagefile : 175048 KB
Page Fault Count : 59566

IO Read : 40370 KB (in 632 reads )
IO Write : 184382 KB (in 2881 writes)
IO Other : 4 KB (in 309 others)

[Gupta]

DeflateTest

lolz was running in the background:D



Process ID : 15300
Thread ID : 46132
Process Exit Code: 0
Thread Exit Code : 0

User Time : 312.703s
Kernel Time : 1.875s
Process Time : 314.578s
Clock Time : 105.089s

Working Set : 365792 KB
Paged Pool : 94 KB
Nonpaged Pool : 12 KB
Pagefile : 392984 KB
Page Fault Count : 472853

IO Read : 915372 KB (in 14304 reads )
IO Write : 2078066 KB (in 32470 writes)
IO Other : 0 KB (in 198 others)


Process ID : 30500
Thread ID : 19324
Process Exit Code: 0
Thread Exit Code : 0

User Time : 243.078s
Kernel Time : 1.421s
Process Time : 244.499s
Clock Time : 75.590s

Working Set : 96028 KB
Paged Pool : 94 KB
Nonpaged Pool : 10 KB
Pagefile : 109796 KB
Page Fault Count : 214348

IO Read : 2078066 KB (in 32471 reads )
IO Write : 915372 KB (in 14303 writes)
IO Other : 0 KB (in 200 others)

Comparing files x64w.rpf and X64W.RPF.RES
FC: no differences encountered




Process ID : 15868
Thread ID : 45756
Process Exit Code: 0
Thread Exit Code : 0

User Time : 295.388s
Kernel Time : 204.062s
Process Time : 499.450s
Clock Time : 653.980s

Working Set : 1070020 KB
Paged Pool : 94 KB
Nonpaged Pool : 11 KB
Pagefile : 1163620 KB
Page Fault Count : 161992419

IO Read : 581265 KB (in 9084 reads )
IO Write : 2173657 KB (in 33964 writes)
IO Other : 0 KB (in 199 others)


Process ID : 51808
Thread ID : 31008
Process Exit Code: 0
Thread Exit Code : 259

User Time : 128.328s
Kernel Time : 13.562s
Process Time : 141.890s
Clock Time : 44.464s

Working Set : 213472 KB
Paged Pool : 94 KB
Nonpaged Pool : 10 KB
Pagefile : 288720 KB
Page Fault Count : 8609857

IO Read : 2173657 KB (in 33965 reads )
IO Write : 581265 KB (in 9083 writes)
IO Other : 0 KB (in 292 others)

Comparing files Books.tar and BOOKS.TAR.RES
FC: no differences encountered

[Razor12911]

Crilayla Test0
NARUTO - Ultimate Ninja Storm (http://www.mediafire.com/file/tb1lp1nd4y8m9j4/NUNS.tar) >> Crilayla Stream but has XTool not identify.
NARUTO SHIPPUDEN - Ultimate Ninja STORM 2 (http://www.mediafire.com/file/tcm3k730nbmip6q/NSUNS2.tar) >> Crilayla Stream but has XTool not identify.


These were not identified because they are encrypted :)
Support for these will be added later on.

[Razor12911]

DeflateTest

lolz was running in the background:D


Page Fault Count : 161992419
Page Fault Count : 8609857


tf is going on here -_- :mad:

[Simorq]

Crilayla Test1
Bayonetta
data00_us.cpk 183 MB >> 738 MB

Process ID : 4248
Thread ID : 4504
Process Exit Code: 0
Thread Exit Code : 0

User Time : 190.800s
Kernel Time : 3.687s
Process Time : 194.487s
Clock Time : 785.673s

Working Set : 214320 KB
Paged Pool : 97 KB
Nonpaged Pool : 9 KB
Pagefile : 249388 KB
Page Fault Count : 93659

IO Read : 187459 KB (in 2931 reads )
IO Write : 756445 KB (in 11820 writes)
IO Other : 4 KB (in 309 others)

Crilayla Test2
Bayonetta
data01.cpk 8.36 MB >> 1.24 GB

Process ID : 7648
Thread ID : 6912
Process Exit Code: 0
Thread Exit Code : 0

User Time : 317.843s
Kernel Time : 2.218s
Process Time : 320.061s
Clock Time : 344.885s

Working Set : 112984 KB
Paged Pool : 96 KB
Nonpaged Pool : 9 KB
Pagefile : 119104 KB
Page Fault Count : 43570

IO Read : 856510 KB (in 13384 reads )
IO Write : 1309975 KB (in 20469 writes)
IO Other : 1 KB (in 190 others)

[Simorq]

DeflateTest1 (GTA V)
x64w.rpf 893 MB >> 1.98 GB >> CRC OK

Process ID : 7648
Thread ID : 7084
Process Exit Code: 0
Thread Exit Code : 0

User Time : 335.578s
Kernel Time : 4.265s
Process Time : 339.843s
Clock Time : 125.620s

Working Set : 343212 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 373892 KB
Page Fault Count : 470257

IO Read : 915372 KB (in 14304 reads )
IO Write : 2078066 KB (in 32470 writes)
IO Other : 2 KB (in 168 others)


Process ID : 7160
Thread ID : 7540
Process Exit Code: 0
Thread Exit Code : 0

User Time : 271.015s
Kernel Time : 2.828s
Process Time : 273.843s
Clock Time : 88.649s

Working Set : 96012 KB
Paged Pool : 96 KB
Nonpaged Pool : 10 KB
Pagefile : 109756 KB
Page Fault Count : 215549

IO Read : 2078066 KB (in 32471 reads )
IO Write : 915372 KB (in 14303 writes)
IO Other : 2 KB (in 170 others)

__________________________________________________ _________
DeflateTest2 (GTA V)
x64g.rpf 2.37 GB >> 4.41 GB >> CRC OK

Process ID : 1408
Thread ID : 7728
Process Exit Code: 0
Thread Exit Code : 0

User Time : 187.472s
Kernel Time : 7.750s
Process Time : 195.222s
Clock Time : 320.236s

Working Set : 312852 KB
Paged Pool : 96 KB
Nonpaged Pool : 12 KB
Pagefile : 347636 KB
Page Fault Count : 424210

IO Read : 2492770 KB (in 38951 reads )
IO Write : 4630875 KB (in 72358 writes)
IO Other : 2 KB (in 168 others)

Process ID : 7464
Thread ID : 8156
Process Exit Code: 0
Thread Exit Code : 0

User Time : 44.581s
Kernel Time : 5.718s
Process Time : 50.299s
Clock Time : 218.178s

Working Set : 103316 KB
Paged Pool : 96 KB
Nonpaged Pool : 10 KB
Pagefile : 112692 KB
Page Fault Count : 243067

IO Read : 4630875 KB (in 72359 reads )
IO Write : 2492770 KB (in 38950 writes)
IO Other : 2 KB (in 170 others)

[Razor12911]

Crilayla Test1
Bayonetta
data00_us.cpk 183 MB >> 738 MB

Process ID : 4248
Thread ID : 4504
Process Exit Code: 0
Thread Exit Code : 0

User Time : 190.800s
Kernel Time : 3.687s
Process Time : 194.487s
Clock Time : 785.673s

Working Set : 214320 KB
Paged Pool : 97 KB
Nonpaged Pool : 9 KB
Pagefile : 249388 KB
Page Fault Count : 93659

IO Read : 187459 KB (in 2931 reads )
IO Write : 756445 KB (in 11820 writes)
IO Other : 4 KB (in 309 others)

Crilayla Test2
Bayonetta
data01.cpk 8.36 MB >> 1.24 GB

Process ID : 7648
Thread ID : 6912
Process Exit Code: 0
Thread Exit Code : 0

User Time : 317.843s
Kernel Time : 2.218s
Process Time : 320.061s
Clock Time : 344.885s

Working Set : 112984 KB
Paged Pool : 96 KB
Nonpaged Pool : 9 KB
Pagefile : 119104 KB
Page Fault Count : 43570

IO Read : 856510 KB (in 13384 reads )
IO Write : 1309975 KB (in 20469 writes)
IO Other : 1 KB (in 190 others)

Crilayla seems to have very less problems apart from its inability to support multiple threads, thanks for the tests.

[Simorq]

DeflateTest (Kingdom Come)
svo.pak 887 MB >> 5.78 GB >> CRC OK

Process ID : 4020
Thread ID : 6096
Process Exit Code: 0
Thread Exit Code : 0

User Time : 173.890s
Kernel Time : 20.515s
Process Time : 194.405s
Clock Time : 117.754s

Working Set : 736760 KB
Paged Pool : 97 KB
Nonpaged Pool : 13 KB
Pagefile : 851568 KB
Page Fault Count : 3658552

IO Read : 909291 KB (in 14209 reads )
IO Write : 6067392 KB (in 94804 writes)
IO Other : 5 KB (in 296 others)


Process ID : 3524
Thread ID : 7160
Process Exit Code: 0
Thread Exit Code : 0

User Time : 74.406s
Kernel Time : 3.953s
Process Time : 78.359s
Clock Time : 72.793s

Working Set : 266276 KB
Paged Pool : 96 KB
Nonpaged Pool : 10 KB
Pagefile : 296100 KB
Page Fault Count : 189961

IO Read : 6067392 KB (in 94805 reads )
IO Write : 909291 KB (in 14208 writes)
IO Other : 2 KB (in 170 others)

[Gupta]

without lolz, file contains like 30 pdfs


Process ID : 10404
Thread ID : 49240
Process Exit Code: 0
Thread Exit Code : 0

User Time : 372.837s
Kernel Time : 162.328s
Process Time : 535.165s
Clock Time : 469.758s

Working Set : 1089552 KB
Paged Pool : 94 KB
Nonpaged Pool : 11 KB
Pagefile : 1187040 KB
Page Fault Count : 162383167

IO Read : 581265 KB (in 9084 reads )
IO Write : 2173605 KB (in 33963 writes)
IO Other : 0 KB (in 198 others)


Process ID : 46140
Thread ID : 14676
Process Exit Code: 0
Thread Exit Code : 0

User Time : 109.453s
Kernel Time : 12.687s
Process Time : 122.140s
Clock Time : 33.501s

Working Set : 213324 KB
Paged Pool : 94 KB
Nonpaged Pool : 10 KB
Pagefile : 288580 KB
Page Fault Count : 8607527

IO Read : 2173605 KB (in 33964 reads )
IO Write : 581265 KB (in 9083 writes)
IO Other : 0 KB (in 292 others)

[Panky]

Zlib/Deflate test

gamedata.pak (KCD)

Process ID : 8376
Thread ID : 5508
Process Exit Code: 0
Thread Exit Code : 0

User Time : 37.093s
Kernel Time : 0.703s
Process Time : 37.796s
Clock Time : 19.000s

Working Set : 323560 KB
Paged Pool : 99 KB
Nonpaged Pool : 13 KB
Pagefile : 338392 KB
Page Fault Count : 264976

IO Read : 262307 KB (in 4100 reads )
IO Write : 501890 KB (in 7843 writes)
IO Other : 2 KB (in 263 others)


Process ID : 1496
Thread ID : 4740
Process Exit Code: 0
Thread Exit Code : 0

User Time : 29.281s
Kernel Time : 0.328s
Process Time : 29.609s
Clock Time : 9.421s

Working Set : 115188 KB
Paged Pool : 99 KB
Nonpaged Pool : 10 KB
Pagefile : 129092 KB
Page Fault Count : 144560

IO Read : 501890 KB (in 7844 reads )
IO Write : 262307 KB (in 4099 writes)
IO Other : 2 KB (in 265 others)

CRC IS OK

[Panky]

Zlib/Deflate test

HearthFires.bsa (Skyrim LE)

Process ID : 2724
Thread ID : 8328
Process Exit Code: 0
Thread Exit Code : 0

User Time : 33.015s
Kernel Time : 3.765s
Process Time : 36.780s
Clock Time : 10.645s

Working Set : 111688 KB
Paged Pool : 99 KB
Nonpaged Pool : 11 KB
Pagefile : 111800 KB
Page Fault Count : 812122

IO Read : 136965 KB (in 2142 reads )
IO Write : 143758 KB (in 2247 writes)
IO Other : 2 KB (in 263 others)


Process ID : 2216
Thread ID : 8032
Process Exit Code: 0
Thread Exit Code : 0

User Time : 7.546s
Kernel Time : 0.125s
Process Time : 7.671s
Clock Time : 2.237s

Working Set : 24012 KB
Paged Pool : 99 KB
Nonpaged Pool : 10 KB
Pagefile : 21932 KB
Page Fault Count : 14834

IO Read : 143758 KB (in 2248 reads )
IO Write : 136965 KB (in 2141 writes)
IO Other : 2 KB (in 265 others)

CRC IS OK

[Panky]

Zlib/Deflate test

SEQ.zip (The Last Express)

Process ID : 10784
Thread ID : 2296
Process Exit Code: 0
Thread Exit Code : 0

User Time : 25.718s
Kernel Time : 0.781s
Process Time : 26.499s
Clock Time : 10.368s

Working Set : 246348 KB
Paged Pool : 99 KB
Nonpaged Pool : 13 KB
Pagefile : 296136 KB
Page Fault Count : 200432

IO Read : 166652 KB (in 2605 reads )
IO Write : 335701 KB (in 5246 writes)
IO Other : 2 KB (in 263 others)


Process ID : 8332
Thread ID : 7896
Process Exit Code: 0
Thread Exit Code : 0

User Time : 21.968s
Kernel Time : 0.421s
Process Time : 22.389s
Clock Time : 5.772s

Working Set : 76664 KB
Paged Pool : 99 KB
Nonpaged Pool : 10 KB
Pagefile : 82660 KB
Page Fault Count : 102346

IO Read : 335701 KB (in 5247 reads )
IO Write : 166652 KB (in 2604 writes)
IO Other : 2 KB (in 265 others)

CRC IS OK

[Panky]

Zlib/Deflate test

textures-s3.dat (This War of Mine)

Process ID : 6888
Thread ID : 10844
Process Exit Code: 0
Thread Exit Code : 0

User Time : 315.565s
Kernel Time : 12.546s
Process Time : 328.111s
Clock Time : 246.563s

Working Set : 440720 KB
Paged Pool : 99 KB
Nonpaged Pool : 13 KB
Pagefile : 465416 KB
Page Fault Count : 6880946

IO Read : 880248 KB (in 13755 reads )
IO Write : 1943962 KB (in 30375 writes)
IO Other : 2 KB (in 529 others)


Process ID : 10936
Thread ID : 196
Process Exit Code: 0
Thread Exit Code : 0

User Time : 151.859s
Kernel Time : 1.921s
Process Time : 153.780s
Clock Time : 52.901s

Working Set : 101616 KB
Paged Pool : 99 KB
Nonpaged Pool : 11 KB
Pagefile : 130124 KB
Page Fault Count : 555441

IO Read : 1943962 KB (in 30376 reads )
IO Write : 880248 KB (in 13754 writes)
IO Other : 2 KB (in 265 others)

CRC IS OK

[Panky]

Crilayla test

b01.cpk (Ni no Kuni II Revenant Kingdom)


Process ID : 6940
Thread ID : 7036
Process Exit Code: 0
Thread Exit Code : 0

User Time : 3.578s
Kernel Time : 0.062s
Process Time : 3.640s
Clock Time : 3.838s

Working Set : 24056 KB
Paged Pool : 100 KB
Nonpaged Pool : 9 KB
Pagefile : 50852 KB
Page Fault Count : 8050

IO Read : 3790 KB (in 61 reads )
IO Write : 7421 KB (in 116 writes)
IO Other : 5 KB (in 529 others)

[Jiva newstone]

xtool also working on xdelta patched, i mean pzlib not working fine, but xtl is working

[Razor12911]

Here's Crilayla test 2

You must just get the same output.

[Simorq]

Crilayla Test
Naruto Revolution data.cpk 1.29 GB >> 3.34 GB

Process ID : 1076
Thread ID : 6320
Process Exit Code: 0
Thread Exit Code : 0

User Time : 405.037s
Kernel Time : 5.171s
Process Time : 410.208s
Clock Time : 1341.003s

Working Set : 156948 KB
Paged Pool : 97 KB
Nonpaged Pool : 9 KB
Pagefile : 169172 KB
Page Fault Count : 56732

IO Read : 1353038 KB (in 21143 reads )
IO Write : 3507341 KB (in 54803 writes)
IO Other : 4 KB (in 310 others)


Process ID : 4476
Thread ID : 5888
Process Exit Code: 0
Thread Exit Code : 0

User Time : 96.638s
Kernel Time : 7.546s
Process Time : 104.184s
Clock Time : 593.379s

Working Set : 430296 KB
Paged Pool : 97 KB
Nonpaged Pool : 15 KB
Pagefile : 474600 KB
Page Fault Count : 186584

IO Read : 1353038 KB (in 21143 reads )
IO Write : 3507341 KB (in 54803 writes)
IO Other : 5 KB (in 229 others)

[Mini]

Metal Gear Rising - Revengeance data106.cpk (106 MB)

data106.cpk.out1 307 MB (322 219 952)
data106.cpk.out2 307 MB (322 219 952)


Process ID : 16504
Thread ID : 16288
Process Exit Code: 0
Thread Exit Code : 0

User Time : 116.656s
Kernel Time : 0.218s
Process Time : 116.874s
Clock Time : 118.159s

Working Set : 132664 KB
Paged Pool : 99 KB
Nonpaged Pool : 9 KB
Pagefile : 145952 KB
Page Fault Count : 49280

IO Read : 102177 KB (in 1598 reads )
IO Write : 314667 KB (in 4917 writes)
IO Other : 6 KB (in 2014 others)


Process ID : 9808
Thread ID : 14012
Process Exit Code: 0
Thread Exit Code : 0

User Time : 142.437s
Kernel Time : 0.359s
Process Time : 142.796s
Clock Time : 46.794s

Working Set : 354384 KB
Paged Pool : 99 KB
Nonpaged Pool : 14 KB
Pagefile : 386788 KB
Page Fault Count : 151306

IO Read : 102177 KB (in 1598 reads )
IO Write : 314667 KB (in 4917 writes)
IO Other : 7 KB (in 908 others)

[Mini]

Metal Gear Rising - Revengeance data001.cpk (1.02 GB)

data001.cpk.out1 1.61 GB (1 738 216 504)
data001.cpk.out2 1.61 GB (1 738 216 504)


Process ID : 744
Thread ID : 13628
Process Exit Code: 0
Thread Exit Code : 0

User Time : 312.843s
Kernel Time : 0.703s
Process Time : 313.546s
Clock Time : 314.184s

Working Set : 162948 KB
Paged Pool : 98 KB
Nonpaged Pool : 9 KB
Pagefile : 174120 KB
Page Fault Count : 58124

IO Read : 1076354 KB (in 16820 reads )
IO Write : 1697477 KB (in 26524 writes)
IO Other : 2 KB (in 1426 others)


Process ID : 2316
Thread ID : 5384
Process Exit Code: 0
Thread Exit Code : 0

User Time : 340.703s
Kernel Time : 0.656s
Process Time : 341.359s
Clock Time : 145.316s

Working Set : 449300 KB
Paged Pool : 98 KB
Nonpaged Pool : 14 KB
Pagefile : 487096 KB
Page Fault Count : 214743

IO Read : 1076354 KB (in 16820 reads )
IO Write : 1697477 KB (in 26524 writes)
IO Other : 3 KB (in 500 others)

[darkwolves]

i want to test crilayla i am in the process of messing with a few games that have it so i have downloaded the file

[Razor12911]

@darkwolves
Sweet

@everyone
thanks for the tests so far, the first release of xtool will come with two codecs, zlib and crilayla. It should be released this week and if no errors are arise then the next codec's development will begin.

[darkwolves]

[QUOTE=Razor12911;470200]@darkwolves
Sweet

its amazing that lz4 was still working on newer games despite not being entirely completed too

[KaktoR]

thanks for the tests so far, the first release of xtool will come with two codecs, zlib and crilayla. It should be released this week and if no errors are arise then the next codec's development will begin.

Nice, can't wait. Then i can finally test KCD + Sims 4 extracting again (ztool stucks)

[KaktoR]

wars_dlc01.cpk (301mb) from Sonic Forces
wars_dlc01.cpk.out1 (331mb)
wars_dlc01.cpk.out2 (331mb)

Process ID : 10716
Thread ID : 8412
Process Exit Code: 0
Thread Exit Code : 0

User Time : 24.312s
Kernel Time : 0.343s
Process Time : 24.655s
Clock Time : 27.187s

Working Set : 97200 KB
Paged Pool : 100 KB
Nonpaged Pool : 9 KB
Pagefile : 108844 KB
Page Fault Count : 34175

IO Read : 309030 KB (in 4830 reads )
IO Write : 339319 KB (in 5302 writes)
IO Other : 5 KB (in 325 others)


Process ID : 4276
Thread ID : 9316
Process Exit Code: 0
Thread Exit Code : 0

User Time : 24.593s
Kernel Time : 0.328s
Process Time : 24.921s
Clock Time : 14.906s

Working Set : 173012 KB
Paged Pool : 101 KB
Nonpaged Pool : 13 KB
Pagefile : 176044 KB
Page Fault Count : 58016

IO Read : 309030 KB (in 4830 reads )
IO Write : 339319 KB (in 5302 writes)
IO Other : 5 KB (in 233 others)

wars_0.cpk (11.4GB) from Sonic Forces
wars_0.cpk.out1 (20.3GB)
wars_0.cpk.out2 (20.3GB)

Process ID : 10356
Thread ID : 6332
Process Exit Code: 0
Thread Exit Code : 0

User Time : 405.599s
Kernel Time : 12.093s
Process Time : 417.692s
Clock Time : 7466.814s

Working Set : 444504 KB
Paged Pool : 99 KB
Nonpaged Pool : 9 KB
Pagefile : 496884 KB
Page Fault Count : 148338

IO Read : 12000054 KB (in 187503 reads )
IO Write : 21291630 KB (in 332682 writes)
IO Other : 2 KB (in 210 others)


Process ID : 3188
Thread ID : 7760
Process Exit Code: 0
Thread Exit Code : 0

User Time : 157.199s
Kernel Time : 23.765s
Process Time : 180.964s
Clock Time : 5138.080s

Working Set : 1042488 KB
Paged Pool : 99 KB
Nonpaged Pool : 13 KB
Pagefile : 1159664 KB
Page Fault Count : 653581

IO Read : 12000053 KB (in 187503 reads )
IO Write : 21291630 KB (in 332682 writes)
IO Other : 2 KB (in 136 others)

[Sergey3695]

del

[Razor12911]

Crilayla test #3

Test this on the Naruto series, some streams are left behind, I have no idea how to decrypt all of them at the moment.

What is tested here is to see if any sort of inflation happens on all the games of the series.

[Andu21]

Razor, why this happens? Shouldn't the compressed output be less?

TombRaider_Beginning.pcf 4.92 GB > 491 MB
TombRaider_Beginning.xtl 4.99 GB > 660 MB

[darkwolves]

Razor, why this happens? Shouldn't the compressed output be less?

TombRaider_Beginning.pcf 4.92 GB > 491 MB
TombRaider_Beginning.xtl 4.99 GB > 660 MB

the pcf looks to be normal precomp format
what are your arc.ini settings for xtool?

[Andu21]

Default settings from xtool 01.

[darkwolves]

Default settings from xtool 01.

[External compressor:xprecomp]
header = 0
packcmd = xtool.exe e:precomp:c32mb,t3 - - <stdin> <stdout>
unpackcmd = xtool.exe d:precomp:t75p - - <stdin> <stdout>

this?

[Andu21]

Yes, but not through FA.

[darkwolves]

Yes, but not through FA.

i am having no problems with it like that i'm using a compressor similar to pankers

[Andu21]

AFAIK the more the inflation the less the final size, not the case now and i wonder why.

[ZakirAhmad]

AFAIK the more the inflation the less the final size, not the case now and i wonder why.

Not the case always.

[darkwolves]

what comes after the inflation is the crucial part i think... some compressors work better for certain games

[ShivShubh]

AFAIK the more the inflation the less the final size, not the case now and i wonder why.

Thats always true, more inflation = more compression by a stronger compressor. But thats not the case here, the extra inflation that you see in these situations is not actually extra inflation but it is actually the loss data, in other ways you can understand it as patch data which is required to restore the new zlib/deflate stream created by the tool to original just like how reflate creates raw files. XTool is able to properly restore streams almost the same as original streams hence it doesn't have much loss data (the extra inflation which it is not) as compared to others.

[elit]

Razor if I may recommend you, have a look at cpk.bms script from standard quickbms scripts set. Reason why is because it handle all kind of formats and coding and using it so far I was able to unpack any cpk file from all kind of games, except one single cpk file. It could help you broad xtool's CRI compatibility further.

[ShivShubh]

Razor if I may recommend you, have a look at cpk.bms script from standard quickbms scripts set. Reason why is because it handle all kind of formats and coding and using it so far I was able to unpack any cpk file from all kind of games, except one single cpk file. It could help you broad xtool's CRI compatibility further.

In which case xtool's crilayla didnt work for you ? I have tested it on more than a few games, it worked everywhere except for the encrypted games which is obvious.

[darkwolves]

the cpk.bms has nothing whatsoever to do with actual inflation it is designed to extract and decompress the file hence the bigger size that is not inflation that is just a result of decompiling the data..

[RamiroCruzo]

the cpk.bms has nothing whatsoever to do with actual inflation it is designed to extract and decompress the file hence the bigger size that is not inflation that is just a result of decompiling the data..

Ouch...You really shouldn't reply when you don't know, each file extracted by cpk.bms is infact a Crilayla stream which is extracted by XTool too, so apart from size of tables in .cpk files, their results are identical unless there is encryption on top of streams.

[darkwolves]

Ouch...You really shouldn't reply when you don't know, each file extracted by cpk.bms is infact a Crilayla stream which is extracted by XTool too, so apart from size of tables in .cpk files, their results are identical unless there is encryption on top of streams.

ok whatever you say i've been using it for a long time i believe what i see from my own personal experiments not what people tell me

[Razor12911]

Razor, why this happens? Shouldn't the compressed output be less?

TombRaider_Beginning.pcf 4.92 GB > 491 MB
TombRaider_Beginning.xtl 4.99 GB > 660 MB

Can you rerun the test using 0.2, thanks in advance

@Shiv, Ramiro and darkwolves
From my personal experience with crilayla, what darkwolves is saying is right but I'm not going through that source, it's exhausting to translate all that stuff to Delphi, however what Ramiro and Shiv are saying is also correct to a certain extent, it's the same thing, it only differs if the the stream do not start with CRILAYLA, either because they are encrypted or it's just missing, there were a few samples I have received where instead of CRILAYLA header you just find a null header, in this case, xtool approach will not work. But keep up with the debate, it's good.

[darkwolves]

Can you rerun the test using 0.2, thanks in advance

@Shiv, Ramiro and darkwolves
From my personal experience with crilayla, what darkwolves is saying is right but I'm not going through that source, it's exhausting to translate all that stuff to Delphi, however what Ramiro and Shiv are saying is also correct to a certain extent, it's the same thing, it only differs if the the stream do not start with CRILAYLA, either because they are encrypted or it's just missing, there were a few samples I have received where instead of CRILAYLA header you just find a null header, in this case, xtool approach will not work. But keep up with the debate, it's good.

i been repacking crilayla games for some time now its funny this is like the third time someone has said i was wrong and razor came in later and told them i was right lol

[ShivShubh]

i been repacking crilayla games for some time now its funny this is like the third time someone has said i was wrong and razor came in later and told them i was right lol

Or maybe there was just a misunderstanding ? Because I can't properly understand what you are trying to say by this :-

the cpk.bms has nothing whatsoever to do with actual inflation it is designed to extract and decompress the file hence the bigger size that is not inflation that is just a result of decompiling the data..

CPK bms by aluigi has the algorithm (comtype cpk) decompression function, the compressed streams are actually decompressed during the extraction process which is equivalent to the inflation done by XTool.

Razor, why this happens? Shouldn't the compressed output be less?

TombRaider_Beginning.pcf 4.92 GB > 491 MB
TombRaider_Beginning.xtl 4.99 GB > 660 MB

Can you rerun the test using 0.2, thanks in advance

Results made using XTool 0.2 on TombRaider_Beginning.pdf (904 MB)

PDF (904 MB) --> precomp (4.92 GB) --> 4x4:b32m:t4:lzma (550 MB)
PDF (904 MB) --> xtool, c32m (4.98 GB) --> 4x4:b32m:t4:lzma (710 MB)
PDF (904 MB) --> xtool, c256m (5.05 GB) --> 4x4:b32m:t4:lzma (686 MB)

Seems like other pdf files will produce similar differences. Precomp has inbuilt PDF scanner so it is able to properly scan and inflate all the PDF streams hence the better output.

[darkwolves]

because it is designed to decompile and extract once extracted it no longer even reads as a cpk file until it is recompressed and encrypted with a cpk tool it would take razor forever to be able to read through all that data and try implementing it into xtool

[ShivShubh]

because it is designed to decompile and extract once extracted it no longer even reads as a cpk file until it is recompressed and encrypted with a cpk tool it would take razor forever to be able to read through all that data and try implementing it into xtool

Why would it be able to read as cpk file after extraction ? Its obvious it wont be read as CPK. And as far as I think XTool's crilayla is based on cpk bms thats why there's no x64 build for it.

[darkwolves]

Why would it be able to read as cpk file after extraction ? Its obvious it wont be read as CPK. And as far as I think XTool's crilayla is based on cpk bms thats why there's no x64 build for it.

if it is extracted it needs to be recompressed by a cpk tool the way razor is making it is were the file inflates not extracts he even said above
how exhausting it is to have to translate all of that

[ShivShubh]

he even said above how exhausting it is to have to translate all of that

You understood that incorrectly, he meant to say it is exhausting to translate CPK's BMS language into Delphi which is a totally different thing.

[darkwolves]

You understood that incorrectly, he meant to say it is exhausting to translate CPK's BMS language into Delphi which is a totally different thing.

oh no i understood that very correctly considering... i have done that enough myself trying to develop my own tool for it his way is faster and its easier so him using a cpk bms would only hinder him and make the project take longer.... and it would only work to a point because some cpk games will not even work with the bms i began using a different tool that extracts all cpk files as well as viewing the content within deeming the cpk bms unreliable

[shazzla]

@Razor12911

its not about ratio ,just for the records :)

XTool V0.2 x64

[External compressor:xprecomp]
header = 0
packcmd = xtool.exe e:precomp:c64mb,t75p:zlib - - <stdin> <stdout>
unpackcmd = xtool.exe d:precomp:t75p - - <stdin> <stdout>

arc a -mxprecomp p:\x.arc "d:\00_win32_audio.zip"

FreeArc 0.67 (March 15 2014) creating archive: p:\x.arc
Compressed 1 file, 1,870,559,418 => 1,870,559,726 bytes. Ratio 100.00%
Compression time: cpu 3.23 sec/real 67.89 sec = 5%. Speed 27.55 mB/s
All OK

arc t p:\x.arc

FreeArc 0.67 (March 15 2014) testing archive: p:\x.arc
Testing 1 file, 1,870,559,418 bytes. Processed 89.0%WARNING: CRC failed in "00_win32_audio.zip".
Tested 1 file, 1,870,559,726 => 1,870,559,418 bytes. Ratio 100.00%
Testing time: cpu 1.78 sec/real 2.32 sec = 77%. Speed 804.75 mB/s
There were 1 warning(s)

dataset : (1.74GB)

https://mega.nz/#!g4phDAiZ!bUFwMndnlw-lZZh_Ufrmp2DR9qf5Z9i3Mnf3PclAKi8

[darkwolves]

@shazzla

i just use msc+srep+lzma on those files and call it a day lol

[ZakirAhmad]

@razor. i tried xtools zlib preprocessor against sevral game files like tomb raider 2013, mass effect 3, splinter cell etc. in all cases, in all cases it produced same result as is produced by precomp042 and also performed better than ztool (speed as well as precompression). hence, i consider it stable.

[Razor12911]

Can someone upload samples from games that utilize Anvil Engine, thank you.

Those
https://en.wikipedia.org/wiki/AnvilNext

[KaktoR]

Which Anvil engine?
Scimitar?
Anvil?
AnvilNext?
AnvilNext 2?

[Razor12911]

Pretty much all of them.

[KaktoR]

Pretty much all of them.

Here are samples from Origins, which is AnvilNext2. The only Anvil files i have atm on hard drive. Have fun

https://www96.zippyshare.com/v/klkEQZ94/file.html

[Razor12911]

Here are samples watch dog https://www106.zippyshare.com/v/lx6ORQ65/file.html

Isn't there a file with the same name but with ".fat" extension that goes with this?

[Razor12911]

all the file's that uploaded work with afr and afr:a2 for unity and syndicate

Great stuff, time to get to work :)

[doofoo24]

i deleted the link's to aviod problem...
i assume you downloaded the samples

[kassane]

Divinity Original Sin 2 - Samples:
GDrive folder (https://drive.google.com/drive/folders/1bfoaZQvf998VXKlJBFoRE75xldLnO1m0?usp=sharing)

[Razor12911]

Divinity Original Sin 2 - Samples:
GDrive folder (https://drive.google.com/drive/folders/1bfoaZQvf998VXKlJBFoRE75xldLnO1m0?usp=sharing)

I did run a test, but forgot to report results but it seems to work with xtool with it being set to zlib.

FG repack has this to report:
Repack uses ZTool library by Razor12911

[Razor12911]

Here's a test for a possible addition to xtool operation option

Try testing the xpreproc method on the entire game, edit the batch file to specify game location.

The game must have zlib streams, best examples would be Forza Horizon 3, GTA and so forth.

Report something like this
Compressed 1 file, 429,620,439 => 294,418,241 bytes. Ratio 68.53%
Compression time: cpu 0.47 sec/real 10.40 sec = 5%. Speed 41.32 mB/s

Only do this test below after the first one if possible, it's just up to you.

You can apply xprecomp if you want to perform further tests then srep afterwards
basically something like this:

xprecomp+srep
vs
xpreproc+xprecomp+srep

I'm interested srep's decompression memory so post the usage of memory for when xpreproc was applied and when it wasn't

[Simorq]

LINKDATA_ENG_LANG.BIN 83 MB
-mxpreproc
Compressed 1 file, 87,191,552 => 77,684,585 bytes. Ratio 89.10%
Compression time: cpu 0.14 sec/real 1.35 sec = 10%. Speed 64.47 mB/s
All OK

-mxprecomp
Compressed 1 file, 87,191,552 => 454,736,069 bytes. Ratio 521.54%
Compression time: cpu 0.14 sec/real 10.47 sec = 1%. Speed 8.33 mB/s
All OK

-mxpreproc+xprecomp
Compressed 1 file, 87,191,552 => 236,598,374 bytes. Ratio 271.35%
Compression time: cpu 0.09 sec/real 9.82 sec = 1%. Speed 8.88 mB/s
All OK

[Simorq]

Mafia3 > fmv_dlc1_0090_goobye_roxy.sds 25.1 MB
-mxpreproc
Compressed 1 file, 26,332,840 => 26,332,840 bytes. Ratio 100.00%
Compression time: cpu 0.02 sec/real 0.46 sec = 3%. Speed 56.86 mB/s
All OK

-mxprecomp
Compressed 1 file, 26,332,840 => 52,460,472 bytes. Ratio 199.22%
Compression time: cpu 0.02 sec/real 2.10 sec = 1%. Speed 12.51 mB/s
All OK

-mxpreproc+xprecomp
Compressed 1 file, 26,332,840 => 52,460,472 bytes. Ratio 199.22%
Compression time: cpu 0.05 sec/real 2.53 sec = 2%. Speed 10.42 mB/s
All OK

[Simorq]

770000620.zip (http://www.mediafire.com/file/83wpabnkn4abemz/770000620.zip) 76.7 MB
-mxpreproc
Compressed 1 file, 80,478,498 => 79,861,084 bytes. Ratio 99.23%
Compression time: cpu 0.11 sec/real 1.72 sec = 6%. Speed 46.77 mB/s
All OK

-mxprecomp
Compressed 1 file, 80,478,498 => 80,478,546 bytes. Ratio 100.00%
Compression time: cpu 0.13 sec/real 24.25 sec = 1%. Speed 3.32 mB/s
All OK

-mxpreproc+xprecomp
Compressed 1 file, 80,478,498 => 79,861,132 bytes. Ratio 99.23%
Compression time: cpu 0.08 sec/real 23.58 sec = 0%. Speed 3.41 mB/s
All OK
:confused:

[Razor12911]

The idea here guys is for xpreproc to give xprecomp a smaller input which speeds up the process while giving roughly the same output as when it wasn't used.

-mxprecomp:
Compressed 225 files, 3,253,110,016 => 5,314,555,404 bytes. Ratio 163.37%
Compression time: cpu 5.31 sec/real 250.34 sec = 2%. Speed 12.99 mB/s

-xpreproc+xprecomp:
Compressed 225 files, 3,253,110,016 => 1,441,325,440 bytes. Ratio 44.31%
Compression time: cpu 5.92 sec/real 132.72 sec = 4%. Speed 24.51 mB/s

if you compress 4.94 GB (5,314,557,896 bytes) output, also the 1.34 GB (1,441,327,942 bytes) output, you should get the same output, the plan here is to reduce srep's memory usage and give a bit of speed and lessen IO whenever srep has created its VM file.

[Mini]

The Crew 2 - Open Beta
xtool not working :(

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

[Razor12911]

The Crew 2 - Open Beta
xtool not working :(

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

The header file (fat) is different from the predecessors however, since this isn't lzo nor lz4, I can detect the streams directly from the dat files meaning no need to add files in xtool.ini

Try testing this program I made, it only detects streams currently, I tried to make a quick update for xtool to add support for this but I'm under a lot of work right now.

Perhaps to help, you could try figuring out what method of oodle was used from the extracted streams because I really don't have time currently.

scenaric.dat
Position: 00000000, Size: 2549827 >> 4194304
Position: 0026E843, Size: 124764 >> 624087
Position: 0028CF9F, Size: 222826 >> 1241117
Position: 002C365F, Size: 2022 >> 7909
...
Position: 00481892, Size: 4014 >> 121496
Position: 00482840, Size: 335 >> 1006
Position: 00482AF2, Size: 313 >> 1006

Overall size: 4724247 >> 15863935

I only tested this file

[Mini]

usa_world.dat
Overall size: 304237021 >> 824626021

works! :)

[Razor12911]

I figured out what they used with the spare break I had, they used the Hydra codec if I'm not mistaken. I didn't add Hydra support in xtool but I will in future.

[Razor12911]

usa_world.dat
Overall size: 304237021 >> 824626021

works! :)

Could you rerun test, I want to see if correct and improved information is found. I played around with the scanner a bit.

@everyone
Participate if possible, try testing this with several games that use oodle, leave out Project CARS 2 out for now since that game is encrypted.

[Mini]

usa_world.dat
Overall size: 314783065 >> 844382444

[Edison007]

The Crew 2 - Open Beta
xtool not working :(

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

[ZakirAhmad]

@Razor: is xtool going to b updated.

[Razor12911]

Hi guys, long time :)

I haven't coded in a while so you can imagine:
26918

I came up with an idea before I went hiatus which was deduplication of streams in xtool. Basically xtool permutes or disregards then removes repeated streams to improve performance. It's the same stuff that srep does only, xtool does it before srep so it improves IO and lessens memory usage.

What I want you guys to do is check out if this thing improves ratio if any and if decompression memory of srep goes down significantly, example. For examples like GTAV and lastly if there are no issues, I did rework the source a bit to make it stable.

My test input is No Man's Sky with the Beyond update (seems like devs have done a good job making sure they don't repeat game resources, wasn't the results I was not expecting...)

xtool 2019:
Compressed 104 files, 9,610,947,737 => 22,079,473,953 bytes. Ratio 229.73%
Compression time: cpu 12.20 sec/real 559.10 sec = 2%. Speed 17.19 mB/s

srep393b:
100%: 22,079,475,924 -> 14,306,151,312: 64.79%. Cpu 123 mb/s (171.078 sec), real 126 mb/s (167.671 sec) = 102%. Remains 00:00
Decompression memory is 699 mb. 6,413,440 matches = 102,615,040 bytes = 0.72% of file

lzturbo:
Compressed 1 file, 14,306,163,456 => 7,573,816,911 bytes. Ratio 52.94%
Compression time: cpu 16.19 sec/real 255.92 sec = 6%. Speed 55.90 mB/s

xtool 2020:
Compressed 104 files, 9,610,947,737 => 18,987,442,809 bytes. Ratio 197.56%
Compression time: cpu 10.94 sec/real 483.97 sec = 2%. Speed 19.86 mB/s

srep393b:
100%: 18,987,444,781 -> 14,276,551,933: 75.19%. Cpu 113 mb/s (160.000 sec), real 114 mb/s (158.234 sec) = 101%. Remains 00:00
Decompression memory is 436 mb. 4,696,348 matches = 75,141,568 bytes = 0.53% of file

lzturbo:
Compressed 1 file, 14,276,560,829 => 7,565,632,525 bytes. Ratio 52.99%
Compression time: cpu 16.86 sec/real 250.04 sec = 7%. Speed 57.10 mB/s

What do these results mean? and why is xtool 2020 producing less output compared to 2019 but somehow after using srep, the output is better?

Well as usual, I always have ideas to try to improve the xtool project for even more speed. I reviewed the project from all possible angles and found that optimisations of the program are nearing its limits meaning I cannot make the program any faster.

However then came an idea to steal some of the work srep does and make xtool involved, this resulted in the idea of making xtool detect and permute repeated streams, as a result. 2020 xtool most of the time will produce less output but when you compare it to 2019's version after using srep, the output is roughly the same or better.

You might ask, what is the point of taking some of srep's work and making it for xtool. Well this is because games over the years get endless updates, example is No Man's Sky and GTAV so there is a high chance that there are several repeated streams and as a result, if xtool doesn't take the job, it will be a job for xtool to deal with when installing a game because srep will require more decompression memory and xtool will process more streams resulting in extended installation times.

Note: I was expecting srep to produce the same result from both 2019 and 2020, because the main idea here is to improve installation times.

TL;DR

Xtool 2020 will produce less output compared to xtool 2019 because it takes some of the work of srep to reduce memory usage when installing a game while also improving speed significantly due to less streams being processed and less IO. (Theoretically at least)

Overall test results (compression):
8.95 GB (original) > 20.5 GB (xtool 2019 / 9 mins 19 secs) > 13.3 GB (srep m3f / 699mb dec memory) > 7.05 GB (lzturbo 32 b128)
8.95 GB (original) > 17.6 GB (xtool 2020 / 8 mins 4 secs) > 13.2 GB (srep m3f / 436mb dec memory) > 7.04 GB (lzturbo 32 b128)

I used "lzturbo 32" to speed up the final compression results (I'm a bit busy), but you will get the idea when you run tests yourselves.

Decompression mode is unavailable, I still have to write the code for it.

If you don't have GTAV or any other game which has live updates or a game with a lot of repetitions, you can just make a copy of the same input then give it to xtool to see what happens.

These optimisations are making me feel like Intel right now... 14nm+++++++++++++++++++++++ :o

[Cesar82]

@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!

[doofoo24]

so i guess only game with zlib for now maybe game like mad max will see improvement last time i compressed the game it was with xtool+srep:m2f+lolz memory usage was way less than srep:m3f but the final game size was 300-400mb more but game install was much more faster...
if i remember correctly m3f dec mem was around 17gb but with m2f around 3-4gb...

[devil777]

To be... honest... desiring a Xtool version... which will work like Pzlib v3 4ever


Thanks razor for all the works

[Razor12911]

@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!

Noted

[Razor12911]

Ok I decided to get Need For Speed The Run because I remembered that the game required a lot of decompression memory for srep to best demonstrate what I want to do with this deduplication idea.

I was surprised by the results

Xtool 2019:
Compressed 120 files, 14,503,034,446 => 29,120,915,487 bytes. Ratio 200.79%
Compression time: cpu 17.25 sec/real 807.42 sec = 2%. Speed 17.96 mB/s

100%: 29,120,917,667 -> 8,068,516,819: 27.71%. Cpu 223 mb/s (124.672 sec), real 225 mb/s (123.527 sec) = 101%. Remains 00:00
Decompression memory is 5660 mb. 2,922,729 matches = 46,763,664 bytes = 0.58% of file

Compressed 1 file, 8,068,559,139 => 4,315,048,356 bytes. Ratio 53.48%
Compression time: cpu 10.02 sec/real 172.58 sec = 6%. Speed 46.75 mB/s

Xtool 2020:
Compressed 120 files, 14,503,034,446 => 12,121,343,984 bytes. Ratio 83.58%
Compression time: cpu 16.13 sec/real 368.86 sec = 4%. Speed 39.32 mB/s

100%: 12,121,346,164 -> 8,010,462,480: 66.09%. Cpu 141 mb/s (81.719 sec), real 145 mb/s (79.890 sec) = 102%. Remains 00:00
Decompression memory is 759 mb. 2,105,722 matches = 33,691,552 bytes = 0.42% of file

Compressed 1 file, 8,010,469,968 => 4,296,282,176 bytes. Ratio 53.63%
Compression time: cpu 9.88 sec/real 159.45 sec = 6%. Speed 50.24 mB/s


Overall test results (compression):
13.5 GB (original) > 27.1 GB (xtool 2019 / 13 mins 27 secs) > 7.51 GB (srep m3f / 5660mb dec memory) > 4.02 GB (lzturbo 32 b128)
13.5 GB (original) > 11.2 GB (xtool 2020 / 6 mins 9 secs) > 7.45 GB (srep m3f / 759mb dec memory) > 4.00 GB (lzturbo 32 b128)

Not only is xtool 2020 faster but it produced a slightly better result while reducing srep's decompression memory. That 13.5 GB > 11.2 GB is not a mistake, xtool did precompress the data but there were too many repeated streams in the input and it removed a lot of them which resulted in negative ratio.

Notes: Xtool does incur additional decoding memory but it will definitely not be 5.6GB, it will far less. Maybe 2GB but if I apply proper memory management, I can reduce it to 1GB saving an additional 3GB of ram for lolz or something else.

[78372]

FreeArc 0.67 (March 15 2014) creating archive: data19.arc
Compressed 1 file, 197,759,488 => 941,714,574 bytes. Ratio 476.19%
Decompression memory is 116 mb. 146,130 matches = 2,338,080 bytes = 0.71% of file

FreeArc 0.67 (March 15 2014) creating archive: data20.arc
Compressed 1 file, 197,759,488 => 734,656,804 bytes. Ratio 371.49%
Decompression memory is 43 mb. 136,880 matches = 2,190,080 bytes = 0.66% of file


Good work ;)

[Cesar82]

@Razor12911
Good work! Great results!

I don't understand practically anything about compression and flow methods, etc., but CIU has XTool (old Xtool, not 2019) and ZTool, and as far as I know some of the flows (I think that's it) are not supported by Xtool.
I would like to know if in this version when it is finished it will support all flows that old ZTool and Xtool support, or it will always be necessary to keep XTool + ZTool in CIU if I use XTool 2020.
It would be great to be able to remove some of the compressors from CIU.
Thank you!

[FitGirl]

Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.

[Gupta]

You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen,

maybe he can introduce the second phase in compression, then he can store forward reference count for a stream that should ideally decrease requirements memory size beyond window size and that should increase compression too.

HDDs are very slow, I recently upgraded to nvme based storage and I can feel the speed.

[panker1992]

Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.

there is also a sorting match feature that can reduce ram needed and that is as follows.

srep does a very good job finding matches that are located far away!
that in order to happen makes a dictionary!
IF you sort the files you feed srep you can actually reduce ram needed and its speed

Sorting preprocession can speedup the process and cost less ram !! and remove IO overhead because NO temps

[Razor12911]

Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.

maybe he can introduce the second phase in compression, then he can store forward reference count for a stream that should ideally decrease requirements memory size beyond window size and that should increase compression too.


HDDs are very slow, I recently upgraded to nvme based storage and I can feel the speed.

there is also a sorting match feature that can reduce ram needed and that is as follows.

srep does a very good job finding matches that are located far away!
that in order to happen makes a dictionary!
IF you sort the files you feed srep you can actually reduce ram needed and its speed

Sorting preprocession can speedup the process and cost less ram !! and remove IO overhead because NO temps


Believe me I have several ideas of how to reduce memory usage before even relying on virtual memory. Optimisation is my middle name.

[bunti_o4u]

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

does it support stdio?

if you add stdio, it would be great..

[panker1992]

xcompress if pure windows 10 compression, and i think it supports it by default.

[Razor12911]

Can you guys test this. I added preflate as an alternative of reflate in case if you are having issues with crc errors when you use reflate or prolonged precompression times such as this:
Dishonored: Death of the Outsider

*.resources - 17 gb > 25.1 gb for 23 minutes on 4 threads

For example. ZTool or Xtool (v0.12) - 30% for 4 hours

Thanks BLACKFIRE69

test unpack

I'm no expert when it comes to C++ and I did my best to compile a library for xtool to use so it may have bugs.

I ran a benchmark and here are the results

benchmark results on "game1.resources" from "Dishonored Death of the Outsiders"
xtool_2019:
8.30 GB >> ?? >> ?? >> ??
precompression time using 4 threads = it's better to watch paint dry (takes hours)

xtool_2020 (WIP):
8.30 GB >> 11.3 GB >> 8.74 GB >> 6.24 GB
precompression time using 2 threads = 8 minutes, 48 seconds
precompression time using 4 threads = 5 minutes, 7 seconds

PrecompX:
8.30 GB >> 11.1 GB >> 8.79 GB >> 6.28 GB
precompression time using 4 threads = 8 minutes, 29 seconds

[dixen]

Dishonored Death of the Outsiders

game1.resources

Compressed 1 file, 8,920,763,754 => 11,774,785,195 bytes. Ratio 131.99%
Compression time: cpu 11.31 sec/real 385.94 sec = 3%. Speed 23.11 mB/s
All OK

When unpack function?)

[Razor12911]

Dishonored Death of the Outsiders

game1.resources



When unpack function?)

Soon, maybe this week. Still busy writing the main code while looking for more ways to speed up processing :)

[lolaya]

what is ztool and xtool and lolz?

[Grumpy]

what is ztool and xtool and lolz?

That information is already available on these forums, search and you will find, you can not expect for others to always hold your hand and spoon feed you all the time. ;)

[lolaya]

Grumpy@

more posts here :D I can't find it easy

[BLACKFIRE69]

Grumpy@

more posts here :D I can't find it easy

that's the why studying hard.... ;)

[doofoo24]

can't wait to test the new xtool :D

[Razor12911]

Here's an early working version of xtool. I'm still busy adding preflate, reflate and other things so currently only the zlib function works which means this will not work on Dishonored 2 or DOOM but it will work on most titles.

Drag and drop game files that are zlib compressed on the bat file

Things to test:

* Stability
* Can you cancel installation when xtool was used in setup
* If you have 16 threads or more, make xtool use 100% of your cpu to see if it can handle it
* if you have 32GB+ ram, set high chunk size, something like -c1gb and see if xtool was able to allocate the necessary memory
* If you have potato pc, don't worry I got a job for you :) check if there are no problems if you are using 1 thread both in encoding and decoding

Whatever you do, don't place preflate_dll.dll near the exe, not yet. else it will fail, the preflate code hasn't been added yet :D

Notes:

The file check must report
FC: no differences encountered
if the test went successfully, if it failed kindly upload the file it failed on.

This xtool uses slightly more memory than the 2019 version, I opted for stability than less memory usage

xtool now accepts maths expressions in command line, if you wanted it to use all cores minus 1, you can just write -t100p-1 or -t100-2, totally up to you

Next release/test focuses on:

+ preflate
+ reflate (reflate actually isn't replaced by preflate, it sometimes gives better output than preflate so I'll keep it just in case you are after best results)
+ depth setting

[Masquerade]

Hello Razor, here is my test (I did make a post before but I had to delete it since I used a reflate files by mistake):

My spec: Ryzen 7 2700 (8c/16t) + 16GB RAM.

Bat file editied for 128mb chunk size.

Here's is XTool using all 16 of my threads:

https://i.imgur.com/jGFDfgB.png

Testing on pak file from Astroneer:

Start size: 2.13GB

End Size: 3.75GB

No differences encountered ;)

Not entirely sure how I'd go about testing it in a setup...

[doofoo24]

tested on just cause2 arc files...
default setting nothing change "ProcProfile64.exe xtool.exe precomp -mzlib -c16mb -t100p-1 %1 %1.out"
system ryzen 3900x with 32gb
*is there arc.ini setting ?

[dixen]

Rayman Orgins
bundle_pc.pak (1.9 gb)

Process ID : 8952
Thread ID : 4228
Process Exit Code: 0
Thread Exit Code : 0

User Time : 149.796s
Kernel Time : 4.375s
Process Time : 154.171s
Clock Time : 74.923s

Working Set : 372312 KB
Paged Pool : 115 KB
Nonpaged Pool : 12 KB
Pagefile : 462572 KB
Page Fault Count : 147577

IO Read : 2046448 KB (in 31978 reads )
IO Write : 3038303 KB (in 47474 writes)
IO Other : 6 KB (in 297 others)

PS. tested on my 2nd pc (fx-4100, 2 cores, 4 threads).

[Razor12911]

*is there arc.ini setting ?

[External compressor:xzlib]
header = 0
packcmd = xtool.exe precomp -c16mb -t100p-1 -mzlib - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

here's preflate test, test on all zlib compressed games + dishonored and doom

[doofoo24]

tested on Dishonored2 with c16m and 512m and set -t100p-1...
work fine on ryzen 3900x and 32gb use all core and ram...

42.1gb to 57.2gb with c16mb
42.1gb to 59.8gb with c512mb

*used on all game file...

tested unpacking with inno
i get ERROR: archive data corrupted...

[Razor12911]

I must have messed up stdio mode during decoding but here is the fixed version.

[doofoo24]

now dec work super fast but i noticed it cpu usage around 50% to 70% of threads unlike compress use 90% to 100% when set to

unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

disk write using ssd nvme mp510 write after burst 400 to 500 mb/s...
nvme bottleneck ?
need ps5 ssd to get the max :D

[Razor12911]

you noticed how much memory xtool uses when compressing compared to when decompressing? that's why. xtool uses less memory when decoding and because of this it can't use up the entire cpu because there are times in between where there is nothing to process from memory. I haven't implemented faster code but in future I will.

[panker1992]

I talked to you about preflate quite some time ago :D

glad to see you have made something out of it, will be returning to testing as soon as i am back home!!


have produced several videos on youtube on advanced compression past weeks.

[Razor12911]

I talked to you about preflate quite some time ago :D


talking to me about it isn't the same as implementing it ;)

@everyone

here's the xtool with reflate support.

I'll once again explain what's the difference. There are 3 deflate precompressors in xtool: zlib, reflate and preflate.

The old xtool/ztool/pzlib all had zlib+reflate but they had so many errors so I added preflate as an alternative if you have issues when you use reflate.

There are other things to this of course. I added all 3 because sometimes one has an advantage such as speed or better size output. Which is up to you to pick.

reflate cannot be used at the same time as preflate or the other way round.

but zlib can be used with reflate or preflate (pick one) like this -mzlib+reflate or -mzlib+preflate

here are my results on games that use the "normal" deflate configuration

using -mzlib:
Compressed 1 file, 1,138,892,800 => 2,611,771,763 bytes. Ratio 229.33%
Compression time: cpu 1.34 sec/real 55.54 sec = 2%. Speed 20.51 mB/s

using -mreflate:
Compressed 1 file, 1,138,892,800 => 2,613,343,052 bytes. Ratio 229.46%
Compression time: cpu 1.28 sec/real 86.86 sec = 1%. Speed 13.11 mB/s

using -mpreflate
Compressed 1 file, 1,138,892,800 => 2,613,650,264 bytes. Ratio 229.49%
Compression time: cpu 1.34 sec/real 90.68 sec = 1%. Speed 12.56 mB/s

[BLACKFIRE69]

talking to me about it isn't the same as implementing it ;)

@everyone

here's the xtool with reflate support.

I'll once again explain what's the difference. There are 3 deflate precompressors in xtool: zlib, reflate and preflate.

The old xtool/ztool/pzlib all had zlib+reflate but they had so many errors so I added preflate as an alternative if you have issues when you use reflate.

There are other things to this of course. I added all 3 because sometimes one has an advantage such as speed or better size output. Which is up to you to pick.

reflate cannot be used at the same time as preflate or the other way round.

but zlib can be used with reflate or preflate (pick one) like this -mzlib+reflate or -mzlib+preflate

here are my results on games that use the "normal" deflate configuration

using -mzlib:
Compressed 1 file, 1,138,892,800 => 2,611,771,763 bytes. Ratio 229.33%
Compression time: cpu 1.34 sec/real 55.54 sec = 2%. Speed 20.51 mB/s

using -mreflate:
Compressed 1 file, 1,138,892,800 => 2,613,343,052 bytes. Ratio 229.46%
Compression time: cpu 1.28 sec/real 86.86 sec = 1%. Speed 13.11 mB/s

using -mpreflate
Compressed 1 file, 1,138,892,800 => 2,613,650,264 bytes. Ratio 229.49%
Compression time: cpu 1.34 sec/real 90.68 sec = 1%. Speed 12.56 mB/s

Razor,
it would be nice if you could update the "Game File Scanner" with the new xtool. ;)

[dixen]

Dishonored. Death of the outsider (yes, again)

game1.resources
game1_001.resources
game1_patch.resources

-mpreflate

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 3 files, 9,663,676,416 => 13,842,292,926 bytes. Ratio 143.24%
Compression time: cpu 12.19 sec/real 498.98 sec = 2%. Speed 19.37 mB/s
All OK

Unpack

Extracted 3 files, 13,842,292,926 => 9,663,676,416 bytes. Ratio 143.24%
Extraction time: cpu 21.22 sec/real 217.84 sec = 10%. Speed 44.36 mB/s
All OK

[Razor12911]

Razor,
it would be nice if you could update the "Game File Scanner" with the new xtool. ;)

I actually have a better idea :rolleyes:

Dishonored. Death of the outsider (yes, again)

game1.resources
game1_001.resources
game1_patch.resources

-mpreflate



Unpack

try with reflate

[BLACKFIRE69]

I actually have a better idea :rolleyes:


sounds great :D

[dixen]

try with reflate-mreflate

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 3 files, 9,663,676,416 => 14,323,974,019 bytes. Ratio 148.22%
Compression time: cpu 11.94 sec/real 273.23 sec = 4%. Speed 35.37 mB/s
All OK

Extracting 3 files, 9,663,676,416 bytes. Processed 92.1%
ERROR: CRC failed in "unpacked\game1.resources". File is broken.


Test on DOOM - later..

[dixen]

DOOM 2016

snap_gameresources.resources
gameresources.resources

-mxtool 2019 (R3)

Extracted 2 files, 5,513,073,128 => 11,189,482,143 bytes. Ratio 49.27%
Extraction time: cpu 20.17 sec/real 1522.22 sec = 1%. Speed 7.35 mB/s
All OK

-mreflate

Decompress

Compressed 2 files, 11,189,482,143 => 21,441,692,530 bytes. Ratio 191.62%
Compression time: cpu 13.69 sec/real 811.30 sec = 2%. Speed 13.79 mB/s
All OK

Unpack

Extracting 2 files, 11,189,482,143 bytes. Processed 78.6%
ERROR: CRC failed in "unpacked\gameresources.resources". File is broken.
arc.exe: wclose: invalid argument (Bad file descriptor)
79.0%

-mpreflate

Compressed 2 files, 11,189,482,143 => 11,189,490,830 bytes. Ratio 100.00%
Compression time: cpu 13.23 sec/real 1371.54 sec = 1%. Speed 8.16 mB/s
All OK

[Razor12911]

If you have Saints Row The Third Remastered, can you run this test for me.

Thanks.