[Dev]XTool

[Razor12911]

Originally Posted by doofoo24

all the file's that uploaded work with afr and afr:a2 for unity and syndicate

Great stuff, time to get to work

[doofoo24]

i deleted the link's to aviod problem...
i assume you downloaded the samples

[kassane]

Divinity Original Sin 2 - Samples:
GDrive folder

[Razor12911]

Originally Posted by kassane

Divinity Original Sin 2 - Samples:
GDrive folder

I did run a test, but forgot to report results but it seems to work with xtool with it being set to zlib.

FG repack has this to report:

Repack uses ZTool library by Razor12911

[Razor12911]

Here's a test for a possible addition to xtool operation option

Try testing the xpreproc method on the entire game, edit the batch file to specify game location.

The game must have zlib streams, best examples would be Forza Horizon 3, GTA and so forth.

Report something like this

Compressed 1 file, 429,620,439 => 294,418,241 bytes. Ratio 68.53%
Compression time: cpu 0.47 sec/real 10.40 sec = 5%. Speed 41.32 mB/s

Only do this test below after the first one if possible, it's just up to you.

You can apply xprecomp if you want to perform further tests then srep afterwards
basically something like this:

xprecomp+srep
vs
xpreproc+xprecomp+srep

I'm interested srep's decompression memory so post the usage of memory for when xpreproc was applied and when it wasn't

[Simorq]

LINKDATA_ENG_LANG.BIN 83 MB

-mxpreproc
Compressed 1 file, 87,191,552 => 77,684,585 bytes. Ratio 89.10%
Compression time: cpu 0.14 sec/real 1.35 sec = 10%. Speed 64.47 mB/s
All OK

-mxprecomp
Compressed 1 file, 87,191,552 => 454,736,069 bytes. Ratio 521.54%
Compression time: cpu 0.14 sec/real 10.47 sec = 1%. Speed 8.33 mB/s
All OK

-mxpreproc+xprecomp
Compressed 1 file, 87,191,552 => 236,598,374 bytes. Ratio 271.35%
Compression time: cpu 0.09 sec/real 9.82 sec = 1%. Speed 8.88 mB/s
All OK

[Simorq]

Mafia3 > fmv_dlc1_0090_goobye_roxy.sds 25.1 MB

-mxpreproc
Compressed 1 file, 26,332,840 => 26,332,840 bytes. Ratio 100.00%
Compression time: cpu 0.02 sec/real 0.46 sec = 3%. Speed 56.86 mB/s
All OK

-mxprecomp
Compressed 1 file, 26,332,840 => 52,460,472 bytes. Ratio 199.22%
Compression time: cpu 0.02 sec/real 2.10 sec = 1%. Speed 12.51 mB/s
All OK

-mxpreproc+xprecomp
Compressed 1 file, 26,332,840 => 52,460,472 bytes. Ratio 199.22%
Compression time: cpu 0.05 sec/real 2.53 sec = 2%. Speed 10.42 mB/s
All OK

[Simorq]

770000620.zip 76.7 MB

-mxpreproc
Compressed 1 file, 80,478,498 => 79,861,084 bytes. Ratio 99.23% 
Compression time: cpu 0.11 sec/real 1.72 sec = 6%. Speed 46.77 mB/s
All OK

-mxprecomp
Compressed 1 file, 80,478,498 => 80,478,546 bytes. Ratio 100.00%
Compression time: cpu 0.13 sec/real 24.25 sec = 1%. Speed 3.32 mB/s
All OK

-mxpreproc+xprecomp
Compressed 1 file, 80,478,498 => 79,861,132 bytes. Ratio 99.23%
Compression time: cpu 0.08 sec/real 23.58 sec = 0%. Speed 3.41 mB/s
All OK

[Razor12911]

The idea here guys is for xpreproc to give xprecomp a smaller input which speeds up the process while giving roughly the same output as when it wasn't used.

-mxprecomp:

Compressed 225 files, 3,253,110,016 => 5,314,555,404 bytes. Ratio 163.37%
Compression time: cpu 5.31 sec/real 250.34 sec = 2%. Speed 12.99 mB/s

-xpreproc+xprecomp:

Compressed 225 files, 3,253,110,016 => 1,441,325,440 bytes. Ratio 44.31%
Compression time: cpu 5.92 sec/real 132.72 sec = 4%. Speed 24.51 mB/s

if you compress 4.94 GB (5,314,557,896 bytes) output, also the 1.34 GB (1,441,327,942 bytes) output, you should get the same output, the plan here is to reduce srep's memory usage and give a bit of speed and lessen IO whenever srep has created its VM file.

[Mini]

The Crew 2 - Open Beta
xtool not working

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

[Razor12911]

Originally Posted by Mini

The Crew 2 - Open Beta
xtool not working

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

The header file (fat) is different from the predecessors however, since this isn't lzo nor lz4, I can detect the streams directly from the dat files meaning no need to add files in xtool.ini

Try testing this program I made, it only detects streams currently, I tried to make a quick update for xtool to add support for this but I'm under a lot of work right now.

Perhaps to help, you could try figuring out what method of oodle was used from the extracted streams because I really don't have time currently.

scenaric.dat

Position: 00000000, Size: 2549827 >> 4194304
Position: 0026E843, Size: 124764 >> 624087
Position: 0028CF9F, Size: 222826 >> 1241117
Position: 002C365F, Size: 2022 >> 7909
...
Position: 00481892, Size: 4014 >> 121496
Position: 00482840, Size: 335 >> 1006
Position: 00482AF2, Size: 313 >> 1006

Overall size: 4724247 >> 15863935

I only tested this file

[Edison007]

Originally Posted by Mini

The Crew 2 - Open Beta
xtool not working

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

[bunti_o4u]

Originally Posted by Edison007

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

does it support stdio?

if you add stdio, it would be great..

[Mini]

usa_world.dat
Overall size: 304237021 >> 824626021

works!

[Razor12911]

Originally Posted by Mini

usa_world.dat
Overall size: 304237021 >> 824626021

works!

Could you rerun test, I want to see if correct and improved information is found. I played around with the scanner a bit.

@everyone
Participate if possible, try testing this with several games that use oodle, leave out Project CARS 2 out for now since that game is encrypted.