[Dev]XTool

[Razor12911]

I must have messed up stdio mode during decoding but here is the fixed version.

[doofoo24]

now dec work super fast but i noticed it cpu usage around 50% to 70% of threads unlike compress use 90% to 100% when set to

unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

disk write using ssd nvme mp510 write after burst 400 to 500 mb/s...
nvme bottleneck ?
need ps5 ssd to get the max

[Razor12911]

you noticed how much memory xtool uses when compressing compared to when decompressing? that's why. xtool uses less memory when decoding and because of this it can't use up the entire cpu because there are times in between where there is nothing to process from memory. I haven't implemented faster code but in future I will.

[panker1992]

I talked to you about preflate quite some time ago

glad to see you have made something out of it, will be returning to testing as soon as i am back home!!


have produced several videos on youtube on advanced compression past weeks.

[Razor12911]

Originally Posted by panker1992

I talked to you about preflate quite some time ago

talking to me about it isn't the same as implementing it

@everyone

here's the xtool with reflate support.

I'll once again explain what's the difference. There are 3 deflate precompressors in xtool: zlib, reflate and preflate.

The old xtool/ztool/pzlib all had zlib+reflate but they had so many errors so I added preflate as an alternative if you have issues when you use reflate.

There are other things to this of course. I added all 3 because sometimes one has an advantage such as speed or better size output. Which is up to you to pick.

reflate cannot be used at the same time as preflate or the other way round.

but zlib can be used with reflate or preflate (pick one) like this -mzlib+reflate or -mzlib+preflate

here are my results on games that use the "normal" deflate configuration

using -mzlib:

Compressed 1 file, 1,138,892,800 => 2,611,771,763 bytes. Ratio 229.33%
Compression time: cpu 1.34 sec/real 55.54 sec = 2%. Speed 20.51 mB/s

using -mreflate:

Compressed 1 file, 1,138,892,800 => 2,613,343,052 bytes. Ratio 229.46%
Compression time: cpu 1.28 sec/real 86.86 sec = 1%. Speed 13.11 mB/s

using -mpreflate

Compressed 1 file, 1,138,892,800 => 2,613,650,264 bytes. Ratio 229.49%
Compression time: cpu 1.34 sec/real 90.68 sec = 1%. Speed 12.56 mB/s

[BLACKFIRE69]

Originally Posted by Razor12911

talking to me about it isn't the same as implementing it

@everyone

here's the xtool with reflate support.

I'll once again explain what's the difference. There are 3 deflate precompressors in xtool: zlib, reflate and preflate.

The old xtool/ztool/pzlib all had zlib+reflate but they had so many errors so I added preflate as an alternative if you have issues when you use reflate.

There are other things to this of course. I added all 3 because sometimes one has an advantage such as speed or better size output. Which is up to you to pick.

reflate cannot be used at the same time as preflate or the other way round.

but zlib can be used with reflate or preflate (pick one) like this -mzlib+reflate or -mzlib+preflate

here are my results on games that use the "normal" deflate configuration

using -mzlib:

Compressed 1 file, 1,138,892,800 => 2,611,771,763 bytes. Ratio 229.33%
Compression time: cpu 1.34 sec/real 55.54 sec = 2%. Speed 20.51 mB/s

using -mreflate:

Compressed 1 file, 1,138,892,800 => 2,613,343,052 bytes. Ratio 229.46%
Compression time: cpu 1.28 sec/real 86.86 sec = 1%. Speed 13.11 mB/s

using -mpreflate

Compressed 1 file, 1,138,892,800 => 2,613,650,264 bytes. Ratio 229.49%
Compression time: cpu 1.34 sec/real 90.68 sec = 1%. Speed 12.56 mB/s

Razor,
it would be nice if you could update the "Game File Scanner" with the new xtool.

[Razor12911]

Originally Posted by BLACKFIRE69

Razor,
it would be nice if you could update the "Game File Scanner" with the new xtool.

I actually have a better idea

Originally Posted by dixen

Dishonored. Death of the outsider (yes, again)

game1.resources
game1_001.resources
game1_patch.resources

-mpreflate



Unpack

try with reflate

[BLACKFIRE69]

Originally Posted by Razor12911

I actually have a better idea

sounds great

[dixen]

Originally Posted by Razor12911

try with reflate

-mreflate

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 3 files, 9,663,676,416 => 14,323,974,019 bytes. Ratio 148.22%
Compression time: cpu 11.94 sec/real 273.23 sec = 4%. Speed 35.37 mB/s
All OK

Extracting 3 files, 9,663,676,416 bytes. Processed 92.1%
ERROR: CRC failed in "unpacked\game1.resources". File is broken.

Test on DOOM - later..

[dixen]

Dishonored. Death of the outsider (yes, again)

game1.resources
game1_001.resources
game1_patch.resources

-mpreflate

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 3 files, 9,663,676,416 => 13,842,292,926 bytes. Ratio 143.24%
Compression time: cpu 12.19 sec/real 498.98 sec = 2%. Speed 19.37 mB/s
All OK

Unpack

Extracted 3 files, 13,842,292,926 => 9,663,676,416 bytes. Ratio 143.24%
Extraction time: cpu 21.22 sec/real 217.84 sec = 10%. Speed 44.36 mB/s
All OK

[dixen]

DOOM 2016

snap_gameresources.resources
gameresources.resources

-mxtool 2019 (R3)

Extracted 2 files, 5,513,073,128 => 11,189,482,143 bytes. Ratio 49.27%
Extraction time: cpu 20.17 sec/real 1522.22 sec = 1%. Speed 7.35 mB/s
All OK

-mreflate

Decompress

Compressed 2 files, 11,189,482,143 => 21,441,692,530 bytes. Ratio 191.62%
Compression time: cpu 13.69 sec/real 811.30 sec = 2%. Speed 13.79 mB/s
All OK

Unpack

Extracting 2 files, 11,189,482,143 bytes. Processed 78.6%
ERROR: CRC failed in "unpacked\gameresources.resources". File is broken.
arc.exe: wclose: invalid argument (Bad file descriptor)
79.0%

-mpreflate

Compressed 2 files, 11,189,482,143 => 11,189,490,830 bytes. Ratio 100.00%
Compression time: cpu 13.23 sec/real 1371.54 sec = 1%. Speed 8.16 mB/s
All OK

[Razor12911]

If you have Saints Row The Third Remastered, can you run this test for me.

Thanks.