XTool 2020 (Main Project)

[shazzla]

If game uses oodle,then you can find its DLLs in game folder somewhere.
Imho : oodle stream scanner is obsolete ?!

[Razor12911]

Originally Posted by Simorq

Hi Razor12911

v: skip verification

This option works for oodle, crilayla, lzo.
Please do not delete this option in the new version.

The option will be there

Originally Posted by dixen

oodle detect, for example

I remember including an oodle scanner, but fine I'll think of something for the scanner.

Originally Posted by KaktoR

This is great.

how about this?

There will be a new feature in xtool called history data, what it does is simply store processed stream information and be able to recall it at the later stage.

If you are wondering how this helps, well there are certain games made by lazy developers who repeat game resources instead of just reusing them, resulting in bigger game size for no absolute reason, apart from that, if they are compressed streams it means that xtool will have to process the same stream over and over again, storing history data allows it to prevent the same stream from being reprocessed instead, it will just look at the history and find out how the previous copy was processed then automatically assume it's the same stream and then apply the same settings, resulting in more speed.

I ran a test on a file I repeated on purpose and these were the results.

old xtool took 196.671 seconds and used 335MB ram
new xtool took 154.174 seconds and used 319MB ram
new xtool with history feature took 72.223 seconds and used 439MB ram

the cost of the feature depends on how much history you want to be stored but with default settings, it should use an additonal 192mb ram usage than without, you can always disable it or order it to store more history, depends on how much memory you have to spare.

Note:
In a practical situation, you can shave off a few seconds but then again, depends on the input

[KaktoR]

This is also good

Did you have to make changes to the libraries too? Or just the source of xtool executable? Just corious

[Razor12911]

Only the source of xtool, all I did was optimize optimize and optimize, the CPU is forever doing something, I tried to minimize the gaps between reads and writes by making xtool process stuff while it is reading and writing so yeah, the threads hardly get rests now.

[Razor12911]

Project status = Welp...

I hope this isn't strike 2 in terms losing my sources, my PC died about 2 days ago, motherboard and GPU got fried this time XD. As for my sources well they were stored on my main PC (the dead one) and I don't know if they are safe since I don't have a spare PC to boot up the drive to grab important files and store them to my backup drive, so I can't really tell if the only components that died were only the board and GPU and not my HDD, but I guess that's a story for some other time. If you are wondering what this means in terms of xtool 2019...

Well since I normally code from desktop because I don't need to go far to do things and etc, it was different for xtool somehow... because I was coding from a backup drive only for that project like I knew some shit like this would happen so yes the source is safe (lucky af), if you are wondering about the development now that my main PC is busted.

Well I just had to dust an old laptop and continued development there, issue is it's incredibly slow but it will have to do. If you are wondering why I only provided x86 binaries, well. OS is 32-bit , somehow can't compile for x64.

Anyways enough about that, this was suppose to be the last test before the first release, but before I uploaded, I noticed some shitty error in made in the reflate code and I have to rewrite some parts of it so after fixing that, I'll post the final test and if all is good, I'll be making a release.

there are 3 codecs in xtool right now (zlib, reflate, rzlib). The bug was in reflate in multi threaded mode when decoding, but if you want to try it out anyways, open the test bat file. replace zlib with reflate or rzlib but use 1 thread to decode.

zlib = xtool uses zlib1.dll/zlibwapi.dll to process data, fails on some streams
reflate = xtool uses reflate dlls to process data, processes all streams but slow
rzlib = xtool blends zlib and reflate dlls to process data, processes all streams but with better size and speed.

PS: Even the archive is zip, don't have 7-Zip installed for my normal 7z attachments
PPS: Nothing is stopping the development of this project XD... I hope cause you never know

[FitGirl]

Shit, mate! You're lucky to have not loosing sources this time! Gotta use some repo soft next time for more security, no?

Anyways, it's nice to see the project is coming to release.

[Harsh ojha]

@Razor12911
Download Windows 10 64 bit Archived file and install

https://youtu.be/zGQ9xXYgVJM
Since i also lost my data

Well we are all depends our pc and laptops

[Razor12911]

First release

Notes

- No idea whether x64 works or not, I just somehow managed to compile. That's up to you to find out.
- Reflate currently is set to use level 6, if you are looking for better output, best to use the old xtool. The reason it's like this is because I'm still working on a way to improve the horrible slow code I used before.
- Skip verification option is currently unavailable.

What I can say, run a lot of tests on this release. I've written some dangerous memory saving code and it may cause errors but if all is well then I can move on to add other codecs such as oodle, lzo, lz4 and the rest. BTW, crilayla will NOT be added, that piece of crap is incredibly slow

[Simorq]

xtool_1907_R1 64Bit

Creating archive: data.arc using zlib
Compressed 1 file, 668,113 => 668,225 bytes. Ratio 100.02%                 
Compression time: cpu 0.02 sec/real 0.18 sec = 9%. Speed 3.71 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 668,225 => 668,113 bytes. Ratio 100.02%                     
Testing time: cpu 0.00 sec/real 0.06 sec = 0%. Speed 10.77 mB/s
All OK

Creating archive: data.arc using reflate
Compressed 1 file, 668,113 => 1,767,491 bytes. Ratio 264.55%               
Compression time: cpu 0.00 sec/real 0.19 sec = 0%. Speed 3.44 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 1,767,491 => 668,113 bytes. Ratio 264.55%                   
Testing time: cpu 0.00 sec/real 0.16 sec = 0%. Speed 4.31 mB/s
All OK

Creating archive: data.arc using rzlib
Compressed 1 file, 668,113 => 1,767,491 bytes. Ratio 264.55%               
Compression time: cpu 0.00 sec/real 0.28 sec = 0%. Speed 2.40 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 1,767,491 => 668,113 bytes. Ratio 264.55%                   
Testing time: cpu 0.00 sec/real 0.16 sec = 0%. Speed 4.25 mB/s
All OK

[Simorq]

Does not work with ba2/csg files.

Fallout4 - Textures7.ba2 / DLCCoast - Geometry.csg

32bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib
64bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib or stuck cpu


-------------------------------------------------------------------

Works with this setting.

[External compressor:rzlib,zlib,reflate]
header    = 0
packcmd   = xtool.exe precomp:rzlib:c32mb,t1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode:t50p - - <stdin> <stdout>

Fallout4 - Textures7.ba2 520 MB

ZLIB

Creating archive: data.arc using zlib
Compressed 1 file, 545,959,220 => 1,237,654,873 bytes. Ratio 226.69%       
Compression time: cpu 0.52 sec/real 84.11 sec = 1%. Speed 6.49 mB/s
All OK

zlib: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%           
Testing time: cpu 0.45 sec/real 9.53 sec = 5%. Speed 57.29 mB/s
All OK

REFLATE

Creating archive: data.arc using reflate
Compressed 1 file, 545,959,220 => 1,237,783,515 bytes. Ratio 226.72%       
Compression time: cpu 0.42 sec/real 113.43 sec = 0%. Speed 4.81 mB/s
All OK

reflate: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,783,515 => 545,959,220 bytes. Ratio 226.72%           
Testing time: cpu 0.36 sec/real 12.23 sec = 3%. Speed 44.62 mB/s
All OK

RZLIB

Creating archive: data.arc using rzlib
Compressed 1 file, 545,959,220 => 1,237,654,873 bytes. Ratio 226.69%       
Compression time: cpu 0.53 sec/real 85.22 sec = 1%. Speed 6.41 mB/s
All OK

rzlib: decode:t2
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%           
Testing time: cpu 0.98 sec/real 73.38 sec = 1%. Speed 7.44 mB/s
All OK

rzlib: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%           
Testing time: cpu 0.77 sec/real 73.54 sec = 1%. Speed 7.42 mB/s
All OK

[Razor12911]

Can you run xtool outside freearc, like via console.

@echo off
xtool.exe precomp:rzlib:c16mb,t100p %1 %1.out
pause

I'm interested in the exception/error message

[Simorq]

@Razor12911
good


-------------------------
I think the main problem is - -.
- - <stdin> <stdout>

[Razor12911]

try this then

@echo off
xtool.exe precomp:rzlib:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res
pause

[Simorq]

@Razor12911
no work
stuck cpu

[Razor12911]

Nice ghost bugs, I'll check what I missed in the source and thanks for the tests.