XTool 2020 (Main Project)

[Razor12911]

XTool is program made specifically repackaging games by providing a full suite of useful features such as data precompression, archiving, encryption and etc.

With that being said, nothing restricts it from being used on everyday files such as documents, pictures and media but with few limitations.

Read the documentation to find out how it works and how to use it.

Link (https://mega.nz/folder/03BkjKQA#WpTxyngxucC-ToBRK9xbkw) for older releases.

[FitGirl]

Съешь ещё этих мягких французских булок, да выпей же чаю.
Google this if you want to know what it is and how it relates to the topic ;)

[Razor12911]

Good thing I know a bit of Russian (and by a bit, I mean a drop in an ocean) but from what I can tell, all Cyrillic letters (Russian) have been used :)

[ShivShubh]

Nice to see you still working for the community, much respect !

[shiyamrrc]

Waiting for the precompressor.
Respect

[FitGirl]

Good thing I know a bit of Russian (and by a bit, I mean a drop in an ocean) but from what I can tell, all Cyrillic letters (Russian) have been used :)
Correct answer, here's your ONE MILLION USD! :)

[Razor12911]

Where??? :D:D:D

[FitGirl]

Where??? :D:D:D
Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

[Grumpy]

Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

It gave me a good laugh :D

[Razor12911]

Yea right... sounds like a scam :(

@everyone
progress of the project is about 80%, should have been done by now if weren't steam sale.

[KaktoR]

steam sale.

I know that feel bro.

And should I tell you something... Most of my games I never played. I just bought them in a sale or from keysellers. I don't know why I buy games and never play them lol.

[doofoo24]

did you fix far cry bug ?
and did you add support for for watch dog/watch dog2/crash bandicoot :o:o:o

[elit]

Here in Lagos, the capital of Nigeria. It's stashed in our Global Repackers Bank, all you need is just send me 500 USD in Western Union and I will gladly help you to get all amount from the bank. Trust me, I never lie!

I almost fell for it back then, was already in the bank opening secondary account for said incoming 1m$, then he said in the phone he need 500$ first for a lawyer to "finish formalities". Being hardcore "Jew" when it comes to money, I said I don't want 1m$ if it cost me 500$ to get them. And that was it.

PS: Razor, your tool broke PornHub. (nuked) :)

[Razor12911]

Here's a test for stability

simply drag and drop a sample file that has deflate streams, do this at least 5 times, the program should print out numbers in ascending order, these are the number of streams, not really important but what you must test is in those 5 tests, the same numbers are produced

As for the project's progress, well all seems good but I'm just making sure xtool never faces errors and if it does it should try to recover before it hangs or quits.

[Harsh ojha]

Thx @Razor12911
I will test

[shazzla]

Did it five times ,as you requested.

Layers of fear 2 - pakchunk2-WindowsNoEditor.pak - OK !
Legacy of the void - data.006 - OK !
X4 Foundations - 05.dat - OK !

Interesting : in all testfiles the last two numbers were the same/file :

05.dat ends : 11372 ,11372.
data.006 : 110507,110507.
pakchunk2-WindowsNoEditor.pak : 34990,34990.
Maybe is it a bug ?!

Thanks for your work !

[KaktoR]

Did it aswell with Kingdom Come Deliverance and a few pdf files.

All seems to be the way it should be!

[Razor12911]

Did it five times ,as you requested.

Layers of fear 2 - pakchunk2-WindowsNoEditor.pak - OK !
Legacy of the void - data.006 - OK !
X4 Foundations - 05.dat - OK !

Interesting : in all testfiles the last two numbers were the same/file :

05.dat ends : 11372 ,11372.
data.006 : 110507,110507.
pakchunk2-WindowsNoEditor.pak : 34990,34990.
Maybe is it a bug ?!

Thanks for your work !
I actually saw those last repeating numbers, I already fixed the problem after uploading the attachment, it's not really a big deal but thanks for the test.

@everyone
there are more tests to come.

[doofoo24]

@everyone
there are more tests to come.

lz4,lzo,zstd...
is there any speed improvement in raw2hif_dll/hif2raw_dll, for exmaple xtool is slow on doom/cod ghost ?

[Razor12911]

well that is the thing, I didn't make those libraries, if they are slow, they are slow nothing I can do unless if you contact the creator here:
https://encode.ru/threads/1399-reflate-a-new-universal-deflate-recompressor

as for doom, well I had plans for that game, there will be a plugin that will come with xtool called idtech6.dll or something similar. The plugin is specially designed to work best for games that use that engine (also dishonored 2, doto), handling zlib specifically for them unlike the universal zlib that tries all possible methods to give you the best output, the plugin will know beforehand what to do and how to handle the streams from those games and speed up the process.

xtool.exe e:precomp:t100p:idtech6 - -
is how you will use it, there are several games that if they require a special touch, they will just come separate from xtool in plugin form like dunia, unity and etc, which is why I created a separate thread here (https://fileforums.com/showthread.php?t=102833) because I reckon there is going to be many.

[Razor12911]

Here's a speed test

prepare two types of inputs, one with deflate streams and one without deflate streams, can be a video file.

drag and drop the test files onto test.bat, there will be a console output, upload those results as test report.

The test is between xtool v012 and the upcoming release.

Note: Your input shouldn't be from the game that use idTech6 engine (DOOM, Dishonored 2 and Death of the Outsider)

[doofoo24]

test on HellbladeSenuasSacrifice (HellbladeGame-WindowsNoEditor.pak)...

Process ID : 5172
Thread ID : 3824
Process Exit Code: 0
Thread Exit Code : 0

User Time : 153.018s
Kernel Time : 28.031s
Process Time : 181.049s
Clock Time : 129.848s

Working Set : 881712 KB
Paged Pool : 109 KB
Nonpaged Pool : 18 KB
Pagefile : 1133300 KB
Page Fault Count : 541119

IO Read : 16928698 KB (in 264520 reads )
IO Write : 26142474 KB (in 3999156 writes)
IO Other : 2 KB (in 130 others)

WOW Super fast 360mb/s with half cpu usage...

[Andu21]

CI_Sc090.mp4

xtool_new.exe:

Process ID : 3920
Thread ID : 2620
Process Exit Code: 0
Thread Exit Code : 0

User Time : 20.779s
Kernel Time : 0.265s
Process Time : 21.044s
Clock Time : 6.797s

Working Set : 104788 KB
Paged Pool : 91 KB
Nonpaged Pool : 7 KB
Pagefile : 163964 KB
Page Fault Count : 26277

IO Read : 354842 KB (in 5548 reads )
IO Write : 354826 KB (in 46 writes)
IO Other : 5 KB (in 240 others)

xtool_old.exe:

Process ID : 4104
Thread ID : 4888
Process Exit Code: 0
Thread Exit Code : 0

User Time : 40.856s
Kernel Time : 0.187s
Process Time : 41.043s
Clock Time : 11.781s

Working Set : 106836 KB
Paged Pool : 93 KB
Nonpaged Pool : 7 KB
Pagefile : 164968 KB
Page Fault Count : 26918

IO Read : 354843 KB (in 5547 reads )
IO Write : 354826 KB (in 5545 writes)
IO Other : 1 KB (in 220 others)

Echo-WindowsNoEditor.pak

xtool_new.exe:

Process ID : 4832
Thread ID : 2644
Process Exit Code: 0
Thread Exit Code : 0

User Time : 196.046s
Kernel Time : 3.322s
Process Time : 199.368s
Clock Time : 489.965s

Working Set : 213032 KB
Paged Pool : 91 KB
Nonpaged Pool : 9 KB
Pagefile : 282924 KB
Page Fault Count : 90883

IO Read : 4233301 KB (in 66149 reads )
IO Write : 4476621 KB (in 171112 writes)
IO Other : 6 KB (in 240 others)

xtool_old.exe:

Process ID : 2736
Thread ID : 4384
Process Exit Code: 0
Thread Exit Code : 259

User Time : 376.648s
Kernel Time : 2.792s
Process Time : 379.440s
Clock Time : 464.932s

Working Set : 214460 KB
Paged Pool : 93 KB
Nonpaged Pool : 9 KB
Pagefile : 287852 KB
Page Fault Count : 100289

IO Read : 4233303 KB (in 66148 reads )
IO Write : 4446197 KB (in 69472 writes)
IO Other : 1 KB (in 232 others)

[dixen]

bundle_pc.ipk (Rayman Origins)

xtool_new.exe:

Process ID : 9272
Thread ID : 1356
Process Exit Code: 0
Thread Exit Code : 0

User Time : 60.453s
Kernel Time : 1.546s
Process Time : 61.999s
Clock Time : 21.767s

Working Set : 448404 KB
Paged Pool : 110 KB
Nonpaged Pool : 11 KB
Pagefile : 540260 KB
Page Fault Count : 211743

IO Read : 2046443 KB (in 31979 reads )
IO Write : 3038292 KB (in 5711 writes)
IO Other : 6 KB (in 202 others)

xtool_old.exe:

Process ID : 1220
Thread ID : 4636
Process Exit Code: 0
Thread Exit Code : 0

User Time : 130.625s
Kernel Time : 2.593s
Process Time : 133.218s
Clock Time : 44.855s

Working Set : 2084988 KB
Paged Pool : 111 KB
Nonpaged Pool : 10 KB
Pagefile : 2291268 KB
Page Fault Count : 1055801

IO Read : 2046448 KB (in 31978 reads )
IO Write : 3038286 KB (in 47474 writes)
IO Other : 3 KB (in 195 others)

[Razor12911]

Thanks for the tests, 95% complete. the new release will first ship with 1 codec (zlib) just to see if everything works as expected

[IgorKolesnik]

Thanks for the tests, 95% complete. the new release will first ship with 1 codec (zlib) just to see if everything works as expected

can update game file scaner? thank

[Razor12911]

can update game file scaner? thank

and add what feature? :rolleyes:

@everyone

here's a benchmark

some file from Pro Evolution Soccer

old xtool = 275MB >> 381MB took 11.63 seconds using 4 threads
new xtool = 275MB >> 381MB took 06.09 seconds using 4 threads

same memory usage and I/O

note: I only reworked the encoder to use the cpu effectively so you should expect roughly x1.25-2 more speed from old version while using same resources and this applies to all codecs reflate, oodle and etc

[Simorq]

Hi Razor12911
v: skip verification
This option works for oodle:D, crilayla:D, lzo:D.
Please do not delete this option in the new version.

[KaktoR]

and add what feature? :rolleyes:
Just fix the hang bug :D
@everyone

here's a benchmark

some file from Pro Evolution Soccer

old xtool = 275MB >> 381MB took 11.63 seconds using 4 threads
new xtool = 275MB >> 381MB took 06.09 seconds using 4 threads

same memory usage and I/O

note: I only reworked the encoder to use the cpu effectively so you should expect roughly x1.25-2 more speed from old version while using same resources and this applies to all codecs reflate, oodle and etc

This is great. :cool:

[dixen]

and add what feature? :rolleyes:

oodle detect, for example:D

[shazzla]

If game uses oodle,then you can find its DLLs in game folder somewhere.
Imho : oodle stream scanner is obsolete ?!

[Razor12911]

Hi Razor12911
v: skip verification
This option works for oodle:D, crilayla:D, lzo:D.
Please do not delete this option in the new version.

The option will be there

oodle detect, for example:D

I remember including an oodle scanner, but fine I'll think of something for the scanner.

This is great. :cool:

how about this?

There will be a new feature in xtool called history data, what it does is simply store processed stream information and be able to recall it at the later stage.

If you are wondering how this helps, well there are certain games made by lazy developers who repeat game resources instead of just reusing them, resulting in bigger game size for no absolute reason, apart from that, if they are compressed streams it means that xtool will have to process the same stream over and over again, storing history data allows it to prevent the same stream from being reprocessed instead, it will just look at the history and find out how the previous copy was processed then automatically assume it's the same stream and then apply the same settings, resulting in more speed.

I ran a test on a file I repeated on purpose and these were the results.

old xtool took 196.671 seconds and used 335MB ram
new xtool took 154.174 seconds and used 319MB ram
new xtool with history feature took 72.223 seconds and used 439MB ram

the cost of the feature depends on how much history you want to be stored but with default settings, it should use an additonal 192mb ram usage than without, you can always disable it or order it to store more history, depends on how much memory you have to spare.

Note:
In a practical situation, you can shave off a few seconds but then again, depends on the input

[KaktoR]

This is also good :D

Did you have to make changes to the libraries too? Or just the source of xtool executable? Just corious :)

[Razor12911]

Only the source of xtool, all I did was optimize optimize and optimize, the CPU is forever doing something, I tried to minimize the gaps between reads and writes by making xtool process stuff while it is reading and writing so yeah, the threads hardly get rests now.

[Razor12911]

Project status = Welp...

I hope this isn't strike 2 in terms losing my sources, my PC died about 2 days ago, motherboard and GPU got fried this time XD. As for my sources well they were stored on my main PC (the dead one) and I don't know if they are safe since I don't have a spare PC to boot up the drive to grab important files and store them to my backup drive, so I can't really tell if the only components that died were only the board and GPU and not my HDD, but I guess that's a story for some other time. If you are wondering what this means in terms of xtool 2019...

Well since I normally code from desktop because I don't need to go far to do things and etc, it was different for xtool somehow... because I was coding from a backup drive only for that project like I knew some shit like this would happen so yes the source is safe (lucky af), if you are wondering about the development now that my main PC is busted.

Well I just had to dust an old laptop and continued development there, issue is it's incredibly slow but it will have to do. If you are wondering why I only provided x86 binaries, well. OS is 32-bit :D, somehow can't compile for x64.

Anyways enough about that, this was suppose to be the last test before the first release, but before I uploaded, I noticed some shitty error in made in the reflate code and I have to rewrite some parts of it so after fixing that, I'll post the final test and if all is good, I'll be making a release.

there are 3 codecs in xtool right now (zlib, reflate, rzlib). The bug was in reflate in multi threaded mode when decoding, but if you want to try it out anyways, open the test bat file. replace zlib with reflate or rzlib but use 1 thread to decode.

zlib = xtool uses zlib1.dll/zlibwapi.dll to process data, fails on some streams
reflate = xtool uses reflate dlls to process data, processes all streams but slow
rzlib = xtool blends zlib and reflate dlls to process data, processes all streams but with better size and speed.

PS: Even the archive is zip, don't have 7-Zip installed for my normal 7z attachments :o
PPS: Nothing is stopping the development of this project XD... I hope cause you never know

[FitGirl]

Shit, mate! You're lucky to have not loosing sources this time! Gotta use some repo soft next time for more security, no? :)

Anyways, it's nice to see the project is coming to release.

[Harsh ojha]

@Razor12911
Download Windows 10 64 bit Archived file and install

https://youtu.be/zGQ9xXYgVJM
Since i also lost my data

Well we are all depends our pc and laptops

[Razor12911]

First release

Notes

- No idea whether x64 works or not, I just somehow managed to compile. That's up to you to find out.
- Reflate currently is set to use level 6, if you are looking for better output, best to use the old xtool. The reason it's like this is because I'm still working on a way to improve the horrible slow code I used before.
- Skip verification option is currently unavailable.

What I can say, run a lot of tests on this release. I've written some dangerous memory saving code and it may cause errors but if all is well then I can move on to add other codecs such as oodle, lzo, lz4 and the rest. BTW, crilayla will NOT be added, that piece of crap is incredibly slow :D

[Simorq]

xtool_1907_R1 64Bit
Creating archive: data.arc using zlib
Compressed 1 file, 668,113 => 668,225 bytes. Ratio 100.02%
Compression time: cpu 0.02 sec/real 0.18 sec = 9%. Speed 3.71 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 668,225 => 668,113 bytes. Ratio 100.02%
Testing time: cpu 0.00 sec/real 0.06 sec = 0%. Speed 10.77 mB/s
All OK

Creating archive: data.arc using reflate
Compressed 1 file, 668,113 => 1,767,491 bytes. Ratio 264.55%
Compression time: cpu 0.00 sec/real 0.19 sec = 0%. Speed 3.44 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 1,767,491 => 668,113 bytes. Ratio 264.55%
Testing time: cpu 0.00 sec/real 0.16 sec = 0%. Speed 4.31 mB/s
All OK
Creating archive: data.arc using rzlib
Compressed 1 file, 668,113 => 1,767,491 bytes. Ratio 264.55%
Compression time: cpu 0.00 sec/real 0.28 sec = 0%. Speed 2.40 mB/s
All OK

Testing archive: data.arc
Tested 1 file, 1,767,491 => 668,113 bytes. Ratio 264.55%
Testing time: cpu 0.00 sec/real 0.16 sec = 0%. Speed 4.25 mB/s
All OK

[Simorq]

Does not work with ba2/csg files.
Fallout4 - Textures7.ba2 / DLCCoast - Geometry.csg
32bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib
64bit= ERROR: write error (disk full?) in compression algorithm zlib/reflate/rzlib or stuck cpu
http://s8.picofile.com/file/8368123326/zlib.jpg

-------------------------------------------------------------------

Works with this setting.
[External compressor:rzlib,zlib,reflate]
header = 0
packcmd = xtool.exe precomp:rzlib:c32mb,t1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode:t50p - - <stdin> <stdout>

Fallout4 - Textures7.ba2 520 MB
ZLIB
Creating archive: data.arc using zlib
Compressed 1 file, 545,959,220 => 1,237,654,873 bytes. Ratio 226.69%
Compression time: cpu 0.52 sec/real 84.11 sec = 1%. Speed 6.49 mB/s
All OK

zlib: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%
Testing time: cpu 0.45 sec/real 9.53 sec = 5%. Speed 57.29 mB/s
All OK
REFLATE
Creating archive: data.arc using reflate
Compressed 1 file, 545,959,220 => 1,237,783,515 bytes. Ratio 226.72%
Compression time: cpu 0.42 sec/real 113.43 sec = 0%. Speed 4.81 mB/s
All OK

reflate: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,783,515 => 545,959,220 bytes. Ratio 226.72%
Testing time: cpu 0.36 sec/real 12.23 sec = 3%. Speed 44.62 mB/s
All OK
RZLIB
Creating archive: data.arc using rzlib
Compressed 1 file, 545,959,220 => 1,237,654,873 bytes. Ratio 226.69%
Compression time: cpu 0.53 sec/real 85.22 sec = 1%. Speed 6.41 mB/s
All OK

rzlib: decode:t2
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%
Testing time: cpu 0.98 sec/real 73.38 sec = 1%. Speed 7.44 mB/s
All OK

rzlib: decode:t100p
Testing archive: data.arc
Tested 1 file, 1,237,654,873 => 545,959,220 bytes. Ratio 226.69%
Testing time: cpu 0.77 sec/real 73.54 sec = 1%. Speed 7.42 mB/s
All OK

[Razor12911]

Can you run xtool outside freearc, like via console.

@echo off
xtool.exe precomp:rzlib:c16mb,t100p %1 %1.out
pause

I'm interested in the exception/error message

[Simorq]

@Razor12911
good
http://s8.picofile.com/file/8368124442/rrrr.jpg

-------------------------
I think the main problem is - -.
- - <stdin> <stdout>:confused:

[Razor12911]

try this then

@echo off
xtool.exe precomp:rzlib:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res
pause

[Simorq]

@Razor12911
no work
stuck cpu
http://s9.picofile.com/file/8368124692/rrr.jpg

[Razor12911]

Nice ghost bugs, I'll check what I missed in the source and thanks for the tests.

[Sergey3695]

I have a problem using t100p.

[Sergey3695]

add pls lvl setting for reflate 1-9. thx.

[Razor12911]

add pls lvl setting for reflate 1-9. thx.

Have patience

[IgorKolesnik]

using these parameters, the expansion reaches 2.4% and freezes.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp:rzlib,zlib,reflate:c16mb,t2 - - <stdin> <stdout>


I am using the x64 version.

[FitGirl]

using these parameters, the expansion reaches 2.4% and freezes.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp:rzlib,zlib,reflate:c16mb,t2 - - <stdin> <stdout>


I am using the x64 version.
And why do you use all three methods at the same time?

[IgorKolesnik]

And why do you use all three methods at the same time?
если я оставлю только один метод, ошибка та же.

----

if I leave only one method, the error is the same.

[IgorKolesnik]

But what is the difference between rzlib and zlib?

[FitGirl]

But what is the difference between rzlib and zlib?
Don't be that lazy.
https://fileforums.com/showpost.php?p=481904&postcount=35

As for decompression error - it's been already reported and Razor answered.

And don't use Russian here, it's English-only forum.

[Razor12911]

There are internal errors I've come across, so for now until next update, don't use xtool. the expansion stops because of memory management issues, xtool keeps increasing memory usage over and over again until you run out of memory, I noticed this when I was running some tests.

As of right now, I have fixed this issue but am running more tests to make sure all is well.

FitGirl mentioned you must pick one codec in the zlib lineup, rzlib is what is best and it's the same one used in the old xtool, I just added the other ones (zlib and reflate) for special cases where a specific codec is required.

[Razor12911]

Update available

1908_R1
- Fixed MT hanging issues
- Fixed reflate issues
- Fixed memory issues
- Improved zlib codec
- Added auto reflate level detector
- Fixed aborting issues when used via stdio

Notes
If there is a file where this xtool fails to precompress better or the same as ztool, pzlib, reflate or precomp, please upload it.

[Simorq]

Streams: ZLib
Middle-Earth - Shadow of Mordor ALL OK
Fallout 4 ALL OK

Wargame Red Dragon
Streams: Reflate
pZLib,ZTool,XTool CRC Error
xtool_1908_R1 Good Work

Log:
Reflate
Creating archive: RG-reflate.Bin using reflate
Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66%
Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s
All OK

Testing archive: RG-reflate.Bin
Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66%
Directory 3,955 => 88,873 bytes. Ratio 4.45%
Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s
All OK

rZLib
Creating archive: RG-rzlib.Bin using rzlib
Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66%
Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s
All OK

Testing archive: RG-rzlib.Bin
Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66%
Directory 3,953 => 88,871 bytes. Ratio 4.45%
Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s
All OK

[Razor12911]

Log:
Reflate
Creating archive: RG-reflate.Bin using reflate
Compressed 2,586 files, 30,541,141,079 => 44,791,613,736 bytes. Ratio 146.66%
Compression time: cpu 22.98 sec/real 1280.30 sec = 2%. Speed 23.85 mB/s
All OK

Testing archive: RG-reflate.Bin
Tested 2,586 files, 44,791,613,736 => 30,541,141,079 bytes. Ratio 146.66%
Directory 3,955 => 88,873 bytes. Ratio 4.45%
Testing time: cpu 16.83 sec/real 924.56 sec = 2%. Speed 33.03 mB/s
All OK

rZLib
Creating archive: RG-rzlib.Bin using rzlib
Compressed 2,586 files, 30,541,141,079 => 44,791,601,829 bytes. Ratio 146.66%
Compression time: cpu 68.80 sec/real 2496.90 sec = 3%. Speed 12.23 mB/s
All OK

Testing archive: RG-rzlib.Bin
Tested 2,586 files, 44,791,601,829 => 30,541,141,079 bytes. Ratio 146.66%
Directory 3,953 => 88,871 bytes. Ratio 4.45%
Testing time: cpu 17.86 sec/real 910.84 sec = 2%. Speed 33.53 mB/s
All OK

Can you test these exes on the same input, thanks

[78372]

FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc
Compressed 1 file, 127,027,167 => 497,974,169 bytes. Ratio 392.02%
Compression time: cpu 0.41 sec/real 76.37 sec = 1%. Speed 1.66 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc
Tested 1 file, 497,974,169 => 127,027,167 bytes. Ratio 392.02%
Testing time: cpu 0.23 sec/real 27.32 sec = 1%. Speed 4.65 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc
Compressed 1 file, 127,027,167 => 436,735,325 bytes. Ratio 343.81%
Compression time: cpu 0.42 sec/real 47.82 sec = 1%. Speed 2.66 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc
Tested 1 file, 436,735,325 => 127,027,167 bytes. Ratio 343.81%
Testing time: cpu 0.14 sec/real 16.94 sec = 1%. Speed 7.50 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc
Compressed 1 file, 127,027,167 => 498,080,370 bytes. Ratio 392.11%
Compression time: cpu 0.53 sec/real 161.65 sec = 0%. Speed 0.79 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc
Tested 1 file, 498,080,370 => 127,027,167 bytes. Ratio 392.11%
Testing time: cpu 0.27 sec/real 60.78 sec = 0%. Speed 2.09 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datarzlib.arc
Compressed 1 file, 111,299,563 => 482,246,557 bytes. Ratio 433.29%
Compression time: cpu 0.45 sec/real 78.94 sec = 1%. Speed 1.41 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datarzlib.arc
Tested 1 file, 482,246,557 => 111,299,563 bytes. Ratio 433.29%
Testing time: cpu 0.23 sec/real 28.13 sec = 1%. Speed 3.96 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datazlib.arc
Compressed 1 file, 111,299,563 => 421,007,713 bytes. Ratio 378.27%
Compression time: cpu 0.39 sec/real 48.68 sec = 1%. Speed 2.29 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datazlib.arc
Tested 1 file, 421,007,713 => 111,299,563 bytes. Ratio 378.27%
Testing time: cpu 0.16 sec/real 16.98 sec = 1%. Speed 6.55 mB/s
All OK

FreeArc 0.67 (March 15 2014) creating archive: datareflate.arc
Compressed 1 file, 111,299,563 => 482,352,758 bytes. Ratio 433.38%
Compression time: cpu 0.33 sec/real 162.03 sec = 0%. Speed 0.69 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: datareflate.arc
Tested 1 file, 482,352,758 => 111,299,563 bytes. Ratio 433.38%
Testing time: cpu 0.09 sec/real 61.75 sec = 0%. Speed 1.80 mB/s
All OK

[Simorq]

@Razor12911

x64
---------------------------
XTool.exe - Application Error
---------------------------
The application was unable to start correctly (0xc000007b). Click OK to close the application.
---------------------------
OK
---------------------------

x86

t100p
ERROR: write error (disk full?) in compression algorithm reflate
ERROR: write error (disk full?) in compression algorithm rzlib
ERROR: write error (disk full?) in compression algorithm zlib

But it works with t1,t2,t3,t4 in pre-compression \ t100p Works on decompression..
AMD RYZEN 5 1600 6-Core

[Simorq]

udun_weather.arch05 2.70 GB

1908_R1 x86 t100p for pre-compression = ERROR: write error (disk full?) in compression algorithm

1908_R1 x86

================================================== =============================
t4
Creating archive: Game_ZLib.Bin using zlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%
Compression time: cpu 1.55 sec/real 80.55 sec = 2%. Speed 36.04 mB/s
All OK

t100p
Testing archive: Game_ZLib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%
Testing time: cpu 1.11 sec/real 32.64 sec = 3%. Speed 88.95 mB/s
All OK
================================================== =============================

================================================== =============================
t4
Creating archive: Game_reflate.Bin using reflate
Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54%
Compression time: cpu 1.88 sec/real 318.25 sec = 1%. Speed 9.12 mB/s
All OK

t100p
Testing archive: Game_reflate.Bin
Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54%
Testing time: cpu 1.61 sec/real 55.21 sec = 3%. Speed 52.58 mB/s
All OK
================================================== =============================

================================================== =============================
t4
Creating archive: Game_rzlib.Bin using rzlib
Compressed 1 file, 2,902,848,160 => 4,223,911,477 bytes. Ratio 145.51%
Compression time: cpu 1.80 sec/real 79.44 sec = 2%. Speed 36.54 mB/s
All OK

t100p
Testing archive: Game_rzlib.Bin
WARNING: CRC failed in "udun_weather.arch05". File is broken.
Tested 1 file, 4,223,911,477 => 2,902,848,160 bytes. Ratio 145.51%
Testing time: cpu 1.59 sec/real 43.39 sec = 4%. Speed 66.90 mB/s
There were 1 warning(s)
================================================== =============================


_binaries_fpc x86

================================================== =============================
t4
Creating archive: Game_ZLib.Bin using zlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%
Compression time: cpu 1.73 sec/real 76.23 sec = 2%. Speed 38.08 mB/s
All OK

t100p
Testing archive: Game_ZLib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%
Testing time: cpu 1.44 sec/real 47.60 sec = 3%. Speed 60.98 mB/s
All OK
================================================== =============================

================================================== =============================
t4
Creating archive: Game_reflate.Bin using reflate
Compressed 1 file, 2,902,848,160 => 4,224,827,147 bytes. Ratio 145.54%
Compression time: cpu 1.56 sec/real 303.38 sec = 1%. Speed 9.57 mB/s
All OK

t100p
Testing archive: Game_reflate.Bin
Tested 1 file, 4,224,827,147 => 2,902,848,160 bytes. Ratio 145.54%
Testing time: cpu 1.59 sec/real 51.68 sec = 3%. Speed 56.17 mB/s
All OK
================================================== =============================

================================================== =============================
t4
Creating archive: Game_rzlib.Bin using rzlib
Compressed 1 file, 2,902,848,160 => 4,223,996,918 bytes. Ratio 145.51%
Compression time: cpu 1.55 sec/real 77.61 sec = 2%. Speed 37.40 mB/s
All OK

t100p
Testing archive: Game_rzlib.Bin
Tested 1 file, 4,223,996,918 => 2,902,848,160 bytes. Ratio 145.51%
Testing time: cpu 1.47 sec/real 42.34 sec = 3%. Speed 68.56 mB/s
All OK
================================================== =============================


__________________________________________________ __
The 1908_R1 x64 version works very stable and well.:D

[Razor12911]

Update available

1908_R2
- Updated history size feature
- Added save/load history file feature
- Improved processing speed

The benefits of the usage history database files

I'll use GTAV as an example, it's a huge game and has a lot of highly compressed streams.

So if you were compressing this game and you messed up the srep/lolz settings or your PC shutdown unexpectedly due to several reasons like powercuts or windows forcing updates and etc, if xtool already stored a database then it will load it up with all the information that was used in previous precompression session on the same input to speed up the process because it will know what needs to be done and what settings should be used.

Here's an example on update.rpf (GTAV)

on first run:
Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48%
Compression time: cpu 1.78 sec/real 77.43 sec = 2%. Speed 10.52 mB/s

since I have processed this input before, xtool loaded up history file:
Compressed 1 file, 814,551,040 => 1,616,722,602 bytes. Ratio 198.48%
Compression time: cpu 2.50 sec/real 26.25 sec = 10%. Speed 31.03 mB/s

The reason I decided to add this feature was because of games compressed with ridiculously high compression settings resulting in precompression being very slow so then I wanted people to save time by allowing at least one of us to process the game then share their database file with everyone so that they can use it to save time on precompressing the same game.

This xtool is already faster than the old one and I just had to find more ways to give you guys more speed. :D

Note: It doesn't matter if the input is different from the one that was used to create a specific history file, an example to explain this, if one used GTAV with latest update to make a history file, a person who has the first release of GTAV without any updates can still use the same history file without problems or visa versa, if a game got updated and you wanted to compress it again with the new files, you can use the old history file.

[Harsh ojha]

Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end :)

[Sergey3695]

reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD

[elit]

There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.

[Sergey3695]

rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?

[Razor12911]

Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end :)

doesn't seem like libraries are loaded :)

reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD

Well up to you, if you know how to use reflate lvls

[Razor12911]

rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?

:eek::eek::eek: and you want me to remove auto detect.

This is why I add it in the first place, especially when it comes to reflate because most people don't know how it works. Low level does not mean you save time and at times, if you choose incorrect levels you end up with negative ratio. Example is GTAV, people used level 1 just because it was faster, resulted in horrible results then at some point people just decided to use level 9 on everything because "most" games are highly compressed, then game mad max which was level 1 compressed, they got shitty results again so yea, if you want level settings, I guess it's not a problem to add it but I added autodetect for a reason, because of people messing up

[Razor12911]

There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.

I'd like to see this method fly :D

especially with the case of xtool because of multi threading, precomp might find this useful because it is single threaded but xtool? no. It's not something I have not thought about before but look at it from this perspective. you're processing DiRT Rally, a game with large streams, so to detect level according to you is to process the first stream, check it level by level. Take note when it comes to deflate, there are not just 9 options (1-9), but there are 81 options, each level has memory settings (9 of them as well) with also affect the checksum of the stream if you are doing trial and error approach, so you're telling me it's best to try 81 levels on the first stream, so the next 9 streams can use the same method, during those 81 trial and errors, the other threads in xtool, what would the other threads be doing during that time? :D, a user selected to use 8 threads but it's still just 1 thread that needs to figure out what options to use and then after that is done, they can proceed. As those threads process the streams using the level they found from the first stream, if it's incorrect, they all now have to go trial and error for all those streams whichever way you look at it, this method has drawbacks in terms of time the other threads spend by not doing anything and just waiting, other drawback is, it's not really guaranteed that all those 81 options you'll be trying will give back perfect checksum so you can be making the other threads waiting for something that will never come.

xtool is fast not just because of multi threading, you can even compare it with precomp using 1 thread. I've thought about level detection and etc.

Here's how xtool works, the scanner itself can even give ideas of what level was used, zlib has a 2 byte header that is generated based on what level was used to compress a specific stream but sometimes this is not enough because some streams are headerless. This is where I decided to add a statistics system that stores information as all threads process the streams, this statistic system I introduced allows xtool to determine which cmp levels popped up the most and which ones were less, so if it were a case of trial and error, it wouldn't be trying from level 1-9, it would be the level that appeared the most using previous stream data(could even be in the order (6,5,9,8,1,2,3,4..). There are more strategies I've added for cmp level detection and they are all quick because they use stats

[Sergey3695]

:eek::eek::eek: and you want me to remove auto detect.

add setting - turn off only/

[elit]

Razor,

you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?

As for my idea,
It don't need to go through all 81 every time, only until match is found and then next stream can be tested with that level found in previous stream first. This means if streams are not mixed there will be no slowdown because first try is a match already. Also your own test implementation on first stream seem fast, why not use same technique, whatever is it doing?

I have never noticed ztool/xtool being slow at beginning, it seems to start immediately. Would there really be such big difference if it had to try 10 streams instead of 1, especially with optimization I mentioned above(starting with previously found level)? Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?

As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long).

You mentioned using stats, that sounds actually very similar to what I am talking about. In fact, if I understand correctly you may have already implemented what I am talking about here :). But interestingly, you mentioned collecting stats from threads, that pretty much equals to cmp levels. So you in fact are collecting from multiple streams, not just first?

[Razor12911]

"you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?"
- it does for all streams, all 81 options but very quickly, takes milliseconds per stream to go through all 81 options.

"It don't need to go through all 81 every time"
- it doesn't go through all 81, sometimes even the first option it tries is correct and it stops, if it isn't correct then the 2nd one is most likely correct because of stats. XTool requests mode (value that appears most often), if that option fails, it requests mode in 2nd order (value that appears the 2nd most often) and so forth... which is why trials on all streams is necessary because it keeps the stats function filled with updated information about the current input.

"next stream can be tested with that level found in previous stream first."
- this method is the same as stat but it stores information limited to one stream which happens to be the previous stream, the approach of xtool is it stores a number of previous stream information, default is 256 streams.
Let's say for example
if the first stream was gave back level 9, the next stream is determined to be level 9, roughly the same as what you suggested, but what if the third stream is level 6? you have go through all the trials, ok fine, now you think then next stream is level 6 because of previous stream but what if that level 6 stream was just png image within the game data but instead the entire game was level 9? you try level 6 on the next stream it fails, then again you have to go through all 81 options again. XTool apporach thanks to stats, is level 9, level 9 and level 6 is what was stored, so for the next stream, it doesn't matter if the last one was level 6, it will try level 9 because it was detected twice compared to level 6, if it wasn't level 9, it will try that level 6 before it tries all 81 options because it was at least detected once, if that fails. it then tries 81 - 2 (6-9 from stats already failed) it does this for the past 256 streams and tries options in mode orders.

"I have never noticed ztool/xtool being slow at beginning, it seems to start immediately"
- It's because of stats plus other methods I've developed that make this possible.

"Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?"
You'll notice not just "few seconds of delay", yes few seconds for a set of streams but minutes of overall delay :D if you were to use this method for big streams, try this on dirt rally or doom 2016 and you'll see some incredibly low CPU utilization when using a lot of threads because other threads are obliged to wait for the first stream.

"As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long)."

with large streams, it is guaranteed to take long as mentioned before, which is why xtool faces too many problems with multi threading, I've instructed the threads to try 81 levels while using stats from all the streams they can find, as a result they end up fighting and crash each other :D it's hilarious but you get the idea, it's just best to try all levels on all streams but let stats decide in what order they should be presented in to speed up the process.

"You mentioned using stats, that sounds actually very similar to what I am talking about."
- Yes with the exception that not the previous stream option is considered but the previous 256 stream options are considered. If from those 256 options, there are 128 level 6s, 10 level 3s , 1 level 5 and 20 level 9s, it will try them in the order: 6 >> 9 >> 3 >> 5, then since all that failed, it begins trials, 1 - 9, it will skip, 3,5,6 and 9 because those failed, after that the new discovered level is added to stats, if it was 2, it will try the new order as 6 >> 9 >> 3 >> 5 >> 2, the reason all streams have to try all options is to get rid of non appearing options, example is when the data out of nowhere started to become level 4 out of nowhere, as stats are added, the number of level 9s will decrease because of these new levels 4s until they reach a certain threshold, then the order will now be 4 >> 6 >> 9 >> 3 >> 5 >> 2, as you can see level 4 over took level 6 but level 6 is still considered because before all of the level 4s there were a lot of level 6s so yes, I really did think deep about how to speed up xtool :)

"I understand correctly you may have already implemented what I am talking about here"
- Yep

"So you in fact are collecting from multiple streams, not just first?"
- Yes, I had to carefully do this else the threads crash themselves while fighting for the same stream or the same information, causes headaches but I find it funny when this happens, this is what causes most problems in pzlib/ztool and old xtool but if you properly implement it, you get to enjoy all the speed that comes with it.

[elit]

Finally, now(!) I am in picture. Thanks for taking your time, it does make a difference when I understand internals.

Believe or not until now I thought (z)xtool simply scan first stream and.. that's it - use same info for *all* rest of data lol :). Reason I thought so was that game packs are likely using single specific level and that could in theory be enough(to pack the game, for which zxtools were designed).

Since that is not the case and your approach is clearly even much better than mine we can rest this case. Thanks again.

[Simorq]

1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work

decode has a bug.

decode:t1
Testing archive: data.arc
Tested 2 files, 4,023,182,817 => 2,147,912,601 bytes. Ratio 187.31%
Testing time: cpu 1.09 sec/real 23.73 sec = 5%. Speed 90.52 mB/s
All OK

decode:t100p
Testing archive: data.arc
Tested 2 files, 4,023,182,817 => 2,147,912,601 bytes. Ratio 187.31%
Testing time: cpu 1.16 sec/real 22.47 sec = 5%. Speed 95.61 mB/s
All OK

Even with t1 it uses all cores!

[doofoo24]

1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work


also with x64 you can get "ERROR: write error (disk full?) in compression algorithm"...
i tested on doom with reflate t100p with c384mb i get ( ERROR: write error (disk full?))...
i changed to t4 it work...

for the decode it work i tested with t1 and t100p, for t1 use 11% of the cpu and for t100p it use all core...
***it's so important to emphasis do not use t100p for decode if you don't have a good cpu cooler, better to use t50p...

[Razor12911]

Guys, I’ll repeat. If you encounter an error, use xtool without fa so it produces an exception message so I know what the problem is

[Simorq]

#############################################
# reflate (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

xtool.exe precomp:reflate:c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1 = OK
Test 2 = OK
Test 3 = OK
Test 4 = OK

XTool x86 Command Line

xtool.exe precomp:reflate:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1

P: OK
D: OK (FC: no differences encountered)

Test 2

P: EAccessViolation: Access violation at address 5378614D in module 'xtool.exe'. Read of address 5378614D
XTool is created by Razor12911

D: EReadError: Stream read error


Test 3

P: Exception EAccessViolation in module xtool.exe at 000BCEC7.
Access violation at address 004BCEC7 in module 'xtool.exe'. Read of address 00611D34.
D: EReadError: Stream read error

reflate works well with c16m in x86.:rolleyes:

[Simorq]

#############################################
# zlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

zlib
xtool.exe precomp:zlib:c32mb and c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res


TEST 1
P: OK
D: CRC Error (0F2C892C: 09 D7)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: EAccessViolation: Access violation at address 00000000 in module 'xtool.exe'. Read of address 00000000

D: EReadError: Stream read error

zlib is very unstable, once successful, failed several times:(

[Simorq]

#############################################
# rzlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

rzlib
xtool.exe precomp:rzlib:c16mb/c32mb/c64mb/c128mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

TEST 1
P: Error (EReadError: Stream read error)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: Error (EReadError: Stream read error)

TEST 5
P: Error (EReadError: Stream read error)
rzlib always fails.:(

[Razor12911]

@Simorq

Looks like I'll have to go through debugging road once more :D

@everyone

Update available

1908_R3
- Compatiblity broken (old history db and inputs cannot work in this version)
- Added plugin support
- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)
- rzlib codec removed (use zlib)
- Added x86 memory limit

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins which can be found here (https://fileforums.com/showthread.php?t=102833)

[ZAZA4EVER]

xtool_1908_R3 X64 ZLIB CODEC
eFootball Pes2020 Demo ... dt00_4K_x64.cpk
FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 15,252,601 => 72,893,019 bytes. Ratio 477.91%
Compression time: cpu 0.05 sec/real 37.50 sec = 0%. Speed 0.41 mB/s
All OK



Zlib Codec is Stable .. i tested it Three Times ...:D
/////////////////////////////////////////////////////////////
xtool_1908_R3 X64 oodle Codec
Wolfenstein Youngblood ... chunkbase_9_pc.resources

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 100,552,642 => 100,552,714 bytes. Ratio 100.00%
Compression time: cpu 0.33 sec/real 3.38 sec = 10%. Speed 29.79 mB/s
All OK


oodle leviathan Stream Dont Work :o

[FitGirl]

oodle leviathan Stream Dont Work :o
They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea :) Also it's slow af in Oodle 2.6.* .

[Razor12911]

oodle leviathan Stream Dont Work :o

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins

Wolfenstein Youngblood ... chunkbase_9_pc.resources

- Could you upload a sample, I need to check something


They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea :) Also it's slow af in Oodle 2.6.* .

Last time I checked the headers did contain size, it was a while ago but I didn't add support because no game used this codec back then

[Simorq]

Hi Razor12911

Assassin's Creed: Origins
Test File: DataPC_extra.forge / oodle4

Creating archive: data.arc using oodle
Compressed 1 file, 1,049,427,968 => 1,049,428,104 bytes. Ratio 100.00%
Compression time: cpu 1.47 sec/real 4.11 sec = 36%. Speed 255.62 mB/s
All OK
__________________________________________________ _____________________________________________
DataPC_SharedGroup_02.forge Sample File (https://www.mediafire.com/file/7320ql6zwytyh1o/DataPC_SharedGroup_02.forge/file)
Oodle_DLL_x86/x64 (https://www.mediafire.com/file/pj0fbo36pmwqoyi/Oodle_DLL_x86-x64.7z/file)

Oodle precompressor (Side project) (https://fileforums.com/showthread.php?t=102453)
Creating archive: data.arc using oo2recm
Compressed 1 file, 105,512,960 => 167,404,655 bytes. Ratio 158.66%
Compression time: cpu 0.17 sec/real 15.59 sec = 1%. Speed 6.77 mB/s
All OK

XTool 1908_R3
Creating archive: data2.arc using oodle
Compressed 1 file, 105,512,960 => 105,513,032 bytes. Ratio 100.00%
Compression time: cpu 0.09 sec/real 0.53 sec = 18%. Speed 199.34 mB/s
All OK

______________________________________

The problem was solved thanks to KaktoR.

[doofoo24]

- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)


"placeholders" :)

[KaktoR]

@Simorq:

Oodle is not inside xtool 2019 yet (they are only "placeholders" for later use). Only zlib and unity lz4 are inside.

[Razor12911]

Yes they are placeholders meant to be used by the plugins, they will get universal support eventually. But the plugins should give the best output since they are specific and not universal and as we know the universal tools I made (side projects) couldn't handle some games properly so I added placeholders to try and give the best precompression via plugins

@everyone, update was delayed. Because I noticed several issues in xtool thanks to Panker, FitGirl and of course you guys, plus I was busy making plugins for some games.

The changelog for next update

- Updated plugin support [DONE]
- Fixed plugin stream detection issues [DONE]
- Added delta/diff support for imperfect streams [0%]
- zlib codec renamed to deflate [DONE]
- Fixed several deflate related issues [DONE]
- Added png stream support [90%]
- Fixed future stream issues [DONE]
- Added low memory usage mode [0%]
- Added zstd universal stream support [DONE]
- Added lzo1x universal stream support (alpha) [DONE]

Plugins done so far:

haemimont, frostbite and anvil and updated unity plugin

[K4miKaZe]

This is looking good.

Many thanks Razor12911 !!!

[ansi]

When packing CS:GO vpk files, xtool 1908 R3 freezes when got to 5%.
Parameters: zlib,lz4:c32mb,t100p
However, there are no freezes on the xtool 1908 R2 version.

[Andu21]

Remove lz4 from the parameters and try again.

[Razor12911]

Update available

1908_R4
- Updated plugin support
- Fixed plugin stream detection issues
- Added delta/diff support for imperfect streams
- zlib codec renamed to deflate
- Fixed several deflate related issues
- Added png stream support
- Fixed future stream issues
- Added low memory usage mode (slower)

Notes

If this update does not fix most issues then my PC crashing has really ruined the development of this project. :(
If you take a look at the change log, it says I renamed zlib codec to deflate, well the reason for that is because zlib codec will be reintroduced later on but it will be mostly use by plugins that find zlib streams in games so as a normal user, use deflate

[doofoo24]

is plugin still placeholder (lz4/lzo...) ?

[Razor12911]

yep, lz4 will forever be a placeholder because I really have no idea how to detect stream, as for lzo well that is a placeholder for that version but next version, it will be there.

[nik1967]

Hello, Razor12911!
What's this? :eek:
The file was downloaded several times.
https://i89.fastpic.ru/big/2019/0822/de/fae4cb09008577c87968304f18ef55de.png

[dixen]

Hello, Razor12911!
What's this? :eek:
The file was downloaded several times.
https://i89.fastpic.ru/big/2019/0822/de/fae4cb09008577c87968304f18ef55de.png
Same bug)

[IgorKolesnik]

When downloading, it says that the archive is corrupted. (

[IgorKolesnik]

use this

thank

[Razor12911]

what is going here lol :D

[Snake288]

Hello,
doofoo24,Razor12911

xtool_1908_R4.7z

The file has been downloaded several times.

file won't open

When downloading, it says that the archive is corrupted.

[Razor12911]

Update available

1908_R5
- Added lzo1x universal stream detection (alpha)

Notes

I've ran a few tests on Far Cry 3 + Blood Dragon and seems like it is working, what you must know is this thing is slow but there is still room for improvement. For each stream detected, there are 2 trials taken and from those trials at least one of them is incorrect furthermore from those 2 trials, there is level detection from level 1-9 so you can imagine the speed deduction but there are ways to improve this somewhere down the line.

Maximum stream support is 4MB, any stream larger than this is not considered.

Streams compressed with lzo1x_999 are the only ones supported.

And don't use this on Unreal Engine games or games that have been compressed with something like lzopro because it will NOT work.

[Grumpy]

@Razor

The same server problem may have corrupted this file as well ;)

[Razor12911]

@Grumpy

hahaha.

@everyone

If you have issues downloading, here is external link
https://mega.nz/#F!03BkjKQA!WpTxyngxucC-ToBRK9xbkw

[Grumpy]

@Razor

I have only just found this out, according to Empire the 4 affected Threads and Files were all posted/uploaded by 'mausschieber' ... so you may have to check your files/uploads afterall. Sorry :o

[Razor12911]

Update available

1908_R6
- Added zlib codec placeholder (zlib)
- Added zstd universal stream detection
- Added kraken, mermaid, selkie universal stream detection
- Fixed plugin issues
- Improved lz4, lzo, zstd, oodle processing speeds
- Removed lzx placeholder (codec complicated)

[Simorq]

http://uupload.ir/files/ubam_1.jpg

CPU Stuck...

Tested with: t1/t50p/t100p - c16m/c32m/c64m/c128m


File Test: ShadowOfWar (snowmtn_lighting.arch06 1.59 GB)

[doofoo24]

yah multi threading stuck during encode :(
but t1 work normal...

[Razor12911]

I only started working on this like today, was busy the whole week and I am completely not surprised about the errors considering I didn't really run any tests :D

While you guys are at it, try the zstd codec, that's also not tested.

Forgot to release notes:

Well I did tweak the detection a bit to detect more streams than oo2reck and its friends

[IgorKolesnik]

when using the xtool R5 reflate version, this error pops up. deflate method

using parameter
packcmd = xtool.exe precomp:deflate:t1,c16mb - - <stdin> <stdout>

[Razor12911]

when using the xtool R5 reflate version, this error pops up. deflate method

using parameter
packcmd = xtool.exe precomp:deflate:t1,c16mb - - <stdin> <stdout>

Someone also reported a similar issue however I need a sample to fix this sort of issue.

[IgorKolesnik]

Someone also reported a similar issue however I need a sample to fix this sort of issue.
tested on Ancestors: The Humankind Odyssey

[doofoo24]

tested (xtool_1908_R6) zstd on MEA xtool.exe remain stuck 0%...
tested (xtool_1908_R6/xtool_1908_R5/xtool_1908_R4) deflate on Spyro Reignited Trilogy pak files always error CRC during install...

[KaktoR]

Razor, if you start working on oodle codec again in future, could you take a look on this sample (https://drive.google.com/open?id=1jqKOcUtddPghPi7rewzm0KlHQWJFWSMn) please? It uses oodle5 dll but your oodle scan detects only [0] for unknown.

[FitGirl]

Razor, if you start working on oodle codec again in future, could you take a look on this sample (https://drive.google.com/open?id=1jqKOcUtddPghPi7rewzm0KlHQWJFWSMn) please? It uses oodle5 dll but your oodle scan detects only [0] for unknown.
Standard TellTale encryption inside.

[ozolt]

Hi Razor12911, can you please rename zstd codec for xtool precomp:{compressor} to something else? I'm asking because [External compressor:zstd] was common entry for standard zstd.exe (for me it works faster than CLS versions). Now we can't use both at the same time since xtool uses same identifier in arc.ini... Yes, I know I can create separate xtool 2019 entry in arc.ini just for that codec but I would rather avoid that if possible.
Regards

[chichi119]

KÃ*nh chúc bác thÃ*nh công trong công việc............................................ ................!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[PSP17]

What combination need to be used for decompression red dead redemption 2 files ?

[ShivShubh]

What combination need to be used for decompression red dead redemption 2 files ?

oo2reck

[FitGirl]

What combination need to be used for decompression red dead redemption 2 files ?
Ash Shiv mentioned below, oo2reck. But keep in mind, that RPFs are partially encrypted, so not all streams will be found and recompressed. It's a new encryption scheme, not the one used in GTA5.

[doofoo24]

@Razor12911 any further development on xtool or the project on halt :(

[Simorq]

[External compressor:deflate,reflate,unity,lzo1c,lzo1x,lzo2a]
header = 0
packcmd = "Razor12911\XTool_R5\XTool.exe" precomp:{compressor}:c32mb,lm,t50p - - <stdin> <stdout>
unpackcmd = "Razor12911\XTool_R5\XTool.exe" decode:t100p - - <stdin> <stdout>
Works well.

The lm option is very important and prevents XTool from getting stuck.
translate.google:p

[Razor12911]

@Razor12911 any further development on xtool or the project on halt :(

Well I had issues handling oodle and every single time I try to fix bugs caused by it I always get errors from nowhere, been doing this for since September last year and I decided to give up on it a few days before new year.

[doofoo24]

:cool:

[traxgaming]

What is Precomp the Files ?
What is the advantage of it ?
How to use it?
What are the methods use it ?

[Razor12911]

Finally an update after a year (08/2019 - 08/2020).

Project is restarted and I hope all goes well this time.

Xtool will be getting frequent updates from now onward since the base code is solid and can allow for new features to be added.

If you have been participating in the [Dev]Xtool thread, thanks a lot. I have managed to fix a lot of bugs and came to a conclusion. Use reflate at your own risk, an alternative has been added which is preflate but this cannot process all streams so choose wisely.

Changes from the Dev Xtool

Added stats monitoring (not visible when using xtool with FA via stdio mode)
Added depth setting (finding streams within streams)
Fixed low memory mode
Speed improvements (when dealing with files with a lot of streams)

Plans for next update
Add stream deduplication

[dixen]

Dishonored. Death of the Outsider

Unpack

Tested 3 files, 13,824,493,421 => 9,663,676,416 bytes. Ratio 143.06%
Testing time: cpu 8.91 sec/real 239.61 sec = 4%. Speed 40.33 mB/s
All OK

[panker1992]

From My POV,

Precompression is give/take antibalanced situation.

Where i see it if you want a game decompressed 100% go ahead and decompress manually all then pack them,
and waste time and space when unpacking to do the same function reversed.

If we can get 98% os the files inputed decompressed we can go ahead and call it a day :D

2% is withing the margin of error :D

[Razor12911]

I was supposed to post an update on the project 2 days ago but then I was greeted by this cloudflare error each time I tried to open the site.

27668

So I just thought the site was down little did I know that either my ISP or the forum blocked my IP address because I used VPN and the site opened.

Perhaps I spend too much time on the forum :D

[Grumpy]

@Razor12911

I quite often get that error when updating the Conversion Index, I just figure it takes to long to update so much text each time ... when I do encounter the error I just open the Index in another window and I find it has updated. ;)

[Masquerade]

Hello, following guidance from the GFS thread, I tried the zlib+preflate method for the dynamic streams located in Automobilista 2, however the files haven't been precompressed. Here is the result:

Compressed 1,438 files, 27,663,572,443 => 27,663,586,865 bytes. Ratio 100.00%
Compression time: cpu 27.78 sec/real 728.72 sec = 4%. Speed 37.96 mB/s
All OK

This makes me think that this method is not correct for this game.

XTool is doing SOMETHING though, since there is CPU and RAM usage, just as if it was processing normal zlib streams.

Any thoughts? I can upload some of the files if anyone is interested.

[shazzla]

I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!

[Masquerade]

I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!

There are oodle libs in game folder, but this game is definitely zlib (as shown in GFS, with dynamic stream detection enabled). Oodle scan detects nothing.

[KaktoR]

Maybe files are crypted?

[shazzla]

If there r oodle libs then those streams r oodle streams imho.
Doesnt matter what scanners says its zlib.
Try with different oodle libs.
Again : imho. But maybe im very wrong...

@Kaktor : +1

[Masquerade]

Maybe files are crypted?

The files are definitely encrypted. Sorry for the problems... :(

500mb sample ---> 450mb when 7z compressed.

[Razor12911]

Next update will have two new features, database and stream deduplication and here are some benchmarks as to what these features can do.

Input is Need for Speed Hot Pursuit\SEACREST

pzlib v2
Compressed 225 files, 3,253,110,016 => 859,932,872 bytes. Ratio 26.43%
Compression time: cpu 5.73 sec/real 300.50 sec = 2%. Speed 10.83 mB/s

pzlib v3
fails, no idea why :o

xtool2018 (v012)
Compressed 225 files, 3,253,110,016 => 722,889,785 bytes. Ratio 22.22%
Compression time: cpu 5.06 sec/real 188.09 sec = 3%. Speed 17.30 mB/s

xtool2019 (1908_R6)
Compressed 225 files, 3,253,110,016 => 722,934,572 bytes. Ratio 22.22%
Compression time: cpu 4.89 sec/real 189.14 sec = 3%. Speed 17.20 mB/s

xtool2020
Compressed 225 files, 3,253,110,016 => 722,927,890 bytes. Ratio 22.22%
Compression time: cpu 4.86 sec/real 165.26 sec = 3%. Speed 19.68 mB/s

xtool2020 with deduplication
Compressed 225 files, 3,253,110,016 => 719,194,176 bytes. Ratio 22.11%
Compression time: cpu 4.08 sec/real 154.45 sec = 3%. Speed 21.06 mB/s

xtool2020 with deduplication + memory database
Compressed 225 files, 3,253,110,016 => 719,195,766 bytes. Ratio 22.11%
Compression time: cpu 4.20 sec/real 75.00 sec = 6%. Speed 43.38 mB/s

xtool2020 with deduplication + imported database
Compressed 225 files, 3,253,110,016 => 719,194,905 bytes. Ratio 22.11%
Compression time: cpu 4.50 sec/real 61.77 sec = 7%. Speed 52.66 mB/s

precomp :(
Compressed 225 files, 3,253,110,016 => 754,538,183 bytes. Ratio 23.19%
Compression time: cpu 6.69 sec/real 547.05 sec = 1%. Speed 5.95 mB/s

In precomp's defence, it uses disk instead of memory to work and stream type was never narrowed to only focus on deflate streams.

All tests ran using 4 threads with 32mb chunk size

NOTE: This input has repeated streams, some games have similar characteristics therefore xtool stores a database in memory to remember the configuration and then removes these repeated streams to give more speed.
It's free speed, there is no penalty in performance if you use the new options at all times however, some tests need to be run due to the possibility of collisions in the dictionary used...

This project has come a long way :)

[FitGirl]

Cool update, waiting impatiently :)
Any predictions on RAM requirements for both imported/memory DBs? I suspect it depends on number of duplicated streams, but it would be good beforehand to know what additional RAM will be required upon recompression. Maybe it's possible for xtool to report it after initial decompression is done?

[Razor12911]

I suspect it depends on number of duplicated streams
you answered yourself :)

but as I said, I have a few ideas to reduce the memory requirements like getting rid of streams from memory that have restored the duplicated streams. I am not sure if srep does this already but if it doesn't then it should be a bonus. GPU memory usage is another thing that will be added but in future. I need start adding support for other codecs, especially the external ones. What I plan is for people to write their own plugins regardless whether they can write code or not. If you can figure out a header structure of a game, just write and ini file and give it to xtool and it will do the rest. This is mostly for lz4 games, encrypted ones or ones that require something special. I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

Here's an example of Saints Row IV Remastered, instead of making another side project, you just write this and give it to xtool.

[stream1]
Codec=lz4hc:l10 // lz4hc compression used, level 10
Header=00BADBEE0FEEDBEE // this is the magic bytes
Structure={Header(8)} {CSize(4)} {DSize(4)} {Stream} // this is the header structure
Condition1=DSize > CSize // the conditions for xtool to accept the stream
Condition2=CSize > 64
Condition3=CSize < 16 x 1024 x 1024

Maybe it's possible for xtool to report it after initial decompression is done?
this I can do


@everyone

here are some benchmarks, I finally written the decompression code, though it needs some tweaking before I can post update

normal xtool
Compressed 1 file, 320,782,304 => 108,018,914 bytes. Ratio 33.67%
Compression time: cpu 0.47 sec/real 22.64 sec = 2%. Speed 14.17 mB/s

Tested 1 file, 108,018,914 => 320,782,304 bytes. Ratio 33.67%
Testing time: cpu 0.38 sec/real 14.91 sec = 3%. Speed 21.51 mB/s

+ memory database
+ deduplication
Compressed 1 file, 320,782,304 => 107,858,828 bytes. Ratio 33.62%
Compression time: cpu 0.44 sec/real 15.98 sec = 3%. Speed 20.08 mB/s

Tested 1 file, 107,858,828 => 320,782,304 bytes. Ratio 33.62%
Testing time: cpu 0.44 sec/real 9.32 sec = 5%. Speed 34.41 mB/s

thus far, the output is slightly better for more speed :)

[FitGirl]

Configuration feature is great! It would be so much easier to repack some obscure formats.

[dixen]

I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)
For LEGO series games this would be fine)

[FitGirl]

For LEGO series games this would be fine)
It will never work with LEGO series, cause there are no public compressors for algos used in those games, and the decompressors used by Luigi are in binary (ripped from DLL) format, so they can only be extracted, not packed back.

[Razor12911]

Any predictions on RAM requirements for both imported/memory DBs?

back on the SEACREST input from Need For Speed Hot Pursuit

I ran a memory usage test as you asked and these were the results.

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

Decompression memory is 1485 mb. 147,537 matches = 2,360,592 bytes = 0.18% of file

= total 1485 MB (without deduplication)

NOTE: Xtool uses memory of its own and this was not factored, about 190 MB
so if you do the math, that's 350+190 = about 540 MB total memory while if xtool doesn't remove deduplicates and lets srep all the work that's 1485+190 = 1675 MB

540 MB vs 1675 MB memory usage, which is better?

when 540 MB ram was used, xtool was decoding at 88.91 MB/s
and when 1675 MB ram was used, xtool decoded at 27.45 MB/s

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine :o)

[Masquerade]

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)
I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?

[FitGirl]

I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?
The idea of new xtool is not replace srep. It's purpose is not to recompress duplicate chunks, but just copy them, which makes less CPU load and faster overall decompression. You still need to run srep after xtool, cause it will deal with unpacked duplicated chunks.

[Masquerade]

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB

https://i.imgur.com/jKD7fvx.png

So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

[FitGirl]

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB

https://i.imgur.com/jKD7fvx.png

So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.
Afair Scribblenauts is encrypted. You are lucky to always choose encrypted games for your tests ;)

[Harsh ojha]

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine :o)

Same here potato pc 😅

FitGirl Repack take more HHD space for installing game ex. Far Cry 3 duology
Need 17 GB HHD space free
But the game Size was 10 GB ( language english.
Without FC3 mapEditor )

*Sorry for my bad English as English is not my mother tongue 😅*

[Masquerade]

You are lucky to always choose encrypted games for your tests ;)
Sucks to be me I guess. I don't knowingly do this. :o

[Razor12911]

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB

https://i.imgur.com/jKD7fvx.png

So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

hmm, final size after compression?

[Masquerade]

hmm, final size after compression?
With xtool r12: 1.38GB

I feel slightly ashamed for not thinking the data could be encrypted.

Didn't test to make a full archive with 2020, I will do this if you wish.

Every time I put xtool 2020 up against xtool r12, r12 always produces a larger size. Both run at 128m chunk size.

[Razor12911]

and the final size of 2020?

[shazzla]

@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !

[humorlesslumber]

Good but can we somehow compress modded android games like dragon city (https://dragoncityzone.com)?

[Razor12911]

@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !

will be waiting for the sample :)

[shazzla]

On it.. maybe 10 hrs from now... maybe.

[FitGirl]

will be waiting for the sample :)
Check Discord for Halo sample ;)

[shazzla]

will be waiting for the sample :)

As i promised :

https://mega.nz/file/11RUTKIb#SCuzYAS627H9Dx9Jyg75lsAtIJ2JSLWlLeP5mgiu6 EM

[Razor12911]

@FitGirl
using

-mzlib+reflate
Tested 1 file, 92,646,762 => 48,529,408 bytes. Ratio 190.91%
Testing time: cpu 0.06 sec/real 2.62 sec = 2%. Speed 18.56 mB/s
-mzlib+preflate
Tested 1 file, 90,385,191 => 48,529,408 bytes. Ratio 186.25%
Testing time: cpu 0.08 sec/real 3.60 sec = 2%. Speed 13.46 mB/s

@shazzla

using

-mzlib+reflate
"localizations.dat". File is broken.
Tested 1 file, 67,329,029 => 9,663,657 bytes. Ratio 696.72%
Testing time: cpu 0.00 sec/real 1.10 sec = 0%. Speed 8.75 mB/s
-mzlib+preflate
Tested 1 file, 9,663,731 => 9,663,657 bytes. Ratio 100.00%
Testing time: cpu 0.02 sec/real 0.59 sec = 3%. Speed 16.34 mB/s

I remember mentioning somewhere that you sometimes need to include reflate/preflate along with zlib because this is no longer automated like in previous iterations.

in the sample FitGirl provided reflate gave better output, not sure if it's because of bigger hifs but this is why preflate doesn't replace reflate and why both are used, you just need to pick one to work with if the default zlib method doesn't process streams by itself.

that said, reflate no longer gets updates so at times, it breaks crc of files like in thr sample provided by shazzla so this means you need to use preflate.

I hope this clears up a few things about the program's usage.

Tests were performed using xtool_2008_R1 (the current version),

arc.ini
[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+preflate -c32mb -t100p-1 -d0 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

Edit:

the next version of xtool will have reflate automatically verify the streams it processes to avoid checksum errors and if it fails but preflate was enabled, it will jump to preflate. may take longer to process but I guess there is no other alternative at this point.

[Razor12911]

Update available

Changes

- fixed command line parser
- updated deflate scanner
- added stream deduplication
- added stream database
- added decompression memory limiter
- added grittibanzli (also handles deflate stream but slow af)

Notes

stream deduplication doesn't allow you to set max decompression memory so it uses VM file, this will be in future versions.
stream deduplication also produces a file that is required for decompression, this file must always exist else xtool will always fail restoring the data

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+preflate -c32mb -t100p-1 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 --dedup=xtool.bin - - <stdin> <stdout>

xtool.bin must be put along with the installation (setup.exe) and extracted to temp like you would with xtool.exe and other decompression exe/dll/cls files

if you have multiple archives, the same xtool.bin file can be used. meaning if you used xtool in data1.bin and data2.bin, xtool.bin can be used for both, the program itself will decide which section of the xtool.bin belongs to which archive. This is better than having xtool1.bin and xtool2.bin for each archive ;)

[nicholas2006]

Thanks for the awesome tool.

It works wonderfully in 99% cases but I recently came across with a game called UnderMine and xtool seems to not do anything..
GFS shows zlib 135 MB > 226 MB and I'm using the following method: -mzlib+preflate -c128mb -t2

Any guidance would be greatly appreciated.

*edit: I used different keywords to search and found my answer in a different thread (issue being it was dynamic stream). But any other input may be helpful!

[Snapppr6]

hi razor12911
in any data type we can use xtool 2008 and 1908

[Masquerade]

Thank you for the dedup mode, it is working well.

Mad Max = 43kb deduplication file made

I expect that with more duplicated streams, the file size of course will increase - is there a limit on how large the file could reach? or could it reach astronomical sizes (20+mb), in which case, packaging it inside a setup would appear irregular.

[Razor12911]

Thanks for the awesome tool.

It works wonderfully in 99% cases but I recently came across with a game called UnderMine and xtool seems to not do anything..
GFS shows zlib 135 MB > 226 MB and I'm using the following method: -mzlib+preflate -c128mb -t2

Any guidance would be greatly appreciated.

*edit: I used different keywords to search and found my answer in a different thread (issue being it was dynamic stream). But any other input may be helpful!

these were probably false positives but you can try using -mzlib+reflate to see what happens.

hi razor12911
in any data type we can use xtool 2008 and 1908

2008 currently only supports deflate compressed data.

Thank you for the dedup mode, it is working well.

Mad Max = 43kb deduplication file made

I expect that with more duplicated streams, the file size of course will increase - is there a limit on how large the file could reach? or could it reach astronomical sizes (20+mb), in which case, packaging it inside a setup would appear irregular.

these deduplication files are actually compressible, try compressing any of the ones you have generated so far and see what happens. Inno Setup compresses resources it uses so... :rolleyes:

[Razor12911]

Update available

Changes

- fixed deduplication memory calculation error
- added virtual memory support for deduplication
- added --mem=# parameter to control deduplication memory usage

the virtual memory files xtool creates look like this and they are placed in same temp directory freearc creates, this way you can change where they are supposed to be created by setting temp/work directory

[Gupta]

good old GTA 5

compressing x64a.rpf...x64k.rpf with xtool+srep+lolz, crc error at x64d.rpf

compress x64a.rpf...x64d.rpf with xtool decompression never finishes

xtool precomp -mzlib+reflate -c64mb -t100p-1 --dbase=gtav.xdb --dedup=xtool.bin

[Razor12911]

try without reflate

[Gupta]

here is offended part of file https://drive.google.com/file/d/1HA-roYeIWyFKHofteoiMm5lSdSKqKUCO/view?usp=sharing

I didn't test much but other method seems to be working fine on this file atleast

[panker1992]

a brief Test was done :D


the files tested are Resident Evil 0 HD Remake PS3 all ARC files in ARC folder

tests was done with dedup activated.

[Razor12911]

Update available

Changes

- added reflate forced verification
- updated deflate scanner
- fixed depthing issues
- fixed low memory mode issues
- fixed hanging issues when encoding

[Razor12911]

some benchmarks of the next version with oodle support

Oodle precompressor (Side project)
Compressed 1 file, 105,360,271 => 351,678,627 bytes. Ratio 333.79%
Compression time: cpu 0.13 sec/real 41.13 sec = 0%. Speed 2.56 mB/s

Xtool 2020
Compressed 1 file, 105,360,271 => 351,298,255 bytes. Ratio 333.43%
Compression time: cpu 0.08 sec/real 7.42 sec = 1%. Speed 14.20 mB/s

If you had problems with the oo2rec series or want features to be added, let me know.

[KaktoR]

Can't wait to test and compare :D

[Razor12911]

Update available

Changes

- added kraken codec
- fixed depthing issues

Notes

This doesn't detect kraken streams as the side project because I opted for speed but this will change in future once I figure out how to properly deal with the codec

Kraken comes with a level option which is used like this "-mkraken:l4" this is only if you know what level was used to speed up the process but if you don't know what level, you can just use kraken plainly like this "-mkraken" and xtool will try all possible levels until it gets the right one. If two levels were used then the method should look like this "-mkraken:l4,l5"

the deduplication applies to all codecs xtool has so if a game was compressed using kraken and it has many repeated streams, that should give you more speed.

the oodle in xtool doesn't require you to rename the dll therefore if "oo2core_9_win64.dll" ever gets released, xtool should be able to support it.

[panker1992]

Well :D name a game that you need to be tested upon.

I think i will try sekiro first wish me luck :D

[dixen]

DOOM Eternal

oo2reck

pack

Compressed 2 files, 180,206,653 => 249,645,165 bytes. Ratio 138.53%
Compression time: cpu 0.28 sec/real 293.17 sec = 0%. Speed 0.61 mB/s
All OK

unpack

Extracted 2 files, 249,645,165 => 180,206,653 bytes. Ratio 138.53%
Extraction time: cpu 0.27 sec/real 4.86 sec = 5%. Speed 37.10 mB/s
All OK

XTOOL 2009 r2

pack

Compressed 2 files, 180,206,653 => 224,968,010 bytes. Ratio 124.84%
Compression time: cpu 0.38 sec/real 22.60 sec = 2%. Speed 9.07 mB/s
All OK

unpack

Extracted 2 files, 224,968,010 => 180,206,653 bytes. Ratio 124.84%
Extraction time: cpu 0.47 sec/real 4.21 sec = 11%. Speed 49.24 mB/s
All OK

Compress with 7z archivator

oo2reck = 99.5 mb
xtool r2 = 99.8 mb

Compress with srep+lolz

oo2reck = 91.1 mb
xtool r2 = 93.9 mb


PS. Tested on my 2nd weak PC with FX-4100:D

[Razor12911]

Update available

Changes

- documentation added

Notes

This release is the same as the one before, I just added a documentation so you know how to properly use this tool.

[shazzla]

@Razor12911 :
Very good job !

Just an idea : It would be nice to see how much data was "saved" during deduplication. :)
Maybe you add this feature later...
Thank you very much !

[dixen]

A little info

I try decompress all *.bff files on Project CARS 2 with XTOOL & oo2reck (both - with BDT)...and..

XTOOL R2 - 45 gb > 67 gb (for 30 min, but with many CRC errors on unpack)
oo2reck - 45 gb > 95 gb (for 3 hour. No error)

[elit]

Greetings Razor, I got few questions if you don't mind.

Regarding stream database, which hash type do you use? I am guessing that size of hash in bits will dictate speed benefit vs collisions # to data size. I think =>256bit should be good enough including for very big data(100+gb) or a lot of streams, regardless of occasional collisions and therefore should be enabled for speed benefits(if that's what you use)?

Thank you for documentation, this was needed. Please consider adding one to command line though. Help file works but gives script errors every time I change page(trying to access internet). I click ignore and it works but its a hassle. Command line help is also more practical, like was in ztool. It should also display version info at very least for user to know what (s)he have(well by (s)he I really mean all the time he + 1x FitGirl :)).

How reliable is latest xtool now compared to ztool? I read some comments in the past where for instance it supposedly inflated only on individual files and not "tarred" ones etc.. Is it at least as reliable as ztool now? PS(I only use zlib/deflate in ztool anyway). I am interested in latest version, I know some stick to older versions of xtool.

I may be wrong but wasn't crilayla in previous xtool versions and is not anymore?

Isn't anything oodle gamepack-format dependent? I recall for API it needed to know both compressed and decompressed size in it's function parameters, meaning it won't be forever compatible if you stop updating the tool(unlike zlib/deflate)(and version differences aside)?

How many bytes are needed to store dedup data in separate file per single stream?

At least for zlib/deflate, is xtool endian neutral? Can it inflate both big and small endian byte streams?


Thank you

[Razor12911]

I have no idea what hash the dictionary uses but likely CRC-32 since it is 32-bit. But I don't think it's that much of an issue, I have yet to find a collision and I have run several tests, once I'm satisfied. I'll make the stream database a default feature.

I have not come across errors myself in the chm help file, I can always compile a pdf help file if there are issues. Perhaps the next release will come in different formats.

The reason the version information is not displayed is because this project is still at alpha stages, as you can see the tests dixen runs, there are still errors but eventually I'll commit to the idea of proper version history along with the program telling users what version it is. This is pretty much why the current version classification is written 2009_R3, where 20 is the year, 09 is the month and R3 means it's the 3rd release on that month, you just need to look at the exe dates to know what version you are using.

Xtool is reliable compared to ztool and should perform better, at least when using the zlib codec.

Crilayla was excluded from xtool because there is no room for slow tools in project plus xtool is now only available in 64-bit and the only crilayla dll around is 32-bit which is incompatible.

oodle is universal like zlib but there are issues with detecting the decompressed size which results in several streams being left behind and longer precompression times. I plan on improving this eventually, I will find a way to fix this.

About 4 bytes per repeated stream to store in dedup data.

endianness isn't something that considered when handling zlib/deflate streams. That's like saying, will a music player be able to play an mp3 player if it encoded using big endian, the standard is the same across all platforms.

[elit]

Thanks. About that help error:
27925

About CRC, you may need to test that on big enough data (300gb+) and containing TONS of very small chunks(64-256k or ~128k) for it to be robust enough for all scenarios. Crc32 may reach iteration limit(collisions start way before that). Good idea is to compare srep m3 vs m5, m3 use VMAC(which is either 64bit or 128bit, dunno which use srep but very likely 128bit) and m5 is re-read bit perfect, so following attachment could help you get some hints regarding collision vs data:
27926

Thanks for that reliability reassuring, from now on I start use it exclusively instead ztool. Crilayla ditching is a disappointment though as this one is very common format in Japanese games and could also help with compressing console roms. You mentioned low speed as a reason but if I recall from past oodle was even slower?

~4 bytes in dedup per stream only?! So then you don't store distance, only single 32bit hash and nothing else me think. You compare with each newly found chunk if there is a crc match right?

About endians, ok but you still do have to search for a byte sequences in order to find something no? For example when I wrote bms script to decompress oodle chunks from xcom2 I searched for a oodle clues, in my case: "\xC1\x83\x2A\x9E" Then it matter if its that or in reverse. I dunno maybe zlib have same header ID order in all endians, but then what about deflate detection? That have no header, just a bytes test..

[FitGirl]

I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.

[elit]

I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.
Very different content and same hash.. that's quite something. This does depend on number of files, their size and especially polynomial of that crc though. Crc32's are multiple versions and polynomial of it is extremely important. I think there were some shitty variants that had low quality and only 2 that were really solid - one use Intel in their CPU's. With those crc32 should be perfectly fine and reliable but still only up to certain counts and data sizes - hence my worry since this tool is used for huge data and games are known to have both size and really big number of files inside their packs = tons of chunks.

As for collisions, he still cannot 100% rely on *any* hash so he have to verify by content before applying dedup regardless, otherwise its too risky. That mean occasional rare collision should not be a big deal to overall size, but also only if chunks are small. If you collide on multiple chunks of 10+mb or even 100+mb then you may get few hundred mb's worse compression.

If minimum chunk size is >= couple of kb's, few extra bytes of hash size should be negligible. I would suggest crc64 or sha128(or even better VMAC that srep use).

[Edison007]

I would suggest crc64 or sha128(or even better VMAC that srep use).
better blake2)

[Masquerade]

Not blake3? ^

[elit]

So I tried xtool on older Telltale game "Back to the future"(gog version) as I "just happen" to be packing it right now. I thought game files should be decrypted first but hey! It inflated them! Specifically I tried on Ep1 file "4_BackToTheFuture101_pc_tx.ttarch" which is 244mb:
27933

^First I tried -mzlib, it fould 10113 streams and inflated to 632mb in 22s global time(I should have marked that one, ignore red underline on wrong time).

27934
^Reflate found same number of streams but inflated them to 638mb, which may or may not be actual data(could be overhead which would make it worse than -mzlib). Time was worse at 35s.

27935
^Preflate processed 3 less streams from all and inflated to 632mb which is same as zlib, but time close to reflate(33s).

27936
^GrittiBanzli. Now this funny name found all the same number of streams as zlib and reflate, but inflated them to 728mb?! WTF? Unfortunately time was horrible at 440sec. Not sure why same number of actually processed(not just found) streams give such a difference in size. I tried more brute options(including depth) on xtool zlib & reflate and even on ztool but they could not get above 638mb no matter what. This thing can inflate about ~20% more. I wonder how would precomp do here...

27937
^Finally, good ol' ztool for reference, better inflation size than xtool and same as reflate, but again it could be tool overhead itself not actual streams - which would mean its worse. Time is on par with xtool -mzlib.


There, don't say I never contributed :).

[Masquerade]

Just another Oodle test:

Death Stranding: Kraken (oo2core_7_win.dll)

Testing On: 968bf82f34e2b499687c901a888e633a.bin

-mdst+oo2reck:

Compressed 1 file, 550,028,020 => 1,216,560,352 bytes. Ratio 221.18%
Compression time: cpu 1.11 sec/real 80.93 sec = 1%. Speed 6.80 mB/s
All OK

-mdst+xtool (no deduplication or database)

Compressed 1 file, 550,028,020 => 1,213,760,805 bytes. Ratio 220.67%
Compression time: cpu 0.98 sec/real 62.03 sec = 2%. Speed 8.87 mB/s
All OK

[Sebazz]

Just another Oodle test:

Death Stranding: Kraken (oo2core_7_win.dll)

Testing On: 968bf82f34e2b499687c901a888e633a.bin

-mdst+oo2reck:

Compressed 1 file, 550,028,020 => 1,216,560,352 bytes. Ratio 221.18%
Compression time: cpu 1.11 sec/real 80.93 sec = 1%. Speed 6.80 mB/s
All OK

-mdst+xtool (no deduplication or database)

Compressed 1 file, 550,028,020 => 1,213,760,805 bytes. Ratio 220.67%
Compression time: cpu 0.98 sec/real 62.03 sec = 2%. Speed 8.87 mB/s
All OK

What is mdst ? What settings you use for xtool oodle (kraken) ?

[dixen]

What is mdst ? What settings you use for xtool oodle (kraken) ?https://fileforums.com/showthread.php?t=103699

[Masquerade]

-m = freearc method parameter

dst = Death Stranding Decrypt Tool

xtool oodle= use -mkraken (-m feature in xtool, can't remember entire packcmd)

[Razor12911]

Update available

Changes

- added zstd codec
- added lz4, lz4hc, lzna, mermaid, selkie, hydra, leviathan codec placeholders
- added configuration support
- added xdelta support to handle crc mismatch streams

Notes

Configuration support is basically telling xtool how to find streams via an ini file. An example is included (-msr3remaster), I'll document its usage in the next release.
The oodle precompressor in xtool is still inferior to the side project, I'll work on it in the next release.

[KaktoR]

zstd does not seem to work ?

[Razor12911]

Fixed

[KaktoR]

Thanks. However, I had to use a different zstd library to make it work actually :)

Compressed 1 file, 50,664,187 => 104,234,410 bytes. Ratio 205.74%
Compression time: cpu 0.08 sec/real 2.19 sec = 4%. Speed 23.10 mB/s
All OK

Extracted 1 file, 104,234,410 => 50,664,187 bytes. Ratio 205.74%
Extraction time: cpu 0.05 sec/real 2.08 sec = 2%. Speed 24.37 mB/s
All OK

PS: I made a little compare with ZSTDRec (Side Project)

Compressed 1 file, 50,664,187 => 105,746,716 bytes. Ratio 208.72%
Compression time: cpu 0.08 sec/real 2.35 sec = 3%. Speed 21.52 mB/s
All OK

Extracted 1 file, 105,746,716 => 50,664,187 bytes. Ratio 208.72%
Extraction time: cpu 0.06 sec/real 1.56 sec = 4%. Speed 32.51 mB/s
All OK

[Sebazz]

Thanks ... This settings are correct ?

[External compressor:xzstd]
header = 0
packcmd = xtool precomp -mzstd -t100p-1 - - <stdin> <stdout>
unpackcmd = xtool decode -t100p-1 - - <stdin> <stdout>

[Razor12911]

yeah

[Sebazz]

i Have rare problem ... using this setting for precomp - - <stdin> <stdout> sometimes works, sometimes dont, and $$arcdatafile$$.tmp $$arcpackedfile$$.tmp works but stuck the compression ... whats wrong ? :(

[Razor12911]

try the same input without using Freearc and see if the error persists and if it does, send over the sample and I'll see what is causing the problem.

[Sebazz]

try the same input without using Freearc and see if the error persists and if it does, send over the sample and I'll see what is causing the problem.

The only setting that works for me. I could solve it :)

[External compressor:xzlib]
header = 0
packcmd = xtool precomp -mzlib -c128mb -t100p-1 $$arcdatafile$$.tmp <stdout>

https://i.imgur.com/tO13PED.jpg

This Setting dont work for me, stopped the conversion.

[External compressor:xzlib]
header = 0
packcmd = xtool precomp -mzlib -c128mb -t100p-1 - - <stdin> <stdout>

https://i.imgur.com/HW5Voxn.jpg

This setting works but freezes the compresion.

[External compressor:xzlib]
header = 0
packcmd = xtool precomp -mzlib -c128mb -t100p-1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

https://i.imgur.com/PTTYyhG.jpg

Mmm rare bug :confused: :D

[Razor12911]

1.3 billion streams? Alrighty then, looks like I have to start working on this bug.

[Sebazz]

[External compressor:xzstd]
header = 0
packcmd = xtool precomp -mzstd -t100p-1 - - <stdin> <stdout>
unpackcmd = xtool decode -t100p-1 - - <stdin> <stdout>

With this setting, the zstd compression is much slower compared with zlib :(.
In my tests > zstd use aprox 150 mb memory / zlib use aprox 450 mb memory
With the new libzstd, what is an optimal setting for use ?

[Razor12911]

naturally zstd compresses slower than zlib especially when xtool tries high levels. optimal use is specifying what level to use furthermore, xdelta steps in for imperfect restoration.
-mzstd:l19 as an example.

[ozolt]

Sorry I must ask...is main project renamed to 2010 so it's now XTool 2010, or is it 2020? ...because if this is not a mistake, this version numbering is very, very wrong. I'm guessing 2010 stands for year 2020 and 10th month but OMG...:rolleyes:

[Razor12911]

It's the same way Microsoft releases versions for its Windows 10. And yes, year 2020 and 10th month. 2010

[Giriplayslegend899]

hey,razor uh i dont know coding and stuff but what does this do ? create repacks or compress game files and how can i use it?

[Giriplayslegend899]

and also what is source code? ( sorry if asking in wrong place) when i see some projects they say an update is out and give some kind of source code. what is that?

[Sebazz]

hey,razor uh i dont know coding and stuff but what does this do ? create repacks or compress game files and how can i use it?

This tools are for compress game files, in the case of XTool has functionally of inflate compatible streams of game engine, then srep compresses it and finally lzma or lolz or lzma2 compresses it even more.

Here are best methods for compress games ... usually i use srep+lzma2 for fast compression, but in some cases i add xtool depend of game.

https://fileforums.com/showthread.php?t=101639

You can get all other tools here.

https://www.fileforums.com/showthread.php?p=478349

[ozolt]

Razor12911, can you please add , (or some other sign) as alternate option to specify combining codecs? I'm actually trying to use arc.exe by specifying codecs from command line rather than modifying arc.ini, so I'm trying to use packcmd in arc.ini like this:
...
[External compressor:xtool]
header = 0
packcmd = xtool precomp -m{option} -c32mb -t100p-1 - - <stdin> <stdout>
..so I can use, for example, arc.exe a -mxtool:zlib+reflate ...
But since you choose to use + sign to combine codecs this will not work....or if somebody have another idea how to specify different codecs that requires + sign from command line and not by editing arc.ini everytime, I would much appreciate advice...

Thanks in advance,
regards

[Giriplayslegend899]

This tools are for compress game files, in the case of XTool has functionally of inflate compatible streams of game engine, then srep compresses it and finally lzma or lolz or lzma2 compresses it even more.

Here are best methods for compress games ... usually i use srep+lzma2 for fast compression, but in some cases i add xtool depend of game.

https://fileforums.com/showthread.php?t=101639

You can get all other tools here.

https://www.fileforums.com/showthread.php?p=478349

A heavy Thanks But i already have razor 's quick archive packer which is okay for me but how can i use custom installer ultimate ( the second link u gave ) . what can i do with it? and also whenever razor or y_thelastknight release an update for a project how can i update it? and what is source code? :confused::confused::confused:

sorry if i am asking too many questions.:D:D:(:(:D

[Masquerade]

Just another oodle test:

Sekiro v1.05 Data4.bdt:

oo2reck (oo2core_4_win64.dll):

Compressed 1 file, 541,402,628 => 1,065,375,629 bytes. Ratio 196.78%
Compression time: cpu 0.63 sec/real 52.87 sec = 1%. Speed 10.24 mB/s
All OK

Xtool Kraken (oo2core_4_win64.dll):

Compressed 1 file, 541,402,628 => 1,045,442,710 bytes. Ratio 193.10%
Compression time: cpu 0.55 sec/real 38.73 sec = 1%. Speed 13.98 mB/s
All OK

[Razor12911]

Razor12911, can you please add , (or some other sign) as alternate option to specify combining codecs? I'm actually trying to use arc.exe by specifying codecs from command line rather than modifying arc.ini, so I'm trying to use packcmd in arc.ini like this:
...
[External compressor:xtool]
header = 0
packcmd = xtool precomp -m{option} -c32mb -t100p-1 - - <stdin> <stdout>
..so I can use, for example, arc.exe a -mxtool:zlib+reflate ...
But since you choose to use + sign to combine codecs this will not work....or if somebody have another idea how to specify different codecs that requires + sign from command line and not by editing arc.ini everytime, I would much appreciate advice...

Thanks in advance,
regards

I'll add "+" as a resource string and if you want to change it as a user, you'd have to do it yourself. in the next update, that is

Update available

Changes

- added database search
- updated zlib scanner
- fixed reflate bug
- fixed 2GB memory limit

Notes
Database search is a feature similar to configuration support, a demonstration on Far Cry 5 and Watch Dogs Legion will be posted to show how it works.

Updated zlib scanner to eliminate some pesky false positives that cause reflate bugs, verification of reflate removed and should work faster now.

2010_R2 update broke reflate, not sure if you noticed but it's now fixed.

Report made by Kaktor on some file that gave outrageous ratios.
https://fileforums.com/showpost.php?p=488366&postcount=559
Turns out the issue was caused by this
https://community.idera.com/developer-tools/programming-languages/f/delphi-language/70406/tmemorystream-2gb-limit

So I created a custom TMemoryStream that can handle more than 2GB data in x64

Compressed 1 file, 26,336,816 => 3,013,659,460 bytes. Ratio 11442.76%
Compression time: cpu 0.19 sec/real 11.60 sec = 2%. Speed 2.27 mB/s

Tested 1 file, 3,013,659,460 => 26,336,816 bytes. Ratio 11442.76%
Testing time: cpu 0.09 sec/real 3.05 sec = 3%. Speed 8.62 mB/s

[Sebazz]

Just another oodle test:

Sekiro v1.05 Data4.bdt:

oo2reck (oo2core_4_win64.dll):

Compressed 1 file, 541,402,628 => 1,065,375,629 bytes. Ratio 196.78%
Compression time: cpu 0.63 sec/real 52.87 sec = 1%. Speed 10.24 mB/s
All OK

Xtool Kraken (oo2core_4_win64.dll):

Compressed 1 file, 541,402,628 => 1,045,442,710 bytes. Ratio 193.10%
Compression time: cpu 0.55 sec/real 38.73 sec = 1%. Speed 13.98 mB/s
All OK

Whats parameters you use in your arc.ini settings ? Thanks

[Masquerade]

Whats parameters you use in your arc.ini settings ? Thanks

[External compressor: xkraken]
header = 0
packcmd = xtool precomp -mkraken -c128mb -t16 - - <stdin> <stdout>

[Gupta]

> Turns out the issue was caused by this
https://community.idera.com/develope...ream-2gb-limit

because of these obscure bugs, I didn't continue with Delphi, especially ide is shit compared with other. You're still holding grounds nice work :)

[ozolt]

I'll add "+" as a resource string and if you want to change it as a user, you'd have to do it yourself. in the next update, that is

Thank you very much:D If I can make a small comment, isn't resource string editing "overkill"? I mean, simple "-mzlib,reflate" syntax is all that is needed...(currently, only possible is, for example, -mzlib+reflate)

Anyway, thanks again,
regards

[Razor12911]

> Turns out the issue was caused by this
https://community.idera.com/develope...ream-2gb-limit

because of these obscure bugs, I didn't continue with Delphi, especially ide is shit compared with other. You're still holding grounds nice work :)

Well I mostly evade the bugs by just sticking to WinAPI functions and a few times by using System functions but then with these I usually have to write custom code that do the same thing just that the one Delphi has is crap. :)

Thank you very much:D If I can make a small comment, isn't resource string editing "overkill"? I mean, simple "-mzlib,reflate" syntax is all that is needed...(currently, only possible is, for example, -mzlib+reflate)

Anyway, thanks again,
regards

the problem is commas are used for something else

example -mzlib:l98,w15+kraken:l9:t128...

if we replace + with , then syntax will look like this -mzlib:l98,w15,kraken:l9,t128..
zlib will start considering everything as one of its parameters. :p

Edit:

This is what you need to change via Resource Hacker in xtool.exe

[Razor12911]

Update available

Changes

- updated search/config support

Notes

You can now use database files (*.xtl) posted in this thread (https://fileforums.com/showthread.php?t=103876) to precompress games which the program does not have a native support for.

An example is Far Cry 5, the game is lz4 compressed and as there is no universal scanner for these streams, you can use a generated database to precompress the game.

Results on farcry5.dat:
Tested 1 file, 16,292,024,225 => 11,422,856,623 bytes. Ratio 142.63%
Testing time: cpu 11.25 sec/real 111.17 sec = 10%. Speed 102.75 mB/s

[ozolt]

if we replace + with , then syntax will look like this -mzlib:l98,w15,kraken:l9,t128..
zlib will start considering everything as one of its parameters. :p


I understand now and I agree comma is not a good choice, but how about & or # sign?...I never asked for , specifically, it does not matter as long as it is not +

Anyway, it is just a suggestion, ResourceHacker editing is not a problem, changed to & and it is working great (it's even more logical to me with & sign) :)...
Thank you very much for your work,regards

[Masquerade]

Greetings Razor

Does the Dunia codec support for Far Cry Primal?

I haven't seen any mention of it here or on the DELZOREC thread on krinkels.org.

I tried using DELZOREC on the HD Texture file, but it certainly didn't work (3GB file somehow unpacked to 1GB).

If anything needs testing, I can upload some small samples. If there's a way you could show me to see what kind of compression is already on the files, that would also be appreciated.

[dixen]

Greetings Razor

Does the Dunia codec support for Far Cry Primal?

I haven't seen any mention of it here or on the DELZOREC thread on krinkels.org.

I tried using DELZOREC on the HD Texture file, but it certainly didn't work (3GB file somehow unpacked to 1GB).

If anything needs testing, I can upload some small samples. If there's a way you could show me to see what kind of compression is already on the files, that would also be appreciated.
Just Srep+LOLZ

[Masquerade]

Just Srep+LOLZ
Yes, but being properly precompressed wil get an even smaller packed version, which is what I'm going for here.

[Razor12911]

Greetings Razor

Does the Dunia codec support for Far Cry Primal?

I haven't seen any mention of it here or on the DELZOREC thread on krinkels.org.

I tried using DELZOREC on the HD Texture file, but it certainly didn't work (3GB file somehow unpacked to 1GB).

If anything needs testing, I can upload some small samples. If there's a way you could show me to see what kind of compression is already on the files, that would also be appreciated.

I had totally forgotten about Primal, I'm only guessing when I say that I think it uses the same stuff that Watch Dogs 2 and Legion uses, so I think after I'm done with these, I'll check it out.

Update available

Changes

- fixed search/config support bug (thanks dixen)

[dixen]

Tested FC5 *.dat with XTool R4 and...It closed on 10-20%
https://i2.imageban.ru/out/2020/11/01/680145010d0686b8df3fe6b05b076a45.jpg

5.5%
ERROR: write error (disk full?) in compression algorithm xt20

[Razor12911]

Well it's not like I didn't say something was wrong and fixed it in the next release... :rolleyes:

[dixen]

But...If tested without SREP or LOLZ - ALL fine unpacked and XT20 process used about 300-400 mb RAM.
UPD. XT20(R5) + SREP = same bug

[Razor12911]

try unpacking with temps (disable cls/stdio), maybe we can see what is causing the problem.

Edit: I may need to make cls of xtool perhaps just to avoid issues.

[dixen]

try unpacking with temps (disable cls/stdio), maybe we can see what is causing the problem.

Edit: I may need to make cls of xtool perhaps just to avoid issues.

20 gb data.arc - same bug.

OFF stdin/stdout - in progress

UPD

[External compressor:xt20]
header = 0
unpackcmd = xtool decode -t100p --mem=75p --dedup=FC.bin $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
With this parameters - same bug - xtool is closed on 2.5-2.7 gb RAM
R4 or R5 - no matter..
Offensively

[Razor12911]

could you try without deduplication, too many points where an error could occur.

[dixen]

could you try without deduplication, too many points where an error could occur.

Of course...waiting
UPD.
Yes without dedup - the test was completed successfully and xt used about 500-600 mb RAM (XT20 R5)
Tested 11 files, 20,205,910,501 => 21,896,534,144 bytes. Ratio 92.28%
Testing time: cpu 68.59 sec/real 365.42 sec = 19%. Speed 59.92 mB/s
All OK

https://i1.imageban.ru/out/2020/11/03/dfd53d5fad23fa166c0da1a3ca83d1e4.jpg

[elit]

You may want to be interested in this.

I was trying to inflate cpk file from Diablo 3 from nintendo switch. Normaly these are crilayla but gfs detected lz. So I tried old ztool, some older xtool(date modified says 26.april 2019) and xtool2009R3.

For reference I tried single file "Act1.cpk" of 814mb size. GFS detected potential of inflation of up to ~1800mb.

Ztool could not inflate past few 10's of mb if any, regardless of setting like m2, m3 etc.

Later xtool2009R3 depended on codec, most were at about 100mb extra only, except reflate which did go to 1.1gb. However final packed size after srep+xlzma was about same which make me think of possible huge overheat?

Finally, that older xtool with :high setting was able to inflate to.. 1.7gb!!
I tried to compress that and got about ~40mb better compression.

General final compression after codecs was around 740mb+, this one went to ~704mb. I wonder if this was real inflated data to 1.7gb or too much overheat. I must also say that older xtool was able to do it only with :high option which unlike ztool is not even know or documented and probably unofficial. Also it took very long time compare to say ztool:high, it seems to work differently.

I have uploaded the file act1.cpk for further research if you are interested:
https://megaup.net/1zEg1/Act1.cpk

[amin fear]

Is there any incompatibility between "CIU 3.x+ZTOOL+XTOOL" extraction process & new AMD CPU series ?

Anybody have tested ZTOOL with below CPU ?

AMD Ryzen™ 9 3900X "12-Core 3.8 GHz" (4.6 GHz Max Boost) Socket AM4 105W

My installer created with "CIU 3.x+SrepInsidex64+ZToolx64+LOLZx64" works pretty well on ALL of mid range CPUs(Corei3+Corei5+Corei7) but when extracting on NEW AMD Ryzen™ 9 3900X, installer stuck on "0.0" percent progress bar & "ZTool.exe" won't launch on task manager!!!

Is "ZTool.exe" incompatible with new 12-Core CPUs ?

Thanks @Razor12911

[KaktoR]

Did you try to unpack with bat file`? If this works then the problem could be inside CIU code somewhere.

[Razor12911]

my tools can handle up to 256 threads, if you have problems it's mainly because you're using ztool instead of the newer xtool which had MT bugs fixed.

[amin fear]

Did you try to unpack with bat file`? If this works then the problem could be inside CIU code somewhere.

Thanks @KaktoR for batch script! i tried that unpack script but it gives me some errors ! can you please take a look at them :

I have a lot of HDD space [200GB] ,but still gives me errors !

https://imgur.com/a/v2DOBcn

I have compressed my files in below names & put them in "OUTPUT" folder of DiskSpan_UNPACK script:

Data1.Bin.001
Data1.Bin.002
Data1.Bin.003

using below compression method :

pzlib:m2+srep:m3f+lolz:dtb1:mtt1:mt4:d128m:mc1023

I only modified archive name in "_DiskSpan_UNPACK (Test Only).bat" , should i change other parameters ? for example compression switch !

[KaktoR]

Try this

Regedit
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Policies\System
EnableLUA > 0
ConsentPromptBehaviorAdmin > 0
FilterAdministratorToken > 0

Restart Computer

[Razor12911]

Update available

Changes

- added temporary libdunia codec (thanks ProFrager and FitGirl)

[Masquerade]

Thanks Razor / ProFrager / FitGirl for the new dunia codec!

Here's a test on primal_main.dat (6.32GB)

Compressed 1 file, 6,796,379,826 => 8,579,219,400 bytes. Ratio 126.23%
Compression time: cpu 6.58 sec/real 299.11 sec = 2%. Speed 22.72 mB/s
All OK

Working fast and good.

[dixen]

WD Legion

london.dat
shadersobj.dat
common.dat
patch.dat
london_preload.dat
london_cache.dat

13 gb > 17.6 gb

Log not saved)) Sorry

[Masquerade]

Having a few problems decompressing with the fcp option, I have tried a few different unpackcmds:

[External compressor: fcp]
header = 0
unpackcmd = xtool decode --dbase=fcp.xtl - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode -mfcp - - <stdin> <stdout>

Yes, fcp.xtl and liblz4.dll are with xtool.exe

Errors like:

G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed 0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed 0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed 0%
ERROR: write error (disk full?) in compression algorithm fcp

This is just on 1 file with no extra compressors.

Is there something wrong with my unpackcmd?

[Masquerade]

Update: srep+lolz is smaller than fcp+srep+lolz

[shazzla]

Guys ,how can i set the

--dedup=FC.bin

parameter in -m chain ?
is it possible ?

[elit]

So I was dealing with crilayla in Persona 5 cpk's. Normally for these I just use bms but for the first time I got errors in some archives. So i tried your older xtool that still had it(in fact that's why I kept it). And man.. this thing is fantastic! I bring it because you said you dropped it because of speed. Well I don't know if I can change your mind but let me try.

I tested 4.4gb cpk with :t4 on my 4960k 4.2ghz, which is pretty normal today. Inflating took about 16min and re-encoding about 14min. This gives the speed of around 4.5-5.5mb\s. Look, this is not bad at all! There are lot of crazy people out there abusing lzma/lolz with outright stupid settings like mc1000+ that can slow you to crawl with no benefits, yet unlike those this tool is extremely beneficial.

Maybe you meant that for decompression its too slow. Well, repacking cpk back with single threaded bms is just as slow if not more + all the work around it, I did it this way till now. I even had to use xdelta because bms compressor write something into header that I have not seen in single game yet(other than zeroes, I think it was offset 16 and ~6 bytes long string).

So your cri xtool is excellent, work well and have reasonable speed including de/compression - considering alternatives. Also future CPU cores are only going to be more, not less.

This is my 2 cents, of course you do what you feel like. But there is no other cri tool like this and compression is probably used second most often after deflate. There already are plenty of deflate tools and libraries, even original ztool is just perfectly fine for this. Its those other codecs that would be beneficial. Crilayla, maybe also lzss and certainly yaz0 to name few.

In fact you already completed the work, why abandon it? So for the love of god I beg you, put it back in latest version!

PS(Zstd and lz4 are temporary and will stop working after time because of their retarded design. Not sure about lzo(that is actually a question, why is standard lzo not working in dunia and unreal engines and need special treatment? I thought its a stable design like zlib?). But crilayla, lzss, yaz0 and such should remain compatible.

[elit]

-grittibanzli inflate completely randomly. Sometimes tested 20mb inflated zip file is 34mb, other times 31.6mb, another time 33mb atd. exactly same setting and just repeating same command on console.

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)

-preflate and reflate both did to same ~27mb, but preflate took ~6sec while reflate did it in ~1sec.

Looks like good old reflate is still the best zlib codec to stick.


EDIT(and ztool is also able to inflate to 27mb without :high or :m2 !) <<UPDATE(BS, scrap that claim, need m2 or high indeed)
EDIT2(ok I checked manual again and learned about zlib+reflate combination benefits. Maybe this is best way. It took ~3sec compared to ~1sec on deflate alone though. Not sure how big those benefits are in real application yet.)
EDIT3(have to explicitly state 'mb' in -c option instead of just 'm', e.g. -c128m vs -c128mb. No big deal but used to ztool, was more flexible. PS: I am really liking the idea of zlib+reflate in one go, before I used to pass twice in test to find out which one to use.)

[FitGirl]

Regarding the crilayla algo - it's speed heavily depends on data and data size. Big files with RAW-like data compress insanely slowly, even with original algo (not xtool). So when xtool+cri is slow - it's not because of xtool, but because of crilayla. And with big files/chunks sometimes only one thread works in xtool at any given second and doesn't start next chunks until the big one is finished.

[elit]

^Yes that is to be expected as threads have to wait for each other at some point to advance It should still be faster than one by one step of bms tool, although using your style where you dump raw files and use ppx to parallel them is of course even better.
But I think in those Persona 5 packs some files are not compressed(in the same file). They(some) don't have "CRILAYLA" header and dealing with all that really add to work. I could just dump ones with header of course as those without are likely uncompresable like audio and videos anyway.
But with xtool this saves so much hassle for just little extra time.


EDIT:I tried another cpk file with FA GUI and c32mb, I am getting about ~9mb/s inflating and ~10mb/s recoding it. ~10mb/s is IMO more than ok and I saw someone's oodle giving only slightly higher speed. Deflate gives me about 20mb/s(decompression) so 10mb/s for crilayla is great.

[elit]

Regarding '+' and ',' characters issues, why not just use ':' for everything and dissect the parameters in the code instead? I mean, for example:
xtool -mzlib:c32mb:reflate:l9^This is perfectly easy to scan because names like zlib, kraken, reflate etc are unique and cannot appear as codec flags. So in your code you would just look for a word that represent a codec in the list and then all other params would be flags until next codec name is detected. So for example:
-mzlib:kraken:{param} program would still know that zlib is with default parameters and kraken with whatever you put behind it simply because it knows about reserved words.

Also regarding versioning, whats wrong with simple v1.032, v1.033, v1.034 or anything like that?

Just a suggestions of course, I don't want to feel ungrateful or anything.

[elit]

Sorry for spamming but another zip file test:
orig zip size: 31.4mb

zlib: 31.4mb
reflate: 85.9mb
preflate: 49.5mb
grittibanzli: 101mb

then srep+xlzma:

reflate: 26.3mb
preflate: 29.8mb
grittibanzli: 25.4mb3.4% better final compression of grittibanzli vs reflate. Preflate look weak so far in every test for me. Looks like zlib+reflate and/or grittibanzli are 2 best general options.

EDIT:tried on another 2g file, grittibanzli took a freaking 20(!) minutes and loaded all cpu cores to max nonstop, pzlib+deflate took ~2min with much smaller cpu activity. Final compression was around 860mb vs 844mb only. That leave pzlib+deflate the only and best viable method for me.

[78372]

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)



Yes, zlib is supposed to handle deflate but here's the catch, most of the standard zip files does not use regular deflate levels. In order to precompress, the zlib codec tries standard deflate compression levels (9levels * 9 memlevels = 81 tries) and not the tuned deflate options. On the other hand, preflate and reflate does not go with the hit&trial method, it has it's own algo for processing deflate streams. Games in general does not use deflate tuning which is why the zlib codec works good for games, but in other cases like zip, preflate/reflate works better.

Using zlib on where it does not need preflate/reflate is better though, as zlib codec does not produce additional hif files which makes compression a bit worse. When you are not sure, zlib+reflate is good to go as zlib will precompress what it can and pass whatever it can't to reflate which will maximize precompression. Except a few complex deflate streams, zlib+preflate is also a good option as hif files produced by preflate are significantly smaller than the files produced by reflate.

[dixen]

Cyberpunk 2077 (Preload)

Test on basegame_1_engine.archive

xtool2020 (mkraken)

pack

Compressed 1 file, 764,993,536 => 1,949,886,395 bytes. Ratio 254.89%
Compression time: cpu 0.83 sec/real 575.53 sec = 0%. Speed 1.33 mB/s
All OK

unpack

Tested 1 file, 1,949,886,395 => 764,993,536 bytes. Ratio 254.89%
Testing time: cpu 0.53 sec/real 25.08 sec = 2%. Speed 30.50 mB/s
All OK

[Masquerade]

@dixen
Which oodle core used? One from game file?

[dixen]

@dixen
which oodle core used? One from game file?

28493

[Razor12911]

Update available

Changes

- added oo2ext* dll support
- updated search support

To everyone

If you left a post and I didn't reply to or acknowledge, I'm a bit busy and I spend way less time on forum.

Regarding the project. I don't plan on making a 2021 release of xtool, the 2020 is more of a final release and it will only receive minor updates and bug fixes with no new features. If there is a game that requires support, you need to check the plugins that are designed for xtool. I'll be focusing more on this. For the next few days I shall be giving out updates until Christmas so if there is a major feature that you need to be added in xtool leave a post and I'll probably add it.

[Masquerade]

I do recall having a few issues with new XTool whereit would stop during decompression of ue4 games.

I will do a test and see if that's still an issue.