:cool:

What is Precomp ?
 

What is Precomp the Files ?
What is the advantage of it ?
How to use it?
What are the methods use it ?

Finally an update after a year (08/2019 - 08/2020).

Project is restarted and I hope all goes well this time.

Xtool will be getting frequent updates from now onward since the base code is solid and can allow for new features to be added.

If you have been participating in the [Dev]Xtool thread, thanks a lot. I have managed to fix a lot of bugs and came to a conclusion. Use reflate at your own risk, an alternative has been added which is preflate but this cannot process all streams so choose wisely.

Changes from the Dev Xtool

Added stats monitoring (not visible when using xtool with FA via stdio mode)
Added depth setting (finding streams within streams)
Fixed low memory mode
Speed improvements (when dealing with files with a lot of streams)

Plans for next update
Add stream deduplication

Dishonored. Death of the Outsider

Unpack

From My POV,

Precompression is give/take antibalanced situation.

Where i see it if you want a game decompressed 100% go ahead and decompress manually all then pack them,
and waste time and space when unpacking to do the same function reversed.

If we can get 98% os the files inputed decompressed we can go ahead and call it a day :D

2% is withing the margin of error :D

I was supposed to post an update on the project 2 days ago but then I was greeted by this cloudflare error each time I tried to open the site.

Attachment 27668

So I just thought the site was down little did I know that either my ISP or the forum blocked my IP address because I used VPN and the site opened.

Perhaps I spend too much time on the forum :D

@Razor12911

I quite often get that error when updating the Conversion Index, I just figure it takes to long to update so much text each time ... when I do encounter the error I just open the Index in another window and I find it has updated. ;)

Hello, following guidance from the GFS thread, I tried the zlib+preflate method for the dynamic streams located in Automobilista 2, however the files haven't been precompressed. Here is the result:

This makes me think that this method is not correct for this game.

XTool is doing SOMETHING though, since there is CPU and RAM usage, just as if it was processing normal zlib streams.

Any thoughts? I can upload some of the files if anyone is interested.

I dont have this game,but are you sure its zlib?! (Ok,gfs.....)
Could it be Oodle?! I dont know,yust a question!

There are oodle libs in game folder, but this game is definitely zlib (as shown in GFS, with dynamic stream detection enabled). Oodle scan detects nothing.

Maybe files are crypted?

If there r oodle libs then those streams r oodle streams imho.
Doesnt matter what scanners says its zlib.
Try with different oodle libs.
Again : imho. But maybe im very wrong...

@Kaktor : +1

Oh dear
 

The files are definitely encrypted. Sorry for the problems... :(

500mb sample ---> 450mb when 7z compressed.

Next update will have two new features, database and stream deduplication and here are some benchmarks as to what these features can do.

Input is Need for Speed Hot Pursuit\SEACREST

pzlib v2
Compressed 225 files, 3,253,110,016 => 859,932,872 bytes. Ratio 26.43%
Compression time: cpu 5.73 sec/real 300.50 sec = 2%. Speed 10.83 mB/s

pzlib v3
fails, no idea why :o

xtool2018 (v012)
Compressed 225 files, 3,253,110,016 => 722,889,785 bytes. Ratio 22.22%
Compression time: cpu 5.06 sec/real 188.09 sec = 3%. Speed 17.30 mB/s

xtool2019 (1908_R6)
Compressed 225 files, 3,253,110,016 => 722,934,572 bytes. Ratio 22.22%
Compression time: cpu 4.89 sec/real 189.14 sec = 3%. Speed 17.20 mB/s

xtool2020
Compressed 225 files, 3,253,110,016 => 722,927,890 bytes. Ratio 22.22%
Compression time: cpu 4.86 sec/real 165.26 sec = 3%. Speed 19.68 mB/s

xtool2020 with deduplication
Compressed 225 files, 3,253,110,016 => 719,194,176 bytes. Ratio 22.11%
Compression time: cpu 4.08 sec/real 154.45 sec = 3%. Speed 21.06 mB/s

xtool2020 with deduplication + memory database
Compressed 225 files, 3,253,110,016 => 719,195,766 bytes. Ratio 22.11%
Compression time: cpu 4.20 sec/real 75.00 sec = 6%. Speed 43.38 mB/s

xtool2020 with deduplication + imported database
Compressed 225 files, 3,253,110,016 => 719,194,905 bytes. Ratio 22.11%
Compression time: cpu 4.50 sec/real 61.77 sec = 7%. Speed 52.66 mB/s

precomp :(
Compressed 225 files, 3,253,110,016 => 754,538,183 bytes. Ratio 23.19%
Compression time: cpu 6.69 sec/real 547.05 sec = 1%. Speed 5.95 mB/s

In precomp's defence, it uses disk instead of memory to work and stream type was never narrowed to only focus on deflate streams.

All tests ran using 4 threads with 32mb chunk size

NOTE: This input has repeated streams, some games have similar characteristics therefore xtool stores a database in memory to remember the configuration and then removes these repeated streams to give more speed.
It's free speed, there is no penalty in performance if you use the new options at all times however, some tests need to be run due to the possibility of collisions in the dictionary used...

This project has come a long way :)

Cool update, waiting impatiently :)
Any predictions on RAM requirements for both imported/memory DBs? I suspect it depends on number of duplicated streams, but it would be good beforehand to know what additional RAM will be required upon recompression. Maybe it's possible for xtool to report it after initial decompression is done?

you answered yourself :)

but as I said, I have a few ideas to reduce the memory requirements like getting rid of streams from memory that have restored the duplicated streams. I am not sure if srep does this already but if it doesn't then it should be a bonus. GPU memory usage is another thing that will be added but in future. I need start adding support for other codecs, especially the external ones. What I plan is for people to write their own plugins regardless whether they can write code or not. If you can figure out a header structure of a game, just write and ini file and give it to xtool and it will do the rest. This is mostly for lz4 games, encrypted ones or ones that require something special. I actually wanted xtool to be able to import stuff from quickbms because it already has a huge library of scripts for games. (too much work though)

Here's an example of Saints Row IV Remastered, instead of making another side project, you just write this and give it to xtool.

this I can do


@everyone

here are some benchmarks, I finally written the decompression code, though it needs some tweaking before I can post update

normal xtool
+ memory database
+ deduplication
thus far, the output is slightly better for more speed :)

Configuration feature is great! It would be so much easier to repack some obscure formats.

For LEGO series games this would be fine)

It will never work with LEGO series, cause there are no public compressors for algos used in those games, and the decompressors used by Luigi are in binary (ripped from DLL) format, so they can only be extracted, not packed back.

back on the SEACREST input from Need For Speed Hot Pursuit

I ran a memory usage test as you asked and these were the results.

xtool-virtual-memory.tmp = 328 MB
Decompression memory is 22 mb. 80,927 matches = 1,294,832 bytes = 0.10% of file

= total 350 MB (with deduplication)

Decompression memory is 1485 mb. 147,537 matches = 2,360,592 bytes = 0.18% of file

= total 1485 MB (without deduplication)

NOTE: Xtool uses memory of its own and this was not factored, about 190 MB
so if you do the math, that's 350+190 = about 540 MB total memory while if xtool doesn't remove deduplicates and lets srep all the work that's 1485+190 = 1675 MB

540 MB vs 1675 MB memory usage, which is better?

when 540 MB ram was used, xtool was decoding at 88.91 MB/s
and when 1675 MB ram was used, xtool decoded at 27.45 MB/s

so yes more speed and less memory usage, potato pcs should benefit greatly from this (like mine :o)

I know you don't have much data to go on, but how does xtool+dedup compare against xtool+srep?

The idea of new xtool is not replace srep. It's purpose is not to recompress duplicate chunks, but just copy them, which makes less CPU load and faster overall decompression. You still need to run srep after xtool, cause it will deal with unpacked duplicated chunks.

Hello, here is more data:

Both Scribblenauts games, +comic book (pdf format):

Original size: 1.84GB

https://i.imgur.com/jKD7fvx.png

So, the xtool 2020 is running with zlib+preflate method, maybe I should have disabled it so it's zlib vs zlib.

Afair Scribblenauts is encrypted. You are lucky to always choose encrypted games for your tests ;)

Same here potato pc 😅

FitGirl Repack take more HHD space for installing game ex. Far Cry 3 duology
Need 17 GB HHD space free
But the game Size was 10 GB ( language english.
Without FC3 mapEditor )

*Sorry for my bad English as English is not my mother tongue 😅*

Sucks to be me I guess. I don't knowingly do this. :o

hmm, final size after compression?

With xtool r12: 1.38GB

I feel slightly ashamed for not thinking the data could be encrypted.

Didn't test to make a full archive with 2020, I will do this if you wish.

Every time I put xtool 2020 up against xtool r12, r12 always produces a larger size. Both run at 128m chunk size.

and the final size of 2020?

@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !

Good but can we somehow compress modded android games like dragon city?

will be waiting for the sample :)

On it.. maybe 10 hrs from now... maybe.

Check Discord for Halo sample ;)

As i promised :

https://mega.nz/file/11RUTKIb#SCuzYA...LWlLeP5mgiu6EM

@FitGirl
using

-mzlib+reflate
-mzlib+preflate
@shazzla

using

-mzlib+reflate
-mzlib+preflate
I remember mentioning somewhere that you sometimes need to include reflate/preflate along with zlib because this is no longer automated like in previous iterations.

in the sample FitGirl provided reflate gave better output, not sure if it's because of bigger hifs but this is why preflate doesn't replace reflate and why both are used, you just need to pick one to work with if the default zlib method doesn't process streams by itself.

that said, reflate no longer gets updates so at times, it breaks crc of files like in thr sample provided by shazzla so this means you need to use preflate.

I hope this clears up a few things about the program's usage.

Tests were performed using xtool_2008_R1 (the current version),

arc.ini
Edit:

the next version of xtool will have reflate automatically verify the streams it processes to avoid checksum errors and if it fails but preflate was enabled, it will jump to preflate. may take longer to process but I guess there is no other alternative at this point.

Update available

Changes

- fixed command line parser
- updated deflate scanner
- added stream deduplication
- added stream database
- added decompression memory limiter
- added grittibanzli (also handles deflate stream but slow af)

Notes

stream deduplication doesn't allow you to set max decompression memory so it uses VM file, this will be in future versions.
stream deduplication also produces a file that is required for decompression, this file must always exist else xtool will always fail restoring the data

xtool.bin must be put along with the installation (setup.exe) and extracted to temp like you would with xtool.exe and other decompression exe/dll/cls files

if you have multiple archives, the same xtool.bin file can be used. meaning if you used xtool in data1.bin and data2.bin, xtool.bin can be used for both, the program itself will decide which section of the xtool.bin belongs to which archive. This is better than having xtool1.bin and xtool2.bin for each archive ;)

Thanks for the awesome tool.

It works wonderfully in 99% cases but I recently came across with a game called UnderMine and xtool seems to not do anything..
GFS shows zlib 135 MB > 226 MB and I'm using the following method: -mzlib+preflate -c128mb -t2

Any guidance would be greatly appreciated.

*edit: I used different keywords to search and found my answer in a different thread (issue being it was dynamic stream). But any other input may be helpful!

????
 

hi razor12911
in any data type we can use xtool 2008 and 1908

Thank you for the dedup mode, it is working well.

Mad Max = 43kb deduplication file made

I expect that with more duplicated streams, the file size of course will increase - is there a limit on how large the file could reach? or could it reach astronomical sizes (20+mb), in which case, packaging it inside a setup would appear irregular.