XTool 2020 (Main Project)

[shazzla]

@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !

[Razor12911]

Originally Posted by shazzla

@Razor12911 :

Few days ago i started playing with XTool 2008. Great job ! Thanks !
But i found this :

Frostpunk - On the edge (Is its streams ZLIB? It must be IMHO)

It cant precompress those .dat files. (For example ,common.dat and languages.dat were tested).
Common.dat nearly a 600 MB file,languages.dat is a much smaller.
In both case ,the output is the same.
Older generation of XTool (AFAIR V0.9) precompresses commond.dat well to ~1.5 GB, languages.dat ,too.
XTOOLs (older) setting was : 128mb
XTOOLs (2008) setting was : 128mb ,d0 and d9 ,after this 256mb. (doesnt matter)

In my other test i faced with this (dont remember what game,maybe Hellbound):
XTool 2008 inflation was 2 gigs AFAIR ,compressed with LOLZ ,after that compressed the game without XTOOL2008 and the same LOLZ parameters and it was smaller with some MBs ! (maybe a special case ,dont know)

ATM im away from my computer for the next...lot of days ,i cant send example files.

Keep up the good work !

will be waiting for the sample

[FitGirl]

Originally Posted by Razor12911

will be waiting for the sample

Check Discord for Halo sample

[shazzla]

Originally Posted by Razor12911

will be waiting for the sample

As i promised :

https://mega.nz/file/11RUTKIb#SCuzYA...LWlLeP5mgiu6EM

[humorlesslumber]

Good but can we somehow compress modded android games like dragon city?

[Masquerade]

Thank you for the dedup mode, it is working well.

Mad Max = 43kb deduplication file made

I expect that with more duplicated streams, the file size of course will increase - is there a limit on how large the file could reach? or could it reach astronomical sizes (20+mb), in which case, packaging it inside a setup would appear irregular.

[Razor12911]

Originally Posted by nicholas2006

Thanks for the awesome tool.

It works wonderfully in 99% cases but I recently came across with a game called UnderMine and xtool seems to not do anything..
GFS shows zlib 135 MB > 226 MB and I'm using the following method: -mzlib+preflate -c128mb -t2

Any guidance would be greatly appreciated.

*edit: I used different keywords to search and found my answer in a different thread (issue being it was dynamic stream). But any other input may be helpful!

these were probably false positives but you can try using -mzlib+reflate to see what happens.

Originally Posted by Snapppr6

hi razor12911
in any data type we can use xtool 2008 and 1908

2008 currently only supports deflate compressed data.

Originally Posted by Masquerade

Thank you for the dedup mode, it is working well.

Mad Max = 43kb deduplication file made

I expect that with more duplicated streams, the file size of course will increase - is there a limit on how large the file could reach? or could it reach astronomical sizes (20+mb), in which case, packaging it inside a setup would appear irregular.

these deduplication files are actually compressible, try compressing any of the ones you have generated so far and see what happens. Inno Setup compresses resources it uses so...

[shazzla]

On it.. maybe 10 hrs from now... maybe.

[Razor12911]

@FitGirl
using

-mzlib+reflate

Tested 1 file, 92,646,762 => 48,529,408 bytes. Ratio 190.91%
Testing time: cpu 0.06 sec/real 2.62 sec = 2%. Speed 18.56 mB/s

-mzlib+preflate

Tested 1 file, 90,385,191 => 48,529,408 bytes. Ratio 186.25%
Testing time: cpu 0.08 sec/real 3.60 sec = 2%. Speed 13.46 mB/s

@shazzla

using

-mzlib+reflate

"localizations.dat". File is broken.
Tested 1 file, 67,329,029 => 9,663,657 bytes. Ratio 696.72%
Testing time: cpu 0.00 sec/real 1.10 sec = 0%. Speed 8.75 mB/s

-mzlib+preflate

Tested 1 file, 9,663,731 => 9,663,657 bytes. Ratio 100.00%
Testing time: cpu 0.02 sec/real 0.59 sec = 3%. Speed 16.34 mB/s

I remember mentioning somewhere that you sometimes need to include reflate/preflate along with zlib because this is no longer automated like in previous iterations.

in the sample FitGirl provided reflate gave better output, not sure if it's because of bigger hifs but this is why preflate doesn't replace reflate and why both are used, you just need to pick one to work with if the default zlib method doesn't process streams by itself.

that said, reflate no longer gets updates so at times, it breaks crc of files like in thr sample provided by shazzla so this means you need to use preflate.

I hope this clears up a few things about the program's usage.

Tests were performed using xtool_2008_R1 (the current version),

arc.ini

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mzlib+preflate -c32mb -t100p-1 -d0 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 - - <stdin> <stdout>

Edit:

the next version of xtool will have reflate automatically verify the streams it processes to avoid checksum errors and if it fails but preflate was enabled, it will jump to preflate. may take longer to process but I guess there is no other alternative at this point.

[Razor12911]

Update available

Changes

- fixed command line parser
- updated deflate scanner
- added stream deduplication
- added stream database
- added decompression memory limiter
- added grittibanzli (also handles deflate stream but slow af)

Notes

stream deduplication doesn't allow you to set max decompression memory so it uses VM file, this will be in future versions.
stream deduplication also produces a file that is required for decompression, this file must always exist else xtool will always fail restoring the data

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mzlib+preflate -c32mb -t100p-1 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p-1 --dedup=xtool.bin - - <stdin> <stdout>

xtool.bin must be put along with the installation (setup.exe) and extracted to temp like you would with xtool.exe and other decompression exe/dll/cls files

if you have multiple archives, the same xtool.bin file can be used. meaning if you used xtool in data1.bin and data2.bin, xtool.bin can be used for both, the program itself will decide which section of the xtool.bin belongs to which archive. This is better than having xtool1.bin and xtool2.bin for each archive

[nicholas2006]

Thanks for the awesome tool.

It works wonderfully in 99% cases but I recently came across with a game called UnderMine and xtool seems to not do anything..
GFS shows zlib 135 MB > 226 MB and I'm using the following method: -mzlib+preflate -c128mb -t2

Any guidance would be greatly appreciated.

*edit: I used different keywords to search and found my answer in a different thread (issue being it was dynamic stream). But any other input may be helpful!

[Snapppr6]

hi razor12911
in any data type we can use xtool 2008 and 1908

[Razor12911]

Update available

Changes

- fixed deduplication memory calculation error
- added virtual memory support for deduplication
- added --mem=# parameter to control deduplication memory usage

the virtual memory files xtool creates look like this and they are placed in same temp directory freearc creates, this way you can change where they are supposed to be created by setting temp/work directory

[Gupta]

good old GTA 5

compressing x64a.rpf...x64k.rpf with xtool+srep+lolz, crc error at x64d.rpf

compress x64a.rpf...x64d.rpf with xtool decompression never finishes

xtool precomp -mzlib+reflate -c64mb -t100p-1 --dbase=gtav.xdb --dedup=xtool.bin

[Razor12911]

try without reflate