these were probably false positives but you can try using -mzlib+reflate to see what happens.

2008 currently only supports deflate compressed data.

these deduplication files are actually compressible, try compressing any of the ones you have generated so far and see what happens. Inno Setup compresses resources it uses so... :rolleyes:

Update available

Changes

- fixed deduplication memory calculation error
- added virtual memory support for deduplication
- added --mem=# parameter to control deduplication memory usage

the virtual memory files xtool creates look like this and they are placed in same temp directory freearc creates, this way you can change where they are supposed to be created by setting temp/work directory

good old GTA 5

compressing x64a.rpf...x64k.rpf with xtool+srep+lolz, crc error at x64d.rpf

compress x64a.rpf...x64d.rpf with xtool decompression never finishes

xtool precomp -mzlib+reflate -c64mb -t100p-1 --dbase=gtav.xdb --dedup=xtool.bin

try without reflate

here is offended part of file https://drive.google.com/file/d/1HA-...ew?usp=sharing

I didn't test much but other method seems to be working fine on this file atleast

a brief Test was done :D


the files tested are Resident Evil 0 HD Remake PS3 all ARC files in ARC folder

tests was done with dedup activated.

Update available

Changes

- added reflate forced verification
- updated deflate scanner
- fixed depthing issues
- fixed low memory mode issues
- fixed hanging issues when encoding

some benchmarks of the next version with oodle support

Oodle precompressor (Side project)
Xtool 2020
If you had problems with the oo2rec series or want features to be added, let me know.

Can't wait to test and compare :D

Update available

Changes

- added kraken codec
- fixed depthing issues

Notes

This doesn't detect kraken streams as the side project because I opted for speed but this will change in future once I figure out how to properly deal with the codec

Kraken comes with a level option which is used like this "-mkraken:l4" this is only if you know what level was used to speed up the process but if you don't know what level, you can just use kraken plainly like this "-mkraken" and xtool will try all possible levels until it gets the right one. If two levels were used then the method should look like this "-mkraken:l4,l5"

the deduplication applies to all codecs xtool has so if a game was compressed using kraken and it has many repeated streams, that should give you more speed.

the oodle in xtool doesn't require you to rename the dll therefore if "oo2core_9_win64.dll" ever gets released, xtool should be able to support it.

Well :D name a game that you need to be tested upon.

I think i will try sekiro first wish me luck :D

DOOM Eternal

oo2reck

pack

unpack

XTOOL 2009 r2

pack

unpack

Compress with 7z archivator

oo2reck = 99.5 mb
xtool r2 = 99.8 mb

Compress with srep+lolz

oo2reck = 91.1 mb
xtool r2 = 93.9 mb


PS. Tested on my 2nd weak PC with FX-4100:D

Update available

Changes

- documentation added

Notes

This release is the same as the one before, I just added a documentation so you know how to properly use this tool.

@Razor12911 :
Very good job !

Just an idea : It would be nice to see how much data was "saved" during deduplication. :)
Maybe you add this feature later...
Thank you very much !

A little info

I try decompress all *.bff files on Project CARS 2 with XTOOL & oo2reck (both - with BDT)...and..

XTOOL R2 - 45 gb > 67 gb (for 30 min, but with many CRC errors on unpack)
oo2reck - 45 gb > 95 gb (for 3 hour. No error)

Greetings Razor, I got few questions if you don't mind.

Regarding stream database, which hash type do you use? I am guessing that size of hash in bits will dictate speed benefit vs collisions # to data size. I think =>256bit should be good enough including for very big data(100+gb) or a lot of streams, regardless of occasional collisions and therefore should be enabled for speed benefits(if that's what you use)?

Thank you for documentation, this was needed. Please consider adding one to command line though. Help file works but gives script errors every time I change page(trying to access internet). I click ignore and it works but its a hassle. Command line help is also more practical, like was in ztool. It should also display version info at very least for user to know what (s)he have(well by (s)he I really mean all the time he + 1x FitGirl :)).

How reliable is latest xtool now compared to ztool? I read some comments in the past where for instance it supposedly inflated only on individual files and not "tarred" ones etc.. Is it at least as reliable as ztool now? PS(I only use zlib/deflate in ztool anyway). I am interested in latest version, I know some stick to older versions of xtool.

I may be wrong but wasn't crilayla in previous xtool versions and is not anymore?

Isn't anything oodle gamepack-format dependent? I recall for API it needed to know both compressed and decompressed size in it's function parameters, meaning it won't be forever compatible if you stop updating the tool(unlike zlib/deflate)(and version differences aside)?

How many bytes are needed to store dedup data in separate file per single stream?

At least for zlib/deflate, is xtool endian neutral? Can it inflate both big and small endian byte streams?


Thank you

I have no idea what hash the dictionary uses but likely CRC-32 since it is 32-bit. But I don't think it's that much of an issue, I have yet to find a collision and I have run several tests, once I'm satisfied. I'll make the stream database a default feature.

I have not come across errors myself in the chm help file, I can always compile a pdf help file if there are issues. Perhaps the next release will come in different formats.

The reason the version information is not displayed is because this project is still at alpha stages, as you can see the tests dixen runs, there are still errors but eventually I'll commit to the idea of proper version history along with the program telling users what version it is. This is pretty much why the current version classification is written 2009_R3, where 20 is the year, 09 is the month and R3 means it's the 3rd release on that month, you just need to look at the exe dates to know what version you are using.

Xtool is reliable compared to ztool and should perform better, at least when using the zlib codec.

Crilayla was excluded from xtool because there is no room for slow tools in project plus xtool is now only available in 64-bit and the only crilayla dll around is 32-bit which is incompatible.

oodle is universal like zlib but there are issues with detecting the decompressed size which results in several streams being left behind and longer precompression times. I plan on improving this eventually, I will find a way to fix this.

About 4 bytes per repeated stream to store in dedup data.

endianness isn't something that considered when handling zlib/deflate streams. That's like saying, will a music player be able to play an mp3 player if it encoded using big endian, the standard is the same across all platforms.

Thanks. About that help error:
Attachment 27925

About CRC, you may need to test that on big enough data (300gb+) and containing TONS of very small chunks(64-256k or ~128k) for it to be robust enough for all scenarios. Crc32 may reach iteration limit(collisions start way before that). Good idea is to compare srep m3 vs m5, m3 use VMAC(which is either 64bit or 128bit, dunno which use srep but very likely 128bit) and m5 is re-read bit perfect, so following attachment could help you get some hints regarding collision vs data:
Attachment 27926

Thanks for that reliability reassuring, from now on I start use it exclusively instead ztool. Crilayla ditching is a disappointment though as this one is very common format in Japanese games and could also help with compressing console roms. You mentioned low speed as a reason but if I recall from past oodle was even slower?

~4 bytes in dedup per stream only?! So then you don't store distance, only single 32bit hash and nothing else me think. You compare with each newly found chunk if there is a crc match right?

About endians, ok but you still do have to search for a byte sequences in order to find something no? For example when I wrote bms script to decompress oodle chunks from xcom2 I searched for a oodle clues, in my case: Then it matter if its that or in reverse. I dunno maybe zlib have same header ID order in all endians, but then what about deflate detection? That have no header, just a bytes test..

I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.

Very different content and same hash.. that's quite something. This does depend on number of files, their size and especially polynomial of that crc though. Crc32's are multiple versions and polynomial of it is extremely important. I think there were some shitty variants that had low quality and only 2 that were really solid - one use Intel in their CPU's. With those crc32 should be perfectly fine and reliable but still only up to certain counts and data sizes - hence my worry since this tool is used for huge data and games are known to have both size and really big number of files inside their packs = tons of chunks.

As for collisions, he still cannot 100% rely on *any* hash so he have to verify by content before applying dedup regardless, otherwise its too risky. That mean occasional rare collision should not be a big deal to overall size, but also only if chunks are small. If you collide on multiple chunks of 10+mb or even 100+mb then you may get few hundred mb's worse compression.

If minimum chunk size is >= couple of kb's, few extra bytes of hash size should be negligible. I would suggest crc64 or sha128(or even better VMAC that srep use).

better blake2)

Not blake3? ^

So I tried xtool on older Telltale game "Back to the future"(gog version) as I "just happen" to be packing it right now. I thought game files should be decrypted first but hey! It inflated them! Specifically I tried on Ep1 file "4_BackToTheFuture101_pc_tx.ttarch" which is 244mb:
Attachment 27933

^First I tried -mzlib, it fould 10113 streams and inflated to 632mb in 22s global time(I should have marked that one, ignore red underline on wrong time).

Attachment 27934
^Reflate found same number of streams but inflated them to 638mb, which may or may not be actual data(could be overhead which would make it worse than -mzlib). Time was worse at 35s.

Attachment 27935
^Preflate processed 3 less streams from all and inflated to 632mb which is same as zlib, but time close to reflate(33s).

Attachment 27936
^GrittiBanzli. Now this funny name found all the same number of streams as zlib and reflate, but inflated them to 728mb?! WTF? Unfortunately time was horrible at 440sec. Not sure why same number of actually processed(not just found) streams give such a difference in size. I tried more brute options(including depth) on xtool zlib & reflate and even on ztool but they could not get above 638mb no matter what. This thing can inflate about ~20% more. I wonder how would precomp do here...

Attachment 27937
^Finally, good ol' ztool for reference, better inflation size than xtool and same as reflate, but again it could be tool overhead itself not actual streams - which would mean its worse. Time is on par with xtool -mzlib.


There, don't say I never contributed :).

Just another Oodle test:

Death Stranding: Kraken (oo2core_7_win.dll)

Testing On: 968bf82f34e2b499687c901a888e633a.bin

-mdst+oo2reck:

-mdst+xtool (no deduplication or database)

What is mdst ? What settings you use for xtool oodle (kraken) ?

-m = freearc method parameter

dst = Death Stranding Decrypt Tool

xtool oodle= use -mkraken (-m feature in xtool, can't remember entire packcmd)

Update available

Changes

- added zstd codec
- added lz4, lz4hc, lzna, mermaid, selkie, hydra, leviathan codec placeholders
- added configuration support
- added xdelta support to handle crc mismatch streams

Notes

Configuration support is basically telling xtool how to find streams via an ini file. An example is included (-msr3remaster), I'll document its usage in the next release.
The oodle precompressor in xtool is still inferior to the side project, I'll work on it in the next release.

zstd does not seem to work ?

Fixed

Thanks. However, I had to use a different zstd library to make it work actually :)

PS: I made a little compare with ZSTDRec (Side Project)

Thanks ... This settings are correct ?

yeah

i Have rare problem ... using this setting for precomp - - <stdin> <stdout> sometimes works, sometimes dont, and $$arcdatafile$$.tmp $$arcpackedfile$$.tmp works but stuck the compression ... whats wrong ? :(

try the same input without using Freearc and see if the error persists and if it does, send over the sample and I'll see what is causing the problem.

The only setting that works for me. I could solve it :)

https://i.imgur.com/tO13PED.jpg

This Setting dont work for me, stopped the conversion.

https://i.imgur.com/HW5Voxn.jpg

This setting works but freezes the compresion.

https://i.imgur.com/PTTYyhG.jpg

Mmm rare bug :confused: :D

1.3 billion streams? Alrighty then, looks like I have to start working on this bug.

With this setting, the zstd compression is much slower compared with zlib :(.
In my tests > zstd use aprox 150 mb memory / zlib use aprox 450 mb memory
With the new libzstd, what is an optimal setting for use ?

naturally zstd compresses slower than zlib especially when xtool tries high levels. optimal use is specifying what level to use furthermore, xdelta steps in for imperfect restoration.
-mzstd:l19 as an example.

Sorry I must ask...is main project renamed to 2010 so it's now XTool 2010, or is it 2020? ...because if this is not a mistake, this version numbering is very, very wrong. I'm guessing 2010 stands for year 2020 and 10th month but OMG...:rolleyes: