XTool 2020 (Main Project)

[panker1992]

a brief Test was done


the files tested are Resident Evil 0 HD Remake PS3 all ARC files in ARC folder

tests was done with dedup activated.

[Razor12911]

Update available

Changes

- added reflate forced verification
- updated deflate scanner
- fixed depthing issues
- fixed low memory mode issues
- fixed hanging issues when encoding

[Razor12911]

some benchmarks of the next version with oodle support

Oodle precompressor (Side project)

Compressed 1 file, 105,360,271 => 351,678,627 bytes. Ratio 333.79%
Compression time: cpu 0.13 sec/real 41.13 sec = 0%. Speed 2.56 mB/s

Xtool 2020

Compressed 1 file, 105,360,271 => 351,298,255 bytes. Ratio 333.43%
Compression time: cpu 0.08 sec/real 7.42 sec = 1%. Speed 14.20 mB/s

If you had problems with the oo2rec series or want features to be added, let me know.

[KaktoR]

Can't wait to test and compare

[Razor12911]

Update available

Changes

- added kraken codec
- fixed depthing issues

Notes

This doesn't detect kraken streams as the side project because I opted for speed but this will change in future once I figure out how to properly deal with the codec

Kraken comes with a level option which is used like this "-mkraken:l4" this is only if you know what level was used to speed up the process but if you don't know what level, you can just use kraken plainly like this "-mkraken" and xtool will try all possible levels until it gets the right one. If two levels were used then the method should look like this "-mkraken:l4,l5"

the deduplication applies to all codecs xtool has so if a game was compressed using kraken and it has many repeated streams, that should give you more speed.

the oodle in xtool doesn't require you to rename the dll therefore if "oo2core_9_win64.dll" ever gets released, xtool should be able to support it.

[panker1992]

Well name a game that you need to be tested upon.

I think i will try sekiro first wish me luck

[dixen]

DOOM Eternal

oo2reck

pack

Compressed 2 files, 180,206,653 => 249,645,165 bytes. Ratio 138.53%
Compression time: cpu 0.28 sec/real 293.17 sec = 0%. Speed 0.61 mB/s
All OK

unpack

Extracted 2 files, 249,645,165 => 180,206,653 bytes. Ratio 138.53%
Extraction time: cpu 0.27 sec/real 4.86 sec = 5%. Speed 37.10 mB/s
All OK

XTOOL 2009 r2

pack

Compressed 2 files, 180,206,653 => 224,968,010 bytes. Ratio 124.84%
Compression time: cpu 0.38 sec/real 22.60 sec = 2%. Speed 9.07 mB/s
All OK

unpack

Extracted 2 files, 224,968,010 => 180,206,653 bytes. Ratio 124.84%
Extraction time: cpu 0.47 sec/real 4.21 sec = 11%. Speed 49.24 mB/s
All OK

Compress with 7z archivator

oo2reck = 99.5 mb
xtool r2 = 99.8 mb

Compress with srep+lolz

oo2reck = 91.1 mb
xtool r2 = 93.9 mb


PS. Tested on my 2nd weak PC with FX-4100

[Razor12911]

Update available

Changes

- documentation added

Notes

This release is the same as the one before, I just added a documentation so you know how to properly use this tool.

[shazzla]

@Razor12911 :
Very good job !

Just an idea : It would be nice to see how much data was "saved" during deduplication.
Maybe you add this feature later...
Thank you very much !

[dixen]

A little info

I try decompress all *.bff files on Project CARS 2 with XTOOL & oo2reck (both - with BDT)...and..

XTOOL R2 - 45 gb > 67 gb (for 30 min, but with many CRC errors on unpack)
oo2reck - 45 gb > 95 gb (for 3 hour. No error)

[elit]

Greetings Razor, I got few questions if you don't mind.

Regarding stream database, which hash type do you use? I am guessing that size of hash in bits will dictate speed benefit vs collisions # to data size. I think =>256bit should be good enough including for very big data(100+gb) or a lot of streams, regardless of occasional collisions and therefore should be enabled for speed benefits(if that's what you use)?

Thank you for documentation, this was needed. Please consider adding one to command line though. Help file works but gives script errors every time I change page(trying to access internet). I click ignore and it works but its a hassle. Command line help is also more practical, like was in ztool. It should also display version info at very least for user to know what (s)he have(well by (s)he I really mean all the time he + 1x FitGirl ).

How reliable is latest xtool now compared to ztool? I read some comments in the past where for instance it supposedly inflated only on individual files and not "tarred" ones etc.. Is it at least as reliable as ztool now? PS(I only use zlib/deflate in ztool anyway). I am interested in latest version, I know some stick to older versions of xtool.

I may be wrong but wasn't crilayla in previous xtool versions and is not anymore?

Isn't anything oodle gamepack-format dependent? I recall for API it needed to know both compressed and decompressed size in it's function parameters, meaning it won't be forever compatible if you stop updating the tool(unlike zlib/deflate)(and version differences aside)?

How many bytes are needed to store dedup data in separate file per single stream?

At least for zlib/deflate, is xtool endian neutral? Can it inflate both big and small endian byte streams?


Thank you

[Razor12911]

I have no idea what hash the dictionary uses but likely CRC-32 since it is 32-bit. But I don't think it's that much of an issue, I have yet to find a collision and I have run several tests, once I'm satisfied. I'll make the stream database a default feature.

I have not come across errors myself in the chm help file, I can always compile a pdf help file if there are issues. Perhaps the next release will come in different formats.

The reason the version information is not displayed is because this project is still at alpha stages, as you can see the tests dixen runs, there are still errors but eventually I'll commit to the idea of proper version history along with the program telling users what version it is. This is pretty much why the current version classification is written 2009_R3, where 20 is the year, 09 is the month and R3 means it's the 3rd release on that month, you just need to look at the exe dates to know what version you are using.

Xtool is reliable compared to ztool and should perform better, at least when using the zlib codec.

Crilayla was excluded from xtool because there is no room for slow tools in project plus xtool is now only available in 64-bit and the only crilayla dll around is 32-bit which is incompatible.

oodle is universal like zlib but there are issues with detecting the decompressed size which results in several streams being left behind and longer precompression times. I plan on improving this eventually, I will find a way to fix this.

About 4 bytes per repeated stream to store in dedup data.

endianness isn't something that considered when handling zlib/deflate streams. That's like saying, will a music player be able to play an mp3 player if it encoded using big endian, the standard is the same across all platforms.

[elit]

Thanks. About that help error:
errhlp.png

About CRC, you may need to test that on big enough data (300gb+) and containing TONS of very small chunks(64-256k or ~128k) for it to be robust enough for all scenarios. Crc32 may reach iteration limit(collisions start way before that). Good idea is to compare srep m3 vs m5, m3 use VMAC(which is either 64bit or 128bit, dunno which use srep but very likely 128bit) and m5 is re-read bit perfect, so following attachment could help you get some hints regarding collision vs data:
srep-readme.zip

Thanks for that reliability reassuring, from now on I start use it exclusively instead ztool. Crilayla ditching is a disappointment though as this one is very common format in Japanese games and could also help with compressing console roms. You mentioned low speed as a reason but if I recall from past oodle was even slower?

~4 bytes in dedup per stream only?! So then you don't store distance, only single 32bit hash and nothing else me think. You compare with each newly found chunk if there is a crc match right?

About endians, ok but you still do have to search for a byte sequences in order to find something no? For example when I wrote bms script to decompress oodle chunks from xcom2 I searched for a oodle clues, in my case:

"\xC1\x83\x2A\x9E"

Then it matter if its that or in reverse. I dunno maybe zlib have same header ID order in all endians, but then what about deflate detection? That have no header, just a bytes test..

[FitGirl]

I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.

[elit]

Originally Posted by FitGirl

I would never use crc32. In my repacking experience I've met three counts of FULL files crc32 match while having absolutely different content and sizes. Please use better/newer algos, otherwise there will be guaranteed collisions meaning corrupted data.

Very different content and same hash.. that's quite something. This does depend on number of files, their size and especially polynomial of that crc though. Crc32's are multiple versions and polynomial of it is extremely important. I think there were some shitty variants that had low quality and only 2 that were really solid - one use Intel in their CPU's. With those crc32 should be perfectly fine and reliable but still only up to certain counts and data sizes - hence my worry since this tool is used for huge data and games are known to have both size and really big number of files inside their packs = tons of chunks.

As for collisions, he still cannot 100% rely on *any* hash so he have to verify by content before applying dedup regardless, otherwise its too risky. That mean occasional rare collision should not be a big deal to overall size, but also only if chunks are small. If you collide on multiple chunks of 10+mb or even 100+mb then you may get few hundred mb's worse compression.

If minimum chunk size is >= couple of kb's, few extra bytes of hash size should be negligible. I would suggest crc64 or sha128(or even better VMAC that srep use).