XTool 2020 (Main Project)

[78372]

Originally Posted by elit

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)

Yes, zlib is supposed to handle deflate but here's the catch, most of the standard zip files does not use regular deflate levels. In order to precompress, the zlib codec tries standard deflate compression levels (9levels * 9 memlevels = 81 tries) and not the tuned deflate options. On the other hand, preflate and reflate does not go with the hit&trial method, it has it's own algo for processing deflate streams. Games in general does not use deflate tuning which is why the zlib codec works good for games, but in other cases like zip, preflate/reflate works better.

Using zlib on where it does not need preflate/reflate is better though, as zlib codec does not produce additional hif files which makes compression a bit worse. When you are not sure, zlib+reflate is good to go as zlib will precompress what it can and pass whatever it can't to reflate which will maximize precompression. Except a few complex deflate streams, zlib+preflate is also a good option as hif files produced by preflate are significantly smaller than the files produced by reflate.

[FitGirl]

Regarding the crilayla algo - it's speed heavily depends on data and data size. Big files with RAW-like data compress insanely slowly, even with original algo (not xtool). So when xtool+cri is slow - it's not because of xtool, but because of crilayla. And with big files/chunks sometimes only one thread works in xtool at any given second and doesn't start next chunks until the big one is finished.

[elit]

^Yes that is to be expected as threads have to wait for each other at some point to advance It should still be faster than one by one step of bms tool, although using your style where you dump raw files and use ppx to parallel them is of course even better.
But I think in those Persona 5 packs some files are not compressed(in the same file). They(some) don't have "CRILAYLA" header and dealing with all that really add to work. I could just dump ones with header of course as those without are likely uncompresable like audio and videos anyway.
But with xtool this saves so much hassle for just little extra time.


EDIT:I tried another cpk file with FA GUI and c32mb, I am getting about ~9mb/s inflating and ~10mb/s recoding it. ~10mb/s is IMO more than ok and I saw someone's oodle giving only slightly higher speed. Deflate gives me about 20mb/s(decompression) so 10mb/s for crilayla is great.

[elit]

Regarding '+' and ',' characters issues, why not just use ':' for everything and dissect the parameters in the code instead? I mean, for example:

xtool -mzlib:c32mb:reflate:l9

^This is perfectly easy to scan because names like zlib, kraken, reflate etc are unique and cannot appear as codec flags. So in your code you would just look for a word that represent a codec in the list and then all other params would be flags until next codec name is detected. So for example:

-mzlib:kraken:{param}

program would still know that zlib is with default parameters and kraken with whatever you put behind it simply because it knows about reserved words.

Also regarding versioning, whats wrong with simple v1.032, v1.033, v1.034 or anything like that?

Just a suggestions of course, I don't want to feel ungrateful or anything.

[Razor12911]

Originally Posted by elit

Regarding '+' and ',' characters issues, why not just use ':' for everything and dissect the parameters in the code instead? I mean, for example:

xtool -mzlib:c32mb:reflate:l9

^This is perfectly easy to scan because names like zlib, kraken, reflate etc are unique and cannot appear as codec flags. So in your code you would just look for a word that represent a codec in the list and then all other params would be flags until next codec name is detected. So for example:

-mzlib:kraken:{param}

program would still know that zlib is with default parameters and kraken with whatever you put behind it simply because it knows about reserved words.

Also regarding versioning, whats wrong with simple v1.032, v1.033, v1.034 or anything like that?

Just a suggestions of course, I don't want to feel ungrateful or anything.

regarding character parsing, I could change it but the thing is, xtool doesn't really know reserved words because there are none but I can change it if that's what people want.

The reason I went with a versioning like this is because the there are already 3 different xtools so to not confuse people, they should just check the YYMM it was released on as the old xtool, 0.0.19... has special editions from the request of repackers so basically the versioning is so confusing so I then decided I'd only adopt it once I think the project is a guaranteed success. 2019 is an example because what I tried to do fail, so I tried again with 2020 to see if it will work with little hope of success but as soon as it seems like the idea of external plugins is possible and can be done without bugs then versions will be brought back.

[elit]

Sorry for spamming but another zip file test:

orig zip size: 31.4mb

zlib: 31.4mb
reflate: 85.9mb
preflate: 49.5mb
grittibanzli: 101mb

then srep+xlzma:

reflate: 26.3mb
preflate: 29.8mb
  grittibanzli: 25.4mb

3.4% better final compression of grittibanzli vs reflate. Preflate look weak so far in every test for me. Looks like zlib+reflate and/or grittibanzli are 2 best general options.

EDIT:tried on another 2g file, grittibanzli took a freaking 20(!) minutes and loaded all cpu cores to max nonstop, pzlib+deflate took ~2min with much smaller cpu activity. Final compression was around 860mb vs 844mb only. That leave pzlib+deflate the only and best viable method for me.

[dixen]

Cyberpunk 2077 (Preload)

Test on basegame_1_engine.archive

xtool2020 (mkraken)

pack

Compressed 1 file, 764,993,536 => 1,949,886,395 bytes. Ratio 254.89%
Compression time: cpu 0.83 sec/real 575.53 sec = 0%. Speed 1.33 mB/s
All OK

unpack

Tested 1 file, 1,949,886,395 => 764,993,536 bytes. Ratio 254.89%
Testing time: cpu 0.53 sec/real 25.08 sec = 2%. Speed 30.50 mB/s
All OK

[Masquerade]

@dixen
Which oodle core used? One from game file?

[dixen]

Originally Posted by masquerade

@dixen
which oodle core used? One from game file?

oo2core_7_win64.zip

[Razor12911]

Update available

Changes

- added oo2ext* dll support
- updated search support

To everyone

If you left a post and I didn't reply to or acknowledge, I'm a bit busy and I spend way less time on forum.

Regarding the project. I don't plan on making a 2021 release of xtool, the 2020 is more of a final release and it will only receive minor updates and bug fixes with no new features. If there is a game that requires support, you need to check the plugins that are designed for xtool. I'll be focusing more on this. For the next few days I shall be giving out updates until Christmas so if there is a major feature that you need to be added in xtool leave a post and I'll probably add it.

[Masquerade]

I do recall having a few issues with new XTool whereit would stop during decompression of ue4 games.

I will do a test and see if that's still an issue.

[Ele]

Mr. Razor
Is lz4, lzo and zstd still not supported for xtool 2020?

[elit]

Razor, pls check my above post No. 243 and consider that idea for parameters parsing to finalize specification. It would make xtool compatible with FA including GUI.
Finally, please consider including all parameters list with their description within command line help instead of external file.

EDIT: Also, provide full documentation on plugin system reserved keywords and their exact meaning(like if offset jump is meant from current position or beginning, if reading string advance position or not and so on), their full list etc.

Thank you for everything.

[Cesar82]

@Razor12911, is there any change from XTool 2020 to have a 32 bit version too?
Perhaps in the thin version l of the XTool 2020, it would be great.
Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it, having to choose to use the old XTool.
Also if the user has a 32-bit system installed he will not be able to use Xtool, even if it is for a game that will be installed on another system in the future.
Thanks!

[elit]

Originally Posted by Cesar82

Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it

Why? What would happen, according to you? 64/32bit xtool have absolutely no relation to 64/32bit games. Only libraries used in xtool, hence why crilayla was dropped.