Xtool:crilayla 4Gb patched

[elit]

So I had an interesting experience with repacking Valkyria Chronicles 4 yesterday. There is a base.cpk file of 22gb size. I still prefer to use older xtool with crilayla support as newer methods produce temporary files and spawn processes. By default I have it set to c32mb chunk size. File was inflated to 32.8gb, yet after compression I only achieved around 2gb better size than fitgirl. However, I recoded videos from 9.9gb to 1.7gb! So knowing I missed something I started experiencing:

t8 + c32mb = 32.8gb
t8 + c64mb = 37.8gb
t8 + c128mb = 39.4gb
t8 + c256mb = ~27gb

I dont remember exactly, but on c256mb t8 inflation simply tanked. Then I tried:

t4 + c256mb = ~32gb

I think size was something like that but I would lie if I claim to know exactly. Point is it certainly was under 40gb. Also, clearly, this game is one of those...

I started testing and watching again for memory usage but did not wait to finish whole file because was late, but at reaching input stream of around 8gb, xtool was inflating to 11gb and stayed just under 2gb usage (~1850mb), remaining there(test was done with lz scan).

32bit exe's by default do not support 4gb memory usage, but only 2gb due to OS constrains, so idea to apply 4gb patch came to my mind. And behold, there was a difference. Already at input of ~4gb, xtool went above 2gb RAM and also inflated more. At input of ~6gb, it was already consuming ~2.5gb memory and inflated to around ~11gb(before it inflated to 11gb when input was already at 8gb). This was with same settings. I did not continue as it was almost morning, but clearly patching exe helped.

Before, there would be no error or anything, tool will silently drop processing chunks and you get smaller size!

With patched exe again:

t4 + c256mb = 42.3gb!

I will probably try experimenting further on this game(like t2 + c512mb), but anyway here is the 4gb patched xtool:crilayla:

xtool_4gb_crilayla.zip

EDIT: Just in case although it should be obvious, all credit goes to Razor for making the tool in the first place!

[elit]

Here is my recommendation on maximum threads/chunk size with patched xtool:crilayla, based on observation from compressing 22gb cpk:

-use c512mb as a maximum and on single thread(t1)
-2t c256mb
-4t c128mb
-8t c64mb
...and so on.

On 2t c256mb I reached ~3.6gb ram usage, hence above recommendation.

Raising chunk size worked for me, I was able to repack Valkyria Chronicles 4 to 7.5gb, which correspond with recoded videos size. Moral of the story is, just because you were able to inflate data does not mean it inflated them all properly. This is especially valid for crilayla and sometimes zlib/deflate, less so for fixed stream chunk oriented compression in games like lz4, zstd, oodle? and so on.

[kj911]

So, for games that use zlib, can you bring an improvement in compressibility, a larger chunk size, more efficient unpacking of data that can be repackaged more efficiently?

[elit]

@kj911 I only patched older xtool that still supported crilayla natively, because it is 32bit app - to utilize full 4gb memory range. New xtool is 64bit in active development and any query for better efficiency shall be addressed to its original developer.

[Razor12911]

Hmm, does the this plugin have the same problem?

[elit]

Originally Posted by Razor12911

Hmm, does the this plugin have the same problem?

Looks like there is a problem with both quickbms or crilayla.exe plugins. It does not inflate base.cpk from Valkyria Profile 4 at all. Older xtool with native cri support does. Although file is 22gb, here is the first 8mb sample:

sample.7z

Now this is how it inflate with older xtool:

sample.png

With new xtool and either of 2 following settings, it does nothing:

[cpk2]
Encode=crilayla.exe d <library>
Decode=crilayla.exe e <library>

[cpk,snappy,rfpk,slz_03]
Encode=quickbms.exe -s "comtype <codec> ; clog <fileout> 0 <insize> <outsize>" "" <filein>
Decode=quickbms.exe -s "comtype <codec>_COMPRESS ; clog <fileout> 0 <insize> <outsize>" "" <filein>

Second option also have: cpk.zip that I generated from bms script.
All files(quickbms.exe, crilayla.exe, xtool.exe + libs + xtool.ini) are in xtool directory. I swear it was working before, maybe I tested on different cpk file/game?

EDIT: I can now confirm problem being new xtool with either method. It does work on another file from same game(dlc_003_en.cpk) but not on base.cpk(or provided sample). Old xtool(one provided here) does work on both files.