XTool 2020 (Plugins)

[Masquerade]

Orcs Must Die! 3 [LZ4HC]


Extract contents of the attached archive and use -mOMD3.
Requires liblz4.dll, I included one that gives a good ratio.
You can discard OMD3.xtl for decoding, however you will still need the same liblz4.dll you used for encoding.

xtool precomp -mOMD3 -c128mb pakchunk42-WindowsNoEditor.pak pakchunk42-WindowsNoEditor.xt_out

XTool is created by Razor12911

Streams: 1892/1892
Time: 00:00:03 (00:00:07)
Memory: 204 MB (204 MB)

[Razor12911]

Deathloop

Usage
Place deathloop.ini near xtool.exe along with required libraries and use -mdeathloop

Notes

The configuration file can be discarded after encoding.

oo2core_7_win64.dll and oo2core_8_win64.dll are required

I'd like to point out that this game has streams that decompress to 500MB+ each, the largest I've seen is around 870MB which is above the threshold xtool sets to ensure decoding memory doesn't get out of hand and because of this, when xtool decodes it will mostly use 1 thread regardless of the number of threads the user has set which means slow installation times if you planned on repacking this game. It's not a bug, it's how xtool was designed and how the game developers decided to make their game use large streams.

I'm leaving this information here in case people in futruee start complaining about slow installation times for this game when they use xtool. Oodle precompressor side protect does not have this memory threshold so you can use that however be sure to use the libraries attached in deathloop oo2core 7 and 8 are required, oo2core 7 is the one that passes some commands to the original library because not only did they use large streams they used non standard compression settings so good luck :)

Maybe I'm doing someting wrong...but how can I use this on, for example, Dragon Ball Xenoverse 2 .cpk files?
Using -mxtool:crilayla works without error but does not expand at all. dbxv2.xlt generated with bms2xtl using cpk.bms script works (using -mxtool:dbxv2) and .cpk IS expanded. But I don't see how to use this external cpk_R1 plugin, nothing is expanded with crilayla.exe, crilayla.dll and xtool.ini in xtool path (I'm using xtool.ini from latest xtool, section name [crilayla]), maybe I'm missing something?
Thanks in advance,
regards


Supported games

To be used by QuickBMS scripts...


cpk doesn't work by itself it has to be used by another plugin and in this case, a database in which you would create using bms2xtl.

[Masquerade]

Project CARS [RC4 + XCompress]


Extract contents of the attached archive and use -mPCARS.
You must use -d1 because this game has two layers to the bff file, the first layer is rc4 encryption and the second is compressed data.
You can discard the PCARS.XTL file and the key files for decoding, but you must have:

xtool.ini
xcompress.exe
xcompress.dll

You must use the following configuration for the XCompress codec for XTool (thanks Razor12911 for explaining this to me):
[xmemcompress]
Encode=xcompress.exe d -window=131072 -partition=524288 <library>
Decode=xcompress.exe e -window=131072 -partition=524288 <library>

Compressing Monterey.bff, 222,888,414 bytes

Compressing 222,888,414 bytes with xtool precomp -mPCARS -c512mb -t16 -d1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911

Streams: 5523/5523
Time: 00:00:22 (00:00:05)
Memory: 525 MB (525 MB)

Compressed 1 file, 222,888,414 => 453,409,235 bytes. Ratio 203.42%
Compression time: cpu 0.27 sec/real 40.10 sec = 1%. Speed 5.56 mB/s
All OK

[KaktoR]

^
This also works with pcars2 and pcars3 (same bms script).

[Masquerade]

^
This also works with pcars2 and pcars3 (same settings for xmemcompress).

PCARS 2 & 3 use Oodle Kraken, not xcompress.

[KaktoR]

Sorry, I wanted to write "same bms script" lol. For some reason I wrote something different

Edit: Corrected my post

[Masquerade]

Automobilista 2 [RC4 + ZLIB]


Extract contents of the attached archive and use -mautomobilista2.
You must use -d1 because this game has two layers to the bff file, the first layer is rc4 encryption and the second is compressed data.
You can discard the Automobilista2.xtl file and the key files for decoding, but you must keep the zlib/reflate libraries.

Compressing BOOTFLOW.bff, 29,921,681 bytes

When using madness.dll XTool plugin:
Compressed 1 file, 29,921,681 => 131,543,892 bytes. Ratio 439.63%
Compression time: cpu 0.00 sec/real 6.44 sec = 0%. Speed 4.64 mB/s
All OK

When using Automobilista 2 Database:
Compressed 1 file, 29,921,681 => 144,766,886 bytes. Ratio 483.82%
Compression time: cpu 0.03 sec/real 7.34 sec = 0%. Speed 4.08 mB/s
All OK

[31/12/22] [Updated to game version v1.4.5.2]

[Masquerade]

Super Robot Wars 30 [DECRYPTION ONLY] [Blowfish]


Extract contents of the attached archive and use -mSRW30.
You can discard the SRW30.XTL file and the key file for decoding, but you must keep the following:

quickbms.exe
xtool.ini

After decrypting, you need to add crilayla precompressor. This XTool database does not do any precompression. I am using crilayla precompressor from XTool v0.12 which is also attached below.
Some streams may be missing from the database due to some stream extraction issue with CriPakTools but most streams, especially the big ones, will definitely be detected.
WARNING! VERY slow :D

Compressing default.cpk, 7,600,688 bytes

FreeArc: -mxt_srw30+Cri

Compressed 1 file, 7,600,688 => 41,476,750 bytes. Ratio 545.70%
Compression time: cpu 0.02 sec/real 14.77 sec = 0%. Speed 0.51 mB/s
All OK

[Masquerade]

Ultra Age [v2] [Oodle]


Extract the contents of the below archive and use -mUltraAge.
Requires oo2core_*_win64.dll, I included one in the archive.
You can discard the UltraAge.xtl for decoding, but you must retain the oodle core.
v2: I made a mistake in first database with the oodle codec. v2 database is correct.

[Masquerade]

Project CARS 3 [RC4 + Kraken]


Extract contents of the attached archive and use -mPCARS3
You must use -d1 because this game has two layers to the bff file, the first layer is rc4 encryption and the second is compressed data.
You can discard the PCARS3.XTL file and the key files for decoding, but you must keep the oodle library.

Compressing Shanghai[Group].bff, 347,349,287 bytes

XTool is created by Razor12911

Streams: 3370/3370
Time: 00:00:17 (00:01:03)
Memory: 533 MB (533 MB)

Compressed 1 file, 347,349,287 => 741,409,774 bytes. Ratio 213.45%
Compression time: cpu 0.44 sec/real 35.27 sec = 1%. Speed 9.85 mB/s
All OK

[Masquerade]

DIRT 5 [LZ4]


Extract the contents of the attached archive and use -mdirt5.
liblz4.dll is required, I included one for you.
You can discard dirt5.xtl for decoding, but you must retain the same liblz4.dll you used for precompression.
Thanks to Razor12911 and spiritovod!

Compressing 64_WIP_DEV.dat, 179,810,694 bytes

XTool is created by Razor12911

Streams: 3986/3986
Time: 00:00:32 (00:00:11)
Memory: 512 MB (512 MB)

Compressed 1 file, 179,810,694 => 522,544,903 bytes. Ratio 290.61%
Compression time: cpu 0.33 sec/real 42.37 sec = 1%. Speed 4.24 mB/s
All OK

[Masquerade]

Castle Of Illusion [LZ4HC]


Extract contents of attached archive and use -mcastleofillusion
liblz4.dll is required. I included one that gives a good ratio.
This game uses LZ4HC at level 9.
You can discard castleofillusion.xtl for decoding, but you most retain the same liblz4 library you used for encoding.

Compressing data.pak, 824,928,625 bytes

XTool is created by Razor12911

Streams: 5524/5539
Time: 00:00:22 (00:00:48)
Memory: 533 MB (533 MB)

Compressed 1 file, 824,928,625 => 1,291,967,429 bytes. Ratio 156.62%
Compression time: cpu 0.86 sec/real 53.64 sec = 2%. Speed 15.38 mB/s
All OK

[Masquerade]

Test Drive: Ferrari Racing Legends [RC4 + XCompress]


Extract contents of the attached archive and use -mTDFRL.
You must use -d1 because this game has two layers to the bff file, the first layer is rc4 encryption and the second is compressed data.
You can discard the TDFRL.XTL file and the key files for decoding, but you must have:

xtool.ini
xcompress.exe
xcompress.dll


XTool is created by Razor12911

Streams: 9108/9108
Time: 00:00:21 (00:00:05)
Memory: 519 MB (519 MB)

100%
Errorlevel=0

Compressed 23 files, 163,883,758 => 448,312,680 bytes. Ratio 273.56%
Compression time: cpu 0.22 sec/real 27.75 sec = 1%. Speed 5.91 mB/s
All OK

[infovs]

Is there any location to download old version of plugins and database makers?
For example, I want to try old version of dunia database maker R3 with xtool 0.3.21.. Because I can't make latest (R5) version to work with FC6 and latest xtool 0.5.3. Yes, I have everything, not new at this at all, successfully created database fc6.xtl, have tried different versions of liblz4.dll with it, tried -d1 in arc.ini, all in same dir, etc etc, no error with -mxtool:fc6 but it simply does not expand. I'm grateful to anyone with any advice on this, maybe I'm missing something.
P.S. btw, there are only _few_ old versions of plugins at "older releases" xtool link.. and none of database maker old releases.

[L0v3craft]

Is there any location to download old version of plugins and database makers?
For example, I want to try old version of dunia database maker R3 with xtool 0.3.21.. Because I can't make latest (R5) version to work with FC6 and latest xtool 0.5.3. Yes, I have everything, not new at this at all, successfully created database fc6.xtl, have tried different versions of liblz4.dll with it, tried -d1 in arc.ini, all in same dir, etc etc, no error with -mxtool:fc6 but it simply does not expand. I'm grateful to anyone with any advice on this, maybe I'm missing something.
P.S. btw, there are only _few_ old versions of plugins at "older releases" xtool link.. and none of database maker old releases.

Doesn't expand because xtool and/or the database maker are bugged. If you create the database only on one .dat file (you must include the same .fat) xtool is going to work. It doesn't work if you create the database on all game files. I have reported this problem to razor already.

[Masquerade]

Updated Orcs Must Die! 3 LZ4HC database to game version v1.2.0.2 / Build 8601683.

[Masquerade]

doboz Plugin for XTool

Just a very simple way of adding the ability to precompress streams compressed using doboz algorithm with XTool.

You cannot use this on its own, just like xmemcompress plugin. It's designed to be used with BMS2XTL (https://fileforums.com/showpost.php?p=490190&postcount=2).

Particularly, the Witcher 3 - which uses a fun compression cоcktail of many algos, one of which is doboz.

Thanks to Attila T. Afra who's doboz compressor is used here. I found this on the wayback machine (https://web.archive.org/web/20110814212314/https://bitbucket.org/attila_afra/doboz/downloads/) when looking for potential source code I could compile.

[dixen]

Particularly, the Witcher 3 - which uses a fun compression cоcktail of many algos, one of which is doboz.
Hmm. It's interesting, thank you
By the way, does anyone know what algorithm the *.str files use in Dead Space 2?

[Masquerade]

Redout 2 [Leviathan]


Extract the contents of the attached archive and use -mredout2.
Requires oo2core_*_win64.dll - I included one that gives a good ratio.
You can delete redout2.xtl for decoding but you must retain the same oodle core used for precompression.

Compressing pakchunk1optional-WindowsNoEditor.pak
Compressing pakchunk6optional-WindowsNoEditor.pak
Compressing pakchunk3optional-WindowsNoEditor.pak

Compressed 3 files, 41,597,331 => 67,153,780 bytes. Ratio 161.44%
Compression time: cpu 0.09 sec/real 6.27 sec = 1%. Speed 6.63 mB/s
All OK

[elit]

Finally I figured out bms->xtool. Was wondering why it doesn't work with:
Encode=quickbms.exe -s "comtype <codec> ; clog <fileout> 0 <insize> <outsize>" "" <filein>
Decode=quickbms.exe -s "comtype <codec>_COMPRESS ; clog <fileout> 0 <insize> <outsize>" "" <filein>

alone, then created xtl using bms2xtl and it did work. The fact that this tool can generate library from the script.. its mind blowing and incredible how much talent this guy have.

I have a question @razor, does this(quickbms method) have any constrains like internal codecs? For example chunk size 'c', does it exist in this realm too or not?

[Razor12911]

I have a question @razor, does this(quickbms method) have any constrains like internal codecs? For example chunk size 'c', does it exist in this realm too or not?

Chunk size exists here as well, sorry for late reply. Been busy :)

[Razor12911]

King Arthur Knight's Tale

Supported games
King Arthur Knight's Tale (requires oo2core_9_win64.dll)

Usage
Place kakt.dll near xtool.exe along with required libraries and use -mkakt

Notes

The library is required when encoding.

Oodle supremacy strikes again: Making repacking obsolete since 2018

[dixen]

UE4 plugin (R6) don't work with Street Fighter V

[Razor12911]

Sample?

[dixen]

Sample?

https://drive.google.com/file/d/180Ta_rCEi-9ILRWBsVedwkRZefxfDMqQ/view?usp=sharing

Key
0x5F6153346D665A4B384D3573354B5743324C7A325673466E 474B4937617A676C

[Razor12911]

Spiderman Remastered

Supported games
Spiderman Remastered (requires liblz4.dll)

Usage
Place spidey.dll near xtool.exe along with required libraries and use -mspidey

Notes

The library is required when encoding.

Might as well compress the game as is, lz4 in fast mode is used.
Tested on small samples provided.

Edit: Hotfix uploaded

[KaktoR]

Thanks, here is a quick comparison

spidey+srep+lzma2
Compressed 1 file, 2,123,501,810 => 1,170,843,056 bytes. Ratio 55.14%
Compression time: cpu 3.23 sec/real 381.94 sec = 1%. Speed 5.56 mB/s
All OK

srep+lzma2
Compressed 1 file, 2,123,501,810 => 1,353,511,940 bytes. Ratio 63.74%
Compression time: cpu 2.81 sec/real 209.88 sec = 1%. Speed 10.12 mB/s
All OK

[Razor12911]

Saints Row 2022

Supported games
Saints Row 2022 (requires liblz4.dll, xdelta_dll.dll)
Saints Row Boss Factory (requires liblz4.dll, xdelta_dll.dll)

Usage
Place sr22.dll near xtool.exe along with required libraries and use -msr22

Notes

The library is required when encoding.

Results
Tested 1 file, 30,616,424,238 => 20,075,203,334 bytes. Ratio 152.51%
Testing time: cpu 18.06 sec/real 250.79 sec = 7%. Speed 80.05 mB/s

[Razor12911]

Plugin update available

Changes

- Fixed issues in unity engine plugin from being able to process multiple streams in the same chunk
- Added feature to allow user to enforce method used.

Notes

Unity plugin was heavily flawed because of a tiny coding mistake on my part, fixed the issue thanks to help of Masquerade and KaktoR.
It was also brought to my attention that newer unity games use level 12 instead of level 10 lz4hc and as such, the user will need to specify the compression algorithm to use including level because that's the only way xdelta works.

What does this mean for novices? If using -munity does not give you desired results, you can enforce the method like this:
-munity,lz4 (this will set the method to lz4, I was given a sample that required this)
-munity,lz4hc,l12 (this will set the method to lz4hc and use level 12, this is for newer unity games)
-munity,lz4hc,l10 (this will set the method to lz4hc and use level 10, this is for older unity games)

You can use verbose mode to see results and always remember than lz4hc compresses better than lz4. You can then play around with the two methods and the levels of lz4hc until you come close to the original size.

[KaktoR]

Here are some games which I could process without any errors and maxed out stream detection

Cult of the Lamb -> lz4hc,l12 (lz4 library 1.8.2 - 1.9.3)
Final Fantasy Pixel Remaster (all 6 games) -> lz4hc,l12
V Rising -> lz4 (lz4 library 1.8.0)
Starship Troopers Terran Command -> lz4hc,l12 (lz4 library 1.9.3 on some small bundle files, not worth in my opinion)
Outward -> lz4hc,l12 (lz4 library 1.8.2 - 1.9.3)

[emadmoner]

Spiderman Remastered

Supported games
Spiderman Remastered (requires liblz4.dll)

Edit: Hotfix uploaded

Thanks for quick hotfix, compress & decompress work perfect

[KaktoR]

Saints Row 2022?

Supported games
Saints Row Boss Factory (requires liblz4.dll, xdelta_dll.dll)

Usage
Place sr22.dll near xtool.exe along with required libraries and use -msr22

Notes

The library is required when encoding.

The development team behind Saints Row released a "game" titled, Saints Row Boss Factory which can be downloaded from Epic Games and it showcases character customisations the game has to offer for people to tryout before the actual game releases. I got the game with the intentions of looking at the game files to create a plugin for Saints Row 2022. This should work with the final release game if nothing changes.

Source code is uploaded if there are any changes.

Results
Tested 1 file, 1,917,017,391 => 1,181,753,626 bytes. Ratio 162.22%
Testing time: cpu 0.97 sec/real 23.70 sec = 4%. Speed 49.86 mB/s

Which lz4 library is required for this?

[Razor12911]

@KaktoR

liblz4 1.9.2

Plugin update available

Changes

- Fixed issues pointed out by dixen in Unreal Engine plugin (hopefully)

[Hexagon123]

Extract contents of the attached archive and use -mNKR2GP --diff=70p.
You can discard the NKR2GP.XTL file for decoding, but you must keep the following:

quickbms.exe
xtool.ini



Tested with Gameplay.apf, 16,511,216 bytes
XTool is created by Razor12911

Streams: 622/622
Time: 00:02:18 (00:00:03)
Memory: 175 MB (175 MB)

[Masquerade]

Saints Row 2022

Working great, thanks Razor!


Compressing engine.vpp_pc
Compressed 1 file, 62,748,440 => 217,935,308 bytes. Ratio 347.32%
Compression time: cpu 0.19 sec/real 15.81 sec = 1%. Speed 3.97 mB/s
All OK

[Razor12911]

Anvil Engine

Usage
Place anvil.dll near xtool.exe along with required libraries and use -manvil

Notes

The library is required both encoding and decoding.

lzo2.dll or oo2core_#_win64.dll is required

Results on AC Valhalla

-manvil
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 1,103,790,080 => 1,978,050,029 bytes. Ratio 179.21%
Compression time: cpu 1.03 sec/real 74.01 sec = 1%. Speed 14.91 mB/s
All OK


-moo2recm
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 1,103,790,080 => 1,776,145,142 bytes. Ratio 160.91%
Compression time: cpu 1.33 sec/real 387.05 sec = 0%. Speed 2.85 mB/s
All OK

Special thanks to KaktoR for running tests

[Razor12911]

Plugin update available

Changes

- Unreal Engine plugin can directly report streams to xtool

Notes

The plugin's original purpose was to decrypt streams so you could get better results however I've decided to extend functionality to allow the user to specify what compression algorithm was used on unreal engine titles even if they are not encrypted so you get similar result as if you were to use unreal tournament 4 script via Bms2Xtl.

Why? Well Xtool's oodle scanner in particular is still lackluster (needs improvements) and at times it will hang for no reason and take too long while still producing underwhelming results.

So instead of normal use of specifying key for decryption -mue4,k0x... you directly specify what algorithm was used, for example, -mue4,kraken

Unreal Engine headers change a lot so I've tried as best I can to mitigate differences between games, just yesterday I noticed a new header structure used in the Back 4 Blood game and I tried to make it so that the user does not specify parameters even they don't know when to use.

Results tests on some Star Wars Jedi Fallen Order

-mmermaid
Compressed 7 files, 126,201,498 => 286,909,553 bytes. Ratio 227.34%
Compression time: cpu 0.16 sec/real 259.98 sec = 0%. Speed 0.49 mB/s // it stalled in the threads doing who knows what :D

-mue4,mermaid
Compressed 7 files, 126,201,498 => 312,346,042 bytes. Ratio 247.50%
Compression time: cpu 0.20 sec/real 21.70 sec = 1%. Speed 5.82 mB/s

[dixen]

Anvil Engine
Thanks but on old AC games (Brotherhood) didn't worked


[0] Confirmed lzo1c stream at 000000000987747B (19327 >> 20919)
[0] Confirmed lzo1c stream at 0000000009948DB3 (26919 >> 29102)
[0] Processing streams on block 0000000008000000 to 00000000099B7FFF (26968064)
[0] Processing lzo1c stream at 000000000987747B (19327 >> 20919 >> 19036) using v999 has failed
[0] - Patching stream at 000000000987747B (19327 >> 19036) [6046] has failed
[0] Processing lzo1c stream at 0000000009948DB3 (26919 >> 29102 >> 23998) using v999 has failed
[0] - Patching stream at 0000000009948DB3 (26919 >> 23998) [4788] has failed

Streams: 0/10
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)


Errorlevel=0

Compressed 1 file, 161,185,792 => 161,185,867 bytes. Ratio 100.00%
Compression time: cpu 0.11 sec/real 0.92 sec = 12%. Speed 174.85 mB/s
All OK

[KaktoR]

Thanks but on old AC games (Brotherhood) didn't worked

It works

DataPC_LGS306_CloacaMaxima.forge
Compressed 1 file, 46,104,576 => 87,396,705 bytes. Ratio 189.56%
Compression time: cpu 0.00 sec/real 3.92 sec = 0%. Speed 11.76 mB/s
All OK

Extracted 1 file, 87,396,705 => 46,104,576 bytes. Ratio 189.56%
Extraction time: cpu 0.03 sec/real 1.81 sec = 2%. Speed 25.47 mB/s
All OK

[0] Processed lzo1x stream at 0000000002BE7A8A (26409 >> 32768 >> 26409) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BEE1B7 (7873 >> 9323 >> 7873) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF0280 (134 >> 150 >> 134) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF0504 (335 >> 578 >> 335) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF0851 (1283 >> 1990 >> 1283) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF0F52 (1164 >> 1858 >> 1164) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF15DC (505 >> 800 >> 505) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF19D3 (290 >> 476 >> 290) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF1CF3 (2030 >> 3180 >> 2030) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF26DF (505 >> 798 >> 505) using l8:v999 successfully
[0] Processed lzo1x stream at 0000000002BF2AD6 (660 >> 1018 >> 660) using l8:v999 successfully

Streams: 2465/2465
Time: 00:00:22 (00:00:07)
Memory: 153 MB (153 MB)

The only game it does not work is AC1 because of lzo pro codec used (maybe AC Rogue aswell, not sure).

[dixen]

It works

Then it's strange..
Which version lzo2.dll are you using?

[KaktoR]

It comes with xtool (crc32 E75ADC9C)

However I have to correct myself.

Here are tests for all older AC games

AC1 - Works
AC2 - Not work
Brotherhood - Works
Revelations - Works
AC3 - Works
AC3Remastered - Not work
Black Flag - Not work
Liberation HD - Not work
Rogue - Not work
Unity - Works
Syndicate - Works

[dixen]

It comes with xtool (crc32 E75ADC9C)
Same version..but ..

Streams: 0/21
Time: 00:00:01 (00:00:01)
Memory: 128 MB (128 MB)


Errorlevel=0

Compressed 1 file, 429,260,800 => 429,261,035 bytes. Ratio 100.00%
Compression time: cpu 0.44 sec/real 2.70 sec = 16%. Speed 158.80 mB/s
All OK
:confused:

[KaktoR]

The library requires msvc2005 and/or 2010, not sure.

xtool.exe precomp -manvil -c16mb -t100p --verbose - - < %1 > %1.out

[Razor12911]

Thanks but on old AC games (Brotherhood) didn't worked


[0] Confirmed lzo1c stream at 000000000987747B (19327 >> 20919)
[0] Confirmed lzo1c stream at 0000000009948DB3 (26919 >> 29102)
[0] Processing streams on block 0000000008000000 to 00000000099B7FFF (26968064)
[0] Processing lzo1c stream at 000000000987747B (19327 >> 20919 >> 19036) using v999 has failed
[0] - Patching stream at 000000000987747B (19327 >> 19036) [6046] has failed
[0] Processing lzo1c stream at 0000000009948DB3 (26919 >> 29102 >> 23998) using v999 has failed
[0] - Patching stream at 0000000009948DB3 (26919 >> 23998) [4788] has failed

Streams: 0/10
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)


Errorlevel=0

Compressed 1 file, 161,185,792 => 161,185,867 bytes. Ratio 100.00%
Compression time: cpu 0.11 sec/real 0.92 sec = 12%. Speed 174.85 mB/s
All OK

could you upload sample

[Masquerade]

Testing on a sample from Ghost Recon Wildlands and there does appear to be a bug because the streams supposedly all decompress to the same size:

https://paste.ec/raw/8DaeUwyt#yx7kpEQUtGZNRmnjM64O9t22pj0FjntunJcRRVMJX zx

Sample: https://www103.zippyshare.com/v/wrkDJuYX/file.html

XTool is created by Razor12911

Streams: 0/1358
Time: 00:00:03 (00:00:24)
Memory: 128 MB (128 MB)

XTool v0.6.3 with xdelta library injected. anvil.dll and lzo2.dll present.

[dixen]

could you upload sample

https://pixeldrain.com/u/62Ra44jU

[Razor12911]

Plugin update available

Changes

- Added support for lzopro streams

@Masquerade
Compressed 1 file, 114,556,928 => 138,193,577 bytes. Ratio 120.63%
Compression time: cpu 0.08 sec/real 1.72 sec = 5%. Speed 66.71 mB/s

@dixen
Compressed 1 file, 229,605,376 => 380,281,233 bytes. Ratio 165.62%
Compression time: cpu 0.17 sec/real 8.43 sec = 2%. Speed 27.23 mB/s

[Cesar82]

Plugin update available

Changes

- Added support for lzopro streams

@Masquerade
Compressed 1 file, 114,556,928 => 138,193,577 bytes. Ratio 120.63%
Compression time: cpu 0.08 sec/real 1.72 sec = 5%. Speed 66.71 mB/s

@dixen
Compressed 1 file, 229,605,376 => 380,281,233 bytes. Ratio 165.62%
Compression time: cpu 0.17 sec/real 8.43 sec = 2%. Speed 27.23 mB/s
Which plugin was updated? The Anvil plugin?

[KaktoR]

Which plugin was updated? The Anvil plugin?

Yes

[Mini]

Razor12911, please make plugin for Evolution Engine (The Darkness II, Dark Sector, Star Trek)
http://aluigi.altervista.org/bms/darkness2.bms

example:
F.Texture.cache + F.Texture.toc (2.13 GB)

unpacked files size 4.17 GB

[Razor12911]

try using this (https://fileforums.com/showpost.php?p=490190&postcount=2)

[Mini]

Razor12911
I already tried it didn't work

Compressing 34 files of 6,515,860,623 bytes: 0.00 sec
Using xtool_reflate+srep:m3f:a2:l256
Memory for compression 0b, decompression 0b
Compressing 0 bytes with xtool.exe precomp -mdarkness+zlib -d1 -c128mb -t100p --diff=1p - -
XTool is created by Razor12911

Streams: 61/317
Time: 00:00:25 (00:02:37)
Memory: 128 MB (128 MB)

100.0%
Errorlevel=0

Compressing 6,515,864,879 bytes with srep64 -m3f -a2 -l256 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

and quickbms does not unpack all files, this program is better https://forum.xentax.com/viewtopic.php?f=17&t=13701 / https://github.com/IcarusXSEC/WF-Evolution-Unpack (+ source)

game files
https://www4.zippyshare.com/v/VXvFa9CU/file.html

[dixen]

game files
https://www4.zippyshare.com/v/VXvFa9CU/file.html

This is NOT ZLIB. in darkness.bms write
comtype evolution

[KaktoR]

doboz Plugin for XTool

Just a very simple way of adding the ability to precompress streams compressed using doboz algorithm with XTool.

You cannot use this on its own, just like xmemcompress plugin. It's designed to be used with BMS2XTL (https://fileforums.com/showpost.php?p=490190&postcount=2).

Particularly, the Witcher 3 - which uses a fun compression cоcktail of many algos, one of which is doboz.

Thanks to Attila T. Afra who's doboz compressor is used here. I found this on the wayback machine (https://web.archive.org/web/20110814212314/https://bitbucket.org/attila_afra/doboz/downloads/) when looking for potential source code I could compile.

Thanks for this. Maybe someone else may find the info usefull.

Like Masquerade said, you have to use it with xtl db (use bms2xtl for this with witcher3.bms script and add lz4=lz4hc to bms2xtl.ini). You have to make the xtl file out of *.bundle files. The script is not compatible with *.cache files for some reason, but it should regarding script info and zenhax forum, but for me it does not work on *.cache files.

For lz4hc use level9. This is the peak for *.bundle files. Some mod files use level3 however.

-mxtool:witcher3:zlib:lz4hc,l9

https://i.imgur.com/hLR8zwe.png

[Mini]

This is NOT ZLIB. in darkness.bms write

how to recompress files? try it ;)

[Razor12911]

@Mini
I don't see any re-encode/compress function in any of these sources.

[Mini]

Razor12911
can you help with this engine? maybe I did something wrong with bms2xtl

http://aluigi.altervista.org/papers/quickbms.txt

evolution, aka lzf or lzfx

i try:

[cpk,snappy,rfpk,evolution,lzf,lzfx]
// http://aluigi.altervista.org/papers/quickbms.txt
Encode=quickbms.exe -s "comtype <codec>; clog <fileout> 0 <insize> <outsize>" "" <filein>
Decode=quickbms.exe -s "comtype <codec>_COMPRESS; clog <fileout> 0 <insize> <outsize>" "" <filein>


precompression not working :(

[dixen]

Shadow Of War (GOG Version, NON-UltraHD Content)

Test file presentations.arch06

Compressed 1 file, 1,310,943,354 => 2,382,102,100 bytes. Ratio 181.71%
Compression time: cpu 1.33 sec/real 83.72 sec = 2%. Speed 15.66 mB/s
All OK

sow.xtl db
https://www50.zippyshare.com/v/cUU4UIeq/file.html

Note.
You need oo2core_5_win64.dll from game folder.
Recommended version XTool - 0.53 (or 0.65 if 0.53 stuck process)

[panker1992]

Shadow Of War (GOG Version, NON-UltraHD Content)

Test file presentations.arch06

Compressed 1 file, 1,310,943,354 => 2,382,102,100 bytes. Ratio 181.71%
Compression time: cpu 1.33 sec/real 83.72 sec = 2%. Speed 15.66 mB/s
All OK

sow.xtl db
https://www50.zippyshare.com/v/cUU4UIeq/file.html

Note.
You need oo2core_5_win64.dll from game folder.
Recommended version XTool - 0.53 (or 0.65 if 0.53 stuck process)

I have been unable to generate a database that works, can you post how you made one ?

[Masquerade]

I have been unable to generate a database that works, can you post how you made one ?

bms2xtl

http://aluigi.altervista.org/bms/shadow_of_mordor.bms

[Masquerade]

Asterigos: Curse of the Stars

Place asterigos.xtl and asterigos.key in the same folder as XTool, use "-masterigos+preflate -d1".

[Snake288]

Hello Masquerade

Asterigos: Curse of the Stars
Place asterigos.xtl and asterigos.key in the same folder as XTool, use "-masterigos+preflate -d1".

File : Asterigos.7z

Asterigos: Curse of the Stars is the mask of the XTool 2020 (Add-ons) game published by Asterigos.7z file
DiskSpan GUI can I get help with adding?

[Wanterlude]

Hello Masquerade

Asterigos: Curse of the Stars
Place asterigos.xtl and asterigos.key in the same folder as XTool, use "-masterigos+preflate -d1".

File : Asterigos.7z

Asterigos: Curse of the Stars is the mask of the XTool 2020 (Add-ons) game published by Asterigos.7z file
DiskSpan GUI can I get help with adding?

DiskSpan Gui
xtool:masterigos:mpreflate:d1
Where is xtool located? (DiskSpan Gui)
COMPRESSOR\Resources\PRE\XTool_2020
Place asterigos.xtl and asterigos.key from the archive to XTool_2020 folder

[Masquerade]

Nickelodeon Kart Racers 3 [Kraken]

Place NickelodeonKart3.xtl in the same folder as XTool, use "-mNickelodeonKart3". Requires oodle library for compression + decompression, but database can be discarded.

Compressing pakchunk49-WindowsNoEditor.pak, 40,495,553 bytes

XTool is created by Razor12911

Streams: 2089/2131
Time: 00:00:12 (00:00:47)
Memory: 317 MB (317 MB)

Compressed 1 file, 40,495,553 => 208,378,061 bytes. Ratio 514.57%
Compression time: cpu 0.13 sec/real 22.64 sec = 1%. Speed 1.79 mB/s
All OK

[Razor12911]

Plugin update available

Changes

- Added support for file version 8 in unity engine plugin???

Notes

R3 attachment is left on main post just in case if I've made a blunder.

Plugin was updated in response to this post (https://fileforums.com/showpost.php?p=498811&postcount=3229)

Compressed 1 file, 60,357,834 => 327,942,992 bytes. Ratio 543.33%
Compression time: cpu 0.06 sec/real 8.31 sec = 1%. Speed 7.26 mB/s

[Hexagon123]

The game, Super Monkey Ball: Banana Mania (2021) uses LZ4 or LZ4HC? doesn't seem to work with XTool, this seems bugged.

Uses version 7.

Use [xtool precomp -munity,lz4hc,l12] with the liblz4.dll (1.9.3)

[KaktoR]

Plugin update available

Changes

- Added support for file version 8 in unity engine plugin???

Notes

R3 attachment is left on main post just in case if I've made a blunder.

Plugin was updated in response to this post (https://fileforums.com/showpost.php?p=498811&postcount=3229)

Compressed 1 file, 60,357,834 => 327,942,992 bytes. Ratio 543.33%
Compression time: cpu 0.06 sec/real 8.31 sec = 1%. Speed 7.26 mB/s

Thanks

Compressed 137 files, 22,499,636,299 => 30,533,946,304 bytes. Ratio 135.71%
Compression time: cpu 9.66 sec/real 669.32 sec = 1%. Speed 33.62 mB/s
All OK

Extracted 137 files, 30,533,946,304 => 22,499,636,299 bytes. Ratio 135.71%
Extraction time: cpu 18.73 sec/real 549.49 sec = 3%. Speed 40.95 mB/s
All OK

Streams: 175306/175306
Time: 00:16:32 (00:12:17)
Memory: 270 MB (270 MB)

[Razor12911]

Plugin update available

Changes

- improved header decoder in unity plugin

Notes

bg_smb1_masters
Compressed 1 file, 6,273,468 => 14,834,634 bytes. Ratio 236.47%
Compression time: cpu 0.00 sec/real 1.07 sec = 0%. Speed 5.84 mB/s

[Razor12911]

Boycott 3

Supported games
Bayonetta 3 Switch (requires oo2core_9_win64.dll)

Usage
Place bn3.dll near xtool.exe along with required libraries and use -mbn3

Notes

A newer oodle library may be required for some streams (either 2.9.7 or 2.9.8)

Results

-mkraken
Compressed 1 file, 127,450,048 => 177,590,706 bytes. Ratio 139.34%
Compression time: cpu 0.17 sec/real 49.27 sec = 0%. Speed 2.59 mB/s

-mbn3
Compressed 1 file, 127,450,048 => 180,417,265 bytes. Ratio 141.56%
Compression time: cpu 0.16 sec/real 21.33 sec = 1%. Speed 5.97 mB/s

[Mini]

hello Razor12911, can you help with this engine?
https://fileforums.com/showpost.php?p=498304&postcount=307

[Masquerade]

^^
Didn't Razor already say there's no recompression code for this algo?
https://www.fileforums.com/showpost.php?p=498301&postcount=306

[Masquerade]

Crysis 2 Remastered [LZ4]


Extract contents of attached archive to XTool directory and use -mcrysis2remastered
Requires liblz4.dll. I included one that gives best ratio.
You can discard crysis2remastered.xtl for decoding, but you must retain the same liblz4.dll used for encoding.

Compressing textures_chk0-part2.pak, 411,058,830 bytes

XTool is created by Razor12911

Streams: 1192/1192
Time: 00:00:07 (00:00:04)
Memory: 357 MB (357 MB)

Compressed 1 file, 411,058,830 => 550,255,834 bytes. Ratio 133.86%
Compression time: cpu 0.52 sec/real 21.70 sec = 2%. Speed 18.94 mB/s
All OK

[kast1450]

I can't get the Unreal Engine decryption to work. The game is 171 and the key is 0xE268A09FA3AB747487935B5E6362BEA968AD7A64795419C0 1D84552D71C011E2. It works fine with UnrealPak however both ue4dt and XTool 2020's Unreal Engine 4 (crypted method) don't decrypt anything. Any advice?

[Razor12911]

Plugin update available

Changes

* Added support for FIFA 23 and NFS Unbound to Frostbite 3 plugin
* Fixed issues with codecs that utilize oodle library

@Mini

I'll take another look at it, could you reupload the sample (My download folder gets cleaned once every week)

@kast1450

Could you upload a sample

[Mini]

Razor12911
B.AnimRetarget.cache + B.AnimRetarget.toc
https://www17.zippyshare.com/v/spfv8aj1/file.html
58 MB

unpacked size 186 MB (with https://github.com/IcarusXSEC/WF-Evolution-Unpack )
cmd: EvolutionUnpack B.AnimRetarget.cache /game

or use
http://aluigi.altervista.org/bms/darkness2.bms

info
1) https://github.com/FromDarkHell/TheDarknessIIDecompiled
2) https://zenhax.com/viewtopic.php?t=4793

[kuyhaa]

In Nightmare

.Bundle

default
lz4hc,l12
lz4hc,l10

all file ratio 100%

sample

[Razor12911]

Plugin update available

Changes

- I don't remember what I changed (unity plugin)

Results

chapter01(12-14)_assets_all.bundle
Compressed 1 file, 8,504,252 => 20,406,552 bytes. Ratio 239.96%
Compression time: cpu 0.00 sec/real 1.20 sec = 0%. Speed 7.10 mB/s

chapter05(12-14)_assets_all.bundle
Compressed 1 file, 14,965,991 => 21,867,535 bytes. Ratio 146.11%
Compression time: cpu 0.03 sec/real 1.24 sec = 3%. Speed 12.05 mB/s

[Masquerade]

The Oregon Trail [2022] [Leviathan]


Extract the contents of the attached archive and use -mtheoregontrail.
Requires the included oo2core_9_win64.dll.
You can delete theoregontrail.xtl for decoding but you must retain the same oodle core used for precompression.


Compressing pakchunk100-WindowsNoEditor.pak, 4,677,614 bytes

XTool is created by Razor12911

Streams: 3246/3246
Time: 00:00:18 (00:01:59)
Memory: 512 MB (512 MB)

Compressed 1 file, 4,677,614 => 825,560,508 bytes. Ratio 17649.18%
Compression time: cpu 0.05 sec/real 27.81 sec = 0%. Speed 0.17 mB/s
All OK

[Wanterlude]

Knights of Honor II Sovereign

AssetBundles
lz4hc,l12
Ratio 100%

I checked the files with the latest unity plugin + lz4 180-194
Maybe I'm doing something wrong? Can someone please check?

[Masquerade]

Knights of Honor II Sovereign

AssetBundles
lz4hc,l12
Ratio 100%

I checked the files with the latest unity plugin + lz4 180-194
Maybe I'm doing something wrong? Can someone please check?

The bundles aren't compressed.

[Razor12911]

Razor12911
B.AnimRetarget.cache + B.AnimRetarget.toc
https://www17.zippyshare.com/v/spfv8aj1/file.html
58 MB

unpacked size 186 MB (with https://github.com/IcarusXSEC/WF-Evolution-Unpack )
cmd: EvolutionUnpack B.AnimRetarget.cache /game

or use
http://aluigi.altervista.org/bms/darkness2.bms

info
1) https://github.com/FromDarkHell/TheDarknessIIDecompiled
2) https://zenhax.com/viewtopic.php?t=4793

I managed to create plugin, decompressing the game is possible but recompressing with matching crc is a different story. :(

Compressed 1 file, 60,893,070 => 203,772,032 bytes. Ratio 334.64%
Compression time: cpu 0.02 sec/real 1.17 sec = 1%. Speed 52.01 mB/s

"B.AnimRetarget.cache". File is broken.
Tested 1 file, 203,772,032 => 60,893,070 bytes. Ratio 334.64%
Testing time: cpu 0.09 sec/real 1.06 sec = 9%. Speed 57.33 mB/s
There were 1 warning(s)

[Razor12911]

Here's an alternative doboz library plugin, it may have bugs as this is the first xtool plugin written in Visual Studio rather than RAD Studio.

via executable
Compressed 1 file, 286,247,872 => 1,095,095,861 bytes. Ratio 382.57%
Compression time: cpu 0.20 sec/real 53.75 sec = 0%. Speed 5.33 mB/s

Tested 1 file, 1,095,095,861 => 286,247,872 bytes. Ratio 382.57%
Testing time: cpu 0.22 sec/real 33.81 sec = 1%. Speed 8.47 mB/s

via library
Compressed 1 file, 286,247,872 => 1,095,095,861 bytes. Ratio 382.57%
Compression time: cpu 0.23 sec/real 2.62 sec = 9%. Speed 109.33 mB/s

Tested 1 file, 1,095,095,861 => 286,247,872 bytes. Ratio 382.57%
Testing time: cpu 0.17 sec/real 27.38 sec = 1%. Speed 10.45 mB/s

Refer to the executable version if you have issues which can be found here (https://fileforums.com/showpost.php?p=497188&postcount=267).

[L33THAK0R]

Here's an alternative doboz library plugin, it may have bugs as this is the first xtool plugin written in Visual Studio rather than RAD Studio.

Do any titles other than "The Witcher 3: Wild Hunt" or titles running on the PS Vita use Doboz? Only done a quick skim through, but cant seem to find many cases.

[L33THAK0R]

I managed to create plugin, decompressing the game is possible but recompressing with matching crc is a different story. :(

Compressed 1 file, 60,893,070 => 203,772,032 bytes. Ratio 334.64%
Compression time: cpu 0.02 sec/real 1.17 sec = 1%. Speed 52.01 mB/s

"B.AnimRetarget.cache". File is broken.
Tested 1 file, 203,772,032 => 60,893,070 bytes. Ratio 334.64%
Testing time: cpu 0.09 sec/real 1.06 sec = 9%. Speed 57.33 mB/s
There were 1 warning(s)

Bit of a silly sounding question but how do you use this plugin? Does xtool.ini need to be modified for this library to be accepted? Interested in seeing if despite the CRC issues anything can be done with the decompressed data either as it is or via an xdelta/hdiffz patch.

[Razor12911]

Do any titles other than "The Witcher 3: Wild Hunt" or titles running on the PS Vita use Doboz? Only done a quick skim through, but cant seem to find many cases.

I don't know either :p

Bit of a silly sounding question but how do you use this plugin? Does xtool.ini need to be modified for this library to be accepted? Interested in seeing if despite the CRC issues anything can be done with the decompressed data either as it is or via an xdelta/hdiffz patch.

The sample he sent over has a bms script, meaning you first need to use bms2xtl, the comtype used in the game is evolution/lzfx so I just make a plugin that acts a evolution precompressor, however no matter what parameters you use for lzfx, you simply cannot recompress the data with matching crc. I revisited the data set again today after updating the library to patch the streams but the diff files are just too big and wouldn't make sense to precompress this game.

[KaktoR]

LEGO Star Wars - The Skywalker Saga

My first xtool plugin.

GAME6.DAT
-mxtool:legoswtws
Compressed 1 file, 1,643,604,387 => 3,766,491,875 bytes. Ratio 229.16%
Compression time: cpu 1.22 sec/real 82.54 sec = 1%. Speed 19.91 mB/s
All OK

-mxtool:kraken,l6
Compressed 1 file, 1,643,604,387 => 3,629,442,699 bytes. Ratio 220.82%
Compression time: cpu 1.23 sec/real 92.98 sec = 1%. Speed 17.68 mB/s
All OK

-moo2reck
Compressed 1 file, 1,643,604,387 => 3,696,758,986 bytes. Ratio 224.92%
Compression time: cpu 1.80 sec/real 434.00 sec = 0%. Speed 3.79 mB/s
All OK

GAME.DAT
-mxtool:legoswtws
Compressed 1 file, 4,062,980,016 => 9,814,174,667 bytes. Ratio 241.55%
Compression time: cpu 3.77 sec/real 229.08 sec = 2%. Speed 17.74 mB/s
All OK

-mxtool:kraken,l6
Compressed 1 file, 4,062,980,016 => 9,451,336,766 bytes. Ratio 232.62%
Compression time: cpu 3.39 sec/real 246.06 sec = 1%. Speed 16.51 mB/s
All OK

-moo2reck
Compressed 1 file, 4,062,980,016 => 9,595,617,928 bytes. Ratio 236.17%
Compression time: cpu 5.73 sec/real 1036.90 sec = 1%. Speed 3.92 mB/s
All OK


Thanks to Razor12911 for the education and sorry for the headaches :)

[Masquerade]

Watch Dogs [XCompress]


Extract the contents of the attached archive to XTool folder and use -mwatchdogs.
Requires XCompress plugin, you can get that here (https://fileforums.com/showpost.php?p=495860&postcount=237).
Previous database shared by Razor12911 does not work with current XTool which is why I am sharing a new database.

Compressing common.dat, 109,476,518 bytes

Compressed 1 file, 109,476,518 => 405,191,786 bytes. Ratio 370.12%
Compression time: cpu 0.11 sec/real 20.00 sec = 1%. Speed 5.47 mB/s
All OK

[Razor12911]

Crilayla

Supported games
Japanese games

Usage
Place crilayla.ini near xtool.exe in conjunction with executable version (https://fileforums.com/showpost.php?p=496275&postcount=248) of the plugin and use -mcrilayla

Persona 3 Portable\data_EN\umd0.cpk
Compressed 1 file, 147,942,064 => 845,153,964 bytes. Ratio 571.27%
Compression time: cpu 0.14 sec/real 41.55 sec = 0%. Speed 3.56 mB/s

[deepshit]

Hey There
Can somebody help me with NFS Shift conversion?

Game uses xmemcompress. I used the bms script to generate xtl db but looks like xtool cant decompress the file.
I also put the required libraries beside xtool.( exe and dll)
sample and script and db are attached.

[Razor12911]

Hey There
Can somebody help me with NFS Shift conversion?

Game uses xmemcompress. I used the bms script to generate xtl db but looks like xtool cant decompress the file.
I also put the required libraries beside xtool.( exe and dll)
sample and script and db are attached.

Refer to the Project Cars 1 (https://fileforums.com/showpost.php?p=496546&postcount=253) configuration by Masquerade for this game, xmemcompress must have these parameters.

[xmemcompress]
Encode=xcompress.exe d -window=131072 -partition=524288 <library>
Decode=xcompress.exe e -window=131072 -partition=524288 <library>

Compressed 1 file, 25,522,432 => 84,558,796 bytes. Ratio 331.31%
Compression time: cpu 0.05 sec/real 4.22 sec = 1%. Speed 6.05 mB/s

[Masquerade]

Bink Video


XTool plugin adding support for Bink Video compression.
This plugin uses Bink Pack library by ProFrager.
Thanks to Razor12911 for the little help.

Advantages of using this plugin versus vanilla Bink Pack

Multithreading support.
No CRC check fail since XTool will patch the videos as they are unpacked.
Incompatible Bink 2 videos (KB2n) are automatically ignored.

Special Notes

For Bink 2 videos, you may need to make use of the --diff=* parameter as some patches between CRC imperfect videos to the original state can be larger than XTool's default threshold for patches.
You may need to use a large chunk size to ensure all videos are processed. See example below.

Example Usage

Extract the attached archive to the XTool folder and use -mbik or -mbk2 depending on your data set.

Test

Test on the videos from Assassin's Creed 2 (27x Bink 1 videos).
xtool precomp -mbik "AC2Videos\"
XTool is created by Razor12911

Streams: 13/13
Time: 00:00:09 (00:00:01)
Memory: 148 MB (148 MB)

Results: 937 MB >> 928 MB
xtool precomp -c128mb -mbik "AC2Videos\"
XTool is created by Razor12911

Streams: 26/26
Time: 00:02:12 (00:00:04)
Memory: 391 MB (391 MB)

Results: 937 MB >> 776 MB
xtool precomp -c256mb -mbik "AC2Videos\"
XTool is created by Razor12911

Streams: 27/27
Time: 00:02:25 (00:00:06)
Memory: 677 MB (677 MB)

Results: 937 MB >> 725 MB


Test on 05_DAM folder from Shadow Warrior 3 Defintive Edition (6x Bink 2 videos).
xtool precomp -c256mb -mbk2 "SW3\Content\Movies\Cutscenes\05_DAM"
XTool is created by Razor12911

Streams: 6/6
Time: 00:01:06 (00:00:03)
Memory: 406 MB (406 MB)

Results: 214 MB >> 187 MB

[Cesar82]

For Bink 2 videos, you may need to make use of the --diff=* parameter as some patches between CRC imperfect videos to the original state can be larger than XTool's default threshold for patches.


For me if I use --diff=* in arc.ini it doesn't work, return:
ERROR: write error (disk full?) in compression algorithm xtool:c256mb:mbk2
Arc.ini

--cache=0
[External compressor:xtool]
header = 0
default = -c64mb -t100p
packcmd = "{compressor}.exe" precomp { -option} --diff=* - - <stdin> <stdout>
unpackcmd = "{compressor}.exe" decode -t100p - - <stdin> <stdout>

pack.bat
arc.exe a -ep1 -dses --dirs -s; -lc- -di -i2 -r -w.\temp -m"xtool:c256mb:mbk2" data.arc "pack\*"

I also tested using it directly from the command line after removing --diff=* from arc.ini:
xtool:c256mb:mbk2:-diff=*
xtool:c256mb:mbk2:df*

Arc.ini

--cache=0
[External compressor:xtool]
header = 0
default = -c64mb -t100p
packcmd = "{compressor}.exe" precomp { -option} - - <stdin> <stdout>
unpackcmd = "{compressor}.exe" decode -t100p - - <stdin> <stdout>

pack.bat
arc.exe a -ep1 -dses --dirs -s; -lc- -di -i2 -r -w.\temp -m"xtool:c256mb:mbk2:df*" data.arc "pack\*"
or
arc.exe a -ep1 -dses --dirs -s; -lc- -di -i2 -r -w.\temp -m"xtool:c256mb:mbk2:-diff=*" data.arc "pack\*"


If I don't use --diff=* it compresses normally.
FreeArc 0.67 (March 15 2014) using additional options: --cache=0
Creating archive: data.arc using xtool:c256mb:mbk2
Memory for compression 0b, decompression 0b, cache 0b
Compressing 5 files, 336,582,396 bytes
Compressing health_screen.bk2
Compressing ms_logo.bk2
Compressing ps_studios_long_strawberry_4k30_006.bk2
Compressing lt_outro_1080p.bk2
Compressing lt_intro_1080p.bk2
Compressed 5 files, 336,582,396 => 291,316,936 bytes. Ratio 86.55%
Compression time: cpu 0.20 sec/real 29.95 sec = 1%. Speed 11.24 mB/s
All OK

Am I doing something wrong?

[Masquerade]

Am I doing something wrong?
The --diff parameter in XTool sets a max size in bytes or percentage size of the stream in order for XTool to accept the stream.

The * character is just a placeholder (a wildcard effectively meaning anything/everything). My explanation meant you'd need to change this to an integer.

By default, XTool will not accept a stream if the generated XDelta patch is greater in size than 5% of the stream, but using --diff= allows you to increase this threshold.

You can use --diff=20p to set the max patch size to be less than or equal to 20% of the stream size for XTool to accept the stream. You can also use an integer to specify a max patch size in bytes (e.g. --diff=50000).

[dixen]

XCOM Enemy Unknown

*.bik *.bk2 (With XTool 0.69 + Bink Plugin)

FreeArc 0.67 (March 15 2014) testing archive: data.arc
Tested 168 files, 6,433,251,603 => 7,859,788,952 bytes. Ratio 81.85%
Testing time: cpu 8.44 sec/real 344.19 sec = 2%. Speed 22.84 mB/s
All OK

[Masquerade]

Labyrinth of Galleria: The Moon Society [LZ4]


Requires LZ4.DLL from the game files. Rename it to liblz4.dll for use with XTool.
Drop the configuration into XTool folder and use -mLoGTMS

By Razor12911

[Masquerade]

Plugin Update

Added STDIN/STDOUT support for Bink Video plugin (thanks to Gehrman for idea).

Thanks to 78372 for cls wrapper.

Usage has not changed, but its significantly faster.

[exorc]

Bink Video


XTool plugin adding support for Bink Video compression.
This plugin uses Bink Pack library by ProFrager.
Thanks to Razor12911 for the little help.

Advantages of using this plugin versus vanilla Bink Pack

Multithreading support.
No CRC check fail since XTool will patch the videos as they are unpacked.
Incompatible Bink 2 videos (KB2n) are automatically ignored.

Special Notes

For Bink 2 videos, you may need to make use of the --diff=* parameter as some patches between CRC imperfect videos to the original state can be larger than XTool's default threshold for patches.
You may need to use a large chunk size to ensure all videos are processed. See example below.

Example Usage

Extract the attached archive to the XTool folder and use -mbik or -mbk2 depending on your data set.

Test

Test on the videos from Assassin's Creed 2 (27x Bink 1 videos).
xtool precomp -mbik "AC2Videos\"
XTool is created by Razor12911

Streams: 13/13
Time: 00:00:09 (00:00:01)
Memory: 148 MB (148 MB)

Results: 937 MB >> 928 MB
xtool precomp -c128mb -mbik "AC2Videos\"
XTool is created by Razor12911

Streams: 26/26
Time: 00:02:12 (00:00:04)
Memory: 391 MB (391 MB)

Results: 937 MB >> 776 MB
xtool precomp -c256mb -mbik "AC2Videos\"
XTool is created by Razor12911

Streams: 27/27
Time: 00:02:25 (00:00:06)
Memory: 677 MB (677 MB)

Results: 937 MB >> 725 MB


Test on 05_DAM folder from Shadow Warrior 3 Defintive Edition (6x Bink 2 videos).
xtool precomp -c256mb -mbk2 "SW3\Content\Movies\Cutscenes\05_DAM"
XTool is created by Razor12911

Streams: 6/6
Time: 00:01:06 (00:00:03)
Memory: 406 MB (406 MB)

Results: 214 MB >> 187 MB

How to use with DiskSpan GUI?

[Masquerade]

NPK2 Archives

Archives ending in .NPK and have a magic no. of NPK2.

Encrypted using Cipherfinal AES_256_CBC which isn't natively supported by XTool.

Examples of games using this scheme:


Tokyo Necro
Demonbane
Sonicomi


Usage

Place npk2.dll with XTool and use
-mnpk2:k###Key in Hexadecimal###+png+preflate -d2

Example for Tokyo Necro [Steam Version]

-mnpk2:k0xAAE616FB7DB7EE84ED424C474AADDAD69A8372799 AD576629601EF9B8938E20F+png+preflate -d2

A file is generated during precomp stage called npk2.bin. This must be retained for decoding as well as npk2.dll. npk2.bin contains the initialisation vectors for re-encrypting the data.

By Razor12911

[dixen]

oo2core_9_win64.dll v2.9.8 & v2.9.9

[Razor12911]

Star Ocean: The Divine Force

Supported games
Star Ocean: The Divine Force (requires libzstd.dll)

Usage
Place sodf.dll near xtool.exe along with required libraries and use -msodf+zstd (game is encrypted), so use -d1 as well

Notes

The library is required both when encoding and decoding.
zstd version 1.5.0 seems to work best
using two instances of xtool without -d1 is faster because some encrypted blocks are large and can only fit in 1 or 2 threads which cannot fully utilise the cpu leading to slower decoding

[Razor12911]

Plugin update available

Changes

- improved header decoder in unity plugin
- added ue5 method in unreal plugin to handle ucas/utoc file format without the need of using database creator

Results

Ghostwire Tokyo\Snowfall\Content\Paks\pakchunk0-WinGDK.ucas

-mue5:k0xC26CC5659A24AE8635585C8687DCE552A205DBD72D C4AA0CBE187C60D88921E2+kraken -d1
Streams: 39249 / 39250
Time: 00:07:54 (CPU 00:46:14)

Size: 4.84 GB >> 11.1 GB

I don't know how it compares with Unreal Engine ucas Database maker but this is an alternative for the newer ucas format as the database maker doesn't support it but this plugin does.

[Razor12911]

PlayStation archive

Supported games
PlayStation games?

Usage
Place psarc.dll near xtool.exe along with required libraries and use -mpsarc

Notes

If you plan on capturing the zlib streams in some of the psarc from The Last of Us, it's advisable to use two instances of xtool because the kraken streams are chunked meaning using -d1 will result in the zlib/reflate part of the precompression receiving incomplete streams.

core.psarc
2,11 GB >> 5.16 GB >> 7.53 GB

[Razor12911]

core.psarc
2,11 GB >> 5.16 GB >> 7.53 GB

Turns out I missed a few streams, R2 uploaded

core.psarc
2.12 GB >> 5.44 GB >> 7.82 GB

Source 2

Supported games
Half-Life: Alyx (requires liblz4.dll)

Usage
Place src2.ini near xtool.exe along with required libraries and use -msrc2

Notes

Sadly I didn't spend too much time on this game and there's really not a lot of lz4 streams and I just totally forgot to upload it after the disappointment. :D

[Masquerade]

TT Isle Of Man Ride on the Edge 3 [LZ4F]

Extract the contents of the attached archive and use -mTTIoM3.
TTIoM3.xtl can be discarded for decoding, but lz4.exe, xtool.ini and xdelta3_dll.dll must be present.
xtool precomp -mTTIoM3 -c128mb -t8 -df20p "WIN32\PKG\CHUNK_0"

XTool is created by Razor12911

Streams: 19100 / 19112
Time: 00:11:36 (CPU 00:03:19)

Size: 8.23 GB >> 13.4 GB

[kuyhaa]

TT Isle Of Man Ride on the Edge 3 [LZ4F]

Extract the contents of the attached archive and use -mTTIoM3.
TTIoM3.xtl can be discarded for decoding, but lz4.exe, xtool.ini and xdelta3_dll.dll must be present.
xtool precomp -mTTIoM3 -c128mb -t8 -df20p "WIN32\PKG\CHUNK_0"

XTool is created by Razor12911

Streams: 19100 / 19112
Time: 00:11:36 (CPU 00:03:19)

Size: 8.23 GB >> 13.4 GB

not work, what's wrong ?

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZURkzxS5B0iSBLuaRx1H3KMZcwEQL2lBbsp7soAB6lN Y7A8dQp2Q4QZSFUiwfrwlzr8Zhwz7uD1PvCL_MSLXUpUQttLb6 XH2K4sS_HflBvP1ncJl8EZ-4lyPilNjxOD0VFpdayMcwq4jY4KsBxJcGxvZo3tyEBjBX7wUfh BeRHHEszXwFDlKMsNatlg/s875/what%20wrong.png

[KaktoR]

Pretty sure you have to use a lz4 library aswell.

[Masquerade]

Pretty sure you have to use a lz4 library aswell.

LZ4 precompression is handled by lz4.exe in this case, plugged in via xtool.ini.

@kuyhaa
Are you trying an updated version? The database was created for the release version of the game.

[kuyhaa]

@kuyhaa
Are you trying an updated version? The database was created for the release version of the game.

i'm using TT Isle Of Man Ride on the Edge 3 update v20230511

if use lz4f:l10 :D

Streams: 5575 / 20181
Time: 00:00:32 (CPU 00:13:10)

Size: 8.23 GB >> 9.73 GB

[Masquerade]

In that case, you will need to make a new database.

You will need VGMToolbox, which you can download here (https://sourceforge.net/projects/vgmtoolbox/files/vgmtoolbox/VGMToolbox%20r1040/).

Copy this text (https://pastebin.com/raw/QBRtdaNj) and save it as PPKG.XML inside the "plugins\AdvancedCutter" folder.

Open the tool and go to Misc Tools ---> Extraction Tools ---> Generic ---> Advanced Cutter / Offset Finder.

In the presets drop down box, select PPKG from the list and then click "Load".

Then, drag the three PKG files into the drag'n'drop box to the left of the preset box.

After all the streams have been extracted, rename the extracted folders to remove the _CUT suffix.

To generate the database, just open a command prompt window in the folder containing the pkg files and do:

for %f in (*.PKG) do xtool generate -c128mb -t100p -mlz4f "%~nf" "%f" TTIoM3.xtl

Now you can delete the folders from VGMToolbox.

[Masquerade]

Divinity: Original Sin [Enhanced Edition]


Extract contents of attached archive and use -mDivinity -df20p.
Divinity.xtl can be discarded for decoding, but liblz4.dll and xdelta3_dll.dll must be present.

Divinity: Original Sin 2 [Definitive Edition]


Extract contents of attached archive and use -mDivinity2 -df20p.
Divinity2.xtl can be discarded for decoding, but liblz4.dll and xdelta3_dll.dll must be present.

[kuyhaa]

In that case, you will need to make a new database.

You will need VGMToolbox, which you can download here (https://sourceforge.net/projects/vgmtoolbox/files/vgmtoolbox/VGMToolbox%20r1040/).

Copy this text (https://pastebin.com/raw/QBRtdaNj) and save it as PPKG.XML inside the "plugins\AdvancedCutter" folder.

Open the tool and go to Misc Tools ---> Extraction Tools ---> Generic ---> Advanced Cutter / Offset Finder.

In the presets drop down box, select PPKG from the list and then click "Load".

Then, drag the three PKG files into the drag'n'drop box to the left of the preset box.

After all the streams have been extracted, rename the extracted folders to remove the _CUT suffix.

To generate the database, just open a command prompt window in the folder containing the pkg files and do:

for %f in (*.PKG) do xtool generate -c128mb -t100p -mlz4f "%~nf" "%f" TTIoM3.xtl

Now you can delete the folders from VGMToolbox.

works, thanks :D

Streams: 19102 / 19113
Time: 00:02:28 (CPU 00:00:55)

Size: 8.23 GB >> 13.4 GB

[soulfit15]

looks like a bug in xtool GUI with unreal plugin
XTool is created by Razor12911

Streams: 1430889 / 1675133
Time: 00:21:48 (CPU 00:16:56)

Size: 5.25 GB >> 5.03 GB


unpacking a game with ue5:key and it gets stuck "used the -d1" and cpu keeps spooling at 16%

[Masquerade]

Orcs Must Die! 3 database updated for game version Build 11248349 / v1.2.1.0 (June 5th 2023).

[kj911]

Bink Video


Extract the attached archive to the XTool folder and use -mbik or -mbk2 depending on your data set.



Whats wrong or am i lame?? (From testing. See two attached pictures.)

Directly use Binkpack.exe or Freearc usaged whole Bink*.exe file and or CLS-based (de)compressors its works in (de)compress in vice versa... Packed 3.07GB huge BIK file correctly to 2.4GB size with Binkpack.
Use Xtool 0.71 directly or Arc with configurations, don't compress smaller.

[KaktoR]

Letting you know here that the playstation archive plugin doesn't work with xtool 0.7.2(HF) on decompression step.

It works with xtool 0.7.1

[Masquerade]

[Proof of Concept / Work In Progress] Yakuza SLLZ v1

This is a little project I've been working on in an attempt to implement the SLLZ algorithm found in Yakuza games as an XTool plugin. The code used for compression and decompression was written by Kaplas80. The attempt was to make it into a standalone executable. I wanted to share it here despite being unfinished. Requires .NET Framework v4.8.

There are a few problems with the plugin in its current state. Decompression of sllz streams seems to work completely fine. However, compression is slow and is missing the 2x 4 bytes of header information containing the compressed size and decompressed size (because for some reason Visual Studio insists on this being 8 byte rather than 4 byte arrays). They are filled as 00 bytes instead for the time being. The compressor is also bugged in other ways when I tested on samples from Yakuza Kiwami. Reading and writing to stdin/stdout hasn't been added yet so it's even slower since data needs to be extracted to disk.

Here's some information about this algorithm from its header:


First 4 bytes, Magic: SLLZ
Next byte, endianness: 00 (little)
Next byte, version: 01 or 02 (compression version - v2 is zlib I think)
Next 2 bytes is header size (from what I've seen, always 0x1000 because the header is always 16 bytes in my findings)
Next 4 bytes is decompressed size.
Next 4 bytes is compressed size.

Total: 16 bytes
[Stream1]
Name=yakuza
Codec=sllz
BigEndian=0
Signature=0x5A4C4C53
Structure=Signature(4),Endianness(1),Version(1),He aderSize(2),CSize(4),DSize(4),Stream
StreamOffset=-16
CompressedSize=CSize
DecompressedSize=DSize
xtool precomp -myakuza -df20p -c128mb -t12 hact.par
XTool is created by Razor12911

Streams: 873 / 3397
Time: 00:11:34 (CPU 00:00:17)

Size: 398 MB >> 404 MB

I will probably drop this project for a bit and return to it sometime in the future :D

[dixen]

Witcher 3: Complete

Used xtool v0.7.3 hotfix

-mzlib+reflate

Streams: 2786 / 2786
Time: 00:01:17 (CPU 00:04:24)
Duplicates: 300 (1.58 MB) [1.80 MB >> 19.1 MB]
Srep decompression memory: 3.00 MB [6.34 MB*]

Size: 508 MB >> 658 MB >> 638 MB >> 607 MB >> 400 MB

100%
Errorlevel=0

Compressed 1 file, 532,191,280 => 419,520,814 bytes. Ratio 78.83%
Compression time: cpu 0.53 sec/real 79.37 sec = 1%. Speed 6.71 mB/s
All OK

-mw3+zlib+reflate

Streams: 3490 / 3854
Time: 00:01:35 (CPU 00:05:40)
Duplicates: 476 (2.20 MB) [3.18 MB >> 226 MB]
Srep decompression memory: 7.00 MB [23.1 MB*]

Size: 508 MB >> 1.12 GB >> 916 MB >> 814 MB >> 303 MB

100%
Errorlevel=0

Compressed 1 file, 532,191,280 => 318,049,218 bytes. Ratio 59.76%
Compression time: cpu 0.47 sec/real 97.06 sec = 0%. Speed 5.48 mB/s
All OK

Needed libliz4.dll (v1.9.3) and doboz.dll near xtool.exe

[Razor12911]

Plugin update available

Changes

- Improved stream detection in unreal engine plugin's ue5 codec (ucas/utoc format)

Notes

I just wished the format of unreal engine file system would stop changing because it's a headache trying to write one library that supports all current and future games and even this plugin has a caveat, mainly the first and last stream left behind and not decrypted because if I tried to grab them using the current method that would mean false positives, I've left R9 just in case R10's detection is worse but good luck.

pakchunk0optional-Windows.ucas (I don't know what game is this but it was a sample I was sent a while ago)

Unreal Engine ucas database maker doesn't work on this file... because they changed they format...

-mue5:k0xCD353CBF6E250AF8ADF4B2B0F79649A27D2BC743AB 0AAAEB202DEF7A7C19B2DB+kraken -d1

R9
Tested 1 file, 437,937,960 => 290,461,744 bytes. Ratio 150.77%
Testing time: cpu 0.19 sec/real 11.54 sec = 2%. Speed 25.17 mB/s

R10
Tested 1 file, 578,518,003 => 290,461,744 bytes. Ratio 199.17%
Testing time: cpu 0.25 sec/real 16.30 sec = 2%. Speed 17.82 mB/s

[Masquerade]

The Legend of Heroes Trails into Reverie


Extract the below archive into XTool folder and use -mReverie.
This database is for assets.pka. The other pkg files are not compressed.
Requires liblz4.dll. I included one that gave a good ratio.
You can discard Reverie.xtl for decoding, but liblz4.dll must be retained.

This database took 13 hours to generate :p

[Razor12911]

Plugin update available

Changes

- Improved stream detection in unreal engine plugin's ue5 codec (ucas/utoc format)

Notes

Using ue5 codec even on pak files yielded streams which should not have been detected meaning ratio is affected one way or another and this release tackles false positive detections.

[Razor12911]

Plugin update available

Changes

- ue3 codec no longer requires m# parameter to work
- ue3 codec supports lzopro streams (requires lzopro executable plugin)
- ue4 codec speed improvement
- ue4 codec improved detection
- ue5 codec minor bug fixes

Notes

This update is primarily made for the next xtool release (0.7.7) but it should work fine for older versions.

Link (https://fileforums.com/showpost.php?p=489801&postcount=25)

[Razor12911]

Insomniac Engine

Supported games
Marvel's Spider-Man Remastered
Marvel's Spider-Man: Miles Morales
Ratchet & Clank: Rift Apart

Usage
Place insomniac.ini near xtool.exe along with required libraries and use -minsomniac

Notes
Tested on small sample provided by dixen for testing, not sure if it works for the whole game. This is the first advanced configuration plugin so also not sure if this feature in xtool works without bugs.

[Cesar82]

@Razor12911, Thanks for the updates.
As always, once again I fill the topic of questions, but here we go..

1) Can I continue to use the m1, m2, and m3 parameters so the DiskSpan GUI knows what to copy into the xtool folder?
Unreal_Engine3_MK11; Method: xtool:mue3,m1;
Unreal_Engine3_MELE; Method: xtool:mue3,m2;
Unreal_Engine3_LZO; Method: xtool:mue3,m3;

2) Could you share which library and version is used for each of the 3 games supported by the new "Insomniac Engine" plugin?

[Razor12911]

Plugin update available

Changes

- Added support for DirectStorage (gdeflate) streams in Insomniac plugin

Notes

Turns out, not only lz4 was used in this game, after investigations made by Fitgirl, she pointed out the plugin was not working in the initial release and after a few mins of research I managed to create a plugin (gdeflate) to add support for the other streams compressed using the Direct Storage technology and as a result of that, to precompress Ratchet & Clank: Rift Apart, you need dstorage.dll, dstoragecore.dll, gdeflate.dll and liblz4.dll.

tex_char_ratchet0 (sample compressed with gdeflate)
-mxtool:insomniac:rcrads
Compressed 1 file, 135,778,928 => 325,814,638 bytes. Ratio 239.96%
Compression time: cpu 0.11 sec/real 1.36 sec = 8%. Speed 99.86 mB/s

IMPORTANT
With the use of DirectStorage, compression and/or installation will only work on Windows 10 or later, Anything older will not work.

@Cesar82

1) Yes
2) It's the same one that comes with xtool (v1.8.3), a lot of games use this version.

[dixen]

Decompression results

Insomniac R1
39 gb > 48 gb

Insomniac R2
39 gb > 63 gb

[KaktoR]

Plugin update available

Changes

- ue3 codec no longer requires m# parameter to work
- ue3 codec supports lzopro streams (requires lzopro executable plugin)
- ue4 codec speed improvement
- ue4 codec improved detection
- ue5 codec minor bug fixes

Notes

This update is primarily made for the next xtool release (0.7.7) but it should work fine for older versions.

Link (https://fileforums.com/showpost.php?p=489801&postcount=25)

Thanks for the update, although it seems that some streems were skipped for ue3+lzopro

Streams: 373 / 1776
Time: 00:00:04 (CPU 00:00:01)

Size: 67.2 MB >> 88.9 MB

[DODI]

Insomniac Engine

Supported games
Marvel's Spider-Man Remastered
Marvel's Spider-Man: Miles Morales
Ratchet & Clank: Rift Apart

Usage
Place insomniac.ini near xtool.exe along with required libraries and use -minsomniac

Notes
Tested on small sample provided by dixen for testing, not sure if it works for the whole game. This is the first advanced configuration plugin so also not sure if this feature in xtool works without bugs.

Thanks :)

Precompression result

Insomniac R2
38.3 gb to 64.5 gb

[Razor12911]

Plugin update available

Changes

- Updated Unreal Engine (https://fileforums.com/showpost.php?p=489801&postcount=25) plugin
- Fixed hanging issue in PlayStation archive (https://fileforums.com/showpost.php?p=500982&postcount=352) plugin

[Razor12911]

Starfield

Supported games
Starfield (requires liblz4.dll)

Usage
Place starfield.dll near xtool.exe along with required libraries and use -mstarfield+reflate (plugin only precompresses lz4 streams)

Notes

liblz4.dll that comes with xtool seems to work just fine (v183).
You will need to use reflate/preflate, the game uses optimised zlib, zlib-ng to be specific and that makes the zlib method not to work.

[Razor12911]

lzopro

Supported games

To be used by old anvil and unreal engine 3 games

Notes

I forgot to post this a while ago but here anyways,

Edit xtool.ini and use -anvil for anvil games and -unreal for unreal engine 3 games while specifying compression level, where level 0 uses lzopro_lzo1x_1_14_compress and level 1..10 uses lzopro_lzo1x_99_compress function for compression.

If level 0 and mostly 6..10 does not work on precompression then you should use uelr as the streams are compressed in such a way that they rely on the work memory buffer left behind by the previously compressed stream (to improve on compression). The streams are compressed in sequence, however it's this improvement that makes it impossible for xtool to work with such streams as because xtool breaks this sequence when it precompresses the streams using multiple threads and if you tried to brute force the work memory to work or if you use xdelta to patch the streams, you will get crc errors.

[Masquerade]

Updated Distant Worlds 2 database for game version v1.1.6.7.

[KaktoR]

Bink Video


XTool plugin adding support for Bink Video compression.
This plugin uses Bink Pack library by ProFrager.
Thanks to Razor12911 for the little help.

Advantages of using this plugin versus vanilla Bink Pack

Multithreading support.
No CRC check fail since XTool will patch the videos as they are unpacked.
Incompatible Bink 2 videos (KB2n) are automatically ignored.

Special Notes

For Bink 2 videos, you may need to make use of the --diff=* parameter as some patches between CRC imperfect videos to the original state can be larger than XTool's default threshold for patches.
You may need to use a large chunk size to ensure all videos are processed. See example below.

Example Usage

Extract the attached archive to the XTool folder and use -mbik or -mbk2 depending on your data set.



Updated plugin.

Basically the 'MessageBoxA' was patched out and replaced with some other function from User32.dll. On some bink2 inputs you had encountered some error messages like 'MotionData_warn' or 'err'. I have included two samples for which it is the case. You shouldn't receive error messages anymore with this patched plugin. Inputs which would give these types of errors are just ignored but processed still, but not compressed with binkpack.

Thanks to masquerade and Razor12911.

[Masquerade]

Forza Motorsport 2023

This game uses XMemCompress in its paks (as denoted by Method: 21).

I didn't download the full game, but here's how to make your own database:

1. Download the attached archive. This contains bms2xtl config and required script as well as XCompress plugin for XTool.

2. Extract the archive, and place the following files into the Forza Motorsport root folder:


bms2xtl.dll
bms2xtl.exe
bms2xtl.ini
quickbms_4gb_files.exe
zip.bms


3. Open command prompt in the root of the Forza Motorsport game folder and paste this command:

for /r %f in (*.zip) do bms2xtl zip.bms "%f" ForzaMotorsport.xtl

4. Once the database is made, use that along with the xcompress plugin supplied to precompress the game data:

CMD> xtool precomp -mForzaMotorsport body01_forzafaithful.zip

XTool is created by Razor12911

Streams: 4 / 4
Time: 00:00:05 (CPU 00:00:00)

Size: 17.0 MB >> 32.0 MB

If you make a full database, consider sharing it here.

[Razor12911]

The config version for the plugin above

I've included 64-bit version of xmemcompress, I compiled it a while ago but I didn't post it here.

Forza Motorsport\media\base\Cinematics.zip

xmemcompress -mfmzip:
Compressed 1 file, 260,317,756 => 1,146,247,029 bytes. Ratio 440.33%
Compression time: cpu 0.08 sec/real 37.12 sec = 0%. Speed 7.01 mB/s

xmemcompress64 -mfmzip:
Compressed 1 file, 260,317,756 => 1,146,247,031 bytes. Ratio 440.33%
Compression time: cpu 0.00 sec/real 28.70 sec = 0%. Speed 9.07 mB/s

It should also work for Forza Motorsport 7,

the older side project can be found here (https://fileforums.com/showthread.php?t=102675)

[Razor12911]

Tony Hawk's Pro Skater 1+2

This only removes the custom encryption used in the game (Not tested)

Drag and drop the game folder and xtool will remove the encryption and give you the decrypted pak files in output folder which will be your new game files, use -mzlib on them for precompression.

Base\Content\Paks\pakchunk1-WindowsNoEditor.pak
Compressed 1 file, 291,695,622 => 503,712,687 bytes. Ratio 172.68%
Compression time: cpu 0.05 sec/real 2.30 sec = 2%. Speed 126.91 mB/s

This is not tested and will all depend on your luck.

[dixen]

Test pakchunk6-WindowsNoEditor.pak

Compressing 1 file, 630,211,718 bytes
Compressing pakchunk6-WindowsNoEditor.pak 58%
Compressing 630,211,718 bytes with xtool1.exe precomp -mthps12+zlib -d3 -c128mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 46582 / 47129
Time: 00:00:14 (CPU 00:01:53)

Size: 601 MB >> 1.40 GB

100%
Errorlevel=0

Compressed 1 file, 630,211,718 => 1,503,292,444 bytes. Ratio 238.54%
Compression time: cpu 0.28 sec/real 20.72 sec = 1%. Speed 30.41 mB/s
All OK

E:\RePack_tools\xtool_0.7.7\Freearc>arc t data.arc
FreeArc 0.67 (March 15 2014) testing archive: data.arc
Testing 1 file, 630,211,718 bytes. Processed 99%WARNING: CRC failed in "pakchunk6-WindowsNoEditor.pak". File is broken.
Tested 1 file, 1,503,292,444 => 630,211,718 bytes. Ratio 238.54%
Testing time: cpu 0.20 sec/real 6.08 sec = 3%. Speed 103.62 mB/s
There were 1 warning(s)

With decrypting PAK - game doesn't launch

[Razor12911]

Welp
https://media.giphy.com/media/qN8mwUPAt7gByge3W1/giphy.gif

[Masquerade]

Test pakchunk6-WindowsNoEditor.pak

ofc CRC error because this library decrypts paks only, not re-encrypts them. Re-encryption of this game is not possible.

[hwang4]

Asterigos: Curse of the Stars

Place asterigos.xtl and asterigos.key in the same folder as XTool, use "-masterigos+preflate -d1".



Masquerade , tried with this one , but cannot get a good compression ratio !thx for help!:)

[Hexagon123]

Masquerade , tried with this one , but cannot get a good compression ratio !thx for help!:)

Looks like this needs an update?

[Masquerade]

DB generator no longer works.

[Masquerade]

XNB
[Stream1]
Name=xnb
Codec=xmemcompress
BigEndian=0
Signature=0x77424E58
Structure=Signature(4),XNBFormatVersion(1),Compres sed(1),FileSize(4),DSize(4),Stream
CompressedSize=FileSize-14
DecompressedSize=DSize
Condition1=Compressed = 0x80

Requires XCompress plugin + correct configuration in xtool.ini.

[Redmist2013]

Tony Hawk's Pro Skater 1+2

This only removes the custom encryption used in the game (Not tested)

Drag and drop the game folder and xtool will remove the encryption and give you the decrypted pak files in output folder which will be your new game files, use -mzlib on them for precompression.

Base\Content\Paks\pakchunk1-WindowsNoEditor.pak
Compressed 1 file, 291,695,622 => 503,712,687 bytes. Ratio 172.68%
Compression time: cpu 0.05 sec/real 2.30 sec = 2%. Speed 126.91 mB/s

This is not tested and will all depend on your luck.

Interesting. It seems to decrypt okay but re-encryption only seems to partially work.

xtool decode pakchunk1-WindowsNoEditor.pak.xt pakchunk1-WindowsNoEditor.pak

...results in about 20% of the bytes in the file matching those in the original file using a binary file comparer.
Fitgirl seems to have made a custom THPS12 encryption tool which gets close and then uses hdiff to fix things up.

[Masquerade]

^^
Did you even read Razor's post?

"This only removes the custom encryption used in the game (Not tested)"

Ofc decode isn't sopprted!

[kuyhaa]

hi, anyone have plugin or other method for Megaton Musashi W ? i try use unity plugin for *.bundle file but not work

thanks

[KaktoR]

Sample

[kuyhaa]

Sample

this is bro

https://pixeldrain.com/u/Ng8j3XAs

[KaktoR]

Either not compressed, crypted or something else. There is no unityfs header, so maybe this is the problem.

[kuyhaa]

hi, anyone have plugin or database creator The Legend of Heroes: Trails through Daybreak

.p3a files

sample


https://pixeldrain.com/u/sFp3YTHU

[KaktoR]

I will quote masquerade, maybe he will tell you how to process this.

A game came out called The Legend of Heroes Trails through Daybreak. They used ZSTD compression on the archives yet because they used dictionary based zstd, xtool can't detect the streams let alone process them.

[Masquerade]

^^
These p3a archives are made with dictionary ZSTD just like Zelda Tears of the Kingdom files.

You can rebuild the archives using p3a_tool and decide for no compression. If you decide on the zstd compression with p3a_tool, you still need to brute force a stream to figure out what version and settings for ZSTD were used.

I think there is more than one dictionary, they can be dumped from memory at runtime but that's about as far as I have got.

[Masquerade]

I cannot believe I was unbelievably stupid to not check the archives themselves for the dictionary. After looking more closely at the syntax for the p3a_tool, there is the option "info". So:

CMD> p3a_tool.exe info scene.p3a

PH3 archive scene.p3a: 916 files, v1200
Extended header available: 16 bytes, 304 bytes per index entry
compression stats: none:0 lz4:0 zStd:0 zStdDict:916
compressed size: 89239105 (15%)
uncompressed size: 578417191
Has dictionary, 112640 bytes

From this info, we learn that the dictionary used in the archive scene.p3a is 112640 bytes in size.
The ZSTD dictionary compression format (https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format) tells us that the magic number for a ZSTD dictionary is 0xEC30A437 (little endian).

So, by searching for this magic number in HXD, sure enough we have our dictionary beginning at location 0x44000 for scene.p3a. Next, select a block and set the length to the dictionary length from earlier - 112640 bytes.

Copy the select bytes and dump them to a new file, I called it scene.zdict.

Now, if you unpack scene.p3a with:

CMD> p3a_tool extract scene.p3a

and then run:

CMD> p3a_tool archive scene.MSQ scene --comp=zstd --dictionary=scene.zdict --

We yield a CRC identical P3A archive:

File: scene.MSQ
CRC-32: 04492de2
MD5: b2c8a40571e8673ed3682df82544281d
SHA-1: 95d40b6e693641ef8c7692531370fecd0378290c
BLAKE3: 79c13b93a662d4789949c228e116a52ac10a47d91562bd3902 3a61cabd09d0ca

File: scene.p3a
CRC-32: 04492de2
MD5: b2c8a40571e8673ed3682df82544281d
SHA-1: 95d40b6e693641ef8c7692531370fecd0378290c
BLAKE3: 79c13b93a662d4789949c228e116a52ac10a47d91562bd3902 3a61cabd09d0ca



So there you go, that's how to repack this game.

[kuyhaa]

i have run HxD application and open the file, but i didn't understand


So, by searching for this magic number in HXD, sure enough we have our dictionary beginning at location 0x44000 for scene.p3a. Next, select a block and set the length to the dictionary length from earlier - 112640 bytes.

Copy the select bytes and dump them to a new file, I called it scene.zdict.


:confused:

by the way, thanks Masquerade for the information

[Masquerade]

^^
In HXD, CTRL+F to search for "37 A4 30 EC" as shown in the image below:

https://i.postimg.cc/BZkH71gj/Untitled.png

Why are they reversed? Endianness.

When you find your result, place your cursor on the "37" byte, change "hex" to "dec" at the top of the screen and press CTRL+E to select a block of data.

Select the radio button for length and paste the length of the dictionary in bytes that you found from the p3a_tool info.

You will now get a selected section of bytes. CTRL+C to copy and then CTRL+N to open a new file. CTRL+V to paste these bytes (they will appear in red) and CTRL+S to save the file. The bytes will go black when saved.

[kuyhaa]

^^
In HXD, CTRL+F to search for "37 A4 30 EC" as shown in the image below:

https://i.postimg.cc/BZkH71gj/Untitled.png

Why are they reversed? Endianness.

When you find your result, place your cursor on the "37" byte, change "hex" to "dec" at the top of the screen and press CTRL+E to select a block of data.

Select the radio button for length and paste the length of the dictionary in bytes that you found from the p3a_tool info.

You will now get a selected section of bytes. CTRL+C to copy and then CTRL+N to open a new file. CTRL+V to paste these bytes (they will appear in red) and CTRL+S to save the file. The bytes will go black when saved.

found it, COOL Masquerade

[Masquerade]

The Legend of Heroes: Trails of Cold Steel III [v1.07] [LZ4HC]


Extract into XTool folder and use -mLoHToCSIII

xtool precomp -mLoHToCSIII I_MG08_CARD.pkg
XTool is created by Razor12911

Version 0.8.9

Streams: 2 / 2
Time: 00:00:11 (CPU 00:00:11)

Size: 15.2 MB >> 85.3 MB

The Legend of Heroes: Trails into Reverie [v1.1.5] [LZ4HC]


Extract into XTool folder and use -mReverie

xtool precomp -mReverie I_MG08_CARD.pkg
XTool is created by Razor12911

Version 0.8.9

Streams: 4 / 4
Time: 00:00:02 (CPU 00:00:05)

Size: 16.2 MB >> 48.0 MB

Merry Christmas ;)