XTool 2020 (Plugins)

[Cesar82]

@Razor12911, Thanks for the updates.
As always, once again I fill the topic of questions, but here we go..

1) Can I continue to use the m1, m2, and m3 parameters so the DiskSpan GUI knows what to copy into the xtool folder?

Unreal_Engine3_MK11;         Method: xtool:mue3,m1;
Unreal_Engine3_MELE;         Method: xtool:mue3,m2;
Unreal_Engine3_LZO;          Method: xtool:mue3,m3;

2) Could you share which library and version is used for each of the 3 games supported by the new "Insomniac Engine" plugin?

[KaktoR]

Originally Posted by Razor12911

Plugin update available

Changes

- ue3 codec no longer requires m# parameter to work
- ue3 codec supports lzopro streams (requires lzopro executable plugin)
- ue4 codec speed improvement
- ue4 codec improved detection
- ue5 codec minor bug fixes

Notes

This update is primarily made for the next xtool release (0.7.7) but it should work fine for older versions.

Link

Thanks for the update, although it seems that some streems were skipped for ue3+lzopro

Streams: 373 / 1776
Time: 00:00:04 (CPU 00:00:01)

Size: 67.2 MB >> 88.9 MB

[Razor12911]

Insomniac Engine

Supported games
Marvel's Spider-Man Remastered
Marvel's Spider-Man: Miles Morales
Ratchet & Clank: Rift Apart

Usage
Place insomniac.ini near xtool.exe along with required libraries and use -minsomniac

Notes
Tested on small sample provided by dixen for testing, not sure if it works for the whole game. This is the first advanced configuration plugin so also not sure if this feature in xtool works without bugs.

[DODI]

Originally Posted by Razor12911

Insomniac Engine

Supported games
Marvel's Spider-Man Remastered
Marvel's Spider-Man: Miles Morales
Ratchet & Clank: Rift Apart

Usage
Place insomniac.ini near xtool.exe along with required libraries and use -minsomniac

Notes
Tested on small sample provided by dixen for testing, not sure if it works for the whole game. This is the first advanced configuration plugin so also not sure if this feature in xtool works without bugs.

Thanks

Precompression result

Insomniac R2
38.3 gb to 64.5 gb

[Razor12911]

Plugin update available

Changes

- Added support for DirectStorage (gdeflate) streams in Insomniac plugin

Notes

Turns out, not only lz4 was used in this game, after investigations made by Fitgirl, she pointed out the plugin was not working in the initial release and after a few mins of research I managed to create a plugin (gdeflate) to add support for the other streams compressed using the Direct Storage technology and as a result of that, to precompress Ratchet & Clank: Rift Apart, you need dstorage.dll, dstoragecore.dll, gdeflate.dll and liblz4.dll.

tex_char_ratchet0 (sample compressed with gdeflate)
-mxtool:insomniac:rcrads

Compressed 1 file, 135,778,928 => 325,814,638 bytes. Ratio 239.96%
Compression time: cpu 0.11 sec/real 1.36 sec = 8%. Speed 99.86 mB/s

IMPORTANT
With the use of DirectStorage, compression and/or installation will only work on Windows 10 or later, Anything older will not work.

@Cesar82

1) Yes
2) It's the same one that comes with xtool (v1.8.3), a lot of games use this version.

[dixen]

Decompression results

Insomniac R1
39 gb > 48 gb

Insomniac R2
39 gb > 63 gb

[Razor12911]

Plugin update available

Changes

- Updated Unreal Engine plugin
- Fixed hanging issue in PlayStation archive plugin

[Razor12911]

Starfield

Supported games
Starfield (requires liblz4.dll)

Usage
Place starfield.dll near xtool.exe along with required libraries and use -mstarfield+reflate (plugin only precompresses lz4 streams)

Notes

liblz4.dll that comes with xtool seems to work just fine (v183).
You will need to use reflate/preflate, the game uses optimised zlib, zlib-ng to be specific and that makes the zlib method not to work.

[Razor12911]

lzopro

Supported games

To be used by old anvil and unreal engine 3 games

Notes

I forgot to post this a while ago but here anyways,

Edit xtool.ini and use -anvil for anvil games and -unreal for unreal engine 3 games while specifying compression level, where level 0 uses lzopro_lzo1x_1_14_compress and level 1..10 uses lzopro_lzo1x_99_compress function for compression.

If level 0 and mostly 6..10 does not work on precompression then you should use uelr as the streams are compressed in such a way that they rely on the work memory buffer left behind by the previously compressed stream (to improve on compression). The streams are compressed in sequence, however it's this improvement that makes it impossible for xtool to work with such streams as because xtool breaks this sequence when it precompresses the streams using multiple threads and if you tried to brute force the work memory to work or if you use xdelta to patch the streams, you will get crc errors.

[Masquerade]

Forza Motorsport 2023

This game uses XMemCompress in its paks (as denoted by Method: 21).

I didn't download the full game, but here's how to make your own database:

1. Download the attached archive. This contains bms2xtl config and required script as well as XCompress plugin for XTool.

2. Extract the archive, and place the following files into the Forza Motorsport root folder:

• bms2xtl.dll
• bms2xtl.exe
• bms2xtl.ini
• quickbms_4gb_files.exe
• zip.bms

3. Open command prompt in the root of the Forza Motorsport game folder and paste this command:

for /r %f in (*.zip) do bms2xtl zip.bms "%f" ForzaMotorsport.xtl

4. Once the database is made, use that along with the xcompress plugin supplied to precompress the game data:

CMD> xtool precomp -mForzaMotorsport body01_forzafaithful.zip

XTool is created by Razor12911

Streams: 4 / 4
Time: 00:00:05 (CPU 00:00:00)

Size: 17.0 MB >> 32.0 MB

If you make a full database, consider sharing it here.

[Razor12911]

The config version for the plugin above

I've included 64-bit version of xmemcompress, I compiled it a while ago but I didn't post it here.

Forza Motorsport\media\base\Cinematics.zip

xmemcompress -mfmzip:

Compressed 1 file, 260,317,756 => 1,146,247,029 bytes. Ratio 440.33%
Compression time: cpu 0.08 sec/real 37.12 sec = 0%. Speed 7.01 mB/s

xmemcompress64 -mfmzip:

Compressed 1 file, 260,317,756 => 1,146,247,031 bytes. Ratio 440.33%
Compression time: cpu 0.00 sec/real 28.70 sec = 0%. Speed 9.07 mB/s

It should also work for Forza Motorsport 7,

the older side project can be found here

[Razor12911]

Tony Hawk's Pro Skater 1+2

This only removes the custom encryption used in the game (Not tested)

Drag and drop the game folder and xtool will remove the encryption and give you the decrypted pak files in output folder which will be your new game files, use -mzlib on them for precompression.

Base\Content\Paks\pakchunk1-WindowsNoEditor.pak

Compressed 1 file, 291,695,622 => 503,712,687 bytes. Ratio 172.68%
Compression time: cpu 0.05 sec/real 2.30 sec = 2%. Speed 126.91 mB/s

This is not tested and will all depend on your luck.

[Redmist2013]

Originally Posted by Razor12911

Tony Hawk's Pro Skater 1+2

This only removes the custom encryption used in the game (Not tested)

Drag and drop the game folder and xtool will remove the encryption and give you the decrypted pak files in output folder which will be your new game files, use -mzlib on them for precompression.

Base\Content\Paks\pakchunk1-WindowsNoEditor.pak

Compressed 1 file, 291,695,622 => 503,712,687 bytes. Ratio 172.68%
Compression time: cpu 0.05 sec/real 2.30 sec = 2%. Speed 126.91 mB/s

This is not tested and will all depend on your luck.

Interesting. It seems to decrypt okay but re-encryption only seems to partially work.

xtool decode pakchunk1-WindowsNoEditor.pak.xt pakchunk1-WindowsNoEditor.pak

...results in about 20% of the bytes in the file matching those in the original file using a binary file comparer.
Fitgirl seems to have made a custom THPS12 encryption tool which gets close and then uses hdiff to fix things up.

[Masquerade]

^^
Did you even read Razor's post?

"This only removes the custom encryption used in the game (Not tested)"

Ofc decode isn't sopprted!

[dixen]

Test pakchunk6-WindowsNoEditor.pak

Compressing 1 file, 630,211,718 bytes
Compressing pakchunk6-WindowsNoEditor.pak 58%
Compressing 630,211,718 bytes with xtool1.exe precomp -mthps12+zlib -d3 -c128mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 46582 / 47129
Time: 00:00:14 (CPU 00:01:53)

Size: 601 MB >> 1.40 GB

100%
Errorlevel=0

Compressed 1 file, 630,211,718 => 1,503,292,444 bytes. Ratio 238.54%
Compression time: cpu 0.28 sec/real 20.72 sec = 1%. Speed 30.41 mB/s
All OK

E:\RePack_tools\xtool_0.7.7\Freearc>arc t data.arc
FreeArc 0.67 (March 15 2014) testing archive: data.arc
Testing 1 file, 630,211,718 bytes. Processed 99%WARNING: CRC failed in "pakchunk6-WindowsNoEditor.pak". File is broken.
Tested 1 file, 1,503,292,444 => 630,211,718 bytes. Ratio 238.54%
Testing time: cpu 0.20 sec/real 6.08 sec = 3%. Speed 103.62 mB/s
There were 1 warning(s)

With decrypting PAK - game doesn't launch