XTool 2020 (Main Project)

[Razor12911]

Originally Posted by elit

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

there are several implementations of the deflate algorithm, it's very flexible. Think of using lzma to compress something, obviously there are compression levels that it comes with 1-9. -mzlib implementation is based around this idea that a user who uses this would have used a common setting such as compression level so for precompression it will decompress the data and then try to recompress it using these common levels, but of course lzma has methods that people use for other purposes like for stronger compression such as "lzma:ultra:a1:mfbt4:d158m: fb273:mc1000000000:lc8" mzlib cannot process such, similarly to deflate streams in png images as the way deflate set in such a way that it's good compressing pixels of an image but also make it quicker for loading/viewing. Reflate and preflate have no problem processing these as they used an advanced method for precompressing them.

You might say that mzlib is therefore useless, actually it still has its uses. mzlib is faster than both reflate and preflate because of its simple implementation and incurring no overhead furthermore mzlib produces better results because there is no additional header information file produced hence why I've preached several times to mix both zlib with reflate or preflate, if zlib fails to process the stream it just passes the stream to reflate/preflate for it to process it.

mzlib:
Compressed 1 file, 86,347,072 => 444,743,922 bytes. Ratio 515.07%
Compression time: cpu 0.16 sec/real 13.37 sec = 1%. Speed 6.46 mB/s

mreflate,l9:
Compressed 1 file, 86,347,072 => 445,179,589 bytes. Ratio 515.57%
Compression time: cpu 0.14 sec/real 14.33 sec = 1%. Speed 6.03 mB/s

mpreflate:
Compressed 1 file, 86,347,072 => 366,023,129 bytes. Ratio 423.90%
Compression time: cpu 0.14 sec/real 10.25 sec = 1%. Speed 8.42 mB/s

mzlib+reflate,l9:
Compressed 1 file, 86,347,072 => 444,941,384 bytes. Ratio 515.29%
Compression time: cpu 0.06 sec/real 11.74 sec = 1%. Speed 7.35 mB/s

mzlib+preflate:
Compressed 1 file, 86,347,072 => 444,744,743 bytes. Ratio 515.07%
Compression time: cpu 0.11 sec/real 11.23 sec = 1%. Speed 7.69 mB/s

As you can see, zlib perform faster when alone but better when you hook it up with reflate/preflate and preflate fails to process some streams. I've given people options, all up to them how they use xtool.

Originally Posted by elit

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

Attachment 31395

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
Attachment 31394

PS(could it be due to window size - 48k ?)

use --verbose mode to see which levels are producing the closest result to the original stream.

[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 17101) using l1 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16535) using l2 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16351) using l3 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15784) using l4 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15711) using l5 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15493) using l6 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l7 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l8 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15444) using l9 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15442) using l10 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15438) using l11 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15437) using l12 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15428) using l13 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14817) using l14 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14733) using l15 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14689) using l16 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l17 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l18 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l20 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l21 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l22 has failed

level 17 is close but level 19 is commonly used, then set -mzstd,l19

[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] - Patched stream at 0000000000000000 (14692 >> 14688) [28] successfully

Compressed 1 file, 14,692 => 48,239 bytes. Ratio 328.34%
Compression time: cpu 0.00 sec/real 0.55 sec = 0%. Speed 0.03 mB/s

Used libzstd130.dll

[elit]

Originally Posted by Razor12911

level 17 is close but level 19 is commonly used, then set -mzstd,l19

Thank you Razor! But wait, why it did not tried that level automatically? Is xtool trying only some levels here as well? Like up to 9, maybe due to speed?

If that is the case, may I suggest that it try to do all of them - up to certain mb of data(user defined). For instance first 100-200mb all levels and if nothing found drop to only speedier levels as is now etc.?

[Razor12911]

Dude it tried all levels, look at the log via verbose (level 1-22) and none of them produced crc perfect stream and you can even see that with level 19 the original stream was 14692 which was decompressed to 48079 and recompressed to 14688.

Xtool applies patches using xdelta only if you have specified what level was used else for zstd as an example which has 22 levels, each stream would have to be recompressed 22 times and 22 patches will exist and then how would xtool decide what level was used, by looking at which one produced the smaller diff file? because if you look at the log again level 17, 18, 19, 20, 21 and 22, yep 6 different levels produced the same result so how would xtool decide what the best level is? Even if you do it for the first 100-200mb, you may even find that zstd compression on some data even taps out at level 10 and any level higher than this will produce the same result. Xtool will still be confused af as to which level is the best which is why I left it to the user.

Xtool in such a scenario will just trust the user specified level and only help out if any of the streams fail to be properly processed by applying xdelta otherwise it will not apply any patches.

Edit: Verbose mode was added just so that the user can see why xtool fails and if there are patterns, like for example this stream "14692 >> 48079 >> 14688", with 4 bytes difference, I assume that all the streams in this game will show a pattern of 4 bytes also missing, which can mean when the game was compressed the developers turned on checksum mode for zstd, meaning a hash sized 4 byte was added for all the streams just to verify if the data is not corrupt in anyway and xtool uses zstd in default mode meaning checksum option is disabled hence why they produce 4 bytes less.

[elit]

@Razor now I understand, thank you! Its clear that internal knowledge of xtool is still important. All make sense now although it also means my "lz scan" cannot be a simple fire & forget proof for all corner cases(well it could but I feel it would be too much to go with each level separately within the script + all the lib versions etc., maybe compiled binary with proper output report would do). It is fine though, as long as users are aware of this.

-------------------------------------------------------------------------------------------------------------------------------

Unrelated to above, I want to give one advice to all users regarding zstd(possibly lz4 as well). For best results it may be important to try all lib versions even if you found one working. During my switch games repacking I found for example that nearby versions and sometimes far versions may get better result(or pickup again) and by results not only I mean size but also speed. ZSTD have a habit to slow down to crawl(~1mb/s) when used on wrong lib version or can have as much as 2-3x speed difference on nearby versions where all of them can inflate very close to!

I dont have data to show you anymore, but just to imagine it was something like this. Lets say we have a random file of 8Mb, then:

libzstd130.dll > 10000k > 6min
libzstd131.dll > 13000k > 5min
libzstd132.dll > 16000k > 3min
libzstd133.dll > 16400k > 1min
libzstd134.dll > 16500k > 20s
libzstd135.dll > 16550k > 15s
libzstd136.dll > 8000k > 4min (back to original size)
...                           (here all following versions did nothing)
libzstd144.dll > 16200k > 1min
libzstd145.dll > 16620k > 13s
libzstd146.dll > 8700k > 4min

This is just example, not exact but yes something like that actually happened.

[elit]

Maybe I found a way to pinpoint zstd level. After you scan with 'verbose' you can see there are several levels same. However, using ':l17' to ':l22'(without 'verbose') separately further reveal which level is real. Only with l19 it shows this:
foundzstdlevel.png
All other levels remain '0/1' even if they show same repacked size in '--verbose' scan. After this it's only matter of figuring out lib version. In this case versions working were 114, 120, 130 and 131. Probably on bigger files time would start showing difference, but I have seen these giving same result before(but also not always) so once they match, any of them should be safe.

[dixen]

[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

[Masquerade]

Originally Posted by dixen

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

Afaik no, because DELZORec requires non solid + 2 input files whereas FA can only deliver 1 input file.

[Razor12911]

Originally Posted by dixen

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

no, it's not possible because xtool can't be made to give two inputs nor accept two outputs

[Razor12911]

Update available

Changes

- generate database feature fixed
- fixed external executable support issues
- fixed lz4f level setting bug

[Razor12911]

Update available

Changes

- updated oodle scanner
- updated external executable support
- updated configuration based plugin support to add depth information
- updated verbose mode

Notes

Given that very small number of people are able to make their own plugins and codecs to use in xtool, I've added an example that imports codecs from QuickBMS after you have generated your database and have no codec to use with it.

For compression:

[cpk,snappy,rfpk]
Encode=quickbms.exe -s "comtype <codec> ; clog <fileout> 0 <insize> <outsize>" "" <filein>
Decode=quickbms.exe -s "comtype <codec>_COMPRESS ; clog <fileout> 0 <insize> <outsize>" "" <filein>

For encryption:

[blowfish,xxtea]
Encode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 0 <ressize>; log <fileout> 0 <insize>" "" <filein>
Decode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 1 <ressize>; log <fileout> 0 <insize>" "" <filein>

You'll need to check http://aluigi.altervista.org/papers/quickbms.txt for the list of supported compressors built in so that you don't pick something like lz2k and expect it to work (quickbms itself has to be able to both compress and decompress). You can add more codecs like I have done here "cpk,snappy,rfpk", if you wanted to add lzss for example, it then becomes "cpk,snappy,rfpk,lzss" then from there you can use Bms2Xtl for scripts that have this comtype and possibly expect xtool to use quickbms with no real issues (hopefully).

Also note that if for whatever reason there is crc mismatch, xtool will still help you by applying xdelta automatically.

Also also note that this generate temps files meaning the process is slower because it has to both read and write to disk therefore, only use this approach if there is nothing else that you can to do to improve the situation.

[elit]

OMG ok with this I think you opened can for a whole lot of more codecs/games to be able to process.

[Razor12911]

Update available

Changes

- fixed issue with storing correct recompression information when stream patching is performed

Notes

This issue was brought up by Masquerade and L0v3craft when they were trying to precompress WRC 10 and upon decoding xtool would produce an error so thanks to them, it has been resolved however I'd like to add more useful information regarding this game and why even with xtool helping you with deltas will not allow you to process all streams based on how xtool was designed.

We all know that xtool patches streams if they cannot be restored correctly however it sets its own boundaries, by default the patch file cannot be 5% larger than the original size else the patch fails as highlighted here with some streams being missed. https://fileforums.com/showpost.php?...&postcount=248

This 5% can be changed by user via the command --diff=5p, with 5p being 5%, so to capture all these other streams, you'll have to increase this percentage.

CHUNK_57.PKG
using -mwrc10

Compressed 1 file, 99,435,664 => 189,185,421 bytes. Ratio 190.26%
Compression time: cpu 0.14 sec/real 3.71 sec = 4%. Speed 26.82 mB/s

using -mwrc10 --diff=20p

Compressed 1 file, 99,435,664 => 263,029,277 bytes. Ratio 264.52%
Compression time: cpu 0.09 sec/real 3.61 sec = 3%. Speed 27.53 mB/s

Verbose information displays

[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed
[0] - Patching stream at 0000000000005476 (514 >> 514) [26] has failed

Patching this 514 byte stream produced a 26 byte patch file, which is 5.1% and that is larger than the 5% limit so increasing this means you are allowing xtool to accept this stream.

--diff=20p

[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed
[0] - Patched stream at 0000000000005476 (514 >> 514) [26] successfully

[Masquerade]

LEGO® Star Wars™: The Skywalker Saga uses Kraken

XTool is created by Razor12911

Streams: 284071/284071
Time: 00:13:43 (00:48:35)
Memory: 512 MB (512 MB)

100.0%
Errorlevel=0

Compressed 1 file, 4,061,948,534 => 9,598,998,234 bytes. Ratio 236.32%
Compression time: cpu 4.22 sec/real 1032.25 sec = 0%. Speed 3.94 mB/s
All OK

[GamerfromPoland]

OMG it's a great tool! Thanks for it!
https://radaryonline.pl/gdzie-jest-burza/

[Razor12911]

Update available

Changes

- added IO functions (erase, replace)
- fixed external executable support bugs

Notes

Xtool has introduced IO functions, which should help you perform file and folder operations such as erase and replace (for now, more will be added). I actually wanted to add these functions a long time ago as they could be useful for repacking however I delayed them time after time because precompression needed more attention but as there have been no bug reports for the past 3 weeks I decided to start working on them.

To summarise, Erase is a feature that fills a given input with zeroes in case you wanted to repack certain data separately and Replace is a feature that replaces existing data within certain files with another.
Xtool will search for locations of the extracted streams and store these positions for when you use decode function to revert the changes (yes, the process is reversible)

Erase

xtool erase extracted_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data

Use cases for erase is the removal of language/videos files from archives in games after using another extraction tool, so rather than manually searching for positions and storing them, erase will make this process automated.

An example is after extracting video files from an archive and then wanting to remove credits video, the syntax would be

xtool erase credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak credits.xtl
xtool decode credits.xtl credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak

decode will fill the zeroes with the original data

Replace

xtool replace old_streams new_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data

The use cases for replace is possibly game file modification.

An example is having several modified files and the original files and you wanted to bulk replace these files, you'd have to prepare two folders with the same file structure and the syntax would be

xtool replace "textures\original\*" "textures\modified\*" "game\*" mod_tex.xtl
xtool decode mod_tex.xtl "textures\original\*" "game\*"

decode will fill the modified data sectors with the original to restore the file its original form.

PS

If you are using these features for a repack in which people would select languages or video credits for example, if the users did not select any of these to be installed then there would be missing files. Xtool's decode command would still work and will try to restore the original data using whatever data that was selected and available without problems.