Sorry, I didn't think of that...

sample?

Smallest bdt is 2.4 gb..Uploading..

https://drive.google.com/file/d/1UDb...ew?usp=sharing

It took even longer. I apologize for the encoding in cmd.

xtool_0.4.2

https://imageup.ru/img119/thumb/113887693.jpg

ELDEN RING

Oodlerec Used
From Masquerade.
use oodle core from Martha is Dead

@dixen

use oo2core_8_win64.dll

Attachment 31310
Attachment 31311


upload the file from the game for me to take a look at.

You can use this oodle core to boost the ratio a little bit more
https://bayfiles.com/55bfieKbx4/oo2core_4_win64_7z
Credits to FitGirl for the dll

From a 100mb sample data0.bdt

LINK ON FILE

xtool_0.3.9 ONLY
https://imageup.ru/img34/thumb/23887702.jpg

Thanks to DIXEN for the hint.

UPD:
https://imageup.ru/img242/thumb/3333887736.jpg

https://imageup.ru/img193/thumb/4443887752.jpg

Update available

Changes

- added verbose mode
- added feature that allows you to enforce a different library to be loaded
- fixed issues related to imperfect stream patching
- fixed issues with old libraries with missing functions that cause xtool to crash on startup
- updated oodle codec
- updated reflate codec
- updated zstd codec

Notes

You can now enforce xtool to use a specific library rather than renaming the library for it to picked up by xtool.

Available parameters for this are:
--zlib=filename, --lz4=filename, --lzo=filename, --zstd=filename, --oodle=filename

You can also use this to run multiple tests to see which zstd or oodle library gives the best output by running these commands on a small sample:
Then you can pick whichever produces the best results

___

Oodle code has been updated, if you having issues with stream detection you can use the parameter n1 (kraken,n1), where the old code will be used. it can also be n2 or n3, depending on how many times each stream should be redetected (for improved results but at a cost of speed)
If this doesn't help then use the side project.

___

Verbose mode was added to give information about input, as to what parameters are used for streams within the input and if you see a common trend then you directly use these parameters to speed up xtool and also give it the opportunity to patch streams if there are any invalid streams.

Verbose mode will enforce xtool to 1 thread thus it should not be used by default as it will slow now precompression.

How to use verbose mode

Drag&Drop a file on batch

ELDEN RING - XTool Update Test (Data3.bdt)

Old XTool - xtool:kraken,l6
New XTool- xtool:kraken,l6
New XTool - xtool:kraken,l6,n1
New XTool - xtool:kraken,l6,n2
Library Used : oo2core_4_win64.dll

Check previous page and download the library from there.

External executable!!!
 

resident evil 4 HD project "*.lfs" Precompressor
https://github.com/emoose/re4-research

Arc.ini
https://upload.cc/i1/2022/02/28/aWcFp5.jpg


--------------------------------------------------------
But it does not work with XTool
XTool.ini
https://upload.cc/i1/2022/02/28/BdNR6z.jpg



Is there a way to solve it?

can you send sample and the exe?

Razor12911
Is it possible to make lz2k-plugin for XTool?

There's no recompression code for this algorithm, not even QuickBMS can compress LZ2K.

How to decompress unity files? What dlls are needed? And if possible examples?

Some use 1.8.0 version of liblz4.dll

SMBBHD works like XTool v12, use -munity,lz4hc,l10 (liblz4.dll - 1.8.0)
Cris Tales, has some LZMA doesn't work due to use of LZ4, use UBT
Sense: ACPGS, has some LZMA doesn't work due to use of LZ4, use UBT
SMBBM, has no effect
Trollhunters, use Unity plugin with -munity,lz4hc,l12 (liblz4.dll - 1.9.3)
Tin & Kuna, not much, use UBT
NSLT, possibly that much unlike XTool v12

I understand the Leviathan codec isn't to be used directly and instead by plugins, so I am attempting to create a database for Wolfenstein Youngblood using a modified BMS script.

The script dumps the leviathan streams to disk and that should be enough to generate a DB, however I recieve an error that the filename of the database I made is not a valid method if I attempt to precompress.

I have attached below a sample file + BMS script if anyone would like to have a look.

The leviathan stream is signified by the 0x8C0C header.

https://www43.zippyshare.com/v/fCNMmXpB/file.html

Thank you for last update, it solved zstd crash for me.
Would it be possible for xtool to additionally print its version when run without parameters? So that it does not rely on file name.

Also, additionally to direct dll loading, I would suggest automatic multi lib query like this:
1_zstd.dll, 2_zstd.dll...
1_lz4.dll, 2_lz4.dll...
1_oodle.dll, 2_oodle.dll...

With those files in xtool directory, *all* of them would be loaded and then sequentially tried up to certain point.
For example at the beginning all compression variation on all dll's would be tried until at least 1 valid chunk is found. Then dll's without result would be discarded and if multiple exist with different size then those with lower size also discarded. Those with same size could still be used up to certain number of chunks in parallel unless either different size is produced or simply all but 1 are discarded in case all kept producing same size.

This way user can just put all different versions of libraries and forget about manual "query and try".

In case different versions use different API, another naming alternative would be agreed naming convention, for example zstd_v1.2.dll, lz4_v1.2.dll or any other, where xtool would be aware of that version(or version range) and which names of API functions specific dll use.

Update available

Changes

- fixed issue of lz4 codec loading incorrect library
- fixed issue with handling endianess via configuration based plugins
- updated framework of library based plugins

@elit
For example at the beginning all compression variation on all dll's would be tried until at least 1 valid chunk is found.
- This cannot be done because versions closer to each other tend to produce the same result and in some streams especially on a small number of them, they produce different results.
- Example, version 1.4.3 and 1.4.4 of zstd, these two libraries can produce same result and would be considered valid chunk using either one of these libraries however this may not be the same for the rest of the input.

Those with same size could still be used up to certain number of chunks in parallel unless either different size is produced or simply all but 1 are discarded in case all kept producing same size.
- The issue with this is considering xtool processes data by chunks, if it processes version by version, it would need to store all results somewhere, most probably in memory and every single library would need its own memory set, coding this would be problematic at least from my end.

@Gehrman

Sorry for late reply, I checked out the sample and the only way this would work is by generating a database using generate function in xtool (which is currently broken), but it will probably be fixed in the next release.
I'll possibly upload the re4lfs configuration example. :)

Thank for update..
But..Metro Exodus lz4 streams still not being searched for((

Metro Exodus lz4 support was never added to xtool, remember you cannot use lz4 codec directly in this program, only via plugins and a plugin for that game was never made.
https://fileforums.com/showthread.php?t=103563

Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.

Razor I think you mentioned in the past that -mzlib is able to detect deflate? I recall mentioning back then that it did not work for me. So here I am attaching set of png images that I just accidentally tested and did not work. Both -mreflate as well as previous ztool:high does work.

Attachment 31371

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

I tried to precompress one of the png files and as you can see, the method -mzlib was able to detect the deflate stream but then it failed to process it.

It's better then nothing I guess, especially if you're not familiar with creating plugins, so thanks for the update :D

Thanks Razor, this will be good for when the generator is working.

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

Attachment 31395

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
Attachment 31394

PS(could it be due to window size - 48k ?)

Sorry for late response but here's an example.

in the generate folder, you'll get the example of how database would be created.

You'll notice that the unpacked data and the original data is the same, that's because this tool accepts game files as they are and doesn't need to extract additional streams.

Note: Chunk size is should be the same size as the largest stream/file

then in precompress folder is where you need to transfer the generated database file from generate folder to precompress folder (lfs.xtl)

Look in xtool.ini to see how the program should be configured based on the information you provided.

Results of the samples you provided

there are several implementations of the deflate algorithm, it's very flexible. Think of using lzma to compress something, obviously there are compression levels that it comes with 1-9. -mzlib implementation is based around this idea that a user who uses this would have used a common setting such as compression level so for precompression it will decompress the data and then try to recompress it using these common levels, but of course lzma has methods that people use for other purposes like for stronger compression such as "lzma:ultra:a1:mfbt4:d158m: fb273:mc1000000000:lc8" mzlib cannot process such, similarly to deflate streams in png images as the way deflate set in such a way that it's good compressing pixels of an image but also make it quicker for loading/viewing. Reflate and preflate have no problem processing these as they used an advanced method for precompressing them.

You might say that mzlib is therefore useless, actually it still has its uses. mzlib is faster than both reflate and preflate because of its simple implementation and incurring no overhead furthermore mzlib produces better results because there is no additional header information file produced hence why I've preached several times to mix both zlib with reflate or preflate, if zlib fails to process the stream it just passes the stream to reflate/preflate for it to process it.

mzlib:
Compressed 1 file, 86,347,072 => 444,743,922 bytes. Ratio 515.07%
Compression time: cpu 0.16 sec/real 13.37 sec = 1%. Speed 6.46 mB/s

mreflate,l9:
Compressed 1 file, 86,347,072 => 445,179,589 bytes. Ratio 515.57%
Compression time: cpu 0.14 sec/real 14.33 sec = 1%. Speed 6.03 mB/s

mpreflate:
Compressed 1 file, 86,347,072 => 366,023,129 bytes. Ratio 423.90%
Compression time: cpu 0.14 sec/real 10.25 sec = 1%. Speed 8.42 mB/s

mzlib+reflate,l9:
Compressed 1 file, 86,347,072 => 444,941,384 bytes. Ratio 515.29%
Compression time: cpu 0.06 sec/real 11.74 sec = 1%. Speed 7.35 mB/s

mzlib+preflate:
Compressed 1 file, 86,347,072 => 444,744,743 bytes. Ratio 515.07%
Compression time: cpu 0.11 sec/real 11.23 sec = 1%. Speed 7.69 mB/s

As you can see, zlib perform faster when alone but better when you hook it up with reflate/preflate and preflate fails to process some streams. I've given people options, all up to them how they use xtool. :)

use --verbose mode to see which levels are producing the closest result to the original stream.
level 17 is close but level 19 is commonly used, then set -mzstd,l19
Used libzstd130.dll

Thank you Razor! But wait, why it did not tried that level automatically? Is xtool trying only some levels here as well? Like up to 9, maybe due to speed?

If that is the case, may I suggest that it try to do all of them - up to certain mb of data(user defined). For instance first 100-200mb all levels and if nothing found drop to only speedier levels as is now etc.?

Dude it tried all levels, look at the log via verbose (level 1-22) and none of them produced crc perfect stream and you can even see that with level 19 the original stream was 14692 which was decompressed to 48079 and recompressed to 14688.

Xtool applies patches using xdelta only if you have specified what level was used else for zstd as an example which has 22 levels, each stream would have to be recompressed 22 times and 22 patches will exist and then how would xtool decide what level was used, by looking at which one produced the smaller diff file? because if you look at the log again level 17, 18, 19, 20, 21 and 22, yep 6 different levels produced the same result so how would xtool decide what the best level is? Even if you do it for the first 100-200mb, you may even find that zstd compression on some data even taps out at level 10 and any level higher than this will produce the same result. Xtool will still be confused af as to which level is the best which is why I left it to the user.

Xtool in such a scenario will just trust the user specified level and only help out if any of the streams fail to be properly processed by applying xdelta otherwise it will not apply any patches.

Edit: Verbose mode was added just so that the user can see why xtool fails and if there are patterns, like for example this stream "14692 >> 48079 >> 14688", with 4 bytes difference, I assume that all the streams in this game will show a pattern of 4 bytes also missing, which can mean when the game was compressed the developers turned on checksum mode for zstd, meaning a hash sized 4 byte was added for all the streams just to verify if the data is not corrupt in anyway and xtool uses zstd in default mode meaning checksum option is disabled hence why they produce 4 bytes less.

generate -c## ;)

good work
Thank you for this wonderful tool

@Razor now I understand, thank you! Its clear that internal knowledge of xtool is still important. All make sense now although it also means my "lz scan" cannot be a simple fire & forget proof for all corner cases(well it could but I feel it would be too much to go with each level separately within the script + all the lib versions etc., maybe compiled binary with proper output report would do). It is fine though, as long as users are aware of this.

-------------------------------------------------------------------------------------------------------------------------------

Unrelated to above, I want to give one advice to all users regarding zstd(possibly lz4 as well). For best results it may be important to try all lib versions even if you found one working. During my switch games repacking I found for example that nearby versions and sometimes far versions may get better result(or pickup again) and by results not only I mean size but also speed. ZSTD have a habit to slow down to crawl(~1mb/s) when used on wrong lib version or can have as much as 2-3x speed difference on nearby versions where all of them can inflate very close to!

I dont have data to show you anymore, but just to imagine it was something like this. Lets say we have a random file of 8Mb, then:
This is just example, not exact but yes something like that actually happened.

Maybe I found a way to pinpoint zstd level. After you scan with 'verbose' you can see there are several levels same. However, using ':l17' to ':l22'(without 'verbose') separately further reveal which level is real. Only with l19 it shows this:
Attachment 31399
All other levels remain '0/1' even if they show same repacked size in '--verbose' scan. After this it's only matter of figuring out lib version. In this case versions working were 114, 120, 130 and 131. Probably on bigger files time would start showing difference, but I have seen these giving same result before(but also not always) so once they match, any of them should be safe.

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat