XTool 2020 (Main Project) - Page 30

Masquerade · #1 01-03-2022, 00:23

Quote:

Originally Posted by dixen

Razor12911
Is it possible to make lz2k-plugin for XTool?

There's no recompression code for this algorithm, not even QuickBMS can compress LZ2K.

Dark0226 · #2 05-03-2022, 18:03

How to decompress unity files? What dlls are needed? And if possible examples?

Hexagon123 · #3 05-03-2022, 20:18

Quote:

Originally Posted by Dark0226

How to decompress unity files? What dlls are needed? And if possible examples?

Some use 1.8.0 version of liblz4.dll

SMBBHD works like XTool v12, use -munity,lz4hc,l10 (liblz4.dll - 1.8.0)
Cris Tales, has some LZMA doesn't work due to use of LZ4, use UBT
Sense: ACPGS, has some LZMA doesn't work due to use of LZ4, use UBT
SMBBM, has no effect
Trollhunters, use Unity plugin with -munity,lz4hc,l12 (liblz4.dll - 1.9.3)
Tin & Kuna, not much, use UBT
NSLT, possibly that much unlike XTool v12

Masquerade · #4 09-03-2022, 08:23

I understand the Leviathan codec isn't to be used directly and instead by plugins, so I am attempting to create a database for Wolfenstein Youngblood using a modified BMS script.

The script dumps the leviathan streams to disk and that should be enough to generate a DB, however I recieve an error that the filename of the database I made is not a valid method if I attempt to precompress.

I have attached below a sample file + BMS script if anyone would like to have a look.

The leviathan stream is signified by the 0x8C0C header.

https://www43.zippyshare.com/v/fCNMmXpB/file.html

elit · #5 09-03-2022, 15:03

Thank you for last update, it solved zstd crash for me.
Would it be possible for xtool to additionally print its version when run without parameters? So that it does not rely on file name.

Also, additionally to direct dll loading, I would suggest automatic multi lib query like this:
1_zstd.dll, 2_zstd.dll...
1_lz4.dll, 2_lz4.dll...
1_oodle.dll, 2_oodle.dll...

With those files in xtool directory, *all* of them would be loaded and then sequentially tried up to certain point.
For example at the beginning all compression variation on all dll's would be tried until at least 1 valid chunk is found. Then dll's without result would be discarded and if multiple exist with different size then those with lower size also discarded. Those with same size could still be used up to certain number of chunks in parallel unless either different size is produced or simply all but 1 are discarded in case all kept producing same size.

This way user can just put all different versions of libraries and forget about manual "query and try".

In case different versions use different API, another naming alternative would be agreed naming convention, for example zstd_v1.2.dll, lz4_v1.2.dll or any other, where xtool would be aware of that version(or version range) and which names of API functions specific dll use.

Razor12911 · #6 10-03-2022, 07:43

Update available

Changes

- fixed issue of lz4 codec loading incorrect library
- fixed issue with handling endianess via configuration based plugins
- updated framework of library based plugins

@elit
For example at the beginning all compression variation on all dll's would be tried until at least 1 valid chunk is found.
- This cannot be done because versions closer to each other tend to produce the same result and in some streams especially on a small number of them, they produce different results.
- Example, version 1.4.3 and 1.4.4 of zstd, these two libraries can produce same result and would be considered valid chunk using either one of these libraries however this may not be the same for the rest of the input.

Those with same size could still be used up to certain number of chunks in parallel unless either different size is produced or simply all but 1 are discarded in case all kept producing same size.
- The issue with this is considering xtool processes data by chunks, if it processes version by version, it would need to store all results somewhere, most probably in memory and every single library would need its own memory set, coding this would be problematic at least from my end.

@Gehrman

Sorry for late reply, I checked out the sample and the only way this would work is by generating a database using generate function in xtool (which is currently broken), but it will probably be fixed in the next release.
I'll possibly upload the re4lfs configuration example.

dixen · #7 10-03-2022, 09:12

Thank for update..
But..Metro Exodus lz4 streams still not being searched for((

Razor12911 · #8 10-03-2022, 09:43

Metro Exodus lz4 support was never added to xtool, remember you cannot use lz4 codec directly in this program, only via plugins and a plugin for that game was never made.
https://fileforums.com/showthread.php?t=103563

Razor12911 · #9 10-03-2022, 15:40

Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

Code:

-mleviathan

If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.

KaktoR · #10 11-03-2022, 01:00

Quote:

Originally Posted by Razor12911

Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

Code:

-mleviathan

If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.

It's better then nothing I guess, especially if you're not familiar with creating plugins, so thanks for the update

Code:

Streams: 354/415
Time: 00:01:18 (00:01:17)
Memory: 423 MB (423 MB)

64,1mb >> 352mb

Masquerade · #11 11-03-2022, 02:50

Thanks Razor, this will be good for when the generator is working.

elit · #12 10-03-2022, 19:19

Razor I think you mentioned in the past that -mzlib is able to detect deflate? I recall mentioning back then that it did not work for me. So here I am attaching set of png images that I just accidentally tested and did not work. Both -mreflate as well as previous ztool:high does work.

pngs.zip

Razor12911 · #13 10-03-2022, 23:48

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

Code:

XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 000000000000A046 (41031)
[0] Actual zlib stream found at 0000000000000053 (40928 >> 87680)

[0] Processing streams on block 0000000000000000 to 000000000000A046 (41031)
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l11,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l12,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l13,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l14,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l15,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l16,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l17,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l18,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l19,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l21,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l22,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l23,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l24,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l25,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l26,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l27,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l28,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l29,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l31,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l32,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l33,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l34,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l35,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l36,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l37,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l38,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l39,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l41,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l42,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l43,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l44,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l45,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l46,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l47,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l48,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l49,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l51,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l52,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l53,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l54,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l55,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l56,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l57,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l58,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l59,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l61,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l62,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l63,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l64,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l65,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l66,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l67,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l68,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l69,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l71,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l72,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l73,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l74,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l75,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 3072) using l76,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l77,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l78,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l79,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l81,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l82,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l83,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l84,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l85,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l86,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l87,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l88,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l89,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l91,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l92,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l93,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l94,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l95,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l96,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l97,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l98,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l99,w15 has failed

Streams: 0/1
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)

I tried to precompress one of the png files and as you can see, the method -mzlib was able to detect the deflate stream but then it failed to process it.

elit · #14 11-03-2022, 05:18

Quote:

Originally Posted by Razor12911

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

elit · #15 16-03-2022, 20:31

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

zstdproblem.png

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
zstdproblem.7z

PS(could it be due to window size - 48k ?)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Dev]XTool	Razor12911	Conversion Tutorials	180	23-10-2020 06:26
Project Cars Digital Edition (3xDVD5) (srep+lzma)	GTX590	PC Games - CD/DVD Conversions	10	28-08-2017 08:34
Project IGI Anthology 1xCD700 CIUV2 2039	mausschieber	PC Games - CD/DVD Conversions	0	24-07-2017 15:12
Space Channel 5 Part 2 Translation Project	Christuserloeser	DC Games	0	21-06-2004 18:16

The Following User Says Thank You to Masquerade For This Useful Post:
dixen (01-03-2022)

The Following 7 Users Say Thank You to Razor12911 For This Useful Post:
dixen (10-03-2022), elit (10-03-2022), ffmla (10-03-2022), Gehrman (10-03-2022), hdneo (10-03-2022), KaktoR (10-03-2022), shazzla (10-03-2022)

The Following 5 Users Say Thank You to Razor12911 For This Useful Post:
elit (10-03-2022), Gehrman (11-03-2022), KaktoR (11-03-2022), L0v3craft (10-03-2022), Masquerade (11-03-2022)

The Following User Says Thank You to Masquerade For This Useful Post:
Gehrman (11-03-2022)

The Following User Says Thank You to Razor12911 For This Useful Post:
Gehrman (11-03-2022)