How to create XTool plugins (configuration based)

KaktoR · #1 15-08-2023, 12:06

Index
Oodle kraken
Oodle kraken with size byte difference
frostbite3 engine (lz4)

KaktoR · #2 15-08-2023, 13:14

Introduction
Sometimes it is better to have a plugin for XTool to catch all the streams, or even speed up the precompression. Here I will show you how to write it and where to start. In this example I will choose a easy one, namely "LEGO Star Wars The Skywalker Saga", which is compressed with oodle kraken. XTool/oo2reck already performs good, but with a plugin it is even better.

Side Notes
Theoretically it is possible to make a plugin for majority of games, but only the minority of them are interestingly enough to make a plugin anyways. Other compression algorithms are a bit harder to understand, and for others it's totally useless to make one (example zlib and zstd). Maybe someone else can make a quick "how to" for algos like lz4/lz4hc.

What do you need?
HxD - https://mh-nexus.de/en/downloads.php?product=HxD20
Notepad (or notepad++)
XTool UI verbose mode

Lets start
First lets take a look on a easy plugin sample:

Code:

[Stream1]
Name=
Codec=
BigEndian=
Signature=
Structure=
StreamOffset=
CompressedSize=
DecompressedSize=

Open XTool UI, select input file, select method, check "verbose" checkbox and start.
Simultaneously open the same file in HxD.

https://i.imgur.com/Z5xmjgt.png

Copy the first offset 135674 and go to it in HxD (CTRL+G). You will see kraken header "8C 06".

So now we have our kraken header "8C 06" at offset 135674.

Again looking at XTool verbose information on the first stream:

Code:

Actual kraken stream found at 0000000000135674 (2139 >> 9287)

We have two more important informations here which will help us alot: CompressedSize (CSize) and DecompressedSize (DSize)

https://i.imgur.com/juCSHtr.png

Now we have to know how such streams are constructed. For the oodle compression family it is mostly a shemata like this (at least for kraken and mermaid):

Code:

CSize - DSize - Header

Where CSize and DSize are 4 bytes long, and the header 2 bytes, which are in total 10 bytes.

https://i.imgur.com/AzO89a7.png

Now we can check if our thoughts are correct. Again we look at our first stream in XTool verbose information:

Code:

Actual kraken stream found at 0000000000135674 (2139 >> 9287)

We see CSize and DSize, and we know the structure is like this:

Code:

CSize (4 bytes) - DSize (4 bytes) - Header (2 bytes)

https://i.imgur.com/xtfzlRh.png

The same with CSize

https://i.imgur.com/Y5usWSO.png

So basically we know now everything what XTool is looking for. Now comes the part what makes a plugin usefull.

But first we note what informations we got until now:
We know the oodle kraken header, which is 2 bytes
We know the DSize for the stream which is 4 bytes
We know the CSize for the stream which is 4 bytes
Which for the Structure key it is like this:

Code:

CSize(number of bytes),DSize(number of bytes),OodleHdr(number of bytes)
CSize(4),DSize(4),OodleHdr(2)

And we know the compression method is kraken (8C 06).

So lets write them right away in our configuration plugin as follow:

Code:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=
Signature=
Structure=CSize(4),DSize(4),OodleHdr(2)
StreamOffset=
CompressedSize=CSize
DecompressedSize=DSize

Now comes the tricky part. We need a signature, so the plugin is usefull at all. In cases like this I pick the first 20 streams starting from the oodle kraken header and paste them in notepad like this:

Code:

A2 0F 00 27 00 14 4F 4F 44 4C 5B 08 00 00 47 24 00 00 8C 06
00 00 00 00 00 00 4F 4F 44 4C B6 19 00 00 00 80 00 00 8C 06
40 48 1C 48 30 24 4F 4F 44 4C BF 17 00 00 00 80 00 00 8C 06
94 88 01 93 91 38 4F 4F 44 4C E8 19 00 00 00 80 00 00 8C 06
01 4F B2 83 03 20 4F 4F 44 4C DC 17 00 00 00 80 00 00 8C 06
3A 61 10 8B 83 2A 4F 4F 44 4C C8 18 00 00 00 80 00 00 8C 06
A8 18 40 2D 18 28 4F 4F 44 4C BE 1A 00 00 00 80 00 00 8C 06
0E 02 05 EA 14 43 4F 4F 44 4C 53 17 00 00 00 80 00 00 8C 06
02 C8 B0 29 42 30 4F 4F 44 4C 11 15 00 00 00 80 00 00 8C 06
...
...
...

Personally I copy between 14 and 20 bytes long for each stream, but that's up to you. Just keep in mind that the first 10 bytes includes CSize, DSize and Header.

In the above table of streams we see a repetitive pattern, our signature. The Signature in this case is "4F 4F 44 4C", so 4 bytes long.

https://i.imgur.com/jRBnaMA.png

Now that we have our repetitive pattern (Signature), we fill in the information again:

Code:

Signature(number of bytes),CSize(number of bytes),DSize(number of bytes),OodleHdr(number of bytes)
Signature(4),CSize(4),DSize(4),OodleHdr(2)

Code:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=
Signature=
Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2)
StreamOffset=
CompressedSize=CSize
DecompressedSize=DSize

In this case you could even use just 2 or 3 bytes as signature, but personally I would recommend to use as much as possible, for reasons.

Also we have to fill the Signature= key with our pattern. Because of endianness and the inclusion of "0x", we have to write the signature "backwards".
"4F 4F 44 4C" will become "4C 44 4F 4F" and adding "0x" at the start will finally become this:

Code:

Signature=0x4C444F4F

So actually we read it from right to left.

Again we fill in the informations:

Code:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=
Signature=0x4C444F4F
Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2)
StreamOffset=
CompressedSize=CSize
DecompressedSize=DSize

Now we need just 2 keys to finalize our plugin: BigEndian and StreamOffset.

The BigEndian key is important because it will tell XTool if a file is Little- or BigEndian. To find out the endianness just mark the CSize or DSize and change the endianness with the radio buttons. If the numbers you see there don't make any sense, then you know that the endianness is wrong.

https://i.imgur.com/py1hfSR.png

We know in this case that it is little endianness, so it should be BigEndian=0 in our plugin.

The other key is StreamOffset. Since our oodle kraken header "8C 06" is part of the entire stream, the offset should be set to "-2" because of the 2 bytes difference. Theoretically you can skip this part and just set is as "0", but XTool have to ensure that this is really a kraken stream, otherwise XTool would think anything beginning from the signature is a kraken stream, even if the signature is something else and there is nothing following. So basically this is just a safety measurement, if you say so.

Now our configuration is complete and it should look like this:

Code:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=0
Signature=0x4C444F4F
Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize

Save it as INI file and name it like whatever you want. Here I name it "legotsws.ini" and place it next to xtool.exe

So finally lets test our plugin.

Code:

xtool:kraken
Compressed 1 file, 4,064,897,269 => 9,465,290,559 bytes. Ratio 232.85%
Compression time: cpu 3.20 sec/real 232.34 sec = 1%. Speed 17.50 mB/s
All OK

xtool:legotsws
Compressed 1 file, 4,064,897,269 => 9,830,656,238 bytes. Ratio 241.84%
Compression time: cpu 3.56 sec/real 210.97 sec = 2%. Speed 19.97 mB/s
All OK

We see the plugin performs better in case of speed and ratio.

-----------------------------------------------

Here is another example for oodle kraken for the game "Shadow of War". In this case making a plugin is useless, because there are many different signatures.

Here are the first 20 streams as 20 bytes long:

Code:

00 00 00 00 00 00 00 00 00 00 D7 0E 00 00 DD 36 00 00 8C 06
FD 0E DE F7 BD DF 86 85 62 76 E5 45 AB 42 AC 2E 9B 00 8C 06
8F 26 43 34 38 10 44 31 91 58 1A 02 00 00 D2 08 00 00 8C 06
72 63 65 73 00 00 00 00 00 00 60 0D 00 00 04 38 00 00 8C 06
00 00 00 00 00 00 00 00 00 58 0C 02 00 00 D6 08 00 00 8C 06
66 66 65 72 14 00 00 00 58 58 46 29 00 00 0E 44 00 00 8C 06
35 70 1A E0 91 4B 44 18 58 58 BC 48 00 00 00 00 01 00 8C 06
80 36 81 F1 57 34 FA 13 32 31 8A 73 00 00 00 00 01 00 8C 06
A0 C1 A3 C9 E9 78 24 16 58 58 4D 53 00 00 00 00 01 00 8C 06
F3 7C 3C CF 87 E7 43 58 58 58 77 50 00 00 00 00 01 00 8C 06
C7 E1 F9 78 3E CF A7 C3 63 58 7E 76 00 00 00 00 01 00 8C 06
AB D6 AA 26 4D 68 34 D1 34 C1 B2 C0 94 24 08 81 A9 D2 8C 06
E1 E9 F9 3C 9F C7 F3 38 58 58 6F 5E 00 00 00 00 01 00 8C 06
63 01 8D CE 8D 57 46 1D 1B 9D 18 7D 1A BE 8D C1 8C 21 8C 06
18 CC 25 73 A9 4C 2A 70 06 58 41 4B 00 00 00 00 01 00 8C 06
5E F7 3A 5A AD 6B 5A CC 63 01 EB 2B 5A D6 63 18 C6 31 8C 06
7F 22 E8 81 08 82 78 4F 58 58 58 2E 5F 00 00 00 00 01 00 8C
1E 68 1C 2B 04 40 E9 2B 58 58 61 69 00 00 00 00 01 00 8C 06
8F 33 CF 23 95 19 68 58 58 58 5B 45 00 00 00 00 01 00 8C 06
F1 38 1C 0D C7 79 9C 07 68 58 A4 43 00 00 00 00 01 00 8C 06

Did you see it? Many signatures are 2 bytes "58 58" or just 1 byte "58". And the other half are just random bytes, which don't follow any pattern.

Testing:

Code:

xtool:kraken
Compressed 1 file, 8,289,538 => 26,917,140 bytes. Ratio 324.71%
Compression time: cpu 0.02 sec/real 2.13 sec = 1%. Speed 3.90 mB/s
All OK

xtool:mesow (signature 0x58)
Compressed 1 file, 8,289,538 => 22,599,836 bytes. Ratio 272.63%
Compression time: cpu 0.02 sec/real 1.46 sec = 1%. Speed 5.68 mB/s
All OK

Code:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=0
Signature=0x58
Structure=Signature(1),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

However, maybe you have a game which has for example just 2 different signatures which you can use. In this case you make [Stream2] inside the same configuration file, like this:

Code:

[Stream1]
Name=kraken
Codec=kraken:l7
BigEndian=0
Signature=0x58
Structure=Signature(1),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

[Stream2]
Name=kraken
Codec=kraken:l7
BigEndian=0
Signature=0x7662
Structure=Signature(2),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

-----------------------------------------------

Did you remember, in the beginning I said that plugins for zlib or zstd are useless? However some time ago I made one for zstd, just for fun and for learning stuff.

Code:

[Stream1]
Name=zstd
Codec=zstd
BigEndian=0
Signature=0x4454535A
Structure=Signature(4),DSize(4),Unknown(8),ZstdHdr(4),Stream
StreamOffset=-4
CompressedSize=0
DecompressedSize=DSize

The Signature was 4 bytes, followed by 4 bytes DSize (DecompressedSize), 8 bytes of unknown trash and 4 bytes zstd header. Maybe some of you would think now, where is CSize? CompressedSize is set to 0, just because the zstd library has a function for it to read the value and tells XTool about this value. But the interesting part here is just the "Unknown(8)" part I wanted to show you, because sometimes you can have something like the above example, were you have a Signature, followed by some unknown bytes, then CSize/DSize and the header.

KaktoR · #3 16-08-2023, 04:09

Oodle kraken with size byte difference

Introduction
Sometimes it could be the case that either CSize or DSize has a byte difference. In this case I will show you how to deal with such things.

Lets start
First lets take a look on the data we have.

https://i.imgur.com/pJAVOGK.png

We have a repetitive pattern (Signature) 7 bytes long, some unknown trash 10 bytes long, DSize 4 bytes, CSize 4 bytes and our kraken header 2 bytes.

Now lets look for the sizes. I can say here that DSize matches, but what happened to CSize?

https://i.imgur.com/THt7MkJ.png

Next stream, same thing.

https://i.imgur.com/0ZYM8ey.png

And again, same thing on the next stream.

https://i.imgur.com/Y1nEpCR.png

So the CSize does not match here. In fact on every stream you have a 12 byte difference. I cannot tell you how that comes, or what happened here technically, but we can fix it within the plugin configuration. All we have to do now is to subtract the 12 bytes from the CSize.

Code:

CompressedSize=CSize - 12

And here is the finished plugin:

Code:

[Stream1]
Name=kraken
Codec=kraken:l6
BigEndian=0
Signature=0x64657070617257
Structure=Signature(7),Unknown(10),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize - 12
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

Testing:

Code:

xtool:kraken
Compressed 1 file, 233,487,310 => 714,300,604 bytes. Ratio 305.93%
Compression time: cpu 0.16 sec/real 30.54 sec = 1%. Speed 7.64 mB/s
All OK

xtool:testplugin
Compressed 1 file, 233,487,310 => 714,358,929 bytes. Ratio 305.95%
Compression time: cpu 0.14 sec/real 23.04 sec = 1%. Speed 11.14 mB/s
All OK

Fairly the ratio did not increase here (just 0.02%), but this example was just a showcase on what to do if you encounter byte differences.

KaktoR · #4 13-09-2023, 07:16

Introduction
This post is about the frostbite 3 engine, how to deal with lz4, zlib, zstd and oodle kraken streams.
It is recommended to read the previous posts, about how to search for offsets and such.

In frostbite3 engine the structure is like this commonly for lz4:

Code:

DSize(#),Signature(#),CSize(#),Stream

Lets start
Beginning with lz4, first make a batch file next to xtool.exe with the following command

Code:

xtool.exe precomp -mlz4:s64k -t100p -v -s - - < %1 > %1.out

and drag&drop a sample onto this batch file. You will see something like this with potentially detected streams.

Code:

XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 0000000000703A01 (7354882)
[0] Actual lz4 stream found at 00000000000BF1EA (21353 >> 65536)
[0] Actual lz4 stream found at 000000000035142B (20177 >> 65536)
[0] Actual lz4 stream found at 00000000004193CE (23693 >> 65536)
[0] Actual lz4 stream found at 000000000046D953 (12877 >> 65536)
[0] Actual lz4 stream found at 000000000052B0B2 (12330 >> 65536)
[0] Actual lz4 stream found at 000000000060A5D6 (26312 >> 65536)
[0] Actual lz4 stream found at 0000000000681013 (23435 >> 65536)

[0] Processing streams on block 0000000000000000 to 0000000000703A01 (7354882)

Streams: 7 / 7
Time: 00:00:01 (CPU 00:00:01)

Size: 7.01 MB >> 7.32 MB

Why "-mlz4:s64k"? Because sizes like 65536 are somewhat common (64 * 1024 = 65536). This way you can speed up the stream detection process, because xtool is just searching for stream sizes which are 65536.

However, go to the first offset (in this case BF1EA). Unlike in the previous examples, this one has no specific header, but we don't need one here.

https://i.imgur.com/BFEKAsp.png

Now we have to go just 2 bytes back before the offset starts and we will see the CSize (you might want to change the endianness).

https://i.imgur.com/R2nZnYo.png

DSize and Signature:

https://i.imgur.com/9RrSGGd.png

We use BigEndian and Streamoffset is set to 0. Now the plugin is nearly complete.

Code:

[Stream1]
Name=lz4
Codec=lz4
BigEndian=1
Signature=0x7009
Structure=DSize(3),Signature(2),CSize(2),Stream
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize

Testing:

Code:

xtool:testplugin
Compressed 1 file, 7,354,882 => 14,408,711 bytes. Ratio 195.91%
Compression time: cpu 0.02 sec/real 1.61 sec = 1%. Speed 4.56 mB/s
All OK

panker1992 · #5 29-09-2023, 10:49

i have tried your method of creating ini based xtool plugins the idea in its core is nice to be able with simple adjustments to gain 2%~ 3% ratio and about 10%~15% in speed. speed only comes in compression mode where compress once decompress many applies!!

generally i have tried 2 games to create with your guide above one is Sekiro which uses kraken and the second one is zstd only for resident evil 2 and resident evil 3 series!! and failed

an average user will fail this easily i will upload samples and an explanation. maybe this idea is very nice if we get it as per engine because as it is, it cant be used!!

KaktoR · #6 29-09-2023, 11:32

Of course this is not the same case for every game or engine. Initially I wanted to point it out on second post, but unfortunatelly I cannot edit it because the forum will kick me back to front page of fileforums.com. Most like it is some bug in the software this forum is using.

As for the game Sekiro, I guess it is not possible here to create ini based plugins, because most likely some of the information is stored in the bhd files. I once looked on the game Armored Core 6, which is using the same engine. All I found was header and csize, nothing more. It looked like this

Code:

header  unk  csize
8C 06 - 00 - 2A 32 - 80 1C 5E 00 00 80 BF 00 00 00 00 30 43 60 0C 9A 40 3E F8 0C 30 9E 9E 88 7B 53 33 29 96 55 95 A1 13 9E 5B 31 65 25 25 55 0B 25 9B 99 28
8C 06 - 00 - 37 B9 - 80 1B 77 00 00 00 00 00 00 00 00 50 45 A4 0B F6 83 40 19 70 05 07 9A 49 A0 8A C6 AC F0 B0 C9 2A B2 24 D3 CC B6 52 58 A6 94 C8 24 D4 A9
8C 06 - 00 - 34 80 - 80 1F AC FF FF 14 00 FF FF FF FF 50 53 FC 0E 10 03 A4 03 0F 88 77 00 10 04 79 9C 39 C8 7B 85 24 82 12 52 12 65 28 24 92 24 A7 4A 27 01
8C 06 - 00 - 35 06 - 80 1F DC FF FF FF FF FF FF FF FF 50 53 D8 0E 21 03 C3 DB 36 98 E8 A1 0A DF 8D D5 8D D5 CA 55 35 DC 96 AB 7A B5 62 94 AA D7 19 56 AE AA
8C 06 - 00 - 33 7A - 80 20 2C 0A 00 00 00 02 00 00 00 50 53 D8 0E 52 02 40 32 40 08 9A 9C 3A 04 2D 53 65 B4 9D 73 2B 04 45 92 95 84 9E AA 54 25 4A D4 A4 4C
8C 06 - 00 - 30 F1 - 80 1D 7D 01 00 00 00 00 00 80 BF 50 4B E8 0C A8 83 40 15 D4 04 27 8A DA 80 20 08 E6 66 14 27 38 42 46 D8 48 24 38 4A 66 53 EC D2 82 49
8C 06 - 00 - 37 23 - 80 1F 40 FF FF 7F 00 7C 00 00 00 50 56 EC 0E 2D 83 C4 67 AE 8A 5B 00 10 04 70 C9 C0 24 66 4C 40 92 26 44 84 82 1D 89 0C 9E 6C B8 CA 59
8C 06 - 00 - 3A 5E - 80 20 72 00 00 00 00 00 00 00 00 50 54 FC 0E 3E 83 30 21 0C 05 6B 40 1C C0 05 20 96 8A 82 19 A4 E2 AA 25 31 11 18 45 89 22 91 25 0B 50
8C 06 - 00 - 24 A2 - 80 15 4B 0E 00 00 00 06 00 00 00 30 2A C8 07 D4 40 29 C4 07 B6 9B 5B 04 79 5B A5 2A 94 9C EA A0 95 45 54 C4 D1 2A 49 59 82 24 AE A5 4A
8C 06 - 00 - 2C 88 - 80 1D 56 00 FF 00 00 00 00 00 00 30 45 E0 0C 3C 40 44 6C 0C 12 9C 3A 44 2B 53 56 A4 95 51 95 A8 A0 A6 56 B6 8C B3 49 56 11 29 55 6A 5A
8C 06 - 00 - 12 A6 - 80 12 A4 00 00 00 00 00 00 00 00 40 29 A0 07 A1 9B 09 C4 79 AA 95 95 4A 4A 47 55 05 11 58 D3 25 8C 92 A4 52 50 92 49 1C A5 5A D2 E5 79
8C 06 - 00 - 40 25 - 80 0D B4 00 00 00 00 00 00 00 00 30 2E 04 07 66 40 29 54 07 0A 96 9A 64 53 04 84 A2 88 B1 94 D8 CC 92 49 09 11 A9 06 11 11 AB 91 37 A9

But you can still try to find something usefull.

panker1992 · #7 30-09-2023, 02:17

i didnt have time last night to post an explanation of my findings so here we go

sample = a chunk of resident evil 2 with 2 zstd streams.

re2.ini
[Stream1]
Name=zstd
Codec=zstd:l11:f0:b0
BigEndian=1
Signature=0xA7DA
Structure=Dict(1),DSize(3),Signature(2),CSize(2),Z StdHdr(4),Stream
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize
Condition1=ZStdHdr = 0x28B52FFD

attachment1= ZStdHdr
attachment2= xtool verbose
attachment3= HEX csize

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
XTool 2020 (Main Project)	Razor12911	Conversion Tutorials	697	Today 04:30
XTool 2020 (Plugins)	Razor12911	Conversion Tutorials	404	13-07-2024 08:52
How to patch languages out of game files (xtool erase decode)	KaktoR	Conversion Tutorials	5	01-11-2022 08:54
XTool 2019 (Plugins)	Razor12911	Conversion Tutorials	50	16-05-2020 06:14
Nero PMA Update failure	angelosbg	CD/DVD Software & Utilities	9	24-09-2008 03:40

The Following 7 Users Say Thank You to KaktoR For This Useful Post:
Cesar82 (15-08-2023), Junior53 (14-09-2023), L0v3craft (16-08-2023), Lord.Freddy (15-08-2023), mausschieber (16-08-2023), Razor12911 (15-08-2023), Wanterlude (15-08-2023)

The Following 14 Users Say Thank You to KaktoR For This Useful Post:
Cesar82 (15-08-2023), Gehrman (15-08-2023), Junior53 (14-09-2023), kenzo34 (15-08-2023), kj911 (16-08-2023), kuyhaa (23-08-2023), L0v3craft (16-08-2023), Lord.Freddy (15-08-2023), Masquerade (15-08-2023), mausschieber (16-08-2023), murphy78 (15-09-2023), rambohazard (21-08-2023), Razor12911 (15-08-2023), Wanterlude (15-08-2023)

The Following 11 Users Say Thank You to KaktoR For This Useful Post:
Gehrman (18-08-2023), Junior53 (14-09-2023), kj911 (16-08-2023), L0v3craft (16-08-2023), Lord.Freddy (16-08-2023), mausschieber (16-08-2023), Pantsi (16-08-2023), rambohazard (21-08-2023), Razor12911 (16-08-2023), shazzla (16-08-2023), Wanterlude (16-08-2023)

The Following 5 Users Say Thank You to KaktoR For This Useful Post:
Junior53 (14-09-2023), L0v3craft (14-09-2023), Lord.Freddy (13-09-2023), mausschieber (13-09-2023), Razor12911 (14-09-2023)