How to create XTool plugins (configuration based)

[KaktoR]

Introduction
Sometimes it is better to have a plugin for XTool to catch all the streams, or even speed up the precompression. Here I will show you how to write it and where to start. In this example I will choose a easy one, namely "LEGO Star Wars The Skywalker Saga", which is compressed with oodle kraken. XTool/oo2reck already performs good, but with a plugin it is even better.

Side Notes
Theoretically it is possible to make a plugin for majority of games, but only the minority of them are interestingly enough to make a plugin anyways. Other compression algorithms are a bit harder to understand, and for others it's totally useless to make one (example zlib and zstd). Maybe someone else can make a quick "how to" for algos like lz4/lz4hc.

What do you need?
HxD - https://mh-nexus.de/en/downloads.php?product=HxD20
Notepad (or notepad++)
XTool UI verbose mode

Lets start
First lets take a look on a easy plugin sample:

[Stream1]
Name=
Codec=
BigEndian=
Signature=
Structure=
StreamOffset=
CompressedSize=
DecompressedSize=

Open XTool UI, select input file, select method, check "verbose" checkbox and start.
Simultaneously open the same file in HxD.

https://i.imgur.com/Z5xmjgt.png


Copy the first offset 135674 and go to it in HxD (CTRL+G). You will see kraken header "8C 06".

So now we have our kraken header "8C 06" at offset 135674.

Again looking at XTool verbose information on the first stream:

Actual kraken stream found at 0000000000135674 (2139 >> 9287)

We have two more important informations here which will help us alot: CompressedSize (CSize) and DecompressedSize (DSize)

https://i.imgur.com/juCSHtr.png


Now we have to know how such streams are constructed. For the oodle compression family it is mostly a shemata like this (at least for kraken and mermaid):

CSize - DSize - Header

Where CSize and DSize are 4 bytes long, and the header 2 bytes, which are in total 10 bytes.

https://i.imgur.com/AzO89a7.png


Now we can check if our thoughts are correct. Again we look at our first stream in XTool verbose information:

Actual kraken stream found at 0000000000135674 (2139 >> 9287)

We see CSize and DSize, and we know the structure is like this:

CSize (4 bytes) - DSize (4 bytes) - Header (2 bytes)

https://i.imgur.com/xtfzlRh.png


The same with CSize

https://i.imgur.com/Y5usWSO.png


So basically we know now everything what XTool is looking for. Now comes the part what makes a plugin usefull.

But first we note what informations we got until now:
We know the oodle kraken header, which is 2 bytes
We know the DSize for the stream which is 4 bytes
We know the CSize for the stream which is 4 bytes
Which for the Structure key it is like this:

CSize(number of bytes),DSize(number of bytes),OodleHdr(number of bytes)
CSize(4),DSize(4),OodleHdr(2)

And we know the compression method is kraken (8C 06).

So lets write them right away in our configuration plugin as follow:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=
Signature=
Structure=CSize(4),DSize(4),OodleHdr(2)
StreamOffset=
CompressedSize=CSize
DecompressedSize=DSize

Now comes the tricky part. We need a signature, so the plugin is usefull at all. In cases like this I pick the first 20 streams starting from the oodle kraken header and paste them in notepad like this:

A2 0F 00 27 00 14 4F 4F 44 4C 5B 08 00 00 47 24 00 00 8C 06
00 00 00 00 00 00 4F 4F 44 4C B6 19 00 00 00 80 00 00 8C 06
40 48 1C 48 30 24 4F 4F 44 4C BF 17 00 00 00 80 00 00 8C 06
94 88 01 93 91 38 4F 4F 44 4C E8 19 00 00 00 80 00 00 8C 06
01 4F B2 83 03 20 4F 4F 44 4C DC 17 00 00 00 80 00 00 8C 06
3A 61 10 8B 83 2A 4F 4F 44 4C C8 18 00 00 00 80 00 00 8C 06
A8 18 40 2D 18 28 4F 4F 44 4C BE 1A 00 00 00 80 00 00 8C 06
0E 02 05 EA 14 43 4F 4F 44 4C 53 17 00 00 00 80 00 00 8C 06
02 C8 B0 29 42 30 4F 4F 44 4C 11 15 00 00 00 80 00 00 8C 06
...
...
...

Personally I copy between 14 and 20 bytes long for each stream, but that's up to you. Just keep in mind that the first 10 bytes includes CSize, DSize and Header.

In the above table of streams we see a repetitive pattern, our signature. The Signature in this case is "4F 4F 44 4C", so 4 bytes long.

https://i.imgur.com/jRBnaMA.png


Now that we have our repetitive pattern (Signature), we fill in the information again:

Signature(number of bytes),CSize(number of bytes),DSize(number of bytes),OodleHdr(number of bytes)
Signature(4),CSize(4),DSize(4),OodleHdr(2)

[Stream1]
Name=kraken
Codec=kraken
BigEndian=
Signature=
Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2)
StreamOffset=
CompressedSize=CSize
DecompressedSize=DSize

In this case you could even use just 2 or 3 bytes as signature, but personally I would recommend to use as much as possible, for reasons.

Also we have to fill the Signature= key with our pattern. Because of endianness and the inclusion of "0x", we have to write the signature "backwards".
"4F 4F 44 4C" will become "4C 44 4F 4F" and adding "0x" at the start will finally become this:

Signature=0x4C444F4F

So actually we read it from right to left.

Again we fill in the informations:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=
Signature=0x4C444F4F
Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2)
StreamOffset=
CompressedSize=CSize
DecompressedSize=DSize

Now we need just 2 keys to finalize our plugin: BigEndian and StreamOffset.

The BigEndian key is important because it will tell XTool if a file is Little- or BigEndian. To find out the endianness just mark the CSize or DSize and change the endianness with the radio buttons. If the numbers you see there don't make any sense, then you know that the endianness is wrong.

https://i.imgur.com/py1hfSR.png


We know in this case that it is little endianness, so it should be BigEndian=0 in our plugin.

The other key is StreamOffset. Since our oodle kraken header "8C 06" is part of the entire stream, the offset should be set to "-2" because of the 2 bytes difference. Theoretically you can skip this part and just set is as "0", but XTool have to ensure that this is really a kraken stream, otherwise XTool would think anything beginning from the signature is a kraken stream, even if the signature is something else and there is nothing following. So basically this is just a safety measurement, if you say so.

Now our configuration is complete and it should look like this:

[Stream1]
Name=kraken
Codec=kraken
BigEndian=0
Signature=0x4C444F4F
Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize

Save it as INI file and name it like whatever you want. Here I name it "legotsws.ini" and place it next to xtool.exe

So finally lets test our plugin.

xtool:kraken
Compressed 1 file, 4,064,897,269 => 9,465,290,559 bytes. Ratio 232.85%
Compression time: cpu 3.20 sec/real 232.34 sec = 1%. Speed 17.50 mB/s
All OK

xtool:legotsws
Compressed 1 file, 4,064,897,269 => 9,830,656,238 bytes. Ratio 241.84%
Compression time: cpu 3.56 sec/real 210.97 sec = 2%. Speed 19.97 mB/s
All OK

We see the plugin performs better in case of speed and ratio.

-----------------------------------------------

Here is another example for oodle kraken for the game "Shadow of War". In this case making a plugin is useless, because there are many different signatures.

Here are the first 20 streams as 20 bytes long:

00 00 00 00 00 00 00 00 00 00 D7 0E 00 00 DD 36 00 00 8C 06
FD 0E DE F7 BD DF 86 85 62 76 E5 45 AB 42 AC 2E 9B 00 8C 06
8F 26 43 34 38 10 44 31 91 58 1A 02 00 00 D2 08 00 00 8C 06
72 63 65 73 00 00 00 00 00 00 60 0D 00 00 04 38 00 00 8C 06
00 00 00 00 00 00 00 00 00 58 0C 02 00 00 D6 08 00 00 8C 06
66 66 65 72 14 00 00 00 58 58 46 29 00 00 0E 44 00 00 8C 06
35 70 1A E0 91 4B 44 18 58 58 BC 48 00 00 00 00 01 00 8C 06
80 36 81 F1 57 34 FA 13 32 31 8A 73 00 00 00 00 01 00 8C 06
A0 C1 A3 C9 E9 78 24 16 58 58 4D 53 00 00 00 00 01 00 8C 06
F3 7C 3C CF 87 E7 43 58 58 58 77 50 00 00 00 00 01 00 8C 06
C7 E1 F9 78 3E CF A7 C3 63 58 7E 76 00 00 00 00 01 00 8C 06
AB D6 AA 26 4D 68 34 D1 34 C1 B2 C0 94 24 08 81 A9 D2 8C 06
E1 E9 F9 3C 9F C7 F3 38 58 58 6F 5E 00 00 00 00 01 00 8C 06
63 01 8D CE 8D 57 46 1D 1B 9D 18 7D 1A BE 8D C1 8C 21 8C 06
18 CC 25 73 A9 4C 2A 70 06 58 41 4B 00 00 00 00 01 00 8C 06
5E F7 3A 5A AD 6B 5A CC 63 01 EB 2B 5A D6 63 18 C6 31 8C 06
7F 22 E8 81 08 82 78 4F 58 58 58 2E 5F 00 00 00 00 01 00 8C
1E 68 1C 2B 04 40 E9 2B 58 58 61 69 00 00 00 00 01 00 8C 06
8F 33 CF 23 95 19 68 58 58 58 5B 45 00 00 00 00 01 00 8C 06
F1 38 1C 0D C7 79 9C 07 68 58 A4 43 00 00 00 00 01 00 8C 06

Did you see it? Many signatures are 2 bytes "58 58" or just 1 byte "58". And the other half are just random bytes, which don't follow any pattern.

Testing:

xtool:kraken
Compressed 1 file, 8,289,538 => 26,917,140 bytes. Ratio 324.71%
Compression time: cpu 0.02 sec/real 2.13 sec = 1%. Speed 3.90 mB/s
All OK

xtool:mesow (signature 0x58)
Compressed 1 file, 8,289,538 => 22,599,836 bytes. Ratio 272.63%
Compression time: cpu 0.02 sec/real 1.46 sec = 1%. Speed 5.68 mB/s
All OK

[Stream1]
Name=kraken
Codec=kraken
BigEndian=0
Signature=0x58
Structure=Signature(1),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

However, maybe you have a game which has for example just 2 different signatures which you can use. In this case you make [Stream2] inside the same configuration file, like this:

[Stream1]
Name=kraken
Codec=kraken:l7
BigEndian=0
Signature=0x58
Structure=Signature(1),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

[Stream2]
Name=kraken
Codec=kraken:l7
BigEndian=0
Signature=0x7662
Structure=Signature(2),CSize(4),DSize(4),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x068C

-----------------------------------------------

Did you remember, in the beginning I said that plugins for zlib or zstd are useless? However some time ago I made one for zstd, just for fun and for learning stuff.

[Stream1]
Name=zstd
Codec=zstd
BigEndian=0
Signature=0x4454535A
Structure=Signature(4),DSize(4),Unknown(8),ZstdHdr(4),Stream
StreamOffset=-4
CompressedSize=0
DecompressedSize=DSize

The Signature was 4 bytes, followed by 4 bytes DSize (DecompressedSize), 8 bytes of unknown trash and 4 bytes zstd header. Maybe some of you would think now, where is CSize? CompressedSize is set to 0, just because the zstd library has a function for it to read the value and tells XTool about this value. But the interesting part here is just the "Unknown(8)" part I wanted to show you, because sometimes you can have something like the above example, were you have a Signature, followed by some unknown bytes, then CSize/DSize and the header.