#1
|
||||
|
||||
How to create XTool plugins (configuration based)
__________________
Haters gonna hate
Last edited by KaktoR; 13-09-2023 at 08:22. |
The Following 7 Users Say Thank You to KaktoR For This Useful Post: | ||
Cesar82 (15-08-2023), Junior53 (14-09-2023), L0v3craft (16-08-2023), Lord.Freddy (15-08-2023), mausschieber (16-08-2023), Razor12911 (15-08-2023), Wanterlude (15-08-2023) |
Sponsored Links |
#2
|
||||
|
||||
Introduction
Sometimes it is better to have a plugin for XTool to catch all the streams, or even speed up the precompression. Here I will show you how to write it and where to start. In this example I will choose a easy one, namely "LEGO Star Wars The Skywalker Saga", which is compressed with oodle kraken. XTool/oo2reck already performs good, but with a plugin it is even better. Side Notes Theoretically it is possible to make a plugin for majority of games, but only the minority of them are interestingly enough to make a plugin anyways. Other compression algorithms are a bit harder to understand, and for others it's totally useless to make one (example zlib and zstd). Maybe someone else can make a quick "how to" for algos like lz4/lz4hc. What do you need? HxD - https://mh-nexus.de/en/downloads.php?product=HxD20 Notepad (or notepad++) XTool UI verbose mode Lets start First lets take a look on a easy plugin sample: Code:
[Stream1] Name= Codec= BigEndian= Signature= Structure= StreamOffset= CompressedSize= DecompressedSize= Simultaneously open the same file in HxD. https://i.imgur.com/Z5xmjgt.png Copy the first offset 135674 and go to it in HxD (CTRL+G). You will see kraken header "8C 06". So now we have our kraken header "8C 06" at offset 135674. Again looking at XTool verbose information on the first stream: Code:
Actual kraken stream found at 0000000000135674 (2139 >> 9287) https://i.imgur.com/juCSHtr.png Now we have to know how such streams are constructed. For the oodle compression family it is mostly a shemata like this (at least for kraken and mermaid): Code:
CSize - DSize - Header https://i.imgur.com/AzO89a7.png Now we can check if our thoughts are correct. Again we look at our first stream in XTool verbose information: Code:
Actual kraken stream found at 0000000000135674 (2139 >> 9287) Code:
CSize (4 bytes) - DSize (4 bytes) - Header (2 bytes) The same with CSize https://i.imgur.com/Y5usWSO.png So basically we know now everything what XTool is looking for. Now comes the part what makes a plugin usefull. But first we note what informations we got until now: We know the oodle kraken header, which is 2 bytes We know the DSize for the stream which is 4 bytes We know the CSize for the stream which is 4 bytes Which for the Structure key it is like this: Code:
CSize(number of bytes),DSize(number of bytes),OodleHdr(number of bytes) CSize(4),DSize(4),OodleHdr(2) So lets write them right away in our configuration plugin as follow: Code:
[Stream1] Name=kraken Codec=kraken BigEndian= Signature= Structure=CSize(4),DSize(4),OodleHdr(2) StreamOffset= CompressedSize=CSize DecompressedSize=DSize Code:
A2 0F 00 27 00 14 4F 4F 44 4C 5B 08 00 00 47 24 00 00 8C 06 00 00 00 00 00 00 4F 4F 44 4C B6 19 00 00 00 80 00 00 8C 06 40 48 1C 48 30 24 4F 4F 44 4C BF 17 00 00 00 80 00 00 8C 06 94 88 01 93 91 38 4F 4F 44 4C E8 19 00 00 00 80 00 00 8C 06 01 4F B2 83 03 20 4F 4F 44 4C DC 17 00 00 00 80 00 00 8C 06 3A 61 10 8B 83 2A 4F 4F 44 4C C8 18 00 00 00 80 00 00 8C 06 A8 18 40 2D 18 28 4F 4F 44 4C BE 1A 00 00 00 80 00 00 8C 06 0E 02 05 EA 14 43 4F 4F 44 4C 53 17 00 00 00 80 00 00 8C 06 02 C8 B0 29 42 30 4F 4F 44 4C 11 15 00 00 00 80 00 00 8C 06 ... ... ... In the above table of streams we see a repetitive pattern, our signature. The Signature in this case is "4F 4F 44 4C", so 4 bytes long. https://i.imgur.com/jRBnaMA.png Now that we have our repetitive pattern (Signature), we fill in the information again: Code:
Signature(number of bytes),CSize(number of bytes),DSize(number of bytes),OodleHdr(number of bytes) Signature(4),CSize(4),DSize(4),OodleHdr(2) Code:
[Stream1] Name=kraken Codec=kraken BigEndian= Signature= Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2) StreamOffset= CompressedSize=CSize DecompressedSize=DSize Also we have to fill the Signature= key with our pattern. Because of endianness and the inclusion of "0x", we have to write the signature "backwards". "4F 4F 44 4C" will become "4C 44 4F 4F" and adding "0x" at the start will finally become this: Code:
Signature=0x4C444F4F Again we fill in the informations: Code:
[Stream1] Name=kraken Codec=kraken BigEndian= Signature=0x4C444F4F Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2) StreamOffset= CompressedSize=CSize DecompressedSize=DSize The BigEndian key is important because it will tell XTool if a file is Little- or BigEndian. To find out the endianness just mark the CSize or DSize and change the endianness with the radio buttons. If the numbers you see there don't make any sense, then you know that the endianness is wrong. https://i.imgur.com/py1hfSR.png We know in this case that it is little endianness, so it should be BigEndian=0 in our plugin. The other key is StreamOffset. Since our oodle kraken header "8C 06" is part of the entire stream, the offset should be set to "-2" because of the 2 bytes difference. Theoretically you can skip this part and just set is as "0", but XTool have to ensure that this is really a kraken stream, otherwise XTool would think anything beginning from the signature is a kraken stream, even if the signature is something else and there is nothing following. So basically this is just a safety measurement, if you say so. Now our configuration is complete and it should look like this: Code:
[Stream1] Name=kraken Codec=kraken BigEndian=0 Signature=0x4C444F4F Structure=Signature(4),CSize(4),DSize(4),OodleHdr(2),Stream StreamOffset=-2 CompressedSize=CSize DecompressedSize=DSize So finally lets test our plugin. Code:
xtool:kraken Compressed 1 file, 4,064,897,269 => 9,465,290,559 bytes. Ratio 232.85% Compression time: cpu 3.20 sec/real 232.34 sec = 1%. Speed 17.50 mB/s All OK xtool:legotsws Compressed 1 file, 4,064,897,269 => 9,830,656,238 bytes. Ratio 241.84% Compression time: cpu 3.56 sec/real 210.97 sec = 2%. Speed 19.97 mB/s All OK ----------------------------------------------- Here is another example for oodle kraken for the game "Shadow of War". In this case making a plugin is useless, because there are many different signatures. Here are the first 20 streams as 20 bytes long: Code:
00 00 00 00 00 00 00 00 00 00 D7 0E 00 00 DD 36 00 00 8C 06 FD 0E DE F7 BD DF 86 85 62 76 E5 45 AB 42 AC 2E 9B 00 8C 06 8F 26 43 34 38 10 44 31 91 58 1A 02 00 00 D2 08 00 00 8C 06 72 63 65 73 00 00 00 00 00 00 60 0D 00 00 04 38 00 00 8C 06 00 00 00 00 00 00 00 00 00 58 0C 02 00 00 D6 08 00 00 8C 06 66 66 65 72 14 00 00 00 58 58 46 29 00 00 0E 44 00 00 8C 06 35 70 1A E0 91 4B 44 18 58 58 BC 48 00 00 00 00 01 00 8C 06 80 36 81 F1 57 34 FA 13 32 31 8A 73 00 00 00 00 01 00 8C 06 A0 C1 A3 C9 E9 78 24 16 58 58 4D 53 00 00 00 00 01 00 8C 06 F3 7C 3C CF 87 E7 43 58 58 58 77 50 00 00 00 00 01 00 8C 06 C7 E1 F9 78 3E CF A7 C3 63 58 7E 76 00 00 00 00 01 00 8C 06 AB D6 AA 26 4D 68 34 D1 34 C1 B2 C0 94 24 08 81 A9 D2 8C 06 E1 E9 F9 3C 9F C7 F3 38 58 58 6F 5E 00 00 00 00 01 00 8C 06 63 01 8D CE 8D 57 46 1D 1B 9D 18 7D 1A BE 8D C1 8C 21 8C 06 18 CC 25 73 A9 4C 2A 70 06 58 41 4B 00 00 00 00 01 00 8C 06 5E F7 3A 5A AD 6B 5A CC 63 01 EB 2B 5A D6 63 18 C6 31 8C 06 7F 22 E8 81 08 82 78 4F 58 58 58 2E 5F 00 00 00 00 01 00 8C 1E 68 1C 2B 04 40 E9 2B 58 58 61 69 00 00 00 00 01 00 8C 06 8F 33 CF 23 95 19 68 58 58 58 5B 45 00 00 00 00 01 00 8C 06 F1 38 1C 0D C7 79 9C 07 68 58 A4 43 00 00 00 00 01 00 8C 06 Testing: Code:
xtool:kraken Compressed 1 file, 8,289,538 => 26,917,140 bytes. Ratio 324.71% Compression time: cpu 0.02 sec/real 2.13 sec = 1%. Speed 3.90 mB/s All OK xtool:mesow (signature 0x58) Compressed 1 file, 8,289,538 => 22,599,836 bytes. Ratio 272.63% Compression time: cpu 0.02 sec/real 1.46 sec = 1%. Speed 5.68 mB/s All OK Code:
[Stream1] Name=kraken Codec=kraken BigEndian=0 Signature=0x58 Structure=Signature(1),CSize(4),DSize(4),OodleHdr(2),Stream StreamOffset=-2 CompressedSize=CSize DecompressedSize=DSize Condition1=OodleHdr = 0x068C Code:
[Stream1] Name=kraken Codec=kraken:l7 BigEndian=0 Signature=0x58 Structure=Signature(1),CSize(4),DSize(4),OodleHdr(2),Stream StreamOffset=-2 CompressedSize=CSize DecompressedSize=DSize Condition1=OodleHdr = 0x068C [Stream2] Name=kraken Codec=kraken:l7 BigEndian=0 Signature=0x7662 Structure=Signature(2),CSize(4),DSize(4),OodleHdr(2),Stream StreamOffset=-2 CompressedSize=CSize DecompressedSize=DSize Condition1=OodleHdr = 0x068C Did you remember, in the beginning I said that plugins for zlib or zstd are useless? However some time ago I made one for zstd, just for fun and for learning stuff. Code:
[Stream1] Name=zstd Codec=zstd BigEndian=0 Signature=0x4454535A Structure=Signature(4),DSize(4),Unknown(8),ZstdHdr(4),Stream StreamOffset=-4 CompressedSize=0 DecompressedSize=DSize
__________________
Haters gonna hate
|
The Following 14 Users Say Thank You to KaktoR For This Useful Post: | ||
Cesar82 (15-08-2023), Gehrman (15-08-2023), Junior53 (14-09-2023), kenzo34 (15-08-2023), kj911 (16-08-2023), kuyhaa (23-08-2023), L0v3craft (16-08-2023), Lord.Freddy (15-08-2023), Masquerade (15-08-2023), mausschieber (16-08-2023), murphy78 (15-09-2023), rambohazard (21-08-2023), Razor12911 (15-08-2023), Wanterlude (15-08-2023) |
#3
|
||||
|
||||
Oodle kraken with size byte difference
Introduction Sometimes it could be the case that either CSize or DSize has a byte difference. In this case I will show you how to deal with such things. Lets start First lets take a look on the data we have. https://i.imgur.com/pJAVOGK.png We have a repetitive pattern (Signature) 7 bytes long, some unknown trash 10 bytes long, DSize 4 bytes, CSize 4 bytes and our kraken header 2 bytes. Now lets look for the sizes. I can say here that DSize matches, but what happened to CSize? https://i.imgur.com/THt7MkJ.png Next stream, same thing. https://i.imgur.com/0ZYM8ey.png And again, same thing on the next stream. https://i.imgur.com/Y1nEpCR.png So the CSize does not match here. In fact on every stream you have a 12 byte difference. I cannot tell you how that comes, or what happened here technically, but we can fix it within the plugin configuration. All we have to do now is to subtract the 12 bytes from the CSize. Code:
CompressedSize=CSize - 12 Code:
[Stream1] Name=kraken Codec=kraken:l6 BigEndian=0 Signature=0x64657070617257 Structure=Signature(7),Unknown(10),CSize(4),DSize(4),OodleHdr(2),Stream StreamOffset=-2 CompressedSize=CSize - 12 DecompressedSize=DSize Condition1=OodleHdr = 0x068C Code:
xtool:kraken Compressed 1 file, 233,487,310 => 714,300,604 bytes. Ratio 305.93% Compression time: cpu 0.16 sec/real 30.54 sec = 1%. Speed 7.64 mB/s All OK xtool:testplugin Compressed 1 file, 233,487,310 => 714,358,929 bytes. Ratio 305.95% Compression time: cpu 0.14 sec/real 23.04 sec = 1%. Speed 11.14 mB/s All OK
__________________
Haters gonna hate
Last edited by KaktoR; 16-08-2023 at 05:17. |
The Following 11 Users Say Thank You to KaktoR For This Useful Post: | ||
Gehrman (18-08-2023), Junior53 (14-09-2023), kj911 (16-08-2023), L0v3craft (16-08-2023), Lord.Freddy (16-08-2023), mausschieber (16-08-2023), Pantsi (16-08-2023), rambohazard (21-08-2023), Razor12911 (16-08-2023), shazzla (16-08-2023), Wanterlude (16-08-2023) |
#4
|
||||
|
||||
Introduction
This post is about the frostbite 3 engine, how to deal with lz4, zlib, zstd and oodle kraken streams. It is recommended to read the previous posts, about how to search for offsets and such. In frostbite3 engine the structure is like this commonly for lz4: Code:
DSize(#),Signature(#),CSize(#),Stream Beginning with lz4, first make a batch file next to xtool.exe with the following command Code:
xtool.exe precomp -mlz4:s64k -t100p -v -s - - < %1 > %1.out Code:
XTool is created by Razor12911 [0] Performing scan from block 0000000000000000 to 0000000000703A01 (7354882) [0] Actual lz4 stream found at 00000000000BF1EA (21353 >> 65536) [0] Actual lz4 stream found at 000000000035142B (20177 >> 65536) [0] Actual lz4 stream found at 00000000004193CE (23693 >> 65536) [0] Actual lz4 stream found at 000000000046D953 (12877 >> 65536) [0] Actual lz4 stream found at 000000000052B0B2 (12330 >> 65536) [0] Actual lz4 stream found at 000000000060A5D6 (26312 >> 65536) [0] Actual lz4 stream found at 0000000000681013 (23435 >> 65536) [0] Processing streams on block 0000000000000000 to 0000000000703A01 (7354882) Streams: 7 / 7 Time: 00:00:01 (CPU 00:00:01) Size: 7.01 MB >> 7.32 MB However, go to the first offset (in this case BF1EA). Unlike in the previous examples, this one has no specific header, but we don't need one here. https://i.imgur.com/BFEKAsp.png Now we have to go just 2 bytes back before the offset starts and we will see the CSize (you might want to change the endianness). https://i.imgur.com/R2nZnYo.png DSize and Signature: https://i.imgur.com/9RrSGGd.png We use BigEndian and Streamoffset is set to 0. Now the plugin is nearly complete. Code:
[Stream1] Name=lz4 Codec=lz4 BigEndian=1 Signature=0x7009 Structure=DSize(3),Signature(2),CSize(2),Stream StreamOffset=0 CompressedSize=CSize DecompressedSize=DSize Testing: Code:
xtool:testplugin Compressed 1 file, 7,354,882 => 14,408,711 bytes. Ratio 195.91% Compression time: cpu 0.02 sec/real 1.61 sec = 1%. Speed 4.56 mB/s All OK
__________________
Haters gonna hate
|
The Following 5 Users Say Thank You to KaktoR For This Useful Post: | ||
Junior53 (14-09-2023), L0v3craft (14-09-2023), Lord.Freddy (13-09-2023), mausschieber (13-09-2023), Razor12911 (14-09-2023) |
#5
|
||||
|
||||
i have tried your method of creating ini based xtool plugins the idea in its core is nice to be able with simple adjustments to gain 2%~ 3% ratio and about 10%~15% in speed. speed only comes in compression mode where compress once decompress many applies!!
generally i have tried 2 games to create with your guide above one is Sekiro which uses kraken and the second one is zstd only for resident evil 2 and resident evil 3 series!! and failed an average user will fail this easily i will upload samples and an explanation. maybe this idea is very nice if we get it as per engine because as it is, it cant be used!!
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
#6
|
||||
|
||||
Of course this is not the same case for every game or engine. Initially I wanted to point it out on second post, but unfortunatelly I cannot edit it because the forum will kick me back to front page of fileforums.com. Most like it is some bug in the software this forum is using.
As for the game Sekiro, I guess it is not possible here to create ini based plugins, because most likely some of the information is stored in the bhd files. I once looked on the game Armored Core 6, which is using the same engine. All I found was header and csize, nothing more. It looked like this Code:
header unk csize 8C 06 - 00 - 2A 32 - 80 1C 5E 00 00 80 BF 00 00 00 00 30 43 60 0C 9A 40 3E F8 0C 30 9E 9E 88 7B 53 33 29 96 55 95 A1 13 9E 5B 31 65 25 25 55 0B 25 9B 99 28 8C 06 - 00 - 37 B9 - 80 1B 77 00 00 00 00 00 00 00 00 50 45 A4 0B F6 83 40 19 70 05 07 9A 49 A0 8A C6 AC F0 B0 C9 2A B2 24 D3 CC B6 52 58 A6 94 C8 24 D4 A9 8C 06 - 00 - 34 80 - 80 1F AC FF FF 14 00 FF FF FF FF 50 53 FC 0E 10 03 A4 03 0F 88 77 00 10 04 79 9C 39 C8 7B 85 24 82 12 52 12 65 28 24 92 24 A7 4A 27 01 8C 06 - 00 - 35 06 - 80 1F DC FF FF FF FF FF FF FF FF 50 53 D8 0E 21 03 C3 DB 36 98 E8 A1 0A DF 8D D5 8D D5 CA 55 35 DC 96 AB 7A B5 62 94 AA D7 19 56 AE AA 8C 06 - 00 - 33 7A - 80 20 2C 0A 00 00 00 02 00 00 00 50 53 D8 0E 52 02 40 32 40 08 9A 9C 3A 04 2D 53 65 B4 9D 73 2B 04 45 92 95 84 9E AA 54 25 4A D4 A4 4C 8C 06 - 00 - 30 F1 - 80 1D 7D 01 00 00 00 00 00 80 BF 50 4B E8 0C A8 83 40 15 D4 04 27 8A DA 80 20 08 E6 66 14 27 38 42 46 D8 48 24 38 4A 66 53 EC D2 82 49 8C 06 - 00 - 37 23 - 80 1F 40 FF FF 7F 00 7C 00 00 00 50 56 EC 0E 2D 83 C4 67 AE 8A 5B 00 10 04 70 C9 C0 24 66 4C 40 92 26 44 84 82 1D 89 0C 9E 6C B8 CA 59 8C 06 - 00 - 3A 5E - 80 20 72 00 00 00 00 00 00 00 00 50 54 FC 0E 3E 83 30 21 0C 05 6B 40 1C C0 05 20 96 8A 82 19 A4 E2 AA 25 31 11 18 45 89 22 91 25 0B 50 8C 06 - 00 - 24 A2 - 80 15 4B 0E 00 00 00 06 00 00 00 30 2A C8 07 D4 40 29 C4 07 B6 9B 5B 04 79 5B A5 2A 94 9C EA A0 95 45 54 C4 D1 2A 49 59 82 24 AE A5 4A 8C 06 - 00 - 2C 88 - 80 1D 56 00 FF 00 00 00 00 00 00 30 45 E0 0C 3C 40 44 6C 0C 12 9C 3A 44 2B 53 56 A4 95 51 95 A8 A0 A6 56 B6 8C B3 49 56 11 29 55 6A 5A 8C 06 - 00 - 12 A6 - 80 12 A4 00 00 00 00 00 00 00 00 40 29 A0 07 A1 9B 09 C4 79 AA 95 95 4A 4A 47 55 05 11 58 D3 25 8C 92 A4 52 50 92 49 1C A5 5A D2 E5 79 8C 06 - 00 - 40 25 - 80 0D B4 00 00 00 00 00 00 00 00 30 2E 04 07 66 40 29 54 07 0A 96 9A 64 53 04 84 A2 88 B1 94 D8 CC 92 49 09 11 A9 06 11 11 AB 91 37 A9
__________________
Haters gonna hate
|
#7
|
||||
|
||||
i didnt have time last night to post an explanation of my findings so here we go
sample = a chunk of resident evil 2 with 2 zstd streams. re2.ini [Stream1] Name=zstd Codec=zstd:l11:f0:b0 BigEndian=1 Signature=0xA7DA Structure=Dict(1),DSize(3),Signature(2),CSize(2),Z StdHdr(4),Stream StreamOffset=0 CompressedSize=CSize DecompressedSize=DSize Condition1=ZStdHdr = 0x28B52FFD attachment1= ZStdHdr attachment2= xtool verbose attachment3= HEX csize
__________________
My projects : Masked Compression, lzma2(xz) on Freearc, Zstd compressor for windows My optimizations : packjpg.exe, zstd, lzham, precomp-dev-0.45. |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
XTool 2020 (Main Project) | Razor12911 | Conversion Tutorials | 697 | 10-10-2024 05:30 |
XTool 2020 (Plugins) | Razor12911 | Conversion Tutorials | 404 | 13-07-2024 09:52 |
How to patch languages out of game files (xtool erase decode) | KaktoR | Conversion Tutorials | 5 | 01-11-2022 09:54 |
XTool 2019 (Plugins) | Razor12911 | Conversion Tutorials | 50 | 16-05-2020 07:14 |
Nero PMA Update failure | angelosbg | CD/DVD Software & Utilities | 9 | 24-09-2008 04:40 |