How to use Diskspan GUI v2+

[Cesar82]

Thank you so much KaktoR for giving this real class.
If you haven't seen it, read the #post

Now let's get to the information I decided to share (again):
I will remind users that although the DiskSpan GUI has a "Masked" mode, I do not recommend using it for compressions that generate a very large Data#.bin file for the following reason.

The "Masked" mode splits the file after creating it, so during compression you will have to have that extra space of the Data#bin file size so you can compress.
So far nothing much, the problem is that a similar process is performed when installing/unpacking the file.

When installing, before extracting the Data#.bin file that was divided, a temporary file will be created on the disk in which all parts of this divided file will be merged and later this temporary file will be extracted.

Example:
Assuming that you compress a 150 GB game and that you use a "Masked" compression (Split Mode), and that after the final compression you get 110 GB (Ratio 73.0%), and choose "BD 50" as media.

When compressing, a single temporary file of 110 GB will be generated, and then this file will be processed, generating 3 new files with sizes of 46.56 GB, 46.56 GB and 16.88 GB respectively.
To compress this game, you will need at least 370 GB of disk space, 150 GB for installing the original game, 110 GB for the temporary file that will be deleted after completion, and 110 GB for the final conversion.

And if you want to test this conversion before uninstalling the original game?
Now the problems begin, as you'll need more space for this as well.

To install using an installer or unpack using the DiskSpan GUI's own features, you will need an additional 260GB.
Of those 260 GB, 150 GB will be occupied by installing/unpacking the game and 110 GB will be occupied by the merged file before the extraction starts.

So in summary:
To create this conversion and test that everything is working before deleting the original game you will need at least: 150 GB (original game size) + 110 GB (Conversion files size) + 110 GB (Temporary merged file size) + 150 GB (Installation test folder size) = 520 GB.

These 520 GB do not necessarily need to be on the same disk or partition. It would take 150GB to install the game on a partition, 220GB on the DiskSpan GUI destination partition to create the conversion, and 260GB (110GB + 150GB) on the chosen partition for the test unpack/installation.

If using DiskSpan mode (Not "Masked/SPLIT" mode), the temporary 110 GB for compression and also the 110 GB for installation/unpacking would not be needed because CLS-DiskSpan.dll (by Razor12911) helps unpack without being necessary merging the Data#.bin file again, and these additional tasks are not time consuming.

Complement:
One way to reduce this inconvenience would be to configure it to create multiple Data#.bin files and use the extension-based or directory-based include/exclude list to specify which files will be in each Data#.bin.

That way it would create smaller Data#.bin files, requiring less space to merge them, or even not being necessary to merge depending on the chosen media.

In the same example above, if you use 3 Data#.bin files, you can configure the first one to occupy a little less than the chosen media size (something like 45 GB) and the second one a little bigger than the chosen media (something like 49 GB), it would only be necessary to merge the second Data#.bin file because the second file would have parts on Disk1, most of it would be on Disk2, and the rest would be on Disk3, and the other 2 files would be whole one on each disk without need to split or merge them in the installation..

So make your choice wisely!

[adrianskiloses]

Whenever I click on Change Method (on Kaktor's guide) for the first step it gives me an error "Could not call proc" and suddenly the program "breaks". If I exit that menu I cannot change included and excluded files.

(edit: nevermind you mentioned that there was the bug.)
But is there a workaround for this? And if you don't mind, can I get access to 2.0.1.8?
Thanks.

[KaktoR]

How to know what compression method a game has

Here I will show you how to know what compression method a game has, so you know what precompression you can apply (or should) to achieve a good ratio.

I will make this more superficial, because I don't really want to dig deeper inside this rabbit hole, I could write a book about this otherwise. Another reason is simply that I don't know all about this. Furthermore I will explain only oodle compression here, just because oodle is nowadays very common.

First things first. In this showcase I will use xtool.exe + xtoolui.dll, just for simplification.

https://i.imgur.com/jtBUHUF.png


In the first example I will work with the game "The Last of Us Part 1", just because it's a current game. However I can tell you that this game use oodle compression, just before I made any efforts in testing with the game files in any way. How can I tell without doing anything? In the past, most games which used oodle compression had a file called "oo2core_x_win64.dll" somewhere in the game folder (replace "x" with an integer starting from 3 up to 9). In this example it is the case aswell. If you look in the root game folder you will notice the file "oo2core_9_win64.dll", so you can see directly that there is somewhere oodle compression used.

Ok, but oodle compression family has several codecs, like kraken, mermaid, selkie, hydra and leviathan. How to know what codec to use? First step is to scan an input, like a game file, with the oodle scanner by Razor12911.

In this example I use the file "common.psarc" as an input to test with. So just drag&drop this file onto "oo2scan_7_win64.exe", which opens a cmd window.

https://i.imgur.com/kG30rO9.png

Now you see stream list with some informations. The important information is the codec used. In the above example it is [1], which means kraken codec is used.

---------------------------

If you ever feel the need to check over, note the offset (Pos: 000xxxxx) and search for this offset in HxD (Search -> Go to... -> ...). As an example, I search for the first oodle stream (Pos: 00013750). So go to "Search" -> "Go to..." and type in as hex "13750".

https://i.imgur.com/kj7YOfU.png


As a side note, kraken header is "8C 06". If you want to dig deeper, you can also find out CSize and DSize (CompressedSize and DecompressedSize for this stream), but this will go too deep into this on this topic.

---------------------------

Anyways, we should move along with the more easier stuff. So we know now that the game is using oodle kraken compression. Now copy the file "oo2core_9_win64.dll" from the game folder in the same folder where xtool.exe is located and open xtool.exe and set the input to the file "common.psarc". For the beginning, you can leave all the settings to default. Now you press the "Configure" button, which will open a new window. There you select the Oodle option and tick the checkbox "Kraken"

https://i.imgur.com/NOBqfy1.png

Then just press "OK" and then press "Start" to begin the process. It looks like this then when it's finished.

https://i.imgur.com/WbgL7OK.png

There you can see some helpful informations, like the Original size and decompressed size of the file, the time it has needed and the amount of streams which were processed. In this case not all the streams were processed, but it is not that much, so in this case we can ignore it. Either it is because an incorrect library (oo2core_x_Win64.dll) was used or something else.

Repeating this step, but in addition with the verbose mode, we will see something like this.

https://i.imgur.com/86AT2Bt.png

In verbose mode we get even more usefull informations about all the streams, for example which level they are using (level means the compression level of the kraken codec: l1 - l9, where l4 is somewhat like medium compression).

The l# option is sometimes usefull to speed up the process a bit, but don't except big time saves here.

Another usefull option is "Number of scan iterations", which will increase the streams that could potentially be found. If set to n128 for example, you will find a bit more streams, which means that the output will be a bit bigger.

Example:

n32 default option
Streams: 14951 / 14974
Time: 00:00:09 (CPU 00:00:42)
Size: 1.10 GB >> 1.66 GB

n128
Streams: 16320 / 16331
Time: 00:00:16 (CPU 00:01:03)
Size: 1.10 GB >> 1.68 GB

As you can see, n128 finds more streams, the output is a bit larger, but the time to process this is also higher. So for a full game you have to ask yourself, if this additional time is worth it.

Another example with n256

Streams: 16679 / 16686
Time: 00:00:23 (CPU 00:01:27)
Size: 1.10 GB >> 1.69 GB

As you can see here, the time between n32 default and n256 has more then doubled to find and process the streams, and the output size just increased marginal. So it is always a consideration.

Now the question could be: is the overall output - after deduplication and compression - worth it to increase the scan iterations?

n32 default - srep m3f - fast lzma2 l6
Streams: 14951 / 14974
Time: 00:00:55 (CPU 00:04:58)
Duplicates: 126 (0.00 MB) [0.01 MB >> 7.04 MB]
Srep decompression memory: 209 MB [210 MB*]

Size: 1.10 GB >> 1.66 GB >> 1.65 GB >> 1.35 GB >> 0.99 GB

n256 - srep m3f - fast lzma2 l6
Streams: 16679 / 16686
Time: 00:01:09 (CPU 00:05:43)
Duplicates: 126 (0.00 MB) [0.01 MB >> 7.04 MB]
Srep decompression memory: 231 MB [232 MB*]

Size: 1.10 GB >> 1.69 GB >> 1.68 GB >> 1.35 GB >> 0.98 GB

Not that much, just a bit. However keep in mind that this was a test with fast-lzma2, so to get meaningful results you should test it with stronger lzma or lolz.

As a side note, this game also has several deflate streams behind the "oodle wall", if you can say so. A simplified example for imagination purposes: Imagine an egg, the eggshell is oodle compression, behind this shell there are some zlib streams, and the inner yellow is the raw data. Or the vise-versa way: some of the raw data are compressed with zlib -> then all data is compressed with oodle.

Example file "core.psarc"

kraken
Streams: 28187 / 28213
Time: 00:00:57 (CPU 00:00:58)

Size: 2.12 GB >> 3.37 GB

kraken+preflate
Streams: 1188 / 1188
Time: 00:01:18 (CPU 00:06:23)

Size: 3.37 GB >> 5.37 GB

This will make a difference in final compression. Let's take a look at this:

kraken - srep m3f - fast lzma2 l6
Size: 2.12 GB >> 3.37 GB >> 3.30 GB >> 2.52 GB >> 1.95 GB

kraken+preflate - srep m3f - fast lzma2 l6
Size: 3.37 GB >> 5.37 GB >> 5.36 GB >> 3.27 GB >> 1.90 GB

50mb saved. But here again, this is a test with fast-lzma2 plugin, so better test such things with stronger lzma or lolz to get meaningful results.

Now that we know something about the main data for this game, we can write a method for DiskSpan_GUI.
Personally I would suggest to use something like this:

xtool:mkraken,l4,n256:core_2.9.9+xtool:mpreflate+srep+...

:core_2.9.9 is important in the method command line, because it will tell DiskSpan_GUI which oodle library to use for this game. 2.9.9 is the library version the game is actually shipped with. In most cases this is always the correct library, but there are also exceptions like "Elden Ring", which is shipped with a different oodle library which will not work correctly processing the streams, or games which even doesn't has any library outside the executable, which will lead us to the next case.

---------------------------

As a side note, "The Last of Us Part 1" has also some video files in bink2 format. In older titles we just used binkpack (bpk) to compress them, but the newer titles use a codec which binkpack cannot process. In those cases you can open the bk2 file in HxD and look directly in the file header for "KB2n". If there is "n", then go ahead and just store this files, or use weak compression with deduplication to get some mb's out of it, if at all tbh.

---------------------------

So, what to do if a game use oodle compression but there is no library inside the game folder? For this example I will use "Assassin's Creed Valhalla", simply because I have installed it at the moment. In those cases it could be useful to know when development of a game has begun, or at least try to guess it. Or instead go for the release date of the game. The oodle development history can help you with this to pick the correct library version, or at least get near enough to it.

Ok so, "Assassin's Creed Valhalla" has no oodle library. The library code is somewhere inside the binaries of the game, so no way we can safely say what library version to use at this point. But we know that the release date of this game was "10. November 2020". So we look at the list I linked above. I would say, oodle library v2.8.X should be used, but which one exactly? Here comes the library checker, which is a part of xtool.

Here again, first check a file with the oodle scanner. I choose "DataPC.forge" for this.

https://i.imgur.com/3wooheG.png

In this case we see [2], which means the oodle mermaid codec is used, So we check xtoolui again and make some settings.

https://i.imgur.com/ZMXfN8h.png

So now we use the library checker. For this just select the drop-down menu and select "Oodle". Now you have to point to a folder, which contains different oodle libraries. For this purpose we have this. Just download the attachment and extract it somewhere.

https://i.imgur.com/U0OOO0l.png

Like I said already, my guess is that this game was compressed with a oodle version 2.8.X, but we don't know which version exactly. Now point to the folder ".\Oodle\v2.8.x" and select it, like in the image above.

Now start the process and wait until it's finished. This will take some time because XTool is now checking the file with different libraries one after another. To know the correct library watch out for the library which gives you the best results. Just compare amount of streams, times and sizes.

https://i.imgur.com/qNDAuDa.png

As you can see here it is very likely that a library from v2.8.0 to v2.8.4 was used for the game. It is now up to you which library to use. Additionaly you can check more game files and if the results are the same, then pick up any library you want. Personally I would use v2.8.4 library because there were some fixes to the oodle code before v2.8.4.

Streams: 4615 / 4615
Time: 00:00:46 (CPU 00:03:58)

Size: 1.03 GB >> 1.54 GB

Some games, as games built with Anvil Forge engine (Assassin's Creed for example) have a xtool plugin you can use.

Here is a example with the anvil plugin on the same file as above

Streams: 60440 / 60440
Time: 00:01:03 (CPU 00:06:10)

Size: 1.03 GB >> 1.84 GB

So here again a command line method for DiskSpan_GUI

xtool:manvil:core_2.8.4+srep+...

I hope I had forgot nothing here to explain.

Edit: Corrected some things.

[Cesar82]

Originally Posted by adrianskiloses

Whenever I click on Change Method (on Kaktor's guide) for the first step it gives me an error "Could not call proc" and suddenly the program "breaks". If I exit that menu I cannot change included and excluded files.

(edit: nevermind you mentioned that there was the bug.)
But is there a workaround for this? And if you don't mind, can I get access to 2.0.1.8?
Thanks.

This is still a beta version (Replace the file).
The final version 2.0.1.8 will have other changes, so when the final version is available delete that version and use the final version.

[adrianskiloses]

Originally Posted by Cesar82

This is still a beta version (Replace the file).
The final version 2.0.1.8 will have other changes, so when the final version is available delete that version and use the final version.

Thank you very much. Unfortunately I found another problem mentioned here.

[random_nik]

I'm trying to repack a game with language selection using this tool. I created a separate .bin files with required languages and created a chekboxes in the installer. Now i can't figure out where in the script to indicate the connection between the language files and chekboxes. I use a script file included with this tool. Can someone help me?

[sailwhist]

Originally Posted by KaktoR

How to set-up configuration with file exclusions

In this example I will choose the game "Hearts of Iron IV", which requires you to exclude a specific file for a given compression method, because of incompatiblities of OGGRE tool.

First run DiskSpan_GUI in developer mode. You have two choices how to make this.

• Hold CTRL key and double click on DiskSpan_GUI.exe
• Run the bat file "_Developer MODE.bat"

I created simple images because I think it is better to understand.

https://i.imgur.com/wMbSoy2.png


https://i.imgur.com/S189iRa.png


https://i.imgur.com/qrnqITj.png


https://i.imgur.com/Tchy5lL.png


https://i.imgur.com/NfM8cuX.png


https://i.imgur.com/ZGplU4N.png


Personally I do all this stuff via notepad, because it is faster for me. Here is a example how it could look like if you follow above instructions.


With the above settings you can compress the game down to ~3GB, while plain packing with 7-zip Ultra preset it is ~3.4GB. So you save around 400mb size.

Hello. Screenshots not working.

[KaktoR]

Determine which libraries should be used by XTool/oo2rec in DiskSpan_GUI

In DiskSpan_GUI (DSG) you have an option to determine which library version the tools should use.

For oodle algorithm you can either use

1. :3..9
   or
2. :core_#.#.#

I. The first option is using predifined libraries, which will not work well in many cases.
Example:

xtool:mkraken:5 = predifined library version for :5 is v2.5.2
oo2recm:7       = predifined library version for :7 is v2.7.0

II. The second option should the way to go in most cases, or if you want to max out detected streams. However because some of you may not know what library to use at all, you should read this, especially the part about the library checker.
Example:

xtool:mkraken:core_2.8.14 = library version v2.8.14 will be used
oo2recm:core_2.6.2        = library version v2.6.2 will be used

There are even examples which using different libraries. Perfect example is the game Elden Ring, which - in fact - is using two different oodle versions. You can determine different libraries for this game to get near 100% of all oodle streams.
Example:

xtool:mkraken:oodleoo2core_5_win64.dll:core_2.5.5+xtool:mkraken:oodleoo2core_8_win64.dll:core_2.8.14

The underlined part refers to the "--oodle=" xtool arc.ini setting.

In those cases start with the lower library version, the second one a higher version. Detecting those rare cases is annoying.


The same can be done with LZ4 (including LZ4HC and LZ4F) and ZSTD algorithms.

In those cases you could use something like this:

xtool:mlz4f:lz4_183
xtool:mzstd:zstd_190

Keep in mind that you cannot use LZ4 or LZ4HC directly without any plugin.

[KaktoR]

A bit out of scope here in this thread, but I don't want to open a new thread just for this...

How to know what library to use for oodle/zstd precompression?

In the following example I will take the game Monster Hunter Rise, which should be a good example of how to find out which library to use. Here again I will use XTool in UI mode (xtool.exe + xtoolui.dll which you can get from latest xtool packages).

Just taking the file "re_chunk_000.pak.patch_001.pak" to test with, we set our settings as follows.

https://i.imgur.com/zn3Xu0V.png


Set "Stop after [10] streams" should be enough in most cases to see if it's the correct library.

In the following output I have already marked all libraries which work as of now.

Library loaded: zstd_114\libzstd.dll

Streams: 10 / 17
Time: 00:00:00 (CPU 00:00:00)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_120\libzstd.dll

Streams: 10 / 17
Time: 00:00:00 (CPU 00:00:00)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_130\libzstd.dll

Streams: 0 / 17
Time: 00:00:08 (CPU 00:00:09)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_132\libzstd.dll

Streams: 0 / 17
Time: 00:00:09 (CPU 00:00:09)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_133\libzstd.dll

Streams: 10 / 17
Time: 00:00:00 (CPU 00:00:00)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_134\libzstd.dll

Streams: 10 / 17
Time: 00:00:00 (CPU 00:00:00)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_135\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_136\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_137\libzstd.dll

Streams: 0 / 17
Time: 00:00:07 (CPU 00:00:07)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_138\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:07)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_140\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_141\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:07)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_142\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_143\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_144\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_145\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_147\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_148\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_149\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_150\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_151\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_152\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_154\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Library loaded: zstd_155\libzstd.dll

Streams: 0 / 17
Time: 00:00:06 (CPU 00:00:06)

Size: 0.00 MB >> 0.00 MB

Done!!!

With library versions 1.1.4, 1.2.0, 1.3.3 and 1.3.4, xtool can process zstd streams at least, were with all other libraries we get 0 streams. Out of 24 libraries, we now have 4 libraries which we can test with further. Things get clearer here.

One would say, just use the newer library, but in this case it is not a good idea. Here is why.

1.3.3 / 1.3.4
Streams: 6949 / 15223
Time: 00:10:01 (CPU 00:41:07)

Size: 4.09 GB >> 5.70 GB

Done!!!

1.1.4 / 1.2.0
Streams: 15223 / 15223
Time: 00:00:41 (CPU 00:03:34)

Size: 4.09 GB >> 6.79 GB

Done!!!

So this is the evidence we need to decide what library to use.

The same principle applies to oodle codecs and LZ4F.

As for LZ4/LZ4HC, things will get a bit complicated, because this codecs cannot be used directly without a plugin.