#16
|
||||
|
||||
Masked/SPLIT mode vs DiskSpan/CLS mode usage disk size
Thank you so much KaktoR for giving this real class.
If you haven't seen it, read the #post Now let's get to the information I decided to share (again): I will remind users that although the DiskSpan GUI has a "Masked" mode, I do not recommend using it for compressions that generate a very large Data#.bin file for the following reason. The "Masked" mode splits the file after creating it, so during compression you will have to have that extra space of the Data#bin file size so you can compress. So far nothing much, the problem is that a similar process is performed when installing/unpacking the file. When installing, before extracting the Data#.bin file that was divided, a temporary file will be created on the disk in which all parts of this divided file will be merged and later this temporary file will be extracted. Example: Assuming that you compress a 150 GB game and that you use a "Masked" compression (Split Mode), and that after the final compression you get 110 GB (Ratio 73.0%), and choose "BD 50" as media. When compressing, a single temporary file of 110 GB will be generated, and then this file will be processed, generating 3 new files with sizes of 46.56 GB, 46.56 GB and 16.88 GB respectively. To compress this game, you will need at least 370 GB of disk space, 150 GB for installing the original game, 110 GB for the temporary file that will be deleted after completion, and 110 GB for the final conversion. And if you want to test this conversion before uninstalling the original game? Now the problems begin, as you'll need more space for this as well. To install using an installer or unpack using the DiskSpan GUI's own features, you will need an additional 260GB. Of those 260 GB, 150 GB will be occupied by installing/unpacking the game and 110 GB will be occupied by the merged file before the extraction starts. So in summary: To create this conversion and test that everything is working before deleting the original game you will need at least: 150 GB (original game size) + 110 GB (Conversion files size) + 110 GB (Temporary merged file size) + 150 GB (Installation test folder size) = 520 GB. These 520 GB do not necessarily need to be on the same disk or partition. It would take 150GB to install the game on a partition, 220GB on the DiskSpan GUI destination partition to create the conversion, and 260GB (110GB + 150GB) on the chosen partition for the test unpack/installation. If using DiskSpan mode (Not "Masked/SPLIT" mode), the temporary 110 GB for compression and also the 110 GB for installation/unpacking would not be needed because CLS-DiskSpan.dll (by Razor12911) helps unpack without being necessary merging the Data#.bin file again, and these additional tasks are not time consuming. Complement: One way to reduce this inconvenience would be to configure it to create multiple Data#.bin files and use the extension-based or directory-based include/exclude list to specify which files will be in each Data#.bin. That way it would create smaller Data#.bin files, requiring less space to merge them, or even not being necessary to merge depending on the chosen media. In the same example above, if you use 3 Data#.bin files, you can configure the first one to occupy a little less than the chosen media size (something like 45 GB) and the second one a little bigger than the chosen media (something like 49 GB), it would only be necessary to merge the second Data#.bin file because the second file would have parts on Disk1, most of it would be on Disk2, and the rest would be on Disk3, and the other 2 files would be whole one on each disk without need to split or merge them in the installation.. So make your choice wisely! Last edited by Cesar82; 09-04-2023 at 06:46. Reason: Added complement |
The Following 4 Users Say Thank You to Cesar82 For This Useful Post: | ||
Sponsored Links |
#17
|
|||
|
|||
Whenever I click on Change Method (on Kaktor's guide) for the first step it gives me an error "Could not call proc" and suddenly the program "breaks". If I exit that menu I cannot change included and excluded files.
(edit: nevermind you mentioned that there was the bug.) But is there a workaround for this? And if you don't mind, can I get access to 2.0.1.8? Thanks. Last edited by adrianskiloses; 09-04-2023 at 09:54. |
#18
|
||||
|
||||
How to know what compression method a game has
Here I will show you how to know what compression method a game has, so you know what precompression you can apply (or should) to achieve a good ratio. I will make this more superficial, because I don't really want to dig deeper inside this rabbit hole, I could write a book about this otherwise. Another reason is simply that I don't know all about this. Furthermore I will explain only oodle compression here, just because oodle is nowadays very common. First things first. In this showcase I will use xtool.exe + xtoolui.dll, just for simplification. https://i.imgur.com/jtBUHUF.png In the first example I will work with the game "The Last of Us Part 1", just because it's a current game. However I can tell you that this game use oodle compression, just before I made any efforts in testing with the game files in any way. How can I tell without doing anything? In the past, most games which used oodle compression had a file called "oo2core_x_win64.dll" somewhere in the game folder (replace "x" with an integer starting from 3 up to 9). In this example it is the case aswell. If you look in the root game folder you will notice the file "oo2core_9_win64.dll", so you can see directly that there is somewhere oodle compression used. Ok, but oodle compression family has several codecs, like kraken, mermaid, selkie, hydra and leviathan. How to know what codec to use? First step is to scan an input, like a game file, with the oodle scanner by Razor12911. In this example I use the file "common.psarc" as an input to test with. So just drag&drop this file onto "oo2scan_7_win64.exe", which opens a cmd window. https://i.imgur.com/kG30rO9.png Now you see stream list with some informations. The important information is the codec used. In the above example it is [1], which means kraken codec is used. --------------------------- If you ever feel the need to check over, note the offset (Pos: 000xxxxx) and search for this offset in HxD (Search -> Go to... -> ...). As an example, I search for the first oodle stream (Pos: 00013750). So go to "Search" -> "Go to..." and type in as hex "13750". https://i.imgur.com/kj7YOfU.png As a side note, kraken header is "8C 06". If you want to dig deeper, you can also find out CSize and DSize (CompressedSize and DecompressedSize for this stream), but this will go too deep into this on this topic. --------------------------- Anyways, we should move along with the more easier stuff. So we know now that the game is using oodle kraken compression. Now copy the file "oo2core_9_win64.dll" from the game folder in the same folder where xtool.exe is located and open xtool.exe and set the input to the file "common.psarc". For the beginning, you can leave all the settings to default. Now you press the "Configure" button, which will open a new window. There you select the Oodle option and tick the checkbox "Kraken" https://i.imgur.com/NOBqfy1.png Then just press "OK" and then press "Start" to begin the process. It looks like this then when it's finished. https://i.imgur.com/WbgL7OK.png There you can see some helpful informations, like the Original size and decompressed size of the file, the time it has needed and the amount of streams which were processed. In this case not all the streams were processed, but it is not that much, so in this case we can ignore it. Either it is because an incorrect library (oo2core_x_Win64.dll) was used or something else. Repeating this step, but in addition with the verbose mode, we will see something like this. https://i.imgur.com/86AT2Bt.png In verbose mode we get even more usefull informations about all the streams, for example which level they are using (level means the compression level of the kraken codec: l1 - l9, where l4 is somewhat like medium compression). The l# option is sometimes usefull to speed up the process a bit, but don't except big time saves here. Another usefull option is "Number of scan iterations", which will increase the streams that could potentially be found. If set to n128 for example, you will find a bit more streams, which means that the output will be a bit bigger. Example: Code:
n32 default option Streams: 14951 / 14974 Time: 00:00:09 (CPU 00:00:42) Size: 1.10 GB >> 1.66 GB n128 Streams: 16320 / 16331 Time: 00:00:16 (CPU 00:01:03) Size: 1.10 GB >> 1.68 GB Another example with n256 Code:
Streams: 16679 / 16686 Time: 00:00:23 (CPU 00:01:27) Size: 1.10 GB >> 1.69 GB Now the question could be: is the overall output - after deduplication and compression - worth it to increase the scan iterations? Code:
n32 default - srep m3f - fast lzma2 l6 Streams: 14951 / 14974 Time: 00:00:55 (CPU 00:04:58) Duplicates: 126 (0.00 MB) [0.01 MB >> 7.04 MB] Srep decompression memory: 209 MB [210 MB*] Size: 1.10 GB >> 1.66 GB >> 1.65 GB >> 1.35 GB >> 0.99 GB Code:
n256 - srep m3f - fast lzma2 l6 Streams: 16679 / 16686 Time: 00:01:09 (CPU 00:05:43) Duplicates: 126 (0.00 MB) [0.01 MB >> 7.04 MB] Srep decompression memory: 231 MB [232 MB*] Size: 1.10 GB >> 1.69 GB >> 1.68 GB >> 1.35 GB >> 0.98 GB As a side note, this game also has several deflate streams behind the "oodle wall", if you can say so. A simplified example for imagination purposes: Imagine an egg, the eggshell is oodle compression, behind this shell there are some zlib streams, and the inner yellow is the raw data. Or the vise-versa way: some of the raw data are compressed with zlib -> then all data is compressed with oodle. Example file "core.psarc" Code:
kraken Streams: 28187 / 28213 Time: 00:00:57 (CPU 00:00:58) Size: 2.12 GB >> 3.37 GB Code:
kraken+preflate Streams: 1188 / 1188 Time: 00:01:18 (CPU 00:06:23) Size: 3.37 GB >> 5.37 GB Code:
kraken - srep m3f - fast lzma2 l6 Size: 2.12 GB >> 3.37 GB >> 3.30 GB >> 2.52 GB >> 1.95 GB Code:
kraken+preflate - srep m3f - fast lzma2 l6 Size: 3.37 GB >> 5.37 GB >> 5.36 GB >> 3.27 GB >> 1.90 GB Now that we know something about the main data for this game, we can write a method for DiskSpan_GUI. Personally I would suggest to use something like this: Code:
xtool:mkraken,l4,n256:core_2.9.9+xtool:mpreflate+srep+... --------------------------- As a side note, "The Last of Us Part 1" has also some video files in bink2 format. In older titles we just used binkpack (bpk) to compress them, but the newer titles use a codec which binkpack cannot process. In those cases you can open the bk2 file in HxD and look directly in the file header for "KB2n". If there is "n", then go ahead and just store this files, or use weak compression with deduplication to get some mb's out of it, if at all tbh. --------------------------- So, what to do if a game use oodle compression but there is no library inside the game folder? For this example I will use "Assassin's Creed Valhalla", simply because I have installed it at the moment. In those cases it could be useful to know when development of a game has begun, or at least try to guess it. Or instead go for the release date of the game. The oodle development history can help you with this to pick the correct library version, or at least get near enough to it. Ok so, "Assassin's Creed Valhalla" has no oodle library. The library code is somewhere inside the binaries of the game, so no way we can safely say what library version to use at this point. But we know that the release date of this game was "10. November 2020". So we look at the list I linked above. I would say, oodle library v2.8.X should be used, but which one exactly? Here comes the library checker, which is a part of xtool. Here again, first check a file with the oodle scanner. I choose "DataPC.forge" for this. https://i.imgur.com/3wooheG.png In this case we see [2], which means the oodle mermaid codec is used, So we check xtoolui again and make some settings. https://i.imgur.com/ZMXfN8h.png So now we use the library checker. For this just select the drop-down menu and select "Oodle". Now you have to point to a folder, which contains different oodle libraries. For this purpose we have this. Just download the attachment and extract it somewhere. https://i.imgur.com/U0OOO0l.png Like I said already, my guess is that this game was compressed with a oodle version 2.8.X, but we don't know which version exactly. Now point to the folder ".\Oodle\v2.8.x" and select it, like in the image above. Now start the process and wait until it's finished. This will take some time because XTool is now checking the file with different libraries one after another. To know the correct library watch out for the library which gives you the best results. Just compare amount of streams, times and sizes. https://i.imgur.com/qNDAuDa.png As you can see here it is very likely that a library from v2.8.0 to v2.8.4 was used for the game. It is now up to you which library to use. Additionaly you can check more game files and if the results are the same, then pick up any library you want. Personally I would use v2.8.4 library because there were some fixes to the oodle code before v2.8.4. Code:
Streams: 4615 / 4615 Time: 00:00:46 (CPU 00:03:58) Size: 1.03 GB >> 1.54 GB Here is a example with the anvil plugin on the same file as above Code:
Streams: 60440 / 60440 Time: 00:01:03 (CPU 00:06:10) Size: 1.03 GB >> 1.84 GB Code:
xtool:manvil:core_2.8.4+srep+... Edit: Corrected some things.
__________________
Haters gonna hate
Last edited by KaktoR; 15-04-2023 at 12:34. |
The Following 9 Users Say Thank You to KaktoR For This Useful Post: | ||
adrianskiloses (09-04-2023), Cesar82 (09-04-2023), jihack (16-07-2023), Junior53 (24-06-2023), kj911 (11-04-2023), mausschieber (09-04-2023), mykhydro (31-01-2024), sajmon83 (16-06-2024), Titeuf (10-04-2023) |
#19
|
||||
|
||||
Quote:
The final version 2.0.1.8 will have other changes, so when the final version is available delete that version and use the final version. |
The Following 4 Users Say Thank You to Cesar82 For This Useful Post: | ||
#20
|
|||
|
|||
Quote:
|
#21
|
|||
|
|||
I'm trying to repack a game with language selection using this tool. I created a separate .bin files with required languages and created a chekboxes in the installer. Now i can't figure out where in the script to indicate the connection between the language files and chekboxes. I use a script file included with this tool. Can someone help me?
|
#22
|
|||
|
|||
Quote:
|
#23
|
||||
|
||||
Determine which libraries should be used by XTool/oo2rec in DiskSpan_GUI
In DiskSpan_GUI (DSG) you have an option to determine which library version the tools should use. For oodle algorithm you can either use
I. The first option is using predifined libraries, which will not work well in many cases. Example: Code:
xtool:mkraken:5 = predifined library version for :5 is v2.5.2 oo2recm:7 = predifined library version for :7 is v2.7.0 Example: Code:
xtool:mkraken:core_2.8.14 = library version v2.8.14 will be used oo2recm:core_2.6.2 = library version v2.6.2 will be used Example: Code:
xtool:mkraken:oodleoo2core_5_win64.dll:core_2.5.5+xtool:mkraken:oodleoo2core_8_win64.dll:core_2.8.14 In those cases start with the lower library version, the second one a higher version. Detecting those rare cases is annoying. The same can be done with LZ4 (including LZ4HC and LZ4F) and ZSTD algorithms. In those cases you could use something like this: Code:
xtool:mlz4f:lz4_183 xtool:mzstd:zstd_190
__________________
Haters gonna hate
|
The Following 2 Users Say Thank You to KaktoR For This Useful Post: | ||
Cesar82 (19-12-2023), mausschieber (19-12-2023) |
#24
|
||||
|
||||
A bit out of scope here in this thread, but I don't want to open a new thread just for this...
How to know what library to use for oodle/zstd precompression? In the following example I will take the game Monster Hunter Rise, which should be a good example of how to find out which library to use. Here again I will use XTool in UI mode (xtool.exe + xtoolui.dll which you can get from latest xtool packages). Just taking the file "re_chunk_000.pak.patch_001.pak" to test with, we set our settings as follows. https://i.imgur.com/zn3Xu0V.png Set "Stop after [10] streams" should be enough in most cases to see if it's the correct library. In the following output I have already marked all libraries which work as of now. Code:
Library loaded: zstd_114\libzstd.dll Streams: 10 / 17 Time: 00:00:00 (CPU 00:00:00) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_120\libzstd.dll Streams: 10 / 17 Time: 00:00:00 (CPU 00:00:00) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_130\libzstd.dll Streams: 0 / 17 Time: 00:00:08 (CPU 00:00:09) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_132\libzstd.dll Streams: 0 / 17 Time: 00:00:09 (CPU 00:00:09) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_133\libzstd.dll Streams: 10 / 17 Time: 00:00:00 (CPU 00:00:00) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_134\libzstd.dll Streams: 10 / 17 Time: 00:00:00 (CPU 00:00:00) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_135\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_136\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_137\libzstd.dll Streams: 0 / 17 Time: 00:00:07 (CPU 00:00:07) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_138\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:07) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_140\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_141\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:07) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_142\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_143\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_144\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_145\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_147\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_148\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_149\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_150\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_151\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_152\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_154\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Library loaded: zstd_155\libzstd.dll Streams: 0 / 17 Time: 00:00:06 (CPU 00:00:06) Size: 0.00 MB >> 0.00 MB Done!!! One would say, just use the newer library, but in this case it is not a good idea. Here is why. Code:
1.3.3 / 1.3.4 Streams: 6949 / 15223 Time: 00:10:01 (CPU 00:41:07) Size: 4.09 GB >> 5.70 GB Done!!! Code:
1.1.4 / 1.2.0 Streams: 15223 / 15223 Time: 00:00:41 (CPU 00:03:34) Size: 4.09 GB >> 6.79 GB Done!!! The same principle applies to oodle codecs and LZ4F. As for LZ4/LZ4HC, things will get a bit complicated, because this codecs cannot be used directly without a plugin.
__________________
Haters gonna hate
|
The Following 5 Users Say Thank You to KaktoR For This Useful Post: | ||
Cesar82 (25-01-2024), Lord.Freddy (26-01-2024), mausschieber (25-01-2024), mykhydro (31-01-2024), sajmon83 (16-06-2024) |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
DiskSpan GUI | Cesar82 | Conversion Tutorials | 431 | 23-11-2024 08:00 |
DiskSpan on Linux | hydefromt70s | Conversion Tutorials | 1 | 15-10-2020 08:12 |
How to use diskspan bat | mausschieber | Conversion Tutorials | 4 | 05-09-2020 13:25 |
DiskSpan FreeArc returns an error | Titeuf | Conversion Tutorials | 2 | 18-07-2020 02:46 |
R.G. Gamers DiskSpan | Simorq | Conversion Tutorials | 1 | 28-10-2017 09:22 |