Best Compression Methods for 'Specific' Games. Q&A

[Wanterlude]

SOUTH PARK: SNOW DAY! [v.70374]
27.3 gb > 16.77 gb

Unreal Engine 4 Plugin (by Razor)+zlib+dd5+lolz
Unreal Key=0x0892FE168A3B47FA0406BB6255A17DE7D1FBA7D651CD BC694728A31DFA99BA1B

Inst.time: 6 min [11400F]

-----

The Callisto Protocol (English VO Only) [Build 13179062]
75.8 gb > 55.8 gb

zlib+dd5+lzma2

Inst.time: 12 min [11400F]

[KaktoR]

Originally Posted by dixen

Which version and settings for srep are you using for Cities Skylines 2? On second pass levelerror=XXXXXXXXX. Srep version and his settings doesn't matter. ZSTDRec or XTool - too.

While the old initial game version worked just fine, I got problems with the new game version 1.1.0.F1.

Then I thought maybe new xtool version was the problem (do not wanted to blame Razor for that lol) so then I tested all xtool versions back to 0.7.9, same error. But then I remembered a problem a while ago about this game with some srep errors, so then I tested just precompressing and srep afterwards. While xtool did not gave me any error messages about srep, I tested srep instead of xtool dedup with srep.exe and this error appears:

Compressing 176,993,820,246 bytes with "srep.exe" -hash- -m3f InFile OutFile
Errorlevel=-1073741819
100.0%
ERROR: general (de)compression error in srep

If using xtool's dd feature using srep.exe, I got no error message at all but compressed data was corrupted like this:

60 GB input  -> 7 GB output, Ratio 11.67%

I even tested all different srep versions I have but all the same. Then I remembered another game where srep had big problems aswell. So after testing a bit again further with the new game version, I found out you have just to use FA's internally old rep before using srep, which will look something like this then:

xtool:zstd+rep+srep+...

Log

Initial version (xtool:zstd:dd3+4x4:lzma)
Overall input size: 55,63 GB
Overall output size: 34,48 GB (Ratio 61.97%)

Version 1.1.0.F1 (xtool:zstd+rep+srep+4x4:lzma)
Overall input size: 59.34 GB
Overall output size: 36.59 GB (Ratio 61.66%)

Extraction time about 20 minutes.

I hope this helps.

[Wanterlude]

Originally Posted by KaktoR

While the old initial game version worked just fine, I got problems with the new game version 1.1.0.F1.

Then I thought maybe new xtool version was the problem (do not wanted to blame Razor for that lol) so then I tested all xtool versions back to 0.7.9, same error. But then I remembered a problem a while ago about this game with some srep errors, so then I tested just precompressing and srep afterwards. While xtool did not gave me any error messages about srep, I tested srep instead of xtool dedup with srep.exe and this error appears:

Compressing 176,993,820,246 bytes with "srep.exe" -hash- -m3f InFile OutFile
Errorlevel=-1073741819
100.0%
ERROR: general (de)compression error in srep

If using xtool's dd feature using srep.exe, I got no error message at all but compressed data was corrupted like this:

60 GB input  -> 7 GB output, Ratio 11.67%

I even tested all different srep versions I have but all the same. Then I remembered another game where srep had big problems aswell. So after testing a bit again further with the new game version, I found out you have just to use FA's internally old rep before using srep, which will look something like this then:

xtool:zstd+rep+srep+...

Log

Initial version (xtool:zstd:dd3+4x4:lzma)
Overall input size: 55,63 GB
Overall output size: 34,48 GB (Ratio 61.97%)

Version 1.1.0.F1 (xtool:zstd+rep+srep+4x4:lzma)
Overall input size: 59.34 GB
Overall output size: 36.59 GB (Ratio 61.66%)

Extraction time about 20 minutes.

I hope this helps.

I just added the srep parameters that dixen18 helped and everything worked:

packcmd   = xtool.exe precomp -t100p -dd5 -spa0:l1024 - - <stdin> <stdout>

[KaktoR]

^
Not for me, so I post an alternative if that does not work for someone

[observer23232]

Hello does anyone know effective compression on Cities Skylines II ?

[dixen]

Originally Posted by observer23232

Hello does anyone know effective compression on Cities Skylines II ?

XTool:zstd оn *.cok files

It will require approximately 170 GB of free space for decompressed data and 90 GB for processing and compression.
To avoid a SREP error, use the -l1024 parameter

[observer23232]

Originally Posted by dixen

XTool:zstd оn *.cok files

It will require approximately 170 GB of free space for decompressed data and 90 GB for processing and compression.
To avoid a SREP error, use the -l1024 parameter

Mate, please show me what parameters would you use?

i use

xzstd+srep:m3f:l1024+lolz:dtb1:d128:mtt0:mt4:mc1023

probably something wrong

[dixen]

Originally Posted by observer23232

Mate, please show me what parameters would you use?

i use

xzstd+srep:m3f:l1024+lolz:dtb1:d128:mtt0:mt4:mc1023

probably something wrong

That's right, but this does not guarantee that there will be no errors from SREP. I got the same error again even when using l1024

[Danziel123]

maybe someone knows the best compression method for the game white day 2? This is the test file:
https://drive.google.com/file/d/19Hb...ew?usp=sharing
Thank you

[KaktoR]

I will give you a hint.

Universal Oodle stream scanner

Created by Razor12911

[0] = Unknown/Invalid
[1] = Kraken/Hydra
[2] = Mermaid/Selkie/Hydra
[3] = Leviathan/Hydra

0001[1]| Pos: 00000049, Size: 503
0002[1]| Pos: 00000849, Size: 1470
0003[1]| Pos: 00001849, Size: 2474
0004[1]| Pos: 00002849, Size: 3324
0005[1]| Pos: 00004049, Size: 2241
0006[1]| Pos: 00005049, Size: 3794
0007[1]| Pos: 00006049, Size: 698
0008[1]| Pos: 0000634C, Size: 36258
0009[1]| Pos: 0000F859, Size: 19065
0010[1]| Pos: 000142D2, Size: 12070
0011[1]| Pos: 00017849, Size: 14203
0012[1]| Pos: 0001B049, Size: 1013
0013[1]| Pos: 0001BC39, Size: 97014
0014[1]| Pos: 0003372F, Size: 119039
0015[1]| Pos: 0005082E, Size: 172366
...
...
...

Streams: 4382 / 4382
Time: 00:00:26 (CPU 00:02:24)

Size: 374 MB >> 1.05 GB

Now see what oodle core library you have to use.

You can do it bro.

[Danziel123]

Originally Posted by KaktoR

I will give you a hint.

Universal Oodle stream scanner

Created by Razor12911

[0] = Unknown/Invalid
[1] = Kraken/Hydra
[2] = Mermaid/Selkie/Hydra
[3] = Leviathan/Hydra

0001[1]| Pos: 00000049, Size: 503
0002[1]| Pos: 00000849, Size: 1470
0003[1]| Pos: 00001849, Size: 2474
0004[1]| Pos: 00002849, Size: 3324
0005[1]| Pos: 00004049, Size: 2241
0006[1]| Pos: 00005049, Size: 3794
0007[1]| Pos: 00006049, Size: 698
0008[1]| Pos: 0000634C, Size: 36258
0009[1]| Pos: 0000F859, Size: 19065
0010[1]| Pos: 000142D2, Size: 12070
0011[1]| Pos: 00017849, Size: 14203
0012[1]| Pos: 0001B049, Size: 1013
0013[1]| Pos: 0001BC39, Size: 97014
0014[1]| Pos: 0003372F, Size: 119039
0015[1]| Pos: 0005082E, Size: 172366
...
...
...

Streams: 4382 / 4382
Time: 00:00:26 (CPU 00:02:24)

Size: 374 MB >> 1.05 GB

Now see what oodle core library you have to use.

You can do it bro.

Use Kraken, right? but don't use something like :core_#.#.#? because I read your post on "How to use Diskspan GUI v2+" but I still don't understand

[Cesar82]

Originally Posted by Danziel123

Use Kraken, right? but don't use something like :core_#.#.#? because I read your post on "How to use Diskspan GUI v2+" but I still don't understand

In the diskspan gui "core_x.x.x" is a fake parameter that will be removed from the command line before compression.
This parameter is used to tell DiskSpan GUI which oodle library (oo2core_#_win64.dll) should be copied to the XTool folder before compression (The same oodle library will be included in Setup.dll).
If you don't know the supported versions, you can open the Oodle_Libs.7z file inside the "..\ADDONS\Libraries\Oodle" folder to find out which versions are currently supported as a fake parameter (The fake parameter is exactly the name of the folder inside the 7-Zip archive such as "core_2.6.3").
Usage example:
NOTES: In the first example, the "393b" parameter is also a fake parameter to inform which version of srep.exe will be used.

xtool:mkraken:core_2.7.6+srep_new:393b+LZMA-MT
xtool:mkraken:core_2.9.9+xtool:dd3+LZMA-MT
xtool:mkraken:core_2.9.5:dd3+LZMA-MT

[Danziel123]

Originally Posted by Cesar82

In the diskspan gui "core_x.x.x" is a fake parameter that will be removed from the command line before compression.
This parameter is used to tell DiskSpan GUI which oodle library (oo2core_#_win64.dll) should be copied to the XTool folder before compression (The same oodle library will be included in Setup.dll).
If you don't know the supported versions, you can open the Oodle_Libs.7z file inside the "..\ADDONS\Libraries\Oodle" folder to find out which versions are currently supported as a fake parameter (The fake parameter is exactly the name of the folder inside the 7-Zip archive such as "core_2.6.3").
Usage example:
NOTES: In the first example, the "393b" parameter is also a fake parameter to inform which version of srep.exe will be used.

xtool:mkraken:core_2.7.6+srep_new:393b+LZMA-MT
xtool:mkraken:core_2.9.9+xtool:dd3+LZMA-MT
xtool:mkraken:core_2.9.5:dd3+LZMA-MT

thanks for the explanation, but I tried putting the .pak file in oodle_ver but it got an error. How do I find out the correct libraby/compression method to use for this game based on posts from KaktoR? i know to use kraken but i don't know which one to use

[KaktoR]

oodle_ver.exe is just for the libraries to know what library version the file is (drop any "oo2core_#_win64.dll" onto oodle_ver.exe will tell you the library version without looking manually with hex editor).

For game files to scan use "oo2scan_7_win64.exe" inside scanner folder from this link:
https://fileforums.com/showthread.php?t=102453
PS: With big files, it could take a while before the process starts. Simply drag&drop the file into "oo2scan_7_win64.exe".

You cannot know what library is used for the game if the game is not shipped with any oo2core library, but the release date of the game is a good start. However for newer titles I suggest starting with library v2.9.5 is a good idea (do not use library v2.9.6 at all because this library has a critical memory bug. This particullar library is just included for completition). If output/speed of processing is bad, increase or decrease library version until you find the matching library.

[Danziel123]

Originally Posted by KaktoR

oodle_ver.exe is just for the libraries to know what library version the file is (drop any "oo2core_#_win64.dll" onto oodle_ver.exe will tell you the library version without looking manually with hex editor).

For game files to scan use "oo2scan_7_win64.exe" inside scanner folder from this link:
https://fileforums.com/showthread.php?t=102453
PS: With big files, it could take a while before the process starts. Simply drag&drop the file into "oo2scan_7_win64.exe".

You cannot know what library is used for the game if the game is not shipped with any oo2core library, but the release date of the game is a good start. However for newer titles I suggest starting with library v2.9.5 is a good idea (do not use library v2.9.6 at all because this library has a critical memory bug. This particullar library is just included for completition). If output/speed of processing is bad, increase or decrease library version until you find the matching library.

I'm starting to understand a little, sorry if I ask again, what application do you use to find out which library you are using can produce a good/bad ratio?