Best Compression Methods for 'Specific' Games. Q&A

[Masquerade]

Compass of Destiny: Istanbul

*.pak/*.ucas

oo2reck+srep+lolz

Everything else:

razor

4.9GB -> 3.6GB

Astronimo

srep+lolz

5.1GB -> 1GB

[KaktoR]

Baldurs Gate 3
v4.1.1.4216792

 17:44:59 - Selected ARC/DS method for Data1a-01.bin was: xtool:mbg3:1+xtool:mzlib:mpreflate+xtool:dd3+4x4:lzma
 17:44:59 - Selected ARC/DS method for Data1b-01.bin was: xtool:c1500mb:mbk2:df20p
 17:44:59 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:lzma
 17:44:59 - Selected ARC/DS method for Data1d-01.bin was: xtool:c128mb:mpreflate:mpng:d1+xtool:dd3+4x4:lzma
 17:44:59 - Selected ARC/DS method for Data1e-01.bin was: 7zdll-pm:m0=mp3det:m1=packmp3c
-------------------------------------------------------------------------------------------------------------------------------------------
 22:12:08 - Overall input size: 139,17 GB
 22:12:08 - Overall output size: 95,69 GB (Ratio 68.76%)
 22:12:08 - Overall conversion time: 04:27:07

SharedSounds.pak was unpacked, 400kb+ wem's were processed and erased out to get additional ratio.

[Dragonis40]

Good morning everybody! I'm trying to compress "Fallout: New Vegas Ultimate Edition" and my mask for *.bsa files and results are:

xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
21 files: 8,588,002,315 byte => 4,776,969,655 bytes. Ratio 55.62%

This site reports the following results:

*.bsa
xtool(zlib)+srep+lolz
(54.5% Ratio)

I suppose the compressed byte are the same bytes as I compressed, but the result is better. Which is the configuration of LOLZ? It seems it isn't reported...

Thanks for your reply!

[Dragonis40]

Originally Posted by Dragonis40

Good morning everybody! I'm trying to compress "Fallout: New Vegas Ultimate Edition" and my mask for *.bsa files and results are:

xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
21 files: 8,588,002,315 byte => 4,776,969,655 bytes. Ratio 55.62%

This site reports the following results:

*.bsa
xtool(zlib)+srep+lolz
(54.5% Ratio)

I suppose the compressed byte are the same bytes as I compressed, but the result is better. Which is the configuration of LOLZ? It seems it isn't reported...

Thanks for your reply!

Any suggestion?

[Dragonis40]

Originally Posted by Dragonis40

Good morning everybody! I'm trying to compress "Fallout: New Vegas Ultimate Edition" and my mask for *.bsa files and results are:

xt_zlib+srep_old:392+lolz:dtb1:d512m:mtb96:ldl5:mc 1023
21 files: 8,588,002,315 byte => 4,776,969,655 bytes. Ratio 55.62%

This site reports the following results:

*.bsa
xtool(zlib)+srep+lolz
(54.5% Ratio)

I suppose the compressed byte are the same bytes as I compressed, but the result is better. Which is the configuration of LOLZ? It seems it isn't reported...

Thanks for your reply!

Any ideas?

[Masquerade]

Very likely high dictionary size, all detections on, fast bytes off etc.

[Dragonis40]

Originally Posted by Masquerade

Very likely high dictionary size, all detections on, fast bytes off etc.

What does exactly mean? Can you post here an example of the parameters you're talking about please?

[KaktoR]

Alan Wake 2
Maybe someone may find this usefull.

I have attached xtl files to extract WEM sounds from both "base-generic-*.rmdblob" and "stream0-generic-*.rmdblob" files (only files which are atleast 100kb in size). The xtl files are compatible with game version 1.0.14.

Processing those WEM files will save you additional 200mb. I removed in total 5 of the WEM files which are not encodable, which leads either to infinite encoding the files (temp files will grow infinitely), or they are just finished instantly, which will lead to broken files if you decode them back to original.

Maybe you have to edit file paths.

Bad WEM files in question:

"base-generic\data_pc\audio\media\152227780.wem"
"base-generic\data_pc\audio\media\526830619.wem"
"base-generic\data_pc\audio\media\577391364.wem"
"base-generic\data_pc\audio\media\880082258.wem"
"base-generic\data_pc\audio\media\908534589.wem"

1. Command to extract WEM files with xtool

:WEM base-generic
if not exist "extracted_data\base-generic" (
mkdir "extracted_data\base-generic" )
xtool extract "aw2_base_generic.xtl" "base-generic-000.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-001.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-002.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-003.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-004.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-005.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-006.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-007.rmdblob" "extracted_data\base-generic"
xtool extract "aw2_base_generic.xtl" "base-generic-008.rmdblob" "extracted_data\base-generic"

:WEM stream0-generic
if not exist "extracted_data\stream0-generic" (
mkdir "extracted_data\stream0-generic" )
xtool extract "aw2_stream_generic.xtl" "stream0-generic-000.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-001.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-002.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-003.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-004.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-005.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-006.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-007.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-008.rmdblob" "extracted_data\stream0-generic"
xtool extract "aw2_stream_generic.xtl" "stream0-generic-009.rmdblob" "extracted_data\stream0-generic"
pause

2. Command to erase WEM files with xtool

:WEM base-generic
xtool erase "base-generic" "base-generic-000.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-001.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-002.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-003.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-004.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-005.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-006.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-007.rmdblob" "aw2_base_generic.xtl"
xtool erase "base-generic" "base-generic-008.rmdblob" "aw2_base_generic.xtl"

:WEM stream0-generic
xtool erase "stream0-generic" "stream0-generic-000.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-001.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-002.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-003.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-004.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-005.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-006.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-007.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-008.rmdblob" "aw2_stream_generic.xtl"
xtool erase "stream0-generic" "stream0-generic-009.rmdblob" "aw2_stream_generic.xtl"
pause

3. Encode with WemTool

WemTool.exe -e

4. Compress files

5. Decode with WemTool

WemTool.exe -d

6. Decode files with xtool (move them back into game files)

:WEM base-generic
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-000.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-001.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-002.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-003.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-004.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-005.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-006.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-007.rmdblob"
xtool decode "aw2_base_generic.xtl" "base-generic" "base-generic-008.rmdblob"

:WEM stream0-generic
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-000.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-001.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-002.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-003.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-004.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-005.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-006.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-007.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-008.rmdblob"
xtool decode "aw2_stream_generic.xtl" "stream0-generic" "stream0-generic-009.rmdblob"
pause

7. Finished. All files are crc perfect at the end.

[KaktoR]

What help you need exactly? Try to be more precise!

[Jahan1373]

://www.mediafire.com/file/gh4um1dsm92xbeg/Help.zip/file
Hello
Can someone help me, I have a problem with this installation
CPU will be 100% after installation
Second, after the installation is finished, the installation icons are opened, the problem is that they are white on the desktop, the game icon is not visible.
Friends, how to solve this problem?

[KaktoR]

Portal Revolution

 21:39:48 - Selected ARC/DS method for Data1a-01.bin was: msc_tak+xtool:dd3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 21:39:48 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 21:53:29 - Overall input size: 10,84 GB
 21:53:29 - Overall output size: 4,23 GB (Ratio 39.07%)
 21:53:29 - Overall conversion time: 00:13:40

[KaktoR]

lolz settings

Data detection options:
-dt [0..1] - enables / disables the pos_ctx / dxt / raw detection. no headers, everything is detected based on the analysis of data statistics. Default: -dt1;
-dtp [0..1] - enables / disables transmission of statistics from previous blocks to subsequent blocks in the detector. Default: -dtp1;
-dtb [0..1] - enables / disables enumeration of all options regardless of heuristics. Default: dtb0;
-dto [0..1] - enables / disables detection of the best positional o1 context. Default: -dto1;
-dtm [0..1] - enables / disables raw graphics multimedia detection. Default: -dtm1;
-dtw [0..1] - enables / disables width detection for raw graphics and dxt textures;
-dtd [0..1] - enables / disables detection of dxt textures;

Multithreading options:
-mtt [0..1] - for multithreading, specifies the operating mode to be used. When set to 0, the dictionary size must be at least 2 times the block size. In this mode, data for each stream will be loaded alternately in block size. In this mode, in most cases, you can achieve better compression than in the second, however, decompression requires as many streams as in compression. When 1, each block is compressed separately, without dependencies on neighboring data, respectively, the compression here is usually worse than in the first mode, but you can specify any number of streams for unpacking. It is for this mode that the options from cls.ini MaxThreadsUsage and MaxMemoryUsage are used. Default: -mtt0;
-mt [1..16] - sets the number of threads to process. With -mt1 and -mtt0, normal sequential lossless compression is obtained by dividing the stream into blocks. Default: -mt1;
-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, respectively, the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;

Main options:
-d [16..2032] - dictionary size in MB. Default: -d64;
-tt [1..256] - the number of paths to consider in the optimal parser. It greatly affects the speed and compression ratio, but not unpacking. You shouldn't ask more than 16, I assure you it's not worth it. Default: -tt4;
-oh [8..14] - sets the maximum number of bytes that the optimal parser will process at a time (2 ^ X). Default: -oh12;
-os [0 .. -oh] - sets the minimum number of bytes that the parser will process at a time (2 ^ X). Default: -os8;
-fba [0..4096] - sets the size of the minimum match, at which the parser will not bother much in the calculations. Compression is decently accelerated (2 times) with a small loss of compression. At 0, these simplifications are disabled. Default: -fba256;
-fbb [0..4096] - THIS OPTION DOES NOT WORK AT THE MOMENT. Asked for even greater simplifications;
-al [0..1] - enables / disables the calculation of the literal price even if rep0 matches. Default: -al1;
-x [0..2] - turns on the slow modes of the parser with the calculation of (almost) all lengths of the found match, as well as the options match + lit + rep0match. Very slow and merciless. And the benefit is very small. It's easier to add -tt; Default: -x0;

Matchfinder options:
-rt [0..2] - THIS OPTION DOES NOT WORK AT THE MOMENT. I set the type of matchfinder - lz, rolz or hybrid mode, but rolz did not meet expectations and I made all the changes without taking it into account, so it does not work now; Default: -rt0;
-mc [2..1023] - Specifies the maximum number of traversals of the binary tree of matches, after which matches for the given position are no longer searched; Default: -mc128;

Model options:
-cm [0..1] - enables / disables a simple context mixer in some critical places, which mixes a couple of models in each place. When enabled, it improves compression, but slows down decompression. Default: -cm1;
-bc [0..8] - sets the level of influence of the previous byte on the mixer; Default: -bc4;
-lm [0..4] - THIS OPTION DOES NOT WORK AT THE MOMENT. Specifies the type of an "elementary" literal. Complex high-order models with cm did not perform well, so I abandoned them, as did rolz. Default: -lm0;
-blo [0..8] - sets the degree of influence of the previous byte on the encoding of the upper part of the literal. Default: -blo8;
-bll [0..8] - sets the degree of influence of the previous byte on the encoding of the lower part of the literal. Default: -bll8;
-blr [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the upper part of the literal. Default: -blr4;
-bm [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the match type flag. Default: -bm4;
-pc [0..4] - sets the positional context for all encoding operations. It is automatically ignored when the detector is on. Default: -pc2;
-dmXY(X [0..3], Y [0..4]) - sets the model for encoding color pairs (X) and alpha channel (Y) pairs. At the maximum value of each parameter, adaptive switching between models is used, while the decompression speed is reduced, but the compression is better in most cases. Default: -dm34;
-gmXY(X [0..2], Y [0..1]) - X - sets the model for encoding raw graphics. at the maximum value, adaptive switching between models is enabled. However, in this case, it is rare to see a gain over the adaptive mode. Generally, mode 0 is in the lead, but its unpacking is 2 times slower than mode 1. Y - enables updating of statistics of models when they were not used (for example, there was a long match). For X0 and X1, it usually gives a small gain in compression, but the speed drops by a factor of 2 (it all depends on the data). In general, the most appropriate is -gm00, he - and the default mode;

[kj911]

Dragonis40: Try it use very slooow settings:

xZLib(old Xtool 0.12)+srep:m3f:a0(srep 3.92!)+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:d512m:fba4096:mc1023:tt16:mt1:ldmf1:ldc0:ldl5

Questions: The *.bsa archives contain goodly compressibble raw/wav audio files? MP3's or any? Videos?

Alternative1: xZLib+srep+delta+lolz
Alternative2: xZLib+msc+srep+delta(used or not)+lolz

[Dragonis40]

Originally Posted by kj911

Dragonis40: Try it use very slooow settings:

xZLib(old Xtool 0.12)+srep:m3f:a0(srep 3.92!)+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:d512m:fba4096:mc1023:tt16:mt1:ldmf1:ldc0:ldl5

Questions: The *.bsa archives contain goodly compressibble raw/wav audio files? MP3's or any? Videos?

Alternative1: xZLib+srep+delta+lolz
Alternative2: xZLib+msc+srep+delta(used or not)+lolz

Thanks for your reply! How can I find out what *.bsa files contain? Do I have to use any tool?

[Masquerade]

@kj911
Don't use LDMF if you are using srep, no point.

@Dragonis40
https://www.nexusmods.com/skyrimspecialedition/mods/974