Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Had a quick look so maybe I'm missing something obvious but are there any lossless compressors that support WaveForm "MS ADPCM" formatted audio? MSC only supports PCM formatted samples, and I got a feeling some substantial gains could be had if this format is able to be properly processed across a range of titles.

[Masquerade]

No afaik. ADPCM is a lossy format. You could try looking into this: https://monkeysaudio.com/

But it works in a similar way to Toms Lossless Audio Kompressor so then again it may do nothing.

[L33THAK0R]

All results include the installer's weight (usually between 4 - 9mb)

Avatar: The Game, James Cameron’s

3.31 GiB >>> 1.98 GiB

xtool 0.6.5 + srep + lolz

Batman: Arkham Knight

51.75 GiB >>> 26.2 GiB (Incl. Optional Videos + Lossless Videos) | 25.3 (Incl. Lossy Videos)

Note: In order to properly create lossy videos you have to extract the cue points from the appropriate USMs'. This process isn't tricky and only takes around a minute in an ideal setup however calculating the offsets and translating them into usable integers can take a hot minute. As I'm sure those who have done a similar job are aware this process can't be automated due to the lack of consistency that cue timings are stored within their appropriate headers, this was definitely more so a proof of concept considering the fact there are ~880 videos that require cue point decoding.

xtool (I forgot to document the version I used, since I did this one back in mid 2020) + uelr + srep + lolz

Far Cry 2 (Fortunes Edition)

3.27 GiB >>> 1.62 GiB

delzorec + srep + lolz

Just Cause 2

4.08 GiB >>> 1.00 GiB

Note: Final size could've been smaller (~950MiB) but I included a couple dozen QoL fixes that added a bit of weight in additional texture data.

xtool 0.3.21 + msc + srep + lolz

Mad Max

32.88 GiB >>> 3.55 GiB (Incl. Lossless Videos) | 3.50 GiB (Incl. Lossy Videos)

Note: Lossy videos were definitley just an experiement, 1080p Bink2 videos rescaled to 720p Bink1, even at 99% the target filesize look like dog-shit. Honestly just wanted to see how small I could get things. There is potential room for improvement if someone can pack the ~900 MiB of dialogue thats packed into one of the Banks, but it's encoded in a format that I can't seem to process, or even losslessly transform to and from a packable format.

xtool 0.3.21 + oggre + srep + lolz

The Surge

21.78 GiB >>> 3.86 GiB (720p Lossless Videos) | 3.72 GiB (720p Lossy Videos)

Note: 1080p + 720p videos both exist in the original game files, I just chose to present 720p for the results as its the smallest, valid, lossless size. Use a LZ4 library to recompress, I got pretty similar results accross a good dozen or so libraries, but some streams were still dropped, I'd encourage people to investigate further but I honestly spent too much time on this series for me to justify spending any more time testing is tricky.

xtool 0.3.21 + oggre + srep + lolz

-----------------------------------------------

I would post more, but these are just the most interesting results I've gotten. I've managed to beat a fair few records posted both here and by other repackers on their own websites but only by using more modern methods and by managing to process troublesome blocks of data by patching out the offending data.

[Masquerade]

@L33THAK0R
Would love it if you could make a post detailing more about the Cue points in the USM videos. I believe it would truly be a valuable resource.

[Carldric Clement]

Original Size: 6.82 GB
Compressed Size: 1.68 GB

Option:

* JPG File   = packjpg
* OGG File   = oggre:s3
* Wav File   = tta
* Rest files = xtool:zlib:reflate+srep:m3f:a2+exe2+7z:d128m:mt2

[KaktoR]

Sid Meier's Civilization VI
v1.0.12.31 (19 January 2023), all DLC, OST ripped

 19:27:51 - Selected ARC/DS method for Data1a-01.bin was: bpk
 19:27:51 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:dd3+lolz:d32:mtt1:mt12:fba4096:mc2
 19:27:51 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:db:dd3+lolz:d32:mtt1:mt12:fba4096:mc2/$deflatepack=xtool:mpng:d1+xtool:mpreflate+xtool:db:dd3+lolz:d32:mtt1:mt12:fba4096:mc2/$oggrepack=oggre
-----------------------------------------------------------------------------------------
 21:02:04 - Overall input size: 21.36 GB
 21:02:04 - Overall output size: 8.13 GB (Ratio 38.08%)
 21:02:04 - Overall conversion time: 01:34:11

Data1a = *.bk2 (exclude "DLC\GranColombia_Maya\Platforms\Windows\Movies\GranColombia_Maya_DefeatClimate.bk2" or use bink patch method for this single file)
Data1b = *.blp
Data1c = rest of the files

PS: I used wemtool by Masquerade for the wem's

[kuyhaa]

hi

i try precomp Steel Division 2

use xtool reflate but still got CRC error when decomp

any info for solve ?

thanks

[Gehrman]

Originally Posted by kuyhaa

hi

i try precomp Steel Division 2

use xtool reflate but still got CRC error when decomp

any info for solve ?

thanks

Try with compatibility with Windows 7

ECHO OFF
Reg.exe add "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe" /t REG_SZ /d "~ DPIUNAWARE WIN7RTM" /f
START Setup.exe
Reg Delete "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe"  /f
Exit

Or use XTool 0.2. (It works well with ZLib's strange streams.)

[kuyhaa]

Originally Posted by Gehrman

Or use XTool 0.2. (It works well with ZLib's strange streams.)

i try still got error bro, stuck in decomp, now i excluded of error files

[Gehrman]

Originally Posted by kuyhaa

i try still got error bro, stuck in decomp, now i excluded of error files

Sorry, the version is wrong
You should use XTOOL 1908 R3.

The "Xatab Repack" also has used the XTOOL 1908 R3 to pre-compress the game.

[shazzla]

Exclude those files from precomp process. I did it,too.

[Gehrman]

30.36 GB (EN)

Sound*

msc:raw=1+srep:m3f:l512+4x4:lzma:a1:mfbt4:32m:fb273:mc10000:lc8

*.DAT

xtool:c256mb:mfc4:dd3+lolz2:d32:dm00:fba0:gm20:mt8:mtt1:mc128

Other Files

Srep+LZMA

Final Size: 12.2 GB

[L33THAK0R]

27.30 GiB >>> 10.9 GiB (Incl. Lossless Videos + Ripped Startup Videos, weighing in at ~40 MiB)

DELZORec + srep + lolz

Note: BPK can't be used as these are the "g" revision of the Bink2 codec. Ripped files include any included redists and non-English loc., as well as the startup video.

It's interesting to see that @Gehrman managed to get MSC working with the by just using the RAW detection, I'll give it a shot myself to see if any further gains can be had.

[Gehrman]

Originally Posted by L33THAK0R

27.30 GiB >>> 10.9 GiB (Incl. Lossless Videos + Ripped Startup Videos, weighing in at ~40 MiB)

DELZORec + srep + lolz

Note: BPK can't be used as these are the "g" revision of the Bink2 codec. Ripped files include any included redists and non-English loc., as well as the startup video.

It's interesting to see that @Gehrman managed to get MSC working with the by just using the RAW detection, I'll give it a shot myself to see if any further gains can be had.

The minimum game size is 30 GB
Most likely your version is without multiplayer files

[L33THAK0R]

Originally Posted by Gehrman

The minimum game size is 30 GB
Most likely your version is without multiplayer files

Ah thanks for that, with multiplayer files I got a final size of 11.5 GiB (the MP files clocked in at ~600 MiB). It's possible and I'd wager likely a smaller size can be achieved, should both the main + multiplayer files be packed together, but I had the multiplayer files as an optional component.

For anyone interested, with MSC I only saw a ~9 MiB reduction. Upon testing I wasn't able to successfully decompress these files. It's hard to say whether this is within variance of running the data through LOLZ, or pure gains due to MSC, however since I wasn't able to get a valid result, I don't think I can comment much on the matter.