Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Try with d1 parameter

[prince55]

Originally Posted by dixen

minsomniac:rcrads+gdeflate

Thanks works perfect

[prince55]

Originally Posted by Wanterlude

use insomniac plugin for xtool
insomniac:rcra or insomniac:rcrads


insomniac:rcrads - uses gdeflate, so its additional use is pointless.

test file: sp-common.psarc (19.8 gb)

insomniac:rcra

Streams: 130981 / 130981
Time: 00:01:12 (CPU 00:05:34)

Size: 19.9 GB >> 31.8 GB

insomniac:rcrads

Streams: 130981 / 130981
Time: 00:01:05 (CPU 00:05:41)

Size: 19.9 GB >> 31.8 GB

insomniac:rcra + gdeflate

Streams: 130981 / 130982
Time: 00:01:08 (CPU 00:05:46)

Size: 19.9 GB >> 31.8 GB

Yeah, i tried all, same result. thanks for help

[L33THAK0R]

GOG Release - Build ID 52358852926631596

Original Size: 3.83 GiB
Final Size: 1.00 GiB (Incl. installer)
Ripped Files: None.
Tools used:

Xtool v0.8.9 (Codecs: zlib, relfate)
Srep
Lolz
Msc
Oggre

Notes: Briefly considered ripping localised audio but all localisations amount to less than 100 MiB. Used a new patching approach that turned out to net a 34.64% reduction (from 1.53 GiB). This involved building the oldPath post-install, rather than ripping one of the audio formats, for separate processing, from an existing oldPath and patching it post-install. A handful of the "*.wav" and ".ogg" audio don't play well with their respective compressor, so (like with many titles), validation tests will be required. I unfortunately didn't save the checksums for the offending files, as this particular title has been scattered across different drives for a handful of weeks now (due to unsatisfactory initial results), but thankfully given the size of the title, the validation tests don't take long.

I'm unsure of room for further gains, the patch file is a close to perfectly optimised as I can currently manage, but if there is room by processing individual asset types separately, I'm not currently aware of them.

[L33THAK0R]

GOG Release - Build ID 57836190042333563

Original Size: 48.5 GiB
Final Size: 15.1 GiB (16,313,675,776 bytes) (Incl. installer)
Ripped Files: None.
Tools used:

Xtool v0.8.9 (Codecs: zlib) (reflate is a touch hit or miss with UE titles, so I opt to play it safe and rely on zlib)
Srep
Lolz
Oggre

Notes: Additional gains were made by patching audio post-install. Unreal Engine 4 titles, in regards to titles that use regular Vorbis OGG audio store it without exception within the "*.uexp" files, archives were first unpacked then using a basic (but fast) python script I threw together, a 10-byte string was searched for within each extracted "*.uexp", resulting in only 772 MiB of writes for an approx. 110 (+/- 10) MiB improvement over current leading results.

Honestly the real challenge was figuring out what I wanted to do for the game icon, I think it turned out alright but the art assets were surprisingly difficult to extend or frame so I went for a compromise using an asset from the official art book as a frame.

I don't want to clog up the forums so I might make a bulk post of the few hundred compressions I've done in the last couple months, but whilst they are quite impressive for the most part in terms of ratio, there's nothing really interesting or unique about the processing methods.

[KaktoR]

Good to see someone else steps in and give such detailed results. I like it, especially the parts where you dig deeper into such formats such as extracting audio files.

Keep it up.

[KaktoR]

Age of Mythology Retold
They switched to lz4. However not much benefit because most of the data seems to be not compressed at all or very weak compression.

[Stream1]
Name=lz4hc
Codec=lz4hc:l9
BigEndian=0
Signature=0x347A6C61
Structure=Signature(4),DSize(4),CSize(4),Unk(4),Stream
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize

[JustFun]

25 to Life - (Retail)

Selected ARC/DS method for Data1a-01.bin was: 
xtool:mpreflate+srep_new:490g+lolz
------------------------------------------------------------------------------------------
Overall input size: 1,43 GB
Overall output size: 589,62 MB (Ratio 40.27%)

[JustFun]

Tom Clancy's Ghost Recon (2001) - (Retail)

Selected ARC/DS method for Data1a-01.bin was: 
xtool:mpreflate+msc_tak+srep_new:490g+lolz
------------------------------------------------------------------------------------------
Overall input size: 0,93 GB
Compressed game size: 334,62 MB (Ratio 35.20%)

[IRAQIGHOST]

Call of Duty Advanced Warfare

Game Size: 44GB
Repack Size: 29.9GB

.ff files

xt_lz4+srep_new:393a+LOLZ_NORMAL

.pak files (Images)

xt_lz4+srep_new:393a+LOLZ_NORMAL

.pak files (Sound)

xt_lz4+msc_tak

Rest files

srep_new:393a+4x4:b96mb:lzma:ultra:bt4:fb273:lc8:m c1000000:96mb

[KaktoR]

Why not combine ff and pak files if you use the same method on them? Pretty sure you missing some duplicates with srep.

[IRAQIGHOST]

Originally Posted by KaktoR

Why not combine ff and pak files if you use the same method on them? Pretty sure you missing some duplicates with srep.

Just to clarify between files (Images) and (Sounds)

[L33THAK0R]

GOG Release - Build ID 58500446910738020

Original Size: 5.07 GiB
Final Size: 2.72 GiB (Incl. installer)
Ripped Files: None.
Tools used:

Xtool v0.8.9 (Codecs: png, reflate, unity, zlib)
Srep
Lolz
Msc
Oggre

Additional notes:

MSC-related streams are decompressed post-install, due to CRC errors when decompressing. To minimise the size of patch files for the relevant containers, organised and efficient sorting of loose files is recommended, a similar approach was taken for my "Seasons after Fall" compression, more details can be found in my write-up for the title.

Quite happy with how this one turned out, given it's a respectable 12.25% smaller than the currently smallest published results. Despite making the post-install process quite fast, the downside is that the relevant containers are effectively written to disk 3 times (~approx. 5.5 GiB); once to build the oldFile, once for patching to the decompressed newFile, and once to restore the decompressed newFile to the original container. I wanted it to be CRC-perfect, but if additional DLC is released and I have to update my results, I'll experiment with decompressing the containers, as I've had success in the past with having titles accept decompressed *.bundle containers.

[KaktoR]

Terminator Dark Fate Defiance
Update 1.06.1107

 16:46:48 - Selected ARC/DS method for Data1a-01.bin was: oggre
 16:46:48 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8
 16:46:48 - Selected ARC/DS method for Data1c-01.bin was: xtool:c128mb:mreflate:mpng:d1:dd3:l10,d128mb
 16:46:48 - Selected ARC/DS method for Data1d-01.bin was: xtool:dd3:l10,d128mb
 16:46:48 - Selected ARC/DS method for Data1e-01.bin was: 1
--------------------------------------------------------------------------
 17:36:23 - Overall input size: 35.26 GB
 17:36:23 - Overall output size: 13.45 GB (Ratio 38.15%)
 17:36:23 - Overall conversion time: 00:46:07

Just a fast compression.
You need to exclude "basis.pak\sounds\dialogs\10_mission\10th_sintu_la st_mission_1_1.wav" from xtool:mflac (or msc) otherwise you will get crc error on this paticular file.

password

oKoo$]bnGTKJLMNBA9A

Recreate paks with

7z.exe a -mx5 -tzip -aoa -r -poKoo$]bnGTKJLMNBA9A

The recreated files are not crc perfect but the game works fine nevertheless.

[KaktoR]

Tempest Rising

0x9A6DF7547946158E6A6210A26F3F81BB7BF33A513BB369A38E41287249246577