Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Scene Release - CPY

Original Size: 2.82 GiB
Final Size: 1.80 GiB (Incl. installer) (1,940,176,780 bytes)
Ripped Files: None.
Tools used:

Xtool v0.8.9 (Codecs: reflate, zlib)
Srep
Lolz
Oggre

Additional notes:

"GzsTool.exe" & "sdp_tool.exe", and WEM processing tools are attached below.

"data_01.g0s" & "data_02.g0s" contain majority of assets, "data_00.g0s" just contains a FMV cut-scene. Archives are XOR-ed, in layman's terms a simple bit-shifting operation is used to encrypt the archive contents.

My solution isn't ideal (data for each relevant archive is written to the disk twice), however I couldn't find a better approach. It should also be noted this isn't a CRC-perfect packing process but it is deterministic from my testing, ensure no additional files are within the target directory for packing, nor is it missing any original files. "GzsTool.exe" is used to unpack the archives, and then repack them post-install. Within these archives are a variety of file-types, notably ".wem" and ".sdp". The former file-type is self explanatory, and can be processed (for those who are unsure) with the following chain

ww2ogg.exe "XXX.wem" --pcb "packed_codebooks_aoTuV_603.bin" -o "XXX.tmp"
oggre_enc.exe "XXX.tmp" "XXX.packed"
oggre_dec.exe "XXX.packed" "XXX.unpacked"
hdiffz -c-lzma2-9 -C-no "XXX.unpacked" "XXX.wem" "XXX.patch"

To eliminate ".wem" files that are inflated by packing, I run a basic test/filter for file diff. that I wrote in python, as I've found the <100KiB rule that I've been using for a good while doesn't always hold true.

".sdp" files are just ".bnk" files with an additional header, they can be extracted quickly using "sbp_tool.exe", there you can isolate the wem files, process them and make a patch. Processing ".sdp" archives only nets around ~30MiB (rough estimate, actual figure is under ~100MiB but I've forgotten the exact number).

I've been trying to make any additional gains to at least match the best publicly available size (CorePack's release coming in at 1.73GiB), but I'm unable to determine where further gains can be made. I do have their "Phantom Pain" release and can confirm for that title they use a similar unpacking/repacking method, however I'm unable to source their "Ground Zeroes" release for further analysis. I feel like I'm missing something glaringly obvious but crawling through community documentation has netted nothing that jumps out at me. 2nd place is a nice consolation prize but it would be nice to at least know what they did for a 70MiB reduction.

I'm currently finalisng my "Metal Gear Solid V: The Phantom Pain" approach, and will publish the results soon.

[KaktoR]

Originally Posted by L33THAK0R

Scene Release - CPY

Original Size: 2.82 GiB
Final Size: 1.80 GiB (Incl. installer) (1,940,176,780 bytes)
Ripped Files: None.
Tools used:

Xtool v0.8.9 (Codecs: reflate, zlib)
Srep
Lolz
Oggre

Additional notes:

"GzsTool.exe" & "sdp_tool.exe", and WEM processing tools are attached below.

"data_01.g0s" & "data_02.g0s" contain majority of assets, "data_00.g0s" just contains a FMV cut-scene. Archives are XOR-ed, in layman's terms a simple bit-shifting operation is used to encrypt the archive contents.

My solution isn't ideal (data for each relevant archive is written to the disk twice), however I couldn't find a better approach. It should also be noted this isn't a CRC-perfect packing process but it is deterministic from my testing, ensure no additional files are within the target directory for packing, nor is it missing any original files. "GzsTool.exe" is used to unpack the archives, and then repack them post-install. Within these archives are a variety of file-types, notably ".wem" and ".sdp". The former file-type is self explanatory, and can be processed (for those who are unsure) with the following chain

ww2ogg.exe "XXX.wem" --pcb "packed_codebooks_aoTuV_603.bin" -o "XXX.tmp"
oggre_enc.exe "XXX.tmp" "XXX.packed"
oggre_dec.exe "XXX.packed" "XXX.unpacked"
hdiffz -c-lzma2-9 -C-no "XXX.unpacked" "XXX.wem" "XXX.patch"

To eliminate ".wem" files that are inflated by packing, I run a basic test/filter for file diff. that I wrote in python, as I've found the <100KiB rule that I've been using for a good while doesn't always hold true.

".sdp" files are just ".bnk" files with an additional header, they can be extracted quickly using "sbp_tool.exe", there you can isolate the wem files, process them and make a patch. Processing ".sdp" archives only nets around ~30MiB (rough estimate, actual figure is under ~100MiB but I've forgotten the exact number).

I've been trying to make any additional gains to at least match the best publicly available size (CorePack's release coming in at 1.73GiB), but I'm unable to determine where further gains can be made. I do have their "Phantom Pain" release and can confirm for that title they use a similar unpacking/repacking method, however I'm unable to source their "Ground Zeroes" release for further analysis. I feel like I'm missing something glaringly obvious but crawling through community documentation has netted nothing that jumps out at me. 2nd place is a nice consolation prize but it would be nice to at least know what they did for a 70MiB reduction.

I'm currently finalisng my "Metal Gear Solid V: The Phantom Pain" approach, and will publish the results soon.

Looks like you cannot use the sbp tool in a for loop with /r option sadly.

[L33THAK0R]

Originally Posted by KaktoR

Looks like you cannot use the sbp tool in a for loop with /r option sadly.

Yeah it's a touch annoying, I just generated a checksum for all the loose files and then isolated the .sbp entries and ran as a batch script. Thankfully it's only needed to extract the initial WEMs, and a hdiffz patch takes care of the rest post-install.