Best Compression Methods for 'Specific' Games. Q&A - Page 192

KaktoR · #1 03-09-2022, 10:00

Middle-Earth: Shadow of Mordor
GOG version

Code:

 16:59:22 - Selected ARC/DS method for Data1a-01.bin was: bpk+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 16:59:22 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzlib+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 18:38:03 - Overall input size: 53.70 GB
 18:38:03 - Overall output size: 23.01 GB (Ratio 42.85%)
 18:38:03 - Overall conversion time: 01:38:39

Code:

-------------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1a-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: Conversion_Output\CONVERSION\Data1a-01.bin
Extracted 338 files, 10,656,657,345 => 12,229,724,967 bytes. Ratio 87.14%
Extraction time: cpu 908.11 sec/real 531.97 sec = 171%. Speed 22.99 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1b-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: Conversion_Output\CONVERSION\Data1b-01.bin
Extracting 66 files, 45,435,263,268 bytes. Processed  99.9%
Extracted 66 files, 14,051,170,767 => 45,435,263,268 bytes. Ratio 30.93%
Extraction time: cpu 815.78 sec/real 646.88 sec = 126%. Speed 70.24 mB/s
All OK

KaktoR · #2 03-09-2022, 14:50

Halo Infinite
Build 9347748, incl. Campaign, no HD pack

Code:

 20:53:02 - Selected ARC/DS method for Data1a-01.bin was: oo2reck:8+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 20:53:02 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzlib+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 20:53:02 - Selected ARC/DS method for Data1c-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 23:32:27 - Overall input size: 42.36 GB
 23:32:27 - Overall output size: 29.04 GB (Ratio 68.57%)
 23:32:27 - Overall conversion time: 02:39:18

Data1a = *.module*
Data1b = package\*
Data1c = rest of the files

Code:

-------------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1a-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: Conversion_Output\CONVERSION\Data1a-01.bin
Extracting 117 files, 28,222,165,341 bytes. Processed  99.8%
Extracted 117 files, 20,616,611,986 => 28,222,165,341 bytes. Ratio 73.05%
Extraction time: cpu 1139.09 sec/real 500.03 sec = 228%. Speed 56.44 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1b-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: Conversion_Output\CONVERSION\Data1b-01.bin
Extracting 88,043 files, 4,710,860,569 bytes. Processed  99.8%
Extracted 88,043 files, 851,140,347 => 4,710,860,569 bytes. Ratio 18.07%
Extraction time: cpu 61.89 sec/real 149.54 sec = 41%. Speed 31.50 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1c-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: Conversion_Output\CONVERSION\Data1c-01.bin
Extracting 1,883 files, 12,549,104,127 bytes. Processed  99.9%
Extracted 1,883 files, 9,714,711,673 => 12,549,104,127 bytes. Ratio 77.41%
Extraction time: cpu 334.20 sec/real 220.25 sec = 152%. Speed 56.98 mB/s
All OK

KaktoR · #3 03-09-2022, 15:01

Halo The Master Chief Collection
Build 9367265, no multiplayer, no extended language packs

Code:

 20:09:49 - Selected ARC/DS method for Data1a-01.bin was: bpk
 20:09:49 - Selected ARC/DS method for Data1b-01.bin was: xt_zlib+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 20:09:49 - Selected ARC/DS method for Data1c-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-----------------------------------------------------------------------------------------
 00:52:22 - Overall input size: 109.60 GB
 00:52:22 - Overall output size: 66.74 GB (Ratio 60.90%)
 00:52:22 - Overall conversion time: 04:42:31

Data1b contents:
halo1\prebuild\shaders_2_cache\prebuild\*.sdc
halo2\prebuild\shaders_2_cache\prebuild\*.sdc
halo2\preload\paks\*.pck
halo3\maps\*.map
halo3odst\maps\*.map
halo4\maps\*.map
haloreach\maps\*.map
mcc\Content\Paks\*.pak
data\ui\Screens\*.gfx

KaktoR · #4 05-09-2022, 08:18

Assassin's Creed Origins
v1.60, 1 language, all DLC

Code:

 13:10:48 - Selected ARC/DS method for Data1a-01.bin was: xtool:manvil:4+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 13:10:48 - Selected ARC/DS method for Data1b-01.bin was: bpk+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 17:08:59 - Overall input size: 69.70 GB
 17:08:59 - Overall output size: 32.56 GB (Ratio 46.71%)
 17:08:59 - Overall conversion time: 03:57:17

Decompression time: 1 hour (6c 12t)

I still had a backup with oo2recm. This time I used xtool anvil plugin to compare the full size. The size is almost the same (oo2recm is ~50mb smaller), but xtool anvil plugin is much faster (~7hrs vs 4hrs). I leave it here just for information on full sizes instead of just samples.

PS: I will do the same for Odyssey and Valhalla soon.

KaktoR · #5 05-09-2022, 13:13

Of course you have to use compressed bink file + diff file to restore the original file.

bink_unpack.exe will fail on decompressing the compressed bink file, but file remains.

Then you use the diff file to patch the decompressed bink file back to original.

In the example above, you have to include BPK2 files and Bink2DIFF files.

L33THAK0R · #6 05-09-2022, 19:24

Recently the title in question got an official PC-port, as it was originally a PS3 exclusive. The compression is, in theory fairly easy to decipher, as it's just Crilayla, however all the assets in their relevant ".cpk" archives are encrypted, the algorithm seems to be the same from a glance however some archives aren't original, so its not clear if a different or multiple private keys were used/replaced.

Attached is the only semi-working (QuickBMS) script I can find (source) that can sort of process some data but falls short very quickly. I'm not very adept at QuickBMS scripting but my guess is the encryption key is incorrect.

I have also attached a sample for anyone interested. For those curious current gains are only ~200MB with the encrypted archives.

Masquerade · #7 06-09-2022, 00:45

Quote:

Originally Posted by L33THAK0R

Attached is the only semi-working (QuickBMS) script I can find (source) that can sort of process some data but falls short very quickly. I'm not very adept at QuickBMS scripting but my guess is the encryption key is incorrect.

You can extract the encrypted data using CriPack Browser v6.7. As for decryption, it's clearly not an XOR implementation like what was seen in Super Robot Wars 30 as all the files are different.

KaktoR · #8 05-09-2022, 23:53

Assassin's Creed Odyssey
v1.5.6, 1 language, all DLC

Code:

 20:41:34 - Selected ARC/DS method for Data1a-01.bin was: xtool:manvil:4+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 20:41:34 - Selected ARC/DS method for Data1b-01.bin was: bpk+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-----------------------------------------------------------------------------------------
 00:44:06 - Overall input size: 99.23 GB
 00:44:06 - Overall output size: 42.15 GB (Ratio 42.47%)
 00:44:06 - Overall conversion time: 04:02:25

oo2recm

KaktoR · #9 07-09-2022, 09:06

Assassin's Creed Valhalla
v1.6.0, 1 language, all DLC

Code:

 11:02:57 - Selected ARC/DS method for Data1a-01.bin was: xtool:manvil:8+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 11:02:57 - Selected ARC/DS method for Data1b-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 18:06:25 - Overall input size: 136.98 GB
 18:06:25 - Overall output size: 49.58 GB (Ratio 36.19%)
 18:06:25 - Overall conversion time: 06:50:10

Masquerade · #10 08-09-2022, 12:19

*.SPK from Steelrising are just zip files with what appears to be no compression. I cannot speak for the larger packs but the locales zips have data just stored.

dixen · #11 09-09-2022, 00:26

Quote:

Originally Posted by Masquerade

*.SPK from Steelrising are just zip files with what appears to be no compression. I cannot speak for the larger packs but the locales zips have data just stored.

Compressing datapc_initial.spk

Quote:

Compressed 1 file, 10,852,967,682 => 18,870,379,505 bytes. Ratio 173.87%
Compression time: cpu 9.08 sec/real 215.30 sec = 4%. Speed 50.41 mB/s
All OK

Files compressed zlib inside SPK

All SPK
66.5 gb > 116 gb.

Question.
datapchighres.spk - is optional data or not?

Masquerade · #12 09-09-2022, 01:52

Quote:

Originally Posted by dixen

Question.
datapchighres.spk - is optional data or not?

Nope, game is made in same engine as Greedfall. The file datapchighres.spk is basically the whole game data.

dixen · #13 09-09-2022, 03:19

Quote:

Originally Posted by Masquerade

Nope, game is made in same engine as Greedfall. The file datapchighres.spk is basically the whole game data.

Really? After all, the game works fine without these files. Especially on weak systems like mine

KaktoR · #14 30-11-2022, 07:33

Assassin's Creed Valhalla
v1.6.2, 1 language, all DLC

Code:

 10:59:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:manvil:8+xtool:db:-dedup=3+4x4:lzma
 10:59:07 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:-dedup=3+4x4:lzma
-------------------------------------------------------------------------------------------------------------------------------------------
 16:29:11 - Overall input size: 143.49 GB
 16:29:11 - Overall output size: 50.14 GB (Ratio 34.94%)
 16:29:11 - Overall conversion time: 05:29:24

KaktoR · #15 09-09-2022, 08:16

In this cases I always look on steamdb for optional depots.
If no info is provided then just rename the files/folders and play a bit and see if it works.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Games Are For Kids	JoyBoy	Chit Chat	83	16-11-2021 14:28
CorePack Compression methods ? Help	ramazan19833	Conversion Tutorials	34	09-08-2018 06:51
i got the email in the mail about copying ps2 games	EVOEvo	PS2 Games	7	21-11-2003 07:56
Self-booting games on an older Japanese DC?	Named	DC Games	1	26-09-2003 17:48
Having Problems Burning Numerous Recent Games....Help is Appreciated	Protosstic	PSX Games	2	15-01-2002 21:27

The Following 3 Users Say Thank You to KaktoR For This Useful Post:
Danziel123 (04-09-2022), Gehrman (03-09-2022), mausschieber (04-09-2022)

The Following 2 Users Say Thank You to KaktoR For This Useful Post:
Gehrman (05-09-2022), JustFun (06-10-2022)

The Following 2 Users Say Thank You to KaktoR For This Useful Post:
Gehrman (03-09-2022), JustFun (06-10-2022)

The Following 5 Users Say Thank You to KaktoR For This Useful Post:
Gehrman (05-09-2022), JustFun (06-10-2022), Razor12911 (05-09-2022), shazzla (05-09-2022), ZAZA4EVER (06-09-2022)

The Following User Says Thank You to L33THAK0R For This Useful Post:
Gehrman (06-09-2022)

The Following User Says Thank You to Masquerade For This Useful Post:
L33THAK0R (06-09-2022)

The Following 2 Users Say Thank You to KaktoR For This Useful Post:
Gehrman (06-09-2022), JustFun (06-10-2022)

The Following 3 Users Say Thank You to KaktoR For This Useful Post:
JustFun (06-10-2022), ScOOt3r (07-09-2022), Wanterlude (07-09-2022)

The Following 2 Users Say Thank You to Masquerade For This Useful Post:
dixen (08-09-2022), ScOOt3r (08-09-2022)

The Following User Says Thank You to dixen For This Useful Post:
Masquerade (09-09-2022)

The Following 2 Users Say Thank You to KaktoR For This Useful Post:
JustFun (11-12-2022), L0v3craft (01-12-2022)