Best Compression Methods for 'Specific' Games. Q&A

[lomesmo]

Bullet Witch

All DLC

Input size: 4.57 GB
Output size: 3.09 GB
Method: xZLib+srep+lzma2

[L33THAK0R]

Hi all,

Has anyone given "Call of Duty: Black Ops III (3)" a shot? It piqued my interest when I saw the QuickBMS script for "xpack" archives[1] mentioned that the "LZO", "LZ4" & "Oodle" compression algorithms were used within these archives. I'm sure testing which libraries for "LZ4" give the best output won't be too tricky, but in regards to the "Oodle" codec(s) used, I'm a bit stumped. As far as I know a "oo2core_#_win64.dll" is required, with either "xtool" or "Oo2Rec#", to pre-compress a given file, however such a file isn't present within the root and sub-directories of the title in question. Does anyone have any idea how to go about pre-compressing the "Oodle" streams for this title? Apologies if I buggered the terminology.

[KaktoR]

Well you can check with hex edit if the game has oodle at all (I don't know), if yes then just try all different libraries.

[KaktoR]

NieR Replicant ver 1 22474487139

Anyone knows what zstd lib to use? It's slow af

[Masquerade]

Originally Posted by KaktoR

NieR Replicant ver 1 22474487139

Anyone knows what zstd lib to use? It's slow af

Unsure, I have checked with latest ZSTD to see if it can list info about the archives, yet it can only seem to load info for few archives. Yet, all files are definitely compressed with zstd.

for %f in (*.arc) do zstd -l "%f"
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0     9.25 MB      30.19 MB  3.263   None  common.arc
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0   690.86 KB    1748.36 KB  2.531   None  info.arc
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
File "init.arc" not compressed by zstd
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
File "lang1.arc" not compressed by zstd
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
File "lang2.arc" not compressed by zstd
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
File "param.arc" not compressed by zstd
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
File "stream.arc" not compressed by zstd

I even tried compressing a demo file with zstd and testing the latest libzstd to see if the precompressor worked, neither zstdrec or xtool could precompress a random file compressed with latest zstd.

[KaktoR]

Yes, the first file contains just 1 block with zstd (28B52FFD).

However, ratio with lzma2 for init.arc is about 70%.

I will investigate it further

[Masquerade]

KaktoR
You can use the following tools to extract all of the compressed ZSTD streams from NieR replicant. It will allow for decompression of 99% of streams.

1. VGM Toolbox and this script. Download latest zstd.exe.

2. Place the XML file in plugins\AdvancedCutter folder of VGMToolbox.

3. Open VGMToolbox, go to Misc. Tools > Extraction Tools > Generic > Advanced Cutter

4. From the drop down menu in the top right, select NieRReplicant and click LOAD.

5. Next, you can drop each of the arc files into the top left box and it will extract the streams. You only really need to do this for the large arc files. No need on the little ones.

6. After extracting the streams, run this python script for each of the files. This will remove the extra padding bytes (@) that the devs added so that each stream begins at an offset of divisible by 16. ZSTD will not decompress files that contain data after the end of the stream.

for %f in (*.MSQ) do trimmer.py "%f"

Ignore that it involves .MSQ extension. You can switch it to .TMP or whatever.

7.

for /r %f in (*.MSQ) do zstd -d "%f" -o "%~df%~pf%~nf.DEC"

This will use zstd to decompress the streams. Afair it fails on about 16 streams out of ~7800, but you can decompress most of the game this way.

[KaktoR]

Question is if it's profitable because with plain srep+lzma i get 13.5GB

[Masquerade]

Originally Posted by KaktoR

Question is if it's profitable because with plain srep+lzma i get 13.5GB

Oh, most certainly not profitable in the slightest. But still good fun!

[hwang4]

Hi!Need A valid unreal key for "Chinese Paladin - Sword & Fairy 7"!
Should i use "ue4dt" or xtool with unreal plugin?thanx!

[KaktoR]

0xE53BEBDBCEC82995F37033B62CFD8243E02C10C03AB12EA682F654DC805F4009

In my tests there were no difference in using ue4dt or xtool.

[dixen]

Keys

Sword and Fairy 7 (cubejoy) 0xB37742E2DF203FAF02D00DD495D3297FAEB4B14D89FBD6C88D874884FAB8EAD5
Sword and Fairy 7 (steam)   0xE53BEBDBCEC82995F37033B62CFD8243E02C10C03AB12EA682F654DC805F4009

[hwang4]

KaktoR , you are right!I tried with both ue4dt & xtool without success!
So ? What's should i use for better compression ?

[KaktoR]

I think the files are not compressed anyway, but I have not checked further.

[KaktoR]

NASCAR 21 Ignition

 15:00:20 - Selected ARC/DS method for Data1a-01.bin was: xtool:mpreflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 15:00:20 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
 16:25:17 - Overall input size: 18.62 GB
 16:25:17 - Overall output size: 12.12 GB (Ratio 65.07%)
 16:25:17 - Overall conversion time: 01:24:52

Data1a *.pak
Data1b rest of the files