Best Compression Methods for 'Specific' Games. Q&A

[Fak Eid]

Thanks for the update. If dd3 can help me compressing the same size as srep and with less extraction time, I'll go ahead with it. My focus is to improve extraction speed at a cost of minimum difference in compression size. I'll try that for sure.

Originally Posted by KaktoR

Because it's faster.

Edit: "3" in dd# just represents the srep m#f setting m[1..5]. You can even use dd5 to get a bit better results in costs of speed.

Edit2: A bit old, but my test in the past when this feature was introduced: https://fileforums.com/showthread.ph...129#post499129

[KaktoR]

ANNO 1800
Update 18.0

 20:37:16 - Selected ARC/DS method for Data1a-01.bin was: xtool:c1500mb:lm:mbk2:df20p+xtool:dd3
 20:37:16 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8+xtool:mzlib:mreflate+xtool:dd3+4x4:lzma
 20:37:16 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+4x4:lzma/$mp3pack=7zdll-pm:m0=mp3det:m1=packmp3c/$bpkpack=bpk
-----------------------------------------------------------------------------------------
 00:12:23 - Overall input size: 101.44 GB
 00:12:23 - Overall output size: 30.63 GB (Ratio 30.20%)
 00:12:23 - Overall conversion time: 03:34:59

Notes: 
- bpk was used on bink1 files which are located in data3.rda
- You have to use -lm for bk2 plugin if you don't have 64GB RAM
- The following files contain bink videos: data3.rda, data11.rda, data12.rda, data13.rda, data16.rda and data18.rda. Use quickbms with "-f "{}.bk2,{}.bik"" file mask to extracrt just bink1/2 files.
- Use XTool erase to create xtl file and zero the files out of the game archives.
- Use XTool decode + xtl file to patch the files back afterwards.
- Some rda files also contain WEM files. You can use wemtool there.
- The language files also contain WEM files, but they are PCM, so you msc/xtool:flac on them.

[L33THAK0R]

I know Masquerade made a database recently for the first "Watch Dogs" title, however I've found myself needing to re-generate a new database in my own conversion, as I've opted to include a variety of mods that restore features left on the cutting room floor as well as providing the end-user access to content previously only accessible through online-only interactions. This restoration overwrites a considerable amount of existing files, as such a new database is required.

My current database seems to be considerably smaller than Masquerade's however, with a final result of 3.84 MiB (versus Masquerade's 14.0 MiB). The only issue I can currently see is when the fat2_fat3.bms script attempts to process the "windy_city.fat/.dat" which gives the following error:

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info:  algorithm   2
       offset      494c62b4
       input size  0x0000632b 25387
       output size 0x00010000 65536
       result      0xffffffff -1

Error: uncompressed data (-1) bigger than allocated buffer (224656)
       It usually means that data is not compressed or uses another algorithm

I'm assuming this is referring to

            elif ZTYPE == 2
                comtype deflate

however my bms2xtl.ini configuration file seems to be fine:

[Config]
SafeMode=0
UserInput=
Exclude=lzma_dynamic,snappy
gzip=zlib
deflate=reflate
zlib_noerror=zlib
deflate_noerror=reflate
zlib=zlib
oodle=leviathan
xmemdecompress=xmemcompress
lz4=lz4hc:l9
zenpxp=zenpxp
lzo1x=lzo1x
lzma_dynamic=reflate

I've ensured I've got the latest version of the xmemcompress plugin present, and its appropriate entry within "xtool.ini" is pretty standard

[xmemcompress]
Encode=xcompress64.exe d -window=131072 -partition=524288 <library>
Decode=xcompress64.exe e -window=131072 -partition=524288 <library>

I'm honestly scratching my head trying to figure out what I've messed up this time, even more so by the fact that the file "windy_city.fat/.dat" is one such file that hasn't been modified. Any tips would be greatly appreciated, its definitely been gnawing away at me.

[Razor12911]

the window and partition for xmemcompress should be 32768 I believe, those configurations are for project cars

[xmemcompress]
Encode=xcompress.exe d -window=32768 -partition=32768 <library>
Decode=xcompress.exe e -window=32768 -partition=32768 <library>

as for the other errors, I'm not so sure

[Masquerade]

DOOM Eternal v6.66 Rev 2.2

• *.streamdb, *.resources, *.lightdb
  

  xtool_kraken:l6_doom7*_dd+xtool_kraken:l6_doom8*_dd+srep+lolz

• *.bk2**
  

  xtool_bink2_dd5_fl2

• *.bik
  

  xtool_bink_dd5_fl2

• Everything else:
  

  razor

* Using these two oodle libraries.

** All bk2 except vs_first.bk2, vs_second.bk2 and doommarinesymbol_wide_fade.bk2.

88.7GB ---> 56GB

[Masquerade]

The Sims 4 [v1.100.147.1030 + All DLCs]

xtool_zlib_dd5+LZMA2

58.7GB ---> 36.4GB

[KaktoR]

Originally Posted by Masquerade

** All bk2 except vs_first.bk2, vs_second.bk2 and doommarinesymbol_wide_fade.bk2.

Why excluding? Does the modified bink plugin go crazy again?

[Masquerade]

Originally Posted by KaktoR

Why excluding? Does the modified bink plugin go crazy again?

The modified import does indeed fix motion data warn but it does not fix err. Trying to compress the folder with those 3 videos present will just leave XTool hanging.

[KaktoR]

Could you send me one of these files? Thanks.

Edit: I looked, and I don't have any problems with the patched bink plugin.

Compressed 3 files, 69,070,112 => 69,070,262 bytes. Ratio 100.00%
Compression time: cpu 0.06 sec/real 1.12 sec = 6%. Speed 61.71 mB/s
All OK

Extracted 3 files, 69,070,262 => 69,070,112 bytes. Ratio 100.00%
Extraction time: cpu 0.05 sec/real 0.60 sec = 8%. Speed 114.63 mB/s
All OK

[KaktoR]

XCOM 2
version 28 March 2022

 20:51:40 - Selected ARC/DS method for Data1a-01.bin was: uelr+xtool:mkraken:core_2.4.1+xtool:mzlib:dd3+4x4:lzma
 20:51:40 - Selected ARC/DS method for Data1b-01.bin was: xtool:lm:c1024mb:mbk2:df20p+xtool:dd3
 20:51:40 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+4x4:lzma/$oggre=oggre/$wemtoolpack=xtool:dd3+zstd
-------------------------------------------------------------------------------------------------------------------------------------------
 00:40:08 - Overall input size: 70.96 GB
 00:40:08 - Overall output size: 25.26 GB (Ratio 35.59%)
 00:40:08 - Overall conversion time: 03:48:26

WEMTool was used for *.wem files.

[L33THAK0R]

Has anyone else experienced some issues with Oggre when processing a large amount of streams? I'm currently working on "Nioh 2", whose audio can be extracted into a readable Vorbis Ogg format. I individually packed every file (~9000) and compared the original file checksums to determine which files were incompatible and removed them accordingly. In the past this process has never failed me and has always without fail resulted in a successful decompression of those particular assets, however this time around it seems I'm left with a bit of a confusing result. Attempting to pack all the streams together results in the installer, presumably due to the CLS Oggre plugin, crashing.

Investigating the issue by creating a large stored archive of all files in their packable format shows a much clearer image:

Original Input, Stored Archive (7.79 GB) -> Processed with Oggre (5.29 GB) -> Unpacked with Oggre (4.67 GB)

My current solution I'll be employing for my conversion is simply post-install, unpacking the stored archive with the "temp" packable files, then using a patch made between the temp files and a stored archive of the original files.

I'll be then putting these 2 files alongside each other into one large archive, whilst praying I can squeeze some small gains but I'm under no illusion it'll almost certainly be a satisfactory result. Definitely not happy about the amount of additional R/W operations that'll be performed on the end-users drive, but its the best solution I can come up with.

I should mention I was able to process the audio from the original "Nioh" without any trouble, which follows the same structure as "Nioh 2", with a pretty decent final result of 20.3 GB for v1.21.04 of the Complete Edition release for "Nioh".

UPDATE: I ran an additional test with the original unmodified audio archives packed with a generic srep + lolz configuration, when alls said and done, srep actually works well with the duplicated data, resulting in an inflated final result of only 390MB. Given the negligible gains compared to the amount of data that would would need to be written (~16gb) for the original patching approach to work, I think for the time being, and in the interests of the end-user, I'll sacrifice this 390MB.

[Masquerade]

Yeah, oggre is a mess.

[L33THAK0R]

Can anyone recommend which library and/or version of XTool to use for processing "Mirror's Edge: Catalyst"'s LZ4 streams? I recently tried running my collection of LZ4 libraries against a 100MB sample, using XTool 0.7.8 however seemingly none of them were of any use (see the following pastebin output). I've had success in the past with another LZ4 Frostbite 3 title, "Battlefield: Hardline", however attempting to use the same library for Catalyst results in 0 streams being detected/processed.

I can only find one user actually succeeding to pre-compress Catalyst, who indicates they used "xtlz4(v12)" but I've no idea if this refers to the library or tool.

Any advice would be greatly appreciated, I'm properly stumped with this one.

[Razor12911]

Originally Posted by L33THAK0R

Can anyone recommend which library and/or version of XTool to use for processing "Mirror's Edge: Catalyst"'s LZ4 streams? I recently tried running my collection of LZ4 libraries against a 100MB sample, using XTool 0.7.8 however seemingly none of them were of any use (see the following pastebin output). I've had success in the past with another LZ4 Frostbite 3 title, "Battlefield: Hardline", however attempting to use the same library for Catalyst results in 0 streams being detected/processed.

I can only find one user actually succeeding to pre-compress Catalyst, who indicates they used "xtlz4(v12)" but I've no idea if this refers to the library or tool.

Any advice would be greatly appreciated, I'm properly stumped with this one.

https://fileforums.com/showpost.php?...82&postcount=2
Use the frostbite 3 plugin for this game -mfrostbite3:lz4

[L33THAK0R]

Originally Posted by Razor12911

https://fileforums.com/showpost.php?...82&postcount=2
Use the frostbite 3 plugin for this game -mfrostbite3:lz4

I should have specified I was having difficulty with finding streams using the frostbite 3 plugin, turns out though I was just testing against a sample that didnt have any streams

Once I tried against a larger sample the plugin worked perfectly, many thanks for your hard-work on it!