Best Compression Methods for 'Specific' Games. Q&A

[Danziel123]

Originally Posted by Fak Eid

xtool:mstarfield+xtool:mreflate+srep_new+lolz/lzma seems correct for all .ba2 files except WWise Sound files. Also as of now, DSG won't be able to extract the files. You need to do it with your installer only.

Thank you for suggestion, change it to xtool:dd now can compress till finish

[KaktoR]

BioShock Infinite
v31.08.2022

 16:54:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:c320mb:mbik
 16:54:07 - Selected ARC/DS method for Data1b-01.bin was: uelr+xtool:c128mb:mzlib+xtool:dd3+4x4:lzma
 16:54:07 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:lzma
------------------------------------------------------------------------------------------
 19:08:31 - Overall input size: 43.20 GB
 19:08:31 - Overall output size: 15.32 GB (Ratio 35.47%)
 19:08:31 - Overall conversion time: 02:14:23

[Fak Eid]

Why did everyone started using xtool:dd3 instead of srep_new:3. It it better?

Originally Posted by KaktoR

BioShock Infinite
v31.08.2022

 16:54:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:c320mb:mbik
 16:54:07 - Selected ARC/DS method for Data1b-01.bin was: uelr+xtool:c128mb:mzlib+xtool:dd3+4x4:lzma
 16:54:07 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:lzma
------------------------------------------------------------------------------------------
 19:08:31 - Overall input size: 43.20 GB
 19:08:31 - Overall output size: 15.32 GB (Ratio 35.47%)
 19:08:31 - Overall conversion time: 02:14:23

[KaktoR]

Because it's faster.

Edit: "3" in dd# just represents the srep m#f setting m[1..5]. You can even use dd5 to get a bit better results in costs of speed.

Edit2: A bit old, but my test in the past when this feature was introduced: https://fileforums.com/showthread.ph...129#post499129

[Masquerade]

@KaktoR did you try using xtool unreal 3 for the lzo? I'm interesred in the performance vs uelr.

[KaktoR]

ANNO 1800
Update 18.0

 20:37:16 - Selected ARC/DS method for Data1a-01.bin was: xtool:c1500mb:lm:mbk2:df20p+xtool:dd3
 20:37:16 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8+xtool:mzlib:mreflate+xtool:dd3+4x4:lzma
 20:37:16 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+4x4:lzma/$mp3pack=7zdll-pm:m0=mp3det:m1=packmp3c/$bpkpack=bpk
-----------------------------------------------------------------------------------------
 00:12:23 - Overall input size: 101.44 GB
 00:12:23 - Overall output size: 30.63 GB (Ratio 30.20%)
 00:12:23 - Overall conversion time: 03:34:59

Notes: 
- bpk was used on bink1 files which are located in data3.rda
- You have to use -lm for bk2 plugin if you don't have 64GB RAM
- The following files contain bink videos: data3.rda, data11.rda, data12.rda, data13.rda, data16.rda and data18.rda. Use quickbms with "-f "{}.bk2,{}.bik"" file mask to extracrt just bink1/2 files.
- Use XTool erase to create xtl file and zero the files out of the game archives.
- Use XTool decode + xtl file to patch the files back afterwards.
- Some rda files also contain WEM files. You can use wemtool there.
- The language files also contain WEM files, but they are PCM, so you msc/xtool:flac on them.

[Fak Eid]

Thanks for the update. If dd3 can help me compressing the same size as srep and with less extraction time, I'll go ahead with it. My focus is to improve extraction speed at a cost of minimum difference in compression size. I'll try that for sure.

Originally Posted by KaktoR

Because it's faster.

Edit: "3" in dd# just represents the srep m#f setting m[1..5]. You can even use dd5 to get a bit better results in costs of speed.

Edit2: A bit old, but my test in the past when this feature was introduced: https://fileforums.com/showthread.ph...129#post499129

[L33THAK0R]

I know Masquerade made a database recently for the first "Watch Dogs" title, however I've found myself needing to re-generate a new database in my own conversion, as I've opted to include a variety of mods that restore features left on the cutting room floor as well as providing the end-user access to content previously only accessible through online-only interactions. This restoration overwrites a considerable amount of existing files, as such a new database is required.

My current database seems to be considerably smaller than Masquerade's however, with a final result of 3.84 MiB (versus Masquerade's 14.0 MiB). The only issue I can currently see is when the fat2_fat3.bms script attempts to process the "windy_city.fat/.dat" which gives the following error:

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info:  algorithm   2
       offset      494c62b4
       input size  0x0000632b 25387
       output size 0x00010000 65536
       result      0xffffffff -1

Error: uncompressed data (-1) bigger than allocated buffer (224656)
       It usually means that data is not compressed or uses another algorithm

I'm assuming this is referring to

            elif ZTYPE == 2
                comtype deflate

however my bms2xtl.ini configuration file seems to be fine:

[Config]
SafeMode=0
UserInput=
Exclude=lzma_dynamic,snappy
gzip=zlib
deflate=reflate
zlib_noerror=zlib
deflate_noerror=reflate
zlib=zlib
oodle=leviathan
xmemdecompress=xmemcompress
lz4=lz4hc:l9
zenpxp=zenpxp
lzo1x=lzo1x
lzma_dynamic=reflate

I've ensured I've got the latest version of the xmemcompress plugin present, and its appropriate entry within "xtool.ini" is pretty standard

[xmemcompress]
Encode=xcompress64.exe d -window=131072 -partition=524288 <library>
Decode=xcompress64.exe e -window=131072 -partition=524288 <library>

I'm honestly scratching my head trying to figure out what I've messed up this time, even more so by the fact that the file "windy_city.fat/.dat" is one such file that hasn't been modified. Any tips would be greatly appreciated, its definitely been gnawing away at me.

[Razor12911]

the window and partition for xmemcompress should be 32768 I believe, those configurations are for project cars

[xmemcompress]
Encode=xcompress.exe d -window=32768 -partition=32768 <library>
Decode=xcompress.exe e -window=32768 -partition=32768 <library>

as for the other errors, I'm not so sure

[Masquerade]

DOOM Eternal v6.66 Rev 2.2

• *.streamdb, *.resources, *.lightdb
  

  xtool_kraken:l6_doom7*_dd+xtool_kraken:l6_doom8*_dd+srep+lolz

• *.bk2**
  

  xtool_bink2_dd5_fl2

• *.bik
  

  xtool_bink_dd5_fl2

• Everything else:
  

  razor

* Using these two oodle libraries.

** All bk2 except vs_first.bk2, vs_second.bk2 and doommarinesymbol_wide_fade.bk2.

88.7GB ---> 56GB

[Masquerade]

The Sims 4 [v1.100.147.1030 + All DLCs]

xtool_zlib_dd5+LZMA2

58.7GB ---> 36.4GB

[KaktoR]

Originally Posted by Masquerade

** All bk2 except vs_first.bk2, vs_second.bk2 and doommarinesymbol_wide_fade.bk2.

Why excluding? Does the modified bink plugin go crazy again?

[Masquerade]

Originally Posted by KaktoR

Why excluding? Does the modified bink plugin go crazy again?

The modified import does indeed fix motion data warn but it does not fix err. Trying to compress the folder with those 3 videos present will just leave XTool hanging.

[KaktoR]

Could you send me one of these files? Thanks.

Edit: I looked, and I don't have any problems with the patched bink plugin.

Compressed 3 files, 69,070,112 => 69,070,262 bytes. Ratio 100.00%
Compression time: cpu 0.06 sec/real 1.12 sec = 6%. Speed 61.71 mB/s
All OK

Extracted 3 files, 69,070,262 => 69,070,112 bytes. Ratio 100.00%
Extraction time: cpu 0.05 sec/real 0.60 sec = 8%. Speed 114.63 mB/s
All OK

[KaktoR]

XCOM 2
version 28 March 2022

 20:51:40 - Selected ARC/DS method for Data1a-01.bin was: uelr+xtool:mkraken:core_2.4.1+xtool:mzlib:dd3+4x4:lzma
 20:51:40 - Selected ARC/DS method for Data1b-01.bin was: xtool:lm:c1024mb:mbk2:df20p+xtool:dd3
 20:51:40 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+4x4:lzma/$oggre=oggre/$wemtoolpack=xtool:dd3+zstd
-------------------------------------------------------------------------------------------------------------------------------------------
 00:40:08 - Overall input size: 70.96 GB
 00:40:08 - Overall output size: 25.26 GB (Ratio 35.59%)
 00:40:08 - Overall conversion time: 03:48:26

WEMTool was used for *.wem files.