Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

STAR WARS BATTLEFRONT II
Celebration Edition

 13:20:52 - Selected ARC method for Data1.bin was: xtool_zstd+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 19:08:26 - Overall input size: 90,27 GB
 19:08:26 - Overall output size: 63,74 GB
 19:08:26 - Overall conversion time: 05:47:31

Note: Use zstd lib v1.3.0

Extracted 451 files, 68,435,068,787 => 96,929,288,702 bytes. Ratio 70.60%
Extraction time: cpu 5274.48 sec/real 2694.36 sec = 196%. Speed 35.97 mB/s
All OK

[Masquerade]

Ripped Soundtrack in WAV format.

Total Start Size: 13.1GB

• Everything:
  

  xtool+srep+lolz

• *.mp3
  

  mpz

Compressed Size: 8.78GB

[Snapppr6]

start size 1.67gb

srep:m3f:a2+7z

compressed 1.02gb

[Hexagon123]

Total start size: 3.5GB

Compress with the following:

• .pak
  

  XTool:c128m,t75p,dm2:zlib+srep:m5f:a2:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:fba4096

• Rest of the Files
  

  srep:m5f:a2:l512+mcm:x11

Total packed size: 1.9GB

[KaktoR]

Sims 4
v1.70.84.1020, nothing ripped

 12:21:15 - Selected ARC method for Data1.bin was: xtool_zlib+xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 15:08:41 - Overall input size: 50.61 GB
 15:08:41 - Overall output size: 28.91 GB
 15:08:41 - Data files creation time: 02:29:14
 15:08:41 - ISO images creation time: 00:18:08
 15:08:41 - Overall conversion time: 02:47:23

--------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1.bin file, Please Wait...
--------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: .\Conversion_Output\Sims 4 DVD1\Data1.bin.001
Extracting 4,268 files, 54,311,277,744 bytes. Processed  99.9%
Extracted 4,268 files, 24,577,323,615 => 54,311,277,744 bytes. Ratio 45.25%
Extraction time: cpu 2466.95 sec/real 1249.89 sec = 197%. Speed 43.45 mB/s
All OK

[hydefromt70s]

The Medium

16:39:58 - Selected ARC method for Data1.bin was: xtool_zlib+srep_new+lolz_mtx
------------------------------------------------------------------------------------
13:43:41 - Overall input size: 37,83 GB
13:43:41 - Overall output size: 15,17 GB
13:43:41 - Overall conversion time: 21:03:41
------------------------------------------------------------------------------------

Tried with lolz mt - took 21h !!! Unpacking now to check if everything is right. Can someone explain me why does it take so much RAM ??

[KaktoR]

Elite Dangerous Horizons
v2021-01-28

 19:21:37 - Selected ARC method for Data1a.bin was: xtool_preflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 20:38:50 - Overall input size: 20.98 GB
 20:38:50 - Overall output size: 14.73 GB
 20:38:50 - Data file creation time: 01:08:28

Extracted 38,187 files, 15,814,328,948 => 22,531,389,592 bytes. Ratio 70.19%
Extraction time: cpu 1189.64 sec/real 800.00 sec = 149%. Speed 28.16 mB/s
All OK

[hydefromt70s]

I did some more tests today on "The Medium". My goal was to find the most efficient algorithm for test file which contains Zlib streams. The winner is XT xZlib (old xtool) - managed to inflate file from 968MB to 1250MB and with srep and lzma compression, final size went down to 283MB. That meets my expectation.

Now is the part where I don't understand things. If file contains Zlib streams (I did check in "Drop and Scan for Zlib" app so I'm sure of that) I was betting on XT2020 Zlib/reflate/preflate that one of them will be the most efficient pre-compressor and they didn't work at all! I thought that new version of xtool 2020 will be better than old one? What I don't understand here is why XT xZlib worked and XT2020 Zlib didn't? Are they not the same algorithms to deal with deflated streams?

Would highly appreciate if someone could explain this to me.

[Cesar82]

Originally Posted by hydefromt70s

I did some more tests today on "The Medium". My goal was to find the most efficient algorithm for test file which contains Zlib streams. The winner is XT xZlib (old xtool) - managed to inflate file from 968MB to 1250MB and with srep and lzma compression, final size went down to 283MB. That meets my expectation.

Now is the part where I don't understand things. If file contains Zlib streams (I did check in "Drop and Scan for Zlib" app so I'm sure of that) I was betting on XT2020 Zlib/reflate/preflate that one of them will be the most efficient pre-compressor and they didn't work at all! I thought that new version of xtool 2020 will be better than old one? What I don't understand here is why XT xZlib worked and XT2020 Zlib didn't? Are they not the same algorithms to deal with deflated streams?

Would highly appreciate if someone could explain this to me.

Maybe it's because old XTool includes reflate automatically if the reflate libraries (raw2hif_dll.dll and hif2raw_dll.dll) are next to xtool.exe
If they were with XTool, try removing raw2hif_dll.dll and hif2raw_dll.dll from the old Xtool, or use XTool 2020 with "zlib+reflate" methods to get an equivalent comparison.
If I reported wrong, please correct me.


Edit:
I did some tests:
sample: pakchunk0-WindowsNoEditor.pak (from game version GOG)
The results were practically the same.

XTool v0.3.9 (2020) (only zlibwapi.dll next to XTool.exe)
packcmd   = XTool.exe precomp -mzlib -c64mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = XTool.exe decode -t100p - - <stdin> <stdout>

pack:
Compressed 1 file, 1,015,774,515 => 1,303,484,741 bytes. Ratio 128.32%
Compression time: cpu 1.03 sec/real 14.51 sec = 7%. Speed 70.00 mB/s

unpack:
Extracted 1 file, 1,303,484,741 => 1,015,774,515 bytes. Ratio 128.32%
Extraction time: cpu 1.84 sec/real 5.74 sec = 32%. Speed 177.02 mB/s

XTool v0.1.2 (OLD) (only zlibwapi.dll next to XTool.exe)
packcmd   = XTool.exe e:precomp:t75p,c128m:zlib - - <stdin> <stdout>
unpackcmd = XTool.exe d:precomp:t90p - - <stdin> <stdout>

pack:
Compressed 1 file, 1,015,774,515 => 1,303,386,164 bytes. Ratio 128.31%
Compression time: cpu 1.05 sec/real 28.62 sec = 4%. Speed 35.50 mB/s

unpack:
Extracted 1 file, 1,303,386,164 => 1,015,774,515 bytes. Ratio 128.31%
Extraction time: cpu 1.50 sec/real 5.83 sec = 26%. Speed 174.33 mB/s

[KaktoR]

You can also combine xtool codecs like -mreflate+zlib

[Razor12911]

Originally Posted by Cesar82

Maybe it's because old XTool includes reflate automatically if the reflate libraries (raw2hif_dll.dll and hif2raw_dll.dll) are next to xtool.exe
If they were with XTool, try removing raw2hif_dll.dll and hif2raw_dll.dll from the old Xtool, or use XTool 2020 with "zlib+reflate" methods to get an equivalent comparison.
If I reported wrong, please correct me.

this is correct but if you look at the time xtool took, one would assume that the either the codec didn't initialize due to missing libraries, failure to load libraries or incorrect command line used. I mean to process 968 MB worth of data taking only 3 secs seems like that's what happened.

Originally Posted by hydefromt70s

Would highly appreciate if someone could explain this to me.

the arc.ini/command line you used for xtool would be helpful, that includes the libraries you placed near xtool 2020

[KaktoR]

Age of Wonders Planetfall
v1.403, nothing ripped

 01:49:01 - Selected ARC method for Data1a.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000/$oggrepack=oggre/$mp3pack=mpz_mtx/$soundpack=srep_new+razor_mtx/$deflatepack=xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 02:25:35 - Overall input size: 16.24 GB
 02:25:35 - Overall output size: 5.27 GB
 02:25:35 - Data file creation time: 00:33:38

[hydefromt70s]

Originally Posted by Razor12911

this is correct but if you look at the time xtool took, one would assume that the either the codec didn't initialize due to missing libraries, failure to load libraries or incorrect command line used. I mean to process 968 MB worth of data taking only 3 secs seems like that's what happened.

the arc.ini/command line you used for xtool would be helpful, that includes the libraries you placed near xtool 2020

And that's exactly what happened. I was using old version of DS GUI with XTools 0.37 and despite problem with loading xtool, GUI was saying everything's fine.

Now with DS GUI 1.0.0.1 and XTool 0.38 everything works as it should.
Thank you all.

[KaktoR]

Dragon Age Inquisition
v1.12, nothing ripped

 13:30:11 - Selected ARC method for Data1a.bin was: xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 15:29:49 - Overall input size: 44.46 GB
 15:29:49 - Overall output size: 28.50 GB
 15:29:49 - Overall conversion time: 01:59:37

[Masquerade]

Immortals Fenyx Rising quick test sample:

Compression used: selkie

FreeArc 0.67 (March 15 2014) Creating archive: __Test\output.MSQ using selkie
Memory for compression 0b, decompression 0b, cache 1mb
Compressing 1 file, 121,110,528 bytes
  Compressing DataPC_Orpheus_gi.forge
Compressed 1 file, 121,110,528 => 180,603,282 bytes. Ratio 149.12%
Compression time: cpu 0.13 sec/real 9.88 sec = 1%. Speed 12.26 mB/s
All OK

Use the oodle core I posted here.

You can also use the oo2core_8_win64.dll that comes with XTool 2020, but you'll need to rename it to oo2core_7_win64.dll in order for oo2recs to work with it!