FileForums - Best Compression Methods for 'Specific' Games. Q&A

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

STAR WARS BATTLEFRONT II
Celebration Edition

Code:

 13:20:52 - Selected ARC method for Data1.bin was: xtool_zstd+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

--------------------------------------------------------------------------------------------------------------------------------------

 19:08:26 - Overall input size: 90,27 GB

 19:08:26 - Overall output size: 63,74 GB

 19:08:26 - Overall conversion time: 05:47:31

Note: Use zstd lib v1.3.0

Code:

Extracted 451 files, 68,435,068,787 => 96,929,288,702 bytes. Ratio 70.60%

Extraction time: cpu 5274.48 sec/real 2694.36 sec = 196%. Speed 35.97 mB/s

All OK

Sniper Ghost Warrior Contracts v20210125 [Digital Deluxe Edition]

Ripped Soundtrack in WAV format.

Total Start Size: 13.1GB

Everything:

Code:

xtool+srep+lolz
*.mp3

Code:

mpz

Compressed Size: 8.78GB

Battlefield 1942 + 2dls

start size 1.67gb

srep:m3f:a2+7z

compressed 1.02gb

Override 2: Super Mech League

Total start size: 3.5GB

Compress with the following:

.pak

Code:

XTool:c128m,t75p,dm2:zlib+srep:m5f:a2:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:fba4096

Rest of the Files

Code:

srep:m5f:a2:l512+mcm:x11

Total packed size: 1.9GB

Sims 4
v1.70.84.1020, nothing ripped

Code:

 12:21:15 - Selected ARC method for Data1.bin was: xtool_zlib+xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

--------------------------------------------------------------------------------------------------------------------------------------

 15:08:41 - Overall input size: 50.61 GB

 15:08:41 - Overall output size: 28.91 GB

 15:08:41 - Data files creation time: 02:29:14

 15:08:41 - ISO images creation time: 00:18:08

 15:08:41 - Overall conversion time: 02:47:23

Code:

--------------------------------------------------------------------------------------------------------------------------------------

 Extracting Data1.bin file, Please Wait...

--------------------------------------------------------------------------------------------------------------------------------------

FreeArc 0.67 (March 15 2014) extracting archive: .\Conversion_Output\Sims 4 DVD1\Data1.bin.001

Extracting 4,268 files, 54,311,277,744 bytes. Processed  99.9%

Extracted 4,268 files, 24,577,323,615 => 54,311,277,744 bytes. Ratio 45.25%

Extraction time: cpu 2466.95 sec/real 1249.89 sec = 197%. Speed 43.45 mB/s

All OK

The Medium

16:39:58 - Selected ARC method for Data1.bin was: xtool_zlib+srep_new+lolz_mtx
------------------------------------------------------------------------------------
13:43:41 - Overall input size: 37,83 GB
13:43:41 - Overall output size: 15,17 GB
13:43:41 - Overall conversion time: 21:03:41
------------------------------------------------------------------------------------

Tried with lolz mt - took 21h !!! Unpacking now to check if everything is right. Can someone explain me why does it take so much RAM ??

Elite Dangerous Horizons
v2021-01-28

Code:

 19:21:37 - Selected ARC method for Data1a.bin was: xtool_preflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

--------------------------------------------------------------------------------------------------------------------------------------

 20:38:50 - Overall input size: 20.98 GB

 20:38:50 - Overall output size: 14.73 GB

 20:38:50 - Data file creation time: 01:08:28

Code:

Extracted 38,187 files, 15,814,328,948 => 22,531,389,592 bytes. Ratio 70.19%

Extraction time: cpu 1189.64 sec/real 800.00 sec = 149%. Speed 28.16 mB/s

All OK

The Medium - compression test and comparision

I did some more tests today on "The Medium". My goal was to find the most efficient algorithm for test file which contains Zlib streams. The winner is XT xZlib (old xtool) - managed to inflate file from 968MB to 1250MB and with srep and lzma compression, final size went down to 283MB. That meets my expectation.

Now is the part where I don't understand things. If file contains Zlib streams (I did check in "Drop and Scan for Zlib" app so I'm sure of that) I was betting on XT2020 Zlib/reflate/preflate that one of them will be the most efficient pre-compressor and they didn't work at all! I thought that new version of xtool 2020 will be better than old one? What I don't understand here is why XT xZlib worked and XT2020 Zlib didn't? Are they not the same algorithms to deal with deflated streams?

Would highly appreciate if someone could explain this to me.

Quote:

Originally Posted by hydefromt70s (Post 490335)

Maybe it's because old XTool includes reflate automatically if the reflate libraries (raw2hif_dll.dll and hif2raw_dll.dll) are next to xtool.exe
If they were with XTool, try removing raw2hif_dll.dll and hif2raw_dll.dll from the old Xtool, or use XTool 2020 with "zlib+reflate" methods to get an equivalent comparison.
If I reported wrong, please correct me.

Edit:
I did some tests:
sample: pakchunk0-WindowsNoEditor.pak (from game version GOG)
The results were practically the same.

Code:

XTool v0.3.9 (2020) (only zlibwapi.dll next to XTool.exe)

packcmd   = XTool.exe precomp -mzlib -c64mb -t100p --dbase - - <stdin> <stdout>

unpackcmd = XTool.exe decode -t100p - - <stdin> <stdout>



pack:

Compressed 1 file, 1,015,774,515 => 1,303,484,741 bytes. Ratio 128.32%

Compression time: cpu 1.03 sec/real 14.51 sec = 7%. Speed 70.00 mB/s



unpack:

Extracted 1 file, 1,303,484,741 => 1,015,774,515 bytes. Ratio 128.32%

Extraction time: cpu 1.84 sec/real 5.74 sec = 32%. Speed 177.02 mB/s



XTool v0.1.2 (OLD) (only zlibwapi.dll next to XTool.exe)

packcmd   = XTool.exe e:precomp:t75p,c128m:zlib - - <stdin> <stdout>

unpackcmd = XTool.exe d:precomp:t90p - - <stdin> <stdout>



pack:

Compressed 1 file, 1,015,774,515 => 1,303,386,164 bytes. Ratio 128.31%

Compression time: cpu 1.05 sec/real 28.62 sec = 4%. Speed 35.50 mB/s



unpack:

Extracted 1 file, 1,303,386,164 => 1,015,774,515 bytes. Ratio 128.31%

Extraction time: cpu 1.50 sec/real 5.83 sec = 26%. Speed 174.33 mB/s

https://i.imgur.com/6oxJUx2.png

You can also combine xtool codecs like -mreflate+zlib

Quote:

Originally Posted by Cesar82 (Post 490336)

this is correct but if you look at the time xtool took, one would assume that the either the codec didn't initialize due to missing libraries, failure to load libraries or incorrect command line used. I mean to process 968 MB worth of data taking only 3 secs seems like that's what happened.

Quote:

Originally Posted by hydefromt70s (Post 490335)

Would highly appreciate if someone could explain this to me.

the arc.ini/command line you used for xtool would be helpful, that includes the libraries you placed near xtool 2020

Age of Wonders Planetfall
v1.403, nothing ripped

Code:

 01:49:01 - Selected ARC method for Data1a.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000/$oggrepack=oggre/$mp3pack=mpz_mtx/$soundpack=srep_new+razor_mtx/$deflatepack=xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

--------------------------------------------------------------------------------------------------------------------------------------

 02:25:35 - Overall input size: 16.24 GB

 02:25:35 - Overall output size: 5.27 GB

 02:25:35 - Data file creation time: 00:33:38

Quote:

Originally Posted by Razor12911 (Post 490339)

And that's exactly what happened. I was using old version of DS GUI with XTools 0.37 and despite problem with loading xtool, GUI was saying everything's fine.

Now with DS GUI 1.0.0.1 and XTool 0.38 everything works as it should.
Thank you all.

Dragon Age Inquisition
v1.12, nothing ripped

Code:

 13:30:11 - Selected ARC method for Data1a.bin was: xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

--------------------------------------------------------------------------------------------------------------------------------------

 15:29:49 - Overall input size: 44.46 GB

 15:29:49 - Overall output size: 28.50 GB

 15:29:49 - Overall conversion time: 01:59:37

Immortals Fenyx Rising quick test sample:

Compression used: selkie

Code:

FreeArc 0.67 (March 15 2014) Creating archive: __Test\output.MSQ using selkie

Memory for compression 0b, decompression 0b, cache 1mb

Compressing 1 file, 121,110,528 bytes

  Compressing DataPC_Orpheus_gi.forge

Compressed 1 file, 121,110,528 => 180,603,282 bytes. Ratio 149.12%

Compression time: cpu 0.13 sec/real 9.88 sec = 1%. Speed 12.26 mB/s

All OK

Use the oodle core I posted here.

You can also use the oo2core_8_win64.dll that comes with XTool 2020, but you'll need to rename it to oo2core_7_win64.dll in order for oo2recs to work with it!