Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Originally Posted by JustFun

LOL, yeah, I hate it too, when decompression fails after long compression time

Though, even worse, once I compressed the game, made installer and everything, and somehow forgot to test installation lol, and I happily burned it to disc˙s.
Some time after, I decided to install that game and play it, and boy was I dissapointed when I got decompression error at 72%

You still burning discs?

[JustFun]

Originally Posted by KaktoR

You still burning discs?

LOL, yeah, it is my favorite way of Backing Up files, games and pretty much everything I would need sometime.

Even though these days I can download games faster then I could install them lol, but still I like to backup on Disc`s, offline backup is the best backup .

[KaktoR]

The Medium

 20:38:35 - Selected ARC method for Data1a.bin was: xtool_preflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 22:36:39 - Overall input size: 38.73 GB
 22:36:39 - Overall output size: 17.75 GB
 22:36:39 - Overall conversion time: 01:58:01
--------------------------------------------------------------------------------------------------------------------------------------

[Snapppr6]

original size 2gb after ripping some mp and unwented sound and video
reflate+serp+lzma2
repack size 1.23gb

[Masquerade]

Ripped Soundtrack in WAV format.

Total Start Size: 13.1GB

• Everything:
  

  xtool+srep+lolz

• *.mp3
  

  mpz

Compressed Size: 8.78GB

[Snapppr6]

start size 1.67gb

srep:m3f:a2+7z

compressed 1.02gb

[KaktoR]

STAR WARS BATTLEFRONT II
Celebration Edition

 13:20:52 - Selected ARC method for Data1.bin was: xtool_zstd+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 19:08:26 - Overall input size: 90,27 GB
 19:08:26 - Overall output size: 63,74 GB
 19:08:26 - Overall conversion time: 05:47:31

Note: Use zstd lib v1.3.0

Extracted 451 files, 68,435,068,787 => 96,929,288,702 bytes. Ratio 70.60%
Extraction time: cpu 5274.48 sec/real 2694.36 sec = 196%. Speed 35.97 mB/s
All OK

[Hexagon123]

Total start size: 3.5GB

Compress with the following:

• .pak
  

  XTool:c128m,t75p,dm2:zlib+srep:m5f:a2:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:fba4096

• Rest of the Files
  

  srep:m5f:a2:l512+mcm:x11

Total packed size: 1.9GB

[KaktoR]

Sims 4
v1.70.84.1020, nothing ripped

 12:21:15 - Selected ARC method for Data1.bin was: xtool_zlib+xtool_reflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 15:08:41 - Overall input size: 50.61 GB
 15:08:41 - Overall output size: 28.91 GB
 15:08:41 - Data files creation time: 02:29:14
 15:08:41 - ISO images creation time: 00:18:08
 15:08:41 - Overall conversion time: 02:47:23

--------------------------------------------------------------------------------------------------------------------------------------
 Extracting Data1.bin file, Please Wait...
--------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: .\Conversion_Output\Sims 4 DVD1\Data1.bin.001
Extracting 4,268 files, 54,311,277,744 bytes. Processed  99.9%
Extracted 4,268 files, 24,577,323,615 => 54,311,277,744 bytes. Ratio 45.25%
Extraction time: cpu 2466.95 sec/real 1249.89 sec = 197%. Speed 43.45 mB/s
All OK

[hydefromt70s]

The Medium

16:39:58 - Selected ARC method for Data1.bin was: xtool_zlib+srep_new+lolz_mtx
------------------------------------------------------------------------------------
13:43:41 - Overall input size: 37,83 GB
13:43:41 - Overall output size: 15,17 GB
13:43:41 - Overall conversion time: 21:03:41
------------------------------------------------------------------------------------

Tried with lolz mt - took 21h !!! Unpacking now to check if everything is right. Can someone explain me why does it take so much RAM ??

[KaktoR]

Elite Dangerous Horizons
v2021-01-28

 19:21:37 - Selected ARC method for Data1a.bin was: xtool_preflate+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
--------------------------------------------------------------------------------------------------------------------------------------
 20:38:50 - Overall input size: 20.98 GB
 20:38:50 - Overall output size: 14.73 GB
 20:38:50 - Data file creation time: 01:08:28

Extracted 38,187 files, 15,814,328,948 => 22,531,389,592 bytes. Ratio 70.19%
Extraction time: cpu 1189.64 sec/real 800.00 sec = 149%. Speed 28.16 mB/s
All OK

[hydefromt70s]

I did some more tests today on "The Medium". My goal was to find the most efficient algorithm for test file which contains Zlib streams. The winner is XT xZlib (old xtool) - managed to inflate file from 968MB to 1250MB and with srep and lzma compression, final size went down to 283MB. That meets my expectation.

Now is the part where I don't understand things. If file contains Zlib streams (I did check in "Drop and Scan for Zlib" app so I'm sure of that) I was betting on XT2020 Zlib/reflate/preflate that one of them will be the most efficient pre-compressor and they didn't work at all! I thought that new version of xtool 2020 will be better than old one? What I don't understand here is why XT xZlib worked and XT2020 Zlib didn't? Are they not the same algorithms to deal with deflated streams?

Would highly appreciate if someone could explain this to me.

[Cesar82]

Originally Posted by hydefromt70s

I did some more tests today on "The Medium". My goal was to find the most efficient algorithm for test file which contains Zlib streams. The winner is XT xZlib (old xtool) - managed to inflate file from 968MB to 1250MB and with srep and lzma compression, final size went down to 283MB. That meets my expectation.

Now is the part where I don't understand things. If file contains Zlib streams (I did check in "Drop and Scan for Zlib" app so I'm sure of that) I was betting on XT2020 Zlib/reflate/preflate that one of them will be the most efficient pre-compressor and they didn't work at all! I thought that new version of xtool 2020 will be better than old one? What I don't understand here is why XT xZlib worked and XT2020 Zlib didn't? Are they not the same algorithms to deal with deflated streams?

Would highly appreciate if someone could explain this to me.

Maybe it's because old XTool includes reflate automatically if the reflate libraries (raw2hif_dll.dll and hif2raw_dll.dll) are next to xtool.exe
If they were with XTool, try removing raw2hif_dll.dll and hif2raw_dll.dll from the old Xtool, or use XTool 2020 with "zlib+reflate" methods to get an equivalent comparison.
If I reported wrong, please correct me.


Edit:
I did some tests:
sample: pakchunk0-WindowsNoEditor.pak (from game version GOG)
The results were practically the same.

XTool v0.3.9 (2020) (only zlibwapi.dll next to XTool.exe)
packcmd   = XTool.exe precomp -mzlib -c64mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = XTool.exe decode -t100p - - <stdin> <stdout>

pack:
Compressed 1 file, 1,015,774,515 => 1,303,484,741 bytes. Ratio 128.32%
Compression time: cpu 1.03 sec/real 14.51 sec = 7%. Speed 70.00 mB/s

unpack:
Extracted 1 file, 1,303,484,741 => 1,015,774,515 bytes. Ratio 128.32%
Extraction time: cpu 1.84 sec/real 5.74 sec = 32%. Speed 177.02 mB/s

XTool v0.1.2 (OLD) (only zlibwapi.dll next to XTool.exe)
packcmd   = XTool.exe e:precomp:t75p,c128m:zlib - - <stdin> <stdout>
unpackcmd = XTool.exe d:precomp:t90p - - <stdin> <stdout>

pack:
Compressed 1 file, 1,015,774,515 => 1,303,386,164 bytes. Ratio 128.31%
Compression time: cpu 1.05 sec/real 28.62 sec = 4%. Speed 35.50 mB/s

unpack:
Extracted 1 file, 1,303,386,164 => 1,015,774,515 bytes. Ratio 128.31%
Extraction time: cpu 1.50 sec/real 5.83 sec = 26%. Speed 174.33 mB/s

[Razor12911]

Originally Posted by Cesar82

Maybe it's because old XTool includes reflate automatically if the reflate libraries (raw2hif_dll.dll and hif2raw_dll.dll) are next to xtool.exe
If they were with XTool, try removing raw2hif_dll.dll and hif2raw_dll.dll from the old Xtool, or use XTool 2020 with "zlib+reflate" methods to get an equivalent comparison.
If I reported wrong, please correct me.

this is correct but if you look at the time xtool took, one would assume that the either the codec didn't initialize due to missing libraries, failure to load libraries or incorrect command line used. I mean to process 968 MB worth of data taking only 3 secs seems like that's what happened.

Originally Posted by hydefromt70s

Would highly appreciate if someone could explain this to me.

the arc.ini/command line you used for xtool would be helpful, that includes the libraries you placed near xtool 2020

[hydefromt70s]

Originally Posted by Razor12911

this is correct but if you look at the time xtool took, one would assume that the either the codec didn't initialize due to missing libraries, failure to load libraries or incorrect command line used. I mean to process 968 MB worth of data taking only 3 secs seems like that's what happened.

the arc.ini/command line you used for xtool would be helpful, that includes the libraries you placed near xtool 2020

And that's exactly what happened. I was using old version of DS GUI with XTools 0.37 and despite problem with loading xtool, GUI was saying everything's fine.

Now with DS GUI 1.0.0.1 and XTool 0.38 everything works as it should.
Thank you all.