Best Compression Methods for 'Specific' Games. Q&A

[Hexagon123]

Total start size (original): 20.4GB with redist / 20.3GB without redist
Compress with the following:

• Contains the following files
  • CoEMusic.fc
  • Music.fc

  oggre

• Contains the following files, since SREP likely crashes, so use XTool dedup instead.
  • AnimationsCompressed.fc
  • CoEAnimationsCompressed.fc
  • CoEFonts.fc
  • CoEGameObjects.fc
  • CoELevels.fc
  • Fonts.fc
  • GameObjects.fc
  • Levels.fc

  xtool:dd3+lolz

• Rest of the files: EXEs, other .FC files, etc. RZ has an dedup option
  

  rz

Total packed size (new): 1.5GB

[Hexagon123]

Total start size (original): 1.4GB with MP3 and FLAC soundtracks / 1.2GB with MP3 soundtrack, FLAC ripped / 1.1GB without soundtracks

Version 1.1.3 [b10631325]

Compress with the following:

• .resS, .assets, .(noextension)
  

  srep+lolz

• Rest of the Files
  

  rz

• .mp3
  

  mpz

Total packed size (new): 244MB with MP3

[wareck]

Doomer simulator

xtool:c32mb:d1:mue5,k0x7100780079007A0081008200830 08A008B008C009300940095009C009D009E00:mreflate+xto ol:mreflate:dd3+LOLZ_NORMAL

xtool:mreflate:dd3+razor_mtx

---------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------- CONVERSION DATE 2025/06/18 --------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------
16:18:11 - Deleting files from the output folder.
16:18:11 - Files have been deleted.
---------------------------------------------------------------------------------------------------------------------------------------------------------
16:18:11 - You have chosen the unlimited type for the output size.
16:18:11 - ISO images will not be created when conversion is complete.
16:18:11 - The computer will remain on when the conversion is complete.
16:18:11 - Selected ARC/DS method for Data1.wrk was: xtool:mue5:k0x7100780079007A008100820083008A008B00 8C009300940095009C009D009E00+xtool:mreflate:mkrake n:mmermaid:core_2.9.10:lib_2+srep_new:490g+lolz:d9 6m:mtb48mb:mt8:ldmf:ldl5:mc1023
16:18:11 - Selected ARC/SPLIT method for Data2.wrk was: xtool:mreflate+srep_new:490g+lolz:d40:mtt1:mt16:mc 1023/$deflatepack=precomp+srep_new:490g+lolz:d40:mtt1:m t16:mc1023/$libpack=xtool:mreflate+srep_new:490g+razor_mtx
---------------------------------------------------------------------------------------------------------------------------------------------------------
16:18:13 - Compression initialized in 18/06/2025 at 16:18:13.
16:18:14 - Starting Data1.wrk archive creation.
17:50:52 - Moving Data1.wrk file to media folder.
17:50:54 - Starting Data2.wrk archive creation.
17:52:59 - Moving Data2.wrk file to media folder.
17:52:59 - Compression finished in 18/06/2025 at 17:52:59.
---------------------------------------------------------------------------------------------------------------------------------------------------------
17:52:59 - Getting size of the game: Doomer simulator
17:52:59 - » Installation folder size: 8,47 GB
17:52:59 - » Data1.wrk file size: 5,49 GB
17:52:59 - » Data2.wrk file size: 96,01 MB
17:52:59 - » Compressed game size: 5,59 GB (Ratio 65.98%)
---------------------------------------------------------------------------------------------------------------------------------------------------------
17:52:59 - Overall input size: 8,47 GB
17:52:59 - Overall output size: 5,59 GB (Ratio 65.98%)
17:52:59 - Overall conversion time: 01:34:45
17:52:59 - Overall conversion media: 1 custom media

[wareck]

Ancestors The Humankind Odyssey:

data1 (pak) :
xtool:mzlib:mreflate:dd3+lolz
data2 (.wem)
msc_tak+srep_new:490g+LZMA-MT
data3 (.bik)
xtool:c1792mb:mbik
data4 (other)
xtool:dd3+lolz

=> arround 3.32gb compressed + installer

[JustFun]

Ben and Ed

Selected ARC/DS method for Data1a-01.bin was: 
srep_new:490g+lolz
------------------------------------------------------------------------------------------
Overall input size: 6.55 GB
Overall output size: 1.07 GB (Ratio 16.36%)

[panker1992]

This is Dedup.exe
part of my archiver i plan to release soon
Please test it

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>

[shazzla]

# arc a -i2 -mdedup p:\ded.arc e:\shdw.xt
FreeArc 0.67 (March 15 2014) creating archive: p:\ded.arc
Compressing 1 file, 31,837,733,330 bytes
Compressing shdw.xt 99.9%
Compressing 31,837,733,330 bytes with dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
100.0%=== DEDUPLICATION (XXH3-128) ===
Pass 1: Scanning for block positions (minimal RAM)
Processing 29.65 GB input file...
Progress: 100.00%
Starting collision-free frequency-based deduplication...
Pass 1 complete: 44006478 blocks, 37991985 unique XXH3 hashes
100.0%Collision-Free Analysis: 6014493 duplicate blocks, 2938568 unique hashes cached
Pass 1.75: Loading ONLY duplicate block data into RAM...
100.0%Loaded 2938568 unique duplicate blocks into RAM
100.0%RAM usage: ~5.3% of original data (duplicates only)
Pass 2: Writing all blocks with XXH3 collision-free hints...
100.0%=== DEDUPLICATION RESULTS (XXH3-128) ===
Original Size: 29.65 GB (31837733330 bytes)
Compressed Size: 26.74 GB (28713904996 bytes)
Compression Ratio: 90.19% (1.11:1)
Space Saved: 2.91 GB (9.8%)
Total Blocks: 44006478
Unique Blocks: 37991985 (86.3%)
Duplicate Blocks: 6014493 (13.7%)
Cached Blocks: 2938568 (7.7% of unique)
Hash Algorithm: XXH3-128 (Collision-Free)
RAM breakdown: Tabs=1.31 GB, FreqMap=869.57 MB, Duplicates=1.58 GB, Total=3.74 GB (12.6% of input)
Collision-free deduplication complete!
100.0%
Errorlevel=0

Compressed 1 file, 31,837,733,330 => 28,713,904,996 bytes. Ratio 90.19%
Compression time: cpu 8.25 sec/real 1506.37 sec = 1%. Speed 21.14 mB/s
All OK

# arc t p:\ded.arc
FreeArc 0.67 (March 15 2014) testing archive: p:\ded.arc
Testing 1 file, 31,837,733,330 bytes. Processed 36.9%
ERROR: write error (disk full?) in compression algorithm dedup

#

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>


Win10 Enterprise 21H2
Ryzen 7 9700x
32 GB RAM

ATM no more free time to testing.
Thanks for the program!
(Maybe later...)

Interesting (another dataset):
Microsoft Windows [Version 10.0.19044.5965]
(c) Microsoft Corporation. All rights reserved.

arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 8.3%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.8%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup

[Dragonis40]

Good morning! I'm trying to compress "Banishers Ghosts Of New Eden" videogame, but i'm having some troubles with .bk2 files. How can i compress thwm? Bpk and Xtool bk2 plugin don't compress at all. What can i compress them? Thank you for you reply!

[KaktoR]

They are kb2n, you cannot compress them.

[panker1992]

Originally Posted by shazzla

# arc a -i2 -mdedup p:\ded.arc e:\shdw.xt
FreeArc 0.67 (March 15 2014) creating archive: p:\ded.arc
Compressing 1 file, 31,837,733,330 bytes
Compressing shdw.xt 99.9%
Compressing 31,837,733,330 bytes with dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
100.0%=== DEDUPLICATION (XXH3-128) ===
Pass 1: Scanning for block positions (minimal RAM)
Processing 29.65 GB input file...
Progress: 100.00%
Starting collision-free frequency-based deduplication...
Pass 1 complete: 44006478 blocks, 37991985 unique XXH3 hashes
100.0%Collision-Free Analysis: 6014493 duplicate blocks, 2938568 unique hashes cached
Pass 1.75: Loading ONLY duplicate block data into RAM...
100.0%Loaded 2938568 unique duplicate blocks into RAM
100.0%RAM usage: ~5.3% of original data (duplicates only)
Pass 2: Writing all blocks with XXH3 collision-free hints...
100.0%=== DEDUPLICATION RESULTS (XXH3-128) ===
Original Size: 29.65 GB (31837733330 bytes)
Compressed Size: 26.74 GB (28713904996 bytes)
Compression Ratio: 90.19% (1.11:1)
Space Saved: 2.91 GB (9.8%)
Total Blocks: 44006478
Unique Blocks: 37991985 (86.3%)
Duplicate Blocks: 6014493 (13.7%)
Cached Blocks: 2938568 (7.7% of unique)
Hash Algorithm: XXH3-128 (Collision-Free)
RAM breakdown: Tabs=1.31 GB, FreqMap=869.57 MB, Duplicates=1.58 GB, Total=3.74 GB (12.6% of input)
Collision-free deduplication complete!
100.0%
Errorlevel=0

Compressed 1 file, 31,837,733,330 => 28,713,904,996 bytes. Ratio 90.19%
Compression time: cpu 8.25 sec/real 1506.37 sec = 1%. Speed 21.14 mB/s
All OK

# arc t p:\ded.arc
FreeArc 0.67 (March 15 2014) testing archive: p:\ded.arc
Testing 1 file, 31,837,733,330 bytes. Processed 36.9%
ERROR: write error (disk full?) in compression algorithm dedup

#

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>


Win10 Enterprise 21H2
Ryzen 7 9700x
32 GB RAM

ATM no more free time to testing.
Thanks for the program!
(Maybe later...)

Interesting (another dataset):
Microsoft Windows [Version 10.0.19044.5965]
(c) Microsoft Corporation. All rights reserved.

arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 8.3%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.8%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup

Hello, i am trying to understand what you are doing ...it's working fine here

[shazzla]

As you see,decompression doesnt work for me.
What can i do?!

[panker1992]

Try this one and tell me

[shazzla]

Its bit-identical with first version.

[wareck]

Originally Posted by shazzla

Its bit-identical with first version.

It works for me
Don't forget to update arc.ini
And you need to have dedup.exe both for compress and decompression.
you may also need to have admin rights to uncompress

[panker1992]

i think when i wrote this i had shared memory technology activated in one module :/ i will check and recompile one more version that doesnt use shared memory as this isnt needed here