Best Compression Methods for 'Specific' Games. Q&A

[JustFun]

Ben and Ed

Selected ARC/DS method for Data1a-01.bin was: 
srep_new:490g+lolz
------------------------------------------------------------------------------------------
Overall input size: 6.55 GB
Overall output size: 1.07 GB (Ratio 16.36%)

[panker1992]

This is Dedup.exe
part of my archiver i plan to release soon
Please test it

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>

[shazzla]

# arc a -i2 -mdedup p:\ded.arc e:\shdw.xt
FreeArc 0.67 (March 15 2014) creating archive: p:\ded.arc
Compressing 1 file, 31,837,733,330 bytes
Compressing shdw.xt 99.9%
Compressing 31,837,733,330 bytes with dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
100.0%=== DEDUPLICATION (XXH3-128) ===
Pass 1: Scanning for block positions (minimal RAM)
Processing 29.65 GB input file...
Progress: 100.00%
Starting collision-free frequency-based deduplication...
Pass 1 complete: 44006478 blocks, 37991985 unique XXH3 hashes
100.0%Collision-Free Analysis: 6014493 duplicate blocks, 2938568 unique hashes cached
Pass 1.75: Loading ONLY duplicate block data into RAM...
100.0%Loaded 2938568 unique duplicate blocks into RAM
100.0%RAM usage: ~5.3% of original data (duplicates only)
Pass 2: Writing all blocks with XXH3 collision-free hints...
100.0%=== DEDUPLICATION RESULTS (XXH3-128) ===
Original Size: 29.65 GB (31837733330 bytes)
Compressed Size: 26.74 GB (28713904996 bytes)
Compression Ratio: 90.19% (1.11:1)
Space Saved: 2.91 GB (9.8%)
Total Blocks: 44006478
Unique Blocks: 37991985 (86.3%)
Duplicate Blocks: 6014493 (13.7%)
Cached Blocks: 2938568 (7.7% of unique)
Hash Algorithm: XXH3-128 (Collision-Free)
RAM breakdown: Tabs=1.31 GB, FreqMap=869.57 MB, Duplicates=1.58 GB, Total=3.74 GB (12.6% of input)
Collision-free deduplication complete!
100.0%
Errorlevel=0

Compressed 1 file, 31,837,733,330 => 28,713,904,996 bytes. Ratio 90.19%
Compression time: cpu 8.25 sec/real 1506.37 sec = 1%. Speed 21.14 mB/s
All OK

# arc t p:\ded.arc
FreeArc 0.67 (March 15 2014) testing archive: p:\ded.arc
Testing 1 file, 31,837,733,330 bytes. Processed 36.9%
ERROR: write error (disk full?) in compression algorithm dedup

#

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>


Win10 Enterprise 21H2
Ryzen 7 9700x
32 GB RAM

ATM no more free time to testing.
Thanks for the program!
(Maybe later...)

Interesting (another dataset):
Microsoft Windows [Version 10.0.19044.5965]
(c) Microsoft Corporation. All rights reserved.

arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 8.3%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.8%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup

[panker1992]

Originally Posted by shazzla

# arc a -i2 -mdedup p:\ded.arc e:\shdw.xt
FreeArc 0.67 (March 15 2014) creating archive: p:\ded.arc
Compressing 1 file, 31,837,733,330 bytes
Compressing shdw.xt 99.9%
Compressing 31,837,733,330 bytes with dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
100.0%=== DEDUPLICATION (XXH3-128) ===
Pass 1: Scanning for block positions (minimal RAM)
Processing 29.65 GB input file...
Progress: 100.00%
Starting collision-free frequency-based deduplication...
Pass 1 complete: 44006478 blocks, 37991985 unique XXH3 hashes
100.0%Collision-Free Analysis: 6014493 duplicate blocks, 2938568 unique hashes cached
Pass 1.75: Loading ONLY duplicate block data into RAM...
100.0%Loaded 2938568 unique duplicate blocks into RAM
100.0%RAM usage: ~5.3% of original data (duplicates only)
Pass 2: Writing all blocks with XXH3 collision-free hints...
100.0%=== DEDUPLICATION RESULTS (XXH3-128) ===
Original Size: 29.65 GB (31837733330 bytes)
Compressed Size: 26.74 GB (28713904996 bytes)
Compression Ratio: 90.19% (1.11:1)
Space Saved: 2.91 GB (9.8%)
Total Blocks: 44006478
Unique Blocks: 37991985 (86.3%)
Duplicate Blocks: 6014493 (13.7%)
Cached Blocks: 2938568 (7.7% of unique)
Hash Algorithm: XXH3-128 (Collision-Free)
RAM breakdown: Tabs=1.31 GB, FreqMap=869.57 MB, Duplicates=1.58 GB, Total=3.74 GB (12.6% of input)
Collision-free deduplication complete!
100.0%
Errorlevel=0

Compressed 1 file, 31,837,733,330 => 28,713,904,996 bytes. Ratio 90.19%
Compression time: cpu 8.25 sec/real 1506.37 sec = 1%. Speed 21.14 mB/s
All OK

# arc t p:\ded.arc
FreeArc 0.67 (March 15 2014) testing archive: p:\ded.arc
Testing 1 file, 31,837,733,330 bytes. Processed 36.9%
ERROR: write error (disk full?) in compression algorithm dedup

#

[External compressor:dedup]
header = 0
packcmd = dedup.exe c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = dedup.exe d - - <stdin> <stdout>


Win10 Enterprise 21H2
Ryzen 7 9700x
32 GB RAM

ATM no more free time to testing.
Thanks for the program!
(Maybe later...)

Interesting (another dataset):
Microsoft Windows [Version 10.0.19044.5965]
(c) Microsoft Corporation. All rights reserved.

arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 8.3%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.8%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup


arc t e:\ixi-dedup.arc
FreeArc 0.67 (March 15 2014) testing archive: e:\ixi-dedup.arc
Testing 178 files, 12,435,931,640 bytes. Processed 15.2%
ERROR: write error (disk full?) in compression algorithm dedup

Hello, i am trying to understand what you are doing ...it's working fine here

[Dragonis40]

Good morning! I'm trying to compress "Banishers Ghosts Of New Eden" videogame, but i'm having some troubles with .bk2 files. How can i compress thwm? Bpk and Xtool bk2 plugin don't compress at all. What can i compress them? Thank you for you reply!

[KaktoR]

They are kb2n, you cannot compress them.

[shazzla]

As you see,decompression doesnt work for me.
What can i do?!

[panker1992]

Try this one and tell me

[shazzla]

Its bit-identical with first version.

[wareck]

Originally Posted by shazzla

Its bit-identical with first version.

It works for me
Don't forget to update arc.ini
And you need to have dedup.exe both for compress and decompression.
you may also need to have admin rights to uncompress

[panker1992]

i think when i wrote this i had shared memory technology activated in one module :/ i will check and recompile one more version that doesnt use shared memory as this isnt needed here

[L0v3craft]

Someone checked "Tony Hawk's Pro Skater 3 + 4" ? The pak files are encrypted. I got the key and tried with "unrealengine_R13" and after zlib/reflate and kraken too. The precompression doesn't work. I guess we need of an updated plugin?

[Masquerade]

^^
Tony Hawk 1+2 used a custom algo.
You could always rebuild the paks like I did - but that's not lossless + CRC perfect

[wareck]

Originally Posted by L0v3craft

Someone checked "Tony Hawk's Pro Skater 3 + 4" ? The pak files are encrypted. I got the key and tried with "unrealengine_R13" and after zlib/reflate and kraken too. The precompression doesn't work. I guess we need of an updated plugin?

Hello
does someone progress on it ?

[Jarmen_Kell]

Any suggestion for "Le Mans Ultimate"?