ADDeX — Advanced Data Deduplication Engine
 

ADDeX — Advanced Data Deduplication Engine

I’m working on a new data deduplication and compression tool called ADDeX (Advanced Data Deduplication Engine).

The goal is to experiment with different levels of duplicate-data detection and compression, with a focus on achieving better compression ratios while allowing the user to control the trade-off between speed, memory usage, and compression efficiency.
Current compression modes

M1 — Standard: General-purpose compression with reasonable speed and memory usage.
M2 — Improved: More aggressive matching for better compression ratios.
M3 — Exhaustive: Performs much more intensive matching for maximum compression potential. This mode is significantly more RAM-intensive and can be considerably slower.

Available options

ADDeX.exe [OPTIONS] <command> <input> <output>

Commands:
-m1 <input> <output> Standard compression mode
-m2 <input> <output> Improved compression mode
-m3 <input> <output> Exhaustive compression mode
-d <input> <output> Decompress a file
-d - - Decompress stdin to stdout
-h Display help

Compression options:
-l<N> Minimum match length
Default: 512 | Min: 64 | Max: 1024

-a<N> Alignment
Default: 0 | Min: 0 | Max: 2

-b<N> Block size in MB
Default: 8 MB

Examples

ADDeX.exe -m1 input.dat output.addex

ADDeX.exe -m2 -l64 -a1 data.bin output.addex

ADDeX.exe -m3 -l512 -b16 data.bin output.addex

ADDeX.exe -d compressed.addex original.bin

type compressed.addex | ADDeX.exe -d - - > output.bin

- can be used as a filename to read from stdin or write to stdout, making ADDeX suitable for command-line pipelines.

Version: 0.1 Alpha
Author: Panker1992

This is an early alpha release, so the format, algorithms, command-line options, and compression behavior are still subject to change. Feedback, testing results, and suggestions are welcome.

For a few days im away from my computer.
So....
Whats the de/compression speed compared to srep? :)

i have no clue :) i never compared to srep, encoding speed should be 15% faster and decode hould be on par

you guys can run this and tell me :)

is it possible to provide a 32 bit cls for freearc. and is the cls working without ADDeX.exe.

32 bit DLL has been included, and yes the DLL can extract alone, but it cant encode...

is has only the decompression code inside.

I will run benchmarks today for people to compare

CLS filters, not works... Max. 16-32MB input data sizes still probably workingly in good scenarios/cases, upto 100MB+ data compressed it, not working.

And different cases, reactions from Win7 x64 SP1:

64bit CLS: Thread blocked indefinitely.
32bit CLS: Few percentage or first ~8MB's or rare cases in below 16MB data extracted, largers not, crashing now after 8MB data readed/writed in.

Only workingly use aDDeX.exe file from testing/decompress. /properly configured comp/decomp commands from arc.ini file./

Tested methods: xZLib+addex:m3+lzma or addex:m3+lzma

Note: xZLib = Old XTool 0.12 32bit versions.

I am working on my own CLS filter written in delphi, this CLS was kindly provided by a friend :)

Will be fixed within the day, you can use the addex to decompress your file, and if you have files that failed in srep you can try to see, rumors said that it worked

Some benchmarks, tests performed on ramdisk.

input: Bright Memory Infinite (zlib precompressed), 20,6 GB (22 175 074 282 bytes)

Srep
Addex
procprofile srep

procprofile addex

I retested again... Now it's good. :confused:

What's certain: The 64bit CLS filter is completely inoperable.

During the afternoon test, during packaging, the 64bit version of XTOOL v0.12 was used (which I wrote as "xZLib"), including the hif/raw*.dll and zlibwapi.dll files. During unpacking, the 32bit version. (From the Xeon PC.)

Furthermore, the unpacking/testing via the "aDDeX.exe" file runs without error, with the 32bit CLS file, although not completely, similarly to the previous case. If there is no xZLib at the beginning of the packaging chain, then there is no error, strange mode. In one of the two smaller images you can see this (test from laptop) + the 64bit CLS error, in the larger image, the other test case.

https://i.kek.sh/dnK6Jkqeh6H.png

https://i.kek.sh/5tMkORTCRRD.png

https://i.kek.sh/hVuCzhmBgAj.png

For those who are experimenting, here is the small prepared package, in the "DATA" folder in the "pack" folder, copy at least 100-500MB of data or even more. Copy/move the created "data.arc" file to the other 3 folders, running the tests one after the other.

For those who don't have enough, you can also copy additional pre- and other compressors to the folders, expanding and editing the "arc.ini" files accordingly, for experimentation. Or add them to the existing test/compressor sets and test them. (eg: Tiny Compressor, Diskspan GUI, etc...)

Plus info: The 32bit CLS-filter, running from XP! (Goood!)

i have added 2 cls filters in native Delphi, exactly in the language the ADDEX is written in :)