MTX - The Universal Accelerator

[BLACKFIRE69]

MTX v0.3
Universal Multi-Threading Accelerator for FreeArc External Compressors
Windows x64 · single exe, no dependencies · by BLACKFIRE69

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

▌ OVERVIEW
Many of the strongest external compressors (precomp, mpz, rz, nz, srep …) are single-threaded: on a modern multi-core CPU they leave most of the machine idle.

MTX sits between FreeArc and the compressor, splits the archive data stream into fixed-size chunks, runs N parallel instances of the compressor on those chunks, and merges the results back into one sequential stream — in guaranteed original order. One config line turns a single-threaded compressor into an all-cores compressor.

▌ TECHNICAL SPECIFICATIONS

• Version — v0.3
• Author — BLACKFIRE69
• Platform — Windows x64, single exe (no dependencies)
• File Size — ~1.5 MB
• Compatibility — FreeArc v0.67 (and forks using arc.ini external compressors); also runs standalone from the command line or any tool that can pipe stdin/stdout

Wrapped-tool I/O modes (auto-detected from the method's command lines):

╭───────────────┬────────────────────────┬───────────────────────────────╮
│ Mode          │ Tool reads from        │ Tool writes to                │
├───────────────┼────────────────────────┼───────────────────────────────┤
│ Full piping   │ <stdin>                │ <stdout>                      │
│ stdin only    │ <stdin>                │ $$arcpackedfile$$ temp file   │
│ stdout only   │ $$arcdatafile$$ file   │ <stdout>                      │
│ Files only    │ $$arcdatafile$$ file   │ $$arcpackedfile$$ temp file   │
╰───────────────┴────────────────────────┴───────────────────────────────╯

▌ PERFORMANCE (v0.3 vs v0.2 vs v0.1)
256 MB of heterogeneous data compressed with zstd -17 as the wrapped single-threaded compressor. v0.3 was run with --progress=off --toolProgress=off; each version round-trips its OWN archive (SHA-256 verified identical).

╭──────────────────────┬─────────┬─────────┬─────────╮
│ Operation            │ v0.1    │ v0.2    │ v0.3    │
├──────────────────────┼─────────┼─────────┼─────────┤
│ encode -t10p  -c32m  │ 24.3 s  │ 24.1 s  │ 23.6 s  │
│ encode -t50p  -c32m  │ 15.7 s  │ 15.4 s  │ 14.4 s  │
│ encode -t100p -c32m  │ 12.7 s  │ 12.7 s  │ 11.1 s  │
│ encode -t100p -c8m   │ 11.9 s  │ 11.8 s  │ 10.6 s  │
│ encode -t100p -c4m   │ 13.3 s  │ 12.4 s  │ 11.0 s  │
│ decode -t100p -c32m  │ 1.1 s   │ 1.3 s   │ 0.7 s   │
╰──────────────────────┴─────────┴─────────┴─────────╯

v0.3 is faster in every configuration, and its lead grows with more threads and smaller chunks: v0.1's batch loop stalls each wave on its slowest chunk — it even slows down at -c4m — while v0.3's continuous reader/worker/writer pipeline never waits and stays flat at ~11 s. Decode is ~1.6× faster, and the margin widens further on machines with more cores.

▌ KEY FEATURES

✦ True Parallel Compression -t#
Continuous reader/workers/writer pipeline — chunks are read, compressed and written simultaneously by N parallel tool instances (-t#, default -t100p = one per logical core). The slowest chunk never stalls the rest.

✦ Guaranteed Stream Order
Output chunks are always written in the original input order, whatever order the workers finish in. FreeArc sees one opaque sequential stream.

✦ Memory Budgeting --mem=
--mem=8g sets a total RAM budget for chunks in flight (default: half of free RAM). --toolMem=2g declares how much RAM one tool instance needs — MTX then starts only as many instances as the machine can hold. A memory-hungry tool like rz can no longer freeze the whole system at -t100p.

✦ Safe Archive Format (MTX2)
Magic + version header, per-chunk frames carrying both sizes and a CRC32 checksum, explicit end marker with chunk count and total size. Damaged, truncated or foreign data is detected and reported — never silently unpacked into garbage.

✦ Independent Thread Counts
Every chunk frame is self-describing, so compression and decompression thread counts are fully independent. Pack with -t2 on a mid-range PC, unpack with -t16 on a high-end gaming PC.

✦ Live Status Display
A gray "#[#]" line above the MTX status shows the wrapped compressor's OWN live progress for the chunk currently gating the output; file-based tools get a per-chunk pseudo-console so they report progress even when piped. --progress=on|auto|off controls the whole display; --toolProgress=auto|on|off controls just the tool line. Errors always print regardless.

✦ Machine-Global Config (MTX.ini)
An optional MTX.ini next to the exe carries per-machine defaults (threads, chunk, mem, toolMem, okcodes, tempPath, basePath, cfgFile, progress, logs). Anything on the command line overrides it, so shared configs stay minimal and machine tuning lives next to the exe.

✦ Strict Error Contract
Tool failures, I/O errors and archive damage reach FreeArc as a nonzero exit code with a clear message: 0 = OK, 1 = runtime error, 2 = usage/config error, 3 = archive damaged. MTX never exits 0 after a failure, so FreeArc never finalizes a broken archive.

✦ Tolerant Tool Handling
--okcodes=0,2 accepts benign nonzero exit codes (precomp exits 2 when it finds nothing to precompress). Tool stderr is passed through, so the wrapped compressor's own messages stay visible.

✦ Collision-Free Temp Handling
Every parallel instance works in its own private temp directory — file-based tools and scratch-file writers (srep) cannot overwrite each other. --tempPath="X:\MyTemp" redirects all temp I/O to a chosen drive.

✦ Per-Method Defaults
In an MTX-only config file, a method section can carry its own defaults: okcodes, mtxthreads, mtxchunk, mtxmem. Command-line flags always win.

✦ Session Logging
--logs appends a timestamped session log (resolved options, per-chunk timings, child command lines) to MTX.log next to the exe.

▌ COMMAND REFERENCE

MTX64.exe <Command> <Options> [Settings] Input Output

Command:   a = encode (compress)   x = decode (extract)
Options:   -mXXX  method name from the config file
           -t#    parallel instances (-t100p = all cores, default)
           -c#    chunk size, e.g. -c64m (encode only; default 64 MB)
Settings:  --basePath=  --cfgFile=  --tempPath=  --mem=  --toolMem=
           --okcodes=  --progress=  --toolProgress=  --logs
Input/Output: file names, or "-" for stdin/stdout.

▌ QUICK START EXAMPLES

Example 1 — Wrap a method in FreeArc's arc.ini

; the plain single-threaded method
[External compressor:rz64]
header    = 0
packcmd   = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

; the MTX-accelerated method
[External compressor:rzmt]
header    = 0
default   = -t100p --progress=on --toolMem=2g --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini"
packcmd   = MTX\Win64\MTX64.exe a -mrz64 -c64m {options} - - <stdin> <stdout>
unpackcmd = MTX\Win64\MTX64.exe x -mrz64 {options} - - <stdin> <stdout>

arc.exe a -m=rzmt archive.arc files\*

Example 2 — Standalone, full piping

MTX64.exe a -mprecomp -c128m -t100p - - <stdin> <stdout>
MTX64.exe x -mprecomp -t100p - - <stdin> <stdout>

Example 3 — Memory-hungry tool, bounded RAM

MTX64.exe a -mrz -c64m -t100p --toolMem=2g --mem=8g - - <stdin> <stdout>
(starts only as many rz instances as 8 GB can hold at ~2 GB each)

Example 4 — Benign nonzero tool exit codes (precomp)

MTX64.exe a -mprecomp -c64m --okcodes=0,2 data.tmp packed.tmp

Example 5 — Per-method defaults in an MTX-only config file (MTX_Config.ini)

[External compressor:rz64]
header     = 0
packcmd    = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd  = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
okcodes    = 0
mtxthreads = 4
mtxchunk   = 32m
mtxmem     = 2g

MTX64.exe a -mrz64 --cfgFile="X:\cfg\MTX_Config.ini" - - <stdin> <stdout>

Keep these keys OUT of FreeArc's own arc.ini — FreeArc rejects unknown keys in its sections. In arc.ini, pass --okcodes= / --toolMem= / … inside the MTX method's "default" line instead, as in Example 1.

Example 6 — Maximum throughput (MTX status on, tool line off)

MTX64.exe a -mzstd -c8m -t100p --progress=on --toolProgress=off - - <stdin> <stdout>

Keeps MTX's own live status but drops the wrapped tool's progress line — and with it the per-chunk pseudo-console that file-based tools need. For a fast compressor split into many small chunks that cost is measurable, so --toolProgress=off runs noticeably faster.

▌ DISTRIBUTION FILES

• MTX64.exe — the tool (single exe, Win64)
• MTX.ini — optional machine-global defaults (place next to the exe)
• arc.ini — example FreeArc config with plain + MTX-wrapped methods (precomp, mpz, rz, nz, zstd, srep …)
• README.txt — full documentation
• WhatsNew.txt — version history

▌ NOTES

• v0.3 archives use the MTX2 format; archives created by v0.1 must be repacked with the v0.3 build before migrating.
• Compression and decompression machines never have to match in cores or RAM.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

Feedback, bug reports, and edge cases are all welcome.


.