FileForums - MTX - The Universal Accelerator

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - MTX - The Universal Accelerator (https://fileforums.com/showthread.php?t=103853)

MTX - The Universal Accelerator

https://i.ibb.co/k2VpTrJj/hh.png

MTX v0.3
Universal Multi-Threading Accelerator for FreeArc External Compressors
Windows x64 · single exe, no dependencies · by BLACKFIRE69

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

▌ OVERVIEW
Many of the strongest external compressors (precomp, mpz, rz, nz, srep …) are single-threaded: on a modern multi-core CPU they leave most of the machine idle.

MTX sits between FreeArc and the compressor, splits the archive data stream into fixed-size chunks, runs N parallel instances of the compressor on those chunks, and merges the results back into one sequential stream — in guaranteed original order. One config line turns a single-threaded compressor into an all-cores compressor.

▌ TECHNICAL SPECIFICATIONS

Version — v0.3
Author — BLACKFIRE69
Platform — Windows x64, single exe (no dependencies)
File Size — ~1.5 MB
Compatibility — FreeArc v0.67 (and forks using arc.ini external compressors); also runs standalone from the command line or any tool that can pipe stdin/stdout

Wrapped-tool I/O modes (auto-detected from the method's command lines):

Code:

╭───────────────┬────────────────────────┬───────────────────────────────╮

│ Mode          │ Tool reads from        │ Tool writes to                │

├───────────────┼────────────────────────┼───────────────────────────────┤

│ Full piping   │ <stdin>                │ <stdout>                      │

│ stdin only    │ <stdin>                │ $$arcpackedfile$$ temp file   │

│ stdout only   │ $$arcdatafile$$ file   │ <stdout>                      │

│ Files only    │ $$arcdatafile$$ file   │ $$arcpackedfile$$ temp file   │

╰───────────────┴────────────────────────┴───────────────────────────────╯

▌ PERFORMANCE (v0.3 vs v0.2 vs v0.1)
256 MB of heterogeneous data compressed with zstd -17 as the wrapped single-threaded compressor. v0.3 was run with --progress=off --toolProgress=off; each version round-trips its OWN archive (SHA-256 verified identical).

Code:

╭──────────────────────┬─────────┬─────────┬─────────╮

│ Operation            │ v0.1    │ v0.2    │ v0.3    │

├──────────────────────┼─────────┼─────────┼─────────┤

│ encode -t10p  -c32m  │ 24.3 s  │ 24.1 s  │ 23.6 s  │

│ encode -t50p  -c32m  │ 15.7 s  │ 15.4 s  │ 14.4 s  │

│ encode -t100p -c32m  │ 12.7 s  │ 12.7 s  │ 11.1 s  │

│ encode -t100p -c8m   │ 11.9 s  │ 11.8 s  │ 10.6 s  │

│ encode -t100p -c4m   │ 13.3 s  │ 12.4 s  │ 11.0 s  │

│ decode -t100p -c32m  │ 1.1 s   │ 1.3 s   │ 0.7 s   │

╰──────────────────────┴─────────┴─────────┴─────────╯

v0.3 is faster in every configuration, and its lead grows with more threads and smaller chunks: v0.1's batch loop stalls each wave on its slowest chunk — it even slows down at -c4m — while v0.3's continuous reader/worker/writer pipeline never waits and stays flat at ~11 s. Decode is ~1.6× faster, and the margin widens further on machines with more cores.

▌ KEY FEATURES

✦ True Parallel Compression -t#
Continuous reader/workers/writer pipeline — chunks are read, compressed and written simultaneously by N parallel tool instances (-t#, default -t100p = one per logical core). The slowest chunk never stalls the rest.

✦ Guaranteed Stream Order
Output chunks are always written in the original input order, whatever order the workers finish in. FreeArc sees one opaque sequential stream.

✦ Memory Budgeting --mem=
--mem=8g sets a total RAM budget for chunks in flight (default: half of free RAM). --toolMem=2g declares how much RAM one tool instance needs — MTX then starts only as many instances as the machine can hold. A memory-hungry tool like rz can no longer freeze the whole system at -t100p.

✦ Safe Archive Format (MTX2)
Magic + version header, per-chunk frames carrying both sizes and a CRC32 checksum, explicit end marker with chunk count and total size. Damaged, truncated or foreign data is detected and reported — never silently unpacked into garbage.

✦ Independent Thread Counts
Every chunk frame is self-describing, so compression and decompression thread counts are fully independent. Pack with -t2 on a mid-range PC, unpack with -t16 on a high-end gaming PC.

✦ Live Status Display
A gray "#[#]" line above the MTX status shows the wrapped compressor's OWN live progress for the chunk currently gating the output; file-based tools get a per-chunk pseudo-console so they report progress even when piped. --progress=on|auto|off controls the whole display; --toolProgress=auto|on|off controls just the tool line. Errors always print regardless.

✦ Machine-Global Config (MTX.ini)
An optional MTX.ini next to the exe carries per-machine defaults (threads, chunk, mem, toolMem, okcodes, tempPath, basePath, cfgFile, progress, logs). Anything on the command line overrides it, so shared configs stay minimal and machine tuning lives next to the exe.

✦ Strict Error Contract
Tool failures, I/O errors and archive damage reach FreeArc as a nonzero exit code with a clear message: 0 = OK, 1 = runtime error, 2 = usage/config error, 3 = archive damaged. MTX never exits 0 after a failure, so FreeArc never finalizes a broken archive.

✦ Tolerant Tool Handling
--okcodes=0,2 accepts benign nonzero exit codes (precomp exits 2 when it finds nothing to precompress). Tool stderr is passed through, so the wrapped compressor's own messages stay visible.

✦ Collision-Free Temp Handling
Every parallel instance works in its own private temp directory — file-based tools and scratch-file writers (srep) cannot overwrite each other. --tempPath="X:\MyTemp" redirects all temp I/O to a chosen drive.

✦ Per-Method Defaults
In an MTX-only config file, a method section can carry its own defaults: okcodes, mtxthreads, mtxchunk, mtxmem. Command-line flags always win.

✦ Session Logging
--logs appends a timestamped session log (resolved options, per-chunk timings, child command lines) to MTX.log next to the exe.

▌ COMMAND REFERENCE

Code:

MTX64.exe <Command> <Options> [Settings] Input Output



Command:   a = encode (compress)   x = decode (extract)

Options:   -mXXX  method name from the config file

           -t#    parallel instances (-t100p = all cores, default)

           -c#    chunk size, e.g. -c64m (encode only; default 64 MB)

Settings:  --basePath=  --cfgFile=  --tempPath=  --mem=  --toolMem=

           --okcodes=  --progress=  --toolProgress=  --logs

Input/Output: file names, or "-" for stdin/stdout.

▌ QUICK START EXAMPLES

Example 1 — Wrap a method in FreeArc's arc.ini

Code:

; the plain single-threaded method

[External compressor:rz64]

header    = 0

packcmd   = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

unpackcmd = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp



; the MTX-accelerated method

[External compressor:rzmt]

header    = 0

default   = -t100p --progress=on --toolMem=2g --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini"

packcmd   = MTX\Win64\MTX64.exe a -mrz64 -c64m {options} - - <stdin> <stdout>

unpackcmd = MTX\Win64\MTX64.exe x -mrz64 {options} - - <stdin> <stdout>



arc.exe a -m=rzmt archive.arc files\*

Example 2 — Standalone, full piping

Code:

MTX64.exe a -mprecomp -c128m -t100p - - <stdin> <stdout>

MTX64.exe x -mprecomp -t100p - - <stdin> <stdout>

Example 3 — Memory-hungry tool, bounded RAM

Code:

MTX64.exe a -mrz -c64m -t100p --toolMem=2g --mem=8g - - <stdin> <stdout>

(starts only as many rz instances as 8 GB can hold at ~2 GB each)

Example 4 — Benign nonzero tool exit codes (precomp)

Code:

MTX64.exe a -mprecomp -c64m --okcodes=0,2 data.tmp packed.tmp

Example 5 — Per-method defaults in an MTX-only config file (MTX_Config.ini)

Code:

[External compressor:rz64]

header     = 0

packcmd    = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

unpackcmd  = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

okcodes    = 0

mtxthreads = 4

mtxchunk   = 32m

mtxmem     = 2g



MTX64.exe a -mrz64 --cfgFile="X:\cfg\MTX_Config.ini" - - <stdin> <stdout>

Keep these keys OUT of FreeArc's own arc.ini — FreeArc rejects unknown keys in its sections. In arc.ini, pass --okcodes= / --toolMem= / … inside the MTX method's "default" line instead, as in Example 1.

Example 6 — Maximum throughput (MTX status on, tool line off)

Code:

MTX64.exe a -mzstd -c8m -t100p --progress=on --toolProgress=off - - <stdin> <stdout>

Keeps MTX's own live status but drops the wrapped tool's progress line — and with it the per-chunk pseudo-console that file-based tools need. For a fast compressor split into many small chunks that cost is measurable, so --toolProgress=off runs noticeably faster.

▌ DISTRIBUTION FILES

MTX64.exe — the tool (single exe, Win64)
MTX.ini — optional machine-global defaults (place next to the exe)
arc.ini — example FreeArc config with plain + MTX-wrapped methods (precomp, mpz, rz, nz, zstd, srep …)
README.txt — full documentation
WhatsNew.txt — version history

▌ NOTES

v0.3 archives use the MTX2 format; archives created by v0.1 must be repacked with the v0.3 build before migrating.
Compression and decompression machines never have to match in cores or RAM.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

Feedback, bug reports, and edge cases are all welcome.

.

If anyone wants an example of "Inno setup" for MTX.exe, just let me know. ;)

MTX console version works perfectly. Also I was tested MCM works perfectly.

MTX.exe supports MASKS (Arc.Groups)

Quote:

example:

[Compression Methods]
MASK = srep+razorx/$zlib=precompx+srep+bcmx/$other=srep+razorx/$lyrfile=precompx+srep+bcmx/$bmp=srep+razorx

Quote:

Originally Posted by BLACKFIRE69 (Post 488133)

MTX.exe supports MASKS (Arc.Groups)

Can you explain about masks?

Quote:

Originally Posted by Balaji007 (Post 488138)

Can you explain about masks?

Masks allow you to use more than one method for different files in an archive

Different methods are specified with groups, which you can write yourself into the file "arc.groups" next to your arc.exe for compression.

Write your group as example:

Code:

$nameofgroup

*.filename

*.otherfile

Then we can use this in our compression method:

Code:

srep+lolz/$nameofgroup=rzx

So here, all of the files defined in the $nameofgroup list will be compressed using rzx, where all other files will be compressed with srep+lolz.

Quote:

Originally Posted by Masquerade (Post 488139)

Code:

$nameofgroup

*.filename

*.otherfile

Then we can use this in our compression method:

Code:

srep+lolz/$nameofgroup=rzx

So here, all of the files defined in the $nameofgroup list will be compressed using rzx, where all other files will be compressed with srep+lolz.

Ok thanks for your information friend

MTX.exe Feature update

MTX.exe Feature update v.3.0.0.0

What's New:

Some improvements.
A new key (MainPath) has been added to MTX.ini.
-- this allows you to set the root directory manually.
-- MTX.exe can be executed within a sub-folder.
There is no need to rename "mtx.ini" (as before).
-- before: MTX64.exe ---> MTX64.ini --> [MTX64]
-- now : MTX64.exe ---> MTX.ini --> [MTX]
Added an example for XLolz.
Added an example for MASKS.

Additional Options For both Compression and the Decompression:

Code:

-----------------------------------------------------------------------------

[MTX]

MainPath=..\..\

Config=arc.ini

//TmpPath=C:\Users\Rexton\Desktop\temptest\

-----------------------------------------------------------------------------







// config := default is arc.ini (current directory).



// config = arc.ini                        <--- MainPath\arc.ini

// config = cfg\config0.ini                <--- MainPath\cfg\config0.ini  







// TmpPath= full path.  ( with or without quotes (") )

Oh, thank you!
Can I have an example or small guide to adapt Inno/ASIS to this please? I'm probably doing something wrong with my frankenstein script lol

Edit/question: Is Xlolz working multithreading via MTX? It doesn't seem for me.
Edit2: It works with fixed thread count and not % like default (?)

Quote:

Originally Posted by Nezha (Post 488197)

You can't use "XLolz" via "MTX" (because both are accelerators). but it supports "MTX + Lolz".

ok, I'll give you some examples for both INNO Setup and Lolz with MTX.

Quote:

Originally Posted by BLACKFIRE69 (Post 488199)

You can't use "XLolz" via "MTX" (because both are accelerators). but it supports "MTX + Lolz".

ok, I'll give you some examples for both INNO Setup and Lolz with MTX.

Oh, yea, my bad: I mean the one inside the default "arc.ini" with MTX, for reference this one:

Quote:

[External compressor:XLolz, LolzMTX, xlolz]
header = 0
packcmd = bin\MTX\MTX.exe a:lolz -c:64m -t:100p - - <stdin> <stdout>
unpackcmd = bin\MTX\MTX.exe x:lolz -t:100p - - <stdin> <stdout>

___

So.. Something works but not well

Using unpack.bat

Quote:

Extracted 6,874 files, 10,021,997,180 => 16,726,226,492 bytes. Ratio 59.92%
Extraction time: cpu 30.00 sec/real 469.39 sec = 6%. Speed 35.63 mB/s
All OK

Using my modified InnoSetup scripts with MTX gives "UnArc.dll -11 error: archive damaged" at ~80% but it actually extracts something

Quote:

Originally Posted by Nezha (Post 488202)

Oh, yea, my bad: I mean the one inside the default "arc.ini" with MTX, for reference this one:

___

So.. Something works but not well

Using unpack.bat

Using my modified InnoSetup scripts with MTX gives "UnArc.dll -11 error: archive damaged" at ~80% but it actually extracts something

ok, just wait for the example.

MTX v.5.0.0.0 - October Update

MTX.ini

Code:

Example:=



exefile:  MTX64.exe

ini:      supports both MTX.ini or MTX64.ini    (next to the exe file)

section:  [MTX] or [MTX64]



exefile:  MTX32.exe

ini:      supports both MTX.ini or MTX32.ini    (next to the exe file)

section:  [MTX] or [MTX32]

Percentage

Code:

Example:=



packcmd   = "MTX.exe" a:bcm -c:64m -t:100p - - <stdin> <stdout>



   or



packcmd   = "MTX.exe" a:bcm -c:64m -t:100% - - <stdin> <stdout>

Supports Custom Input/Output

Code:

Example:=



[External compressor:bcm, bcm32]

header    = 0 

default   = -9



packcmd   = "Bcm\{compressor}" {options} MyInput.in MyOutput.out

unpackcmd = "Bcm\{compressor}" -d MyOutput.out MyInput.in

datafile   = MyInput.in

packedfile = MyOutput.out

New Options

Code:



1. --help   (Show long help)

2. --logs   (Generate a log file)

3. --mainpath=< main path >

4. --config=< config file >

5. --tmppath=< temp path >





Example:=  (When not using MTX.ini.)





[External compressor:BscX, BscMTX, bscx]

header = 0

default   = --logs --mainpath=".\..\" --config="%MainPath%\confis\arc.ini" --tmppath="%ExePath%\TEMP"

packcmd   = MTX\MTX.exe a:bsc32 -c:64m -t:100% {options} - - <stdin> <stdout>

unpackcmd = MTX\MTX.exe x:bsc32 -t:100p  {options} - - <stdin> <stdout>

MTX.ini Configuration

Code:

 MTX Variables:

    1. %MainPath%

         -- default is the directory that containing MTX.exe.

            ex: C:\Testing\mtxTest\Res\MTX\MTX.exe  then %MainPath% = C:\Testing\mtxTest\Res\MTX\



         -- Users can change manually.



    2. %ExePath%

         -- the directory that containing MTX.exe.

            ex: C:\Testing\mtxTest\Res\MTX\MTX.exe  then %ExePath% = C:\Testing\mtxTest\Res\MTX\





 Other

    1. MainPath

        -- default is         %ExePath%



        -- Users can change manually.



        -- MainPath = (full path)

            ex:        MainPath=C:\Testing\mtxTest\Res\



        -- MainPath = (short path)

            ex:        MainPath=.\..\

              > MainPath is centered on ExePath. (for short paths)

              > If ExePath = C:\Testing\mtxTest\Res\MTX\  then  MainPath = C:\Testing\mtxTest\Res\  (MainPath=.\..\)



        -- MainPath = (alternatively)

            ex:         MainPath=%ExePath%





    2. Config

        -- default is        %ExePath%\Arc.ini



        -- Users can change manually.



        -- Config = (full path)

            ex:        Config=C:\Testing\mtxTest\Res\Arc.ini



        -- Config = (short path)

            ex:        Config=.\..\Arc.ini

               > Config is centered on MainPath. (for short paths)

               > If MainPath = C:\Testing\mtxTest\Res\MTX\  then  Config = C:\Testing\mtxTest\Res\Arc.ini  (Config=.\..\Arc.ini)



        -- Config = (alternatively)

            ex:        Config=%MainPath%\Arc.ini

            ex:        Config=%ExePath%\Arc.ini





    3. TmpPath

        -- default is        %ExePath%



        -- Users can change manually.



        -- TmpPath = (full path)

            ex:        TmpPath=C:\Testing\mtxTest\Res\TEMP\



        -- TmpPath = (short path)

            ex:        TmpPath=.\..\TEMP\

               > TmpPath is centered on MainPath. (for short paths)

               > If MainPath = C:\Testing\mtxTest\Res\MTX\  then  Config = C:\Testing\mtxTest\Res\TEMP\  (TmpPath=.\..\TEMP\)



        -- TmpPath = (alternatively)

            ex:        TmpPath=%MainPath%\TEMP\

            ex:        TmpPath=%ExePath%\TEMP\

Will show some errors in nanozip decompression with MTX.exe. therefore it is better to use cls-nzx.dll instead of MTX.exe. ;)

guys, I don't have enough space to upload it here. so use this link....

MTX v5.0.0.0 + Full Examples.rar

_

@BLACKFIRE69, very good the new key default options in example 5.
It will be very useful for the need to use MRX.ini.
Thanks for the great job.

If you have both configurations, which one will be used?
I mean if you have MTX.ini configured in one way and use the key default with --mainpath, etc. in another way, which one will be used?

Quote:

Originally Posted by Cesar82 (Post 488276)

first configured file (MTX.ini) then checks the default keys (--mainpath , ...).

means,

If MTX.ini is not configured, it uses the default keys.