FileForums - MTX - The Universal Accelerator

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - MTX - The Universal Accelerator (https://fileforums.com/showthread.php?t=103853)

@Blackfire69

I've hbeen having issues with both MTX.exe and RazorX trying to compress a few files, leading to an FreeArc error at 99.1% in both tools (bad file descriptor error).

I use srep before running. Will this cause issues during decompression? I've never encountered that before. I'm currently testing that myself. Arc says decompression error in razorx/mtx though, and not srep

Quote:

Originally Posted by Masquerade (Post 490695)

ok, more info please.

how many threads did you use? (for the compression and the decompression)
did you run this test on the same cpu?

in general, MTX returns errors because the number of threads doesn't match.
I mean for example, MTX can't control this situation if you use 8 threads for compression and 16 threads for decompression. this's because the number of threads for decompression should always be less than or equal to the number of threads you used in compression.

MSVCP140.dll error

how do I fix this? (MSVCP140.dll)

solution: Install VC++ 2015-2017-2019 package (both x86 and x64)

latest Visual C++ downloads

I'd argue this is a better solution :D
https://github.com/abbodi1406/vcredist/releases

Anyways, I solved my issues, it appears the files being extracted were simply too small (1-2kb) and something just went wrong.

I played around a bit and managed to reach a correct extraction:

Code:

FreeArc 0.67 (March 15 2014) extracting archive: MASQUERADE-Data_01.MSQ

Extracting 41 files, 5,071,054,255 bytes. Processed   2.6%

Extracted 41 files, 1,955,615,101 => 5,071,054,255 bytes. Ratio 38.56%

Extraction time: cpu 28.59 sec/real 158.83 sec = 18%. Speed 31.93 mB/s

All OK

MTX v5.0.0.2 New Update

What's new:

Code:

1. Fixed config file parsing bug. (Thanks to @Cesar82)

 

2. Added support of MTX.ini.

    -- When you use MTX.ini next to the EXE file, 

       the settings "--basePath", "--cfgfile" and "--tmpPath" are overridden. 

       This simply means that if you use MTX.ini, you no longer need the  "--basePath",

       "--cfgfile" and "--tmpPath" setting in the commandline.

    

    -- If you use both MTX.ini and the commandline settings, the MTX will ignore

       the commandline settings ("--basePath", "--cfgfile" and "--tmpPath").

 

3. Compatible with the previous version (v5.0.0.1).

MTX.ini

Code:

[MTX64]

MainPath=".\"

Config="$basePath$\arc.ini"

TmpPath="$basePath$\temp64"

Code:

Section name

        Filename>> MTX32.exe      then [MTX32]

        Filename>> MTX64.exe      then [MTX64]

        Filename>> MTX.exe        then [MTX]

        Filename>> FileName.exe   then [FileName]



MainPath

        MainPath="$exePath$"  or

        MainPath=".\..\..\"   or

        MainPath="C:\test"



Config

        Config="$basePath$\arc.ini"  or

        Config="$exePath$\arc.ini"   or

        Config=".\..\..\arc.ini"     or

        Config="C:\test\arc.ini"



TmpPath

        TmpPath="$basePath$\temp"  or

        TmpPath="$exePath$\temp"   or

        TmpPath=".\..\..\temp"     or

        TmpPath="C:\test\temp"

MTX v5.0.0.2 Improved version

What's new:

Fixed a few minor bugs.

Improved exception handling.

Returns an error code for each situation.

* Check the first post for more info & downloads.

.

@BLACKFIRE69

MTX crashes on when decompressing on systems with strange numbers of cores/threads (e.g. 6c/6t and 6c/12t)

Are you able to fix this? (issues in both EXE and CLS versions)

I can only decompress with mtx if t100p is set in packcmd in arc.ini.

If I set t75p (or any other number other than 100) freearc can't decompress the archive.

Is this a bug or am I missing something?

Edit: encode and decode have to be equal. I missed it. If you used t100p in encode you have to use t100p in decode too. Sorry :D

BLACKFIRE69, you could remove the ":" from the parameters in your MTX.
Example, how can I send 4 threads (-t4) as a parameter from pack.bat. It is not allowed to send "t:4" because the ":" are already FreeArc parameter separators.

Quote:

Originally Posted by Cesar82 (Post 496366)

sure,

Quote:

[External compressor:precompx,mtx_precomp]
header = 0
default = -t100p --logs --basePath=".\..\.." --cfgfile="$basePath$\arc.ini" --tmpPath="$exePath$\temp"

packcmd = MTX\Win64\MTX.exe a -mprecomp048 -c256m {option} - - <stdin> <stdout>
unpackcmd = MTX\Win64\MTX.exe x -mprecomp048 {option} - - <stdin> <stdout>

Quote:

Originally Posted by KaktoR (Post 496364)

an equal or lesser value is also valid.
ex:
encode: 16 (Threads)
decode: 16, 8, 6, 4, 2, ...

if any user has entered an invalid value for the decoding threads, i'll improve MTX so that it can be corrected by MTX itself in a future update.
(since i'm a bit busy, give me some time for that ;))

in the meantime you can test out the beta version of new RazorX. i've used 50p for encoding and 100p for decoding. although this is usually invalid, it's automatically corrected by RazorX.

Code:

[External compressor:rzmt]

header = 0

packcmd   = RazorX.exe a -c:64m -t:50p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = RazorX.exe x -t:100p - - <stdin> <stdout>

(read Important.txt before use)
.

Quote:

Originally Posted by BLACKFIRE69 (Post 496388)

Code:

[External compressor:rzmt]

header = 0

packcmd   = RazorX.exe a -c:64m -t:50p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = RazorX.exe x -t:100p - - <stdin> <stdout>

(read Important.txt before use)
.

For you to better understand what happened.
Me and KaktoR were doing tests with RAZOR MTX.

The configuration used was:

Code:

packcmd   = MTX.exe a -m:rz {option} -c:64m -t:100p - - <stdin> <stdout>

unpackcmd = MTX.exe x -m:rz {option} -t:100p - - <stdin> <stdout>

As his processor is 6/12 and my processor is 12/24
If it compressed and sent me the file I couldn't extract it because it would have been compressed using 12 threads and I would extract using 24 threads.
So if you make a game backup and upgrade the CPU to one with more threads, there will be an extraction error.
I know I could set it to t2 for extraction, but that loses the meaning of being multi threaded (use only 2 of 24).

The workaround is to set it to 100p use send thread number (get from system) as method parameter and when extract use {option} to set -t parameter.

Thanks for answering!

Quote:

Originally Posted by Cesar82 (Post 496390)

that won't happen in the next update of MTX. checkout the RazorX beta above,
the following config works without any problems.

Code:

[External compressor:rzmt]

header = 0

packcmd   = RazorX.exe a -c:64m -t:12 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = RazorX.exe x -t:24 - - <stdin> <stdout>

OR

KaktoR can use it to encode as,

Code:

packcmd = RazorX.exe a -c:64m -t:100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

also, cesar you can use the same to decode, -t:100p

Code:

unpackcmd = RazorX.exe x -t:100p - - <stdin> <stdout>

MTX v5.0.0.3 Beta - Update

Update!

Code:

USAGE:



[External compressor:zstdmt,zstdx,mtx_zstd]

header = 0

default   = -t100p --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini" 



packcmd   = MTX\Win64\MTX64.exe a -mzstd64 -c128m {option} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = MTX\Win64\MTX64.exe x -mzstd64 {option} $$arcpackedfile$$.tmp $$arcdatafile$$.tmp





[External compressor:zstdmt,zstdx,mtx_zstd]

header = 0

default   = --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini" 



packcmd   = MTX\Win64\MTX64.exe a -mzstd64 -c128m -t8 {option} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = MTX\Win64\MTX64.exe x -mzstd64 -t8 {option} $$arcpackedfile$$.tmp $$arcdatafile$$.tmp





[External compressor:zstdmt,zstdx,mtx_zstd]

header = 0

default   = --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini" 



packcmd   = MTX\Win64\MTX64.exe a -mzstd64 -c128m -t4 {option} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = MTX\Win64\MTX64.exe x -mzstd64 -t8 {option} $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

MTX v5.0.0.3 beta Update 1

Code:

-- Fixed a minor issue.

Thanks @Cesar82 for pointing this out.

MTX v5.0.0.3_beta Update 01 _ 64-Bit.rar

MTX - The Universal Accelerator - 2023

Code:

1. Created MTX2023 from scratch and optimized for speed and efficiency.

  - As a result, it isn't compatible with any previous versions.



2. The bugs experienced in previous versions have been addressed and fixed.



3. MTX can handle scenarios where the #userThreads > #compressionThreads.



4. Simultaneous reading and writing of data takes place.

  - Previous versions required extra waiting time that's no longer necessary in this version.

  - Make sure to specify the '-ds' parameter with the 'arc.exe a'. (arc.exe a -ds ...)

Code:

Remark:



1. '--TempPath=' parameter has been removed.

  - Tmp files will now be created in the freearc temp dir instead.



2. '%' sign has been removed from threads (100%).

3. Support for "MTX32.ini" and "MTX64.ini" files has been discontinued.

4. There won't be a 32-bit version of MTX.

Code:

1. Recommended setting (stdio mode) but no info will be displayed.



packcmd   = MTX64.exe a -mNz64 -c32m {options} - - <stdin> <stdout>

unpackcmd = MTX64.exe x -mNz64 {options} - - <stdin> <stdout>





2. Info will only be displayed for packing.



packcmd   = MTX64.exe a -mNz64 -c32m {options} - $$arcpackedfile$$.tmp <stdin>

unpackcmd = MTX64.exe x -mNz64 {options} - - <stdin> <stdout>





3. Info will only be displayed for unpacking.



packcmd   = MTX64.exe a -mNz64 -c32m {options} - - <stdin> <stdout>

unpackcmd = MTX64.exe x -mNz64 {options} - $$arcdatafile$$.tmp <stdin>





4. Info will be displayed for both packing and unpacking. 



packcmd   = MTX64.exe a -mNz64 -c32m {options} - $$arcpackedfile$$.tmp <stdin>

unpackcmd = MTX64.exe x -mNz64 {options} - $$arcdatafile$$.tmp <stdin>

Tested MTX 2023 with razor. It is a disaster, freezes my computer. Never happened with the accelerator razorx.

Looks like it has created 190k files in the temp folder of freearc, when I have compressed one single file of 2.77GB.

Quote:

Originally Posted by BLACKFIRE69 (Post 497654)

MTX v5.0.0.3 beta Update 1

Code:

-- Fixed a minor issue.

Thanks @Cesar82 for pointing this out.

MTX v5.0.0.3_beta Update 01 _ 64-Bit.rar

Could you get me the 32-bit release on UPX of this version 5.0.0.3?

Quote:

Originally Posted by Cesar82 (Post 502552)

Could you get me the 32-bit release on UPX of this version 5.0.0.3?

The MTX project has been rebooted, and it is now starting from version 0.1. The new version of MTX is compatible with both cls-diskspan.dll and DiskSpan-GUI.

The first post has been updated.

.

Quote:

Originally Posted by BLACKFIRE69 (Post 502560)

1) The --tmpPath= parameter and the $tmpPath$ variable no longer exist?
2) There is no longer support for external configurations in MTX.ini?

Quote:

Originally Posted by Cesar82 (Post 502561)

1) The --tmpPath= parameter and the $tmpPath$ variable no longer exist?
2) There is no longer support for external configurations in MTX.ini?

no.

Does it still require the "-ds" parameter in FreeArc?

can you repost a link for the new versions of MTX ?

MTX v0.3 - Update

https://i.ibb.co/k2VpTrJj/hh.png

MTX v0.3
Universal Multi-Threading Accelerator for FreeArc External Compressors
Windows x64 · single exe, no dependencies · by BLACKFIRE69

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

▌ OVERVIEW
Many of the strongest external compressors (precomp, mpz, rz, nz, srep …) are single-threaded: on a modern multi-core CPU they leave most of the machine idle.

MTX sits between FreeArc and the compressor, splits the archive data stream into fixed-size chunks, runs N parallel instances of the compressor on those chunks, and merges the results back into one sequential stream — in guaranteed original order. One config line turns a single-threaded compressor into an all-cores compressor.

▌ TECHNICAL SPECIFICATIONS

Version — v0.3
Author — BLACKFIRE69
Platform — Windows x64, single exe (no dependencies)
File Size — ~1.5 MB
Compatibility — FreeArc v0.67 (and forks using arc.ini external compressors); also runs standalone from the command line or any tool that can pipe stdin/stdout

Wrapped-tool I/O modes (auto-detected from the method's command lines):

Code:

╭───────────────┬────────────────────────┬───────────────────────────────╮

│ Mode          │ Tool reads from        │ Tool writes to                │

├───────────────┼────────────────────────┼───────────────────────────────┤

│ Full piping   │ <stdin>                │ <stdout>                      │

│ stdin only    │ <stdin>                │ $$arcpackedfile$$ temp file   │

│ stdout only   │ $$arcdatafile$$ file   │ <stdout>                      │

│ Files only    │ $$arcdatafile$$ file   │ $$arcpackedfile$$ temp file   │

╰───────────────┴────────────────────────┴───────────────────────────────╯

▌ PERFORMANCE (v0.3 vs v0.2 vs v0.1)
256 MB of heterogeneous data compressed with zstd -17 as the wrapped single-threaded compressor. v0.3 was run with --progress=off --toolProgress=off; each version round-trips its OWN archive (SHA-256 verified identical).

Code:

╭──────────────────────┬─────────┬─────────┬─────────╮

│ Operation            │ v0.1    │ v0.2    │ v0.3    │

├──────────────────────┼─────────┼─────────┼─────────┤

│ encode -t10p  -c32m  │ 24.3 s  │ 24.1 s  │ 23.6 s  │

│ encode -t50p  -c32m  │ 15.7 s  │ 15.4 s  │ 14.4 s  │

│ encode -t100p -c32m  │ 12.7 s  │ 12.7 s  │ 11.1 s  │

│ encode -t100p -c8m   │ 11.9 s  │ 11.8 s  │ 10.6 s  │

│ encode -t100p -c4m   │ 13.3 s  │ 12.4 s  │ 11.0 s  │

│ decode -t100p -c32m  │ 1.1 s   │ 1.3 s   │ 0.7 s   │

╰──────────────────────┴─────────┴─────────┴─────────╯

v0.3 is faster in every configuration, and its lead grows with more threads and smaller chunks: v0.1's batch loop stalls each wave on its slowest chunk — it even slows down at -c4m — while v0.3's continuous reader/worker/writer pipeline never waits and stays flat at ~11 s. Decode is ~1.6× faster, and the margin widens further on machines with more cores.

▌ KEY FEATURES

✦ True Parallel Compression -t#
Continuous reader/workers/writer pipeline — chunks are read, compressed and written simultaneously by N parallel tool instances (-t#, default -t100p = one per logical core). The slowest chunk never stalls the rest.

✦ Guaranteed Stream Order
Output chunks are always written in the original input order, whatever order the workers finish in. FreeArc sees one opaque sequential stream.

✦ Memory Budgeting --mem=
--mem=8g sets a total RAM budget for chunks in flight (default: half of free RAM). --toolMem=2g declares how much RAM one tool instance needs — MTX then starts only as many instances as the machine can hold. A memory-hungry tool like rz can no longer freeze the whole system at -t100p.

✦ Safe Archive Format (MTX2)
Magic + version header, per-chunk frames carrying both sizes and a CRC32 checksum, explicit end marker with chunk count and total size. Damaged, truncated or foreign data is detected and reported — never silently unpacked into garbage.

✦ Independent Thread Counts
Every chunk frame is self-describing, so compression and decompression thread counts are fully independent. Pack with -t2 on a mid-range PC, unpack with -t16 on a high-end gaming PC.

✦ Live Status Display
A gray "#[#]" line above the MTX status shows the wrapped compressor's OWN live progress for the chunk currently gating the output; file-based tools get a per-chunk pseudo-console so they report progress even when piped. --progress=on|auto|off controls the whole display; --toolProgress=auto|on|off controls just the tool line. Errors always print regardless.

✦ Machine-Global Config (MTX.ini)
An optional MTX.ini next to the exe carries per-machine defaults (threads, chunk, mem, toolMem, okcodes, tempPath, basePath, cfgFile, progress, logs). Anything on the command line overrides it, so shared configs stay minimal and machine tuning lives next to the exe.

✦ Strict Error Contract
Tool failures, I/O errors and archive damage reach FreeArc as a nonzero exit code with a clear message: 0 = OK, 1 = runtime error, 2 = usage/config error, 3 = archive damaged. MTX never exits 0 after a failure, so FreeArc never finalizes a broken archive.

✦ Tolerant Tool Handling
--okcodes=0,2 accepts benign nonzero exit codes (precomp exits 2 when it finds nothing to precompress). Tool stderr is passed through, so the wrapped compressor's own messages stay visible.

✦ Collision-Free Temp Handling
Every parallel instance works in its own private temp directory — file-based tools and scratch-file writers (srep) cannot overwrite each other. --tempPath="X:\MyTemp" redirects all temp I/O to a chosen drive.

✦ Per-Method Defaults
In an MTX-only config file, a method section can carry its own defaults: okcodes, mtxthreads, mtxchunk, mtxmem. Command-line flags always win.

✦ Session Logging
--logs appends a timestamped session log (resolved options, per-chunk timings, child command lines) to MTX.log next to the exe.

▌ COMMAND REFERENCE

Code:

MTX64.exe <Command> <Options> [Settings] Input Output



Command:   a = encode (compress)   x = decode (extract)

Options:   -mXXX  method name from the config file

           -t#    parallel instances (-t100p = all cores, default)

           -c#    chunk size, e.g. -c64m (encode only; default 64 MB)

Settings:  --basePath=  --cfgFile=  --tempPath=  --mem=  --toolMem=

           --okcodes=  --progress=  --toolProgress=  --logs

Input/Output: file names, or "-" for stdin/stdout.

▌ QUICK START EXAMPLES

Example 1 — Wrap a method in FreeArc's arc.ini

Code:

; the plain single-threaded method

[External compressor:rz64]

header    = 0

packcmd   = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

unpackcmd = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp



; the MTX-accelerated method

[External compressor:rzmt]

header    = 0

default   = -t100p --progress=on --toolMem=2g --basePath=".\..\..\" --cfgfile="$basePath$\arc.ini"

packcmd   = MTX\Win64\MTX64.exe a -mrz64 -c64m {options} - - <stdin> <stdout>

unpackcmd = MTX\Win64\MTX64.exe x -mrz64 {options} - - <stdin> <stdout>



arc.exe a -m=rzmt archive.arc files\*

Example 2 — Standalone, full piping

Code:

MTX64.exe a -mprecomp -c128m -t100p - - <stdin> <stdout>

MTX64.exe x -mprecomp -t100p - - <stdin> <stdout>

Example 3 — Memory-hungry tool, bounded RAM

Code:

MTX64.exe a -mrz -c64m -t100p --toolMem=2g --mem=8g - - <stdin> <stdout>

(starts only as many rz instances as 8 GB can hold at ~2 GB each)

Example 4 — Benign nonzero tool exit codes (precomp)

Code:

MTX64.exe a -mprecomp -c64m --okcodes=0,2 data.tmp packed.tmp

Example 5 — Per-method defaults in an MTX-only config file (MTX_Config.ini)

Code:

[External compressor:rz64]

header     = 0

packcmd    = Rz\rz.exe a -d 128m $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

unpackcmd  = Rz\rz.exe x $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

okcodes    = 0

mtxthreads = 4

mtxchunk   = 32m

mtxmem     = 2g



MTX64.exe a -mrz64 --cfgFile="X:\cfg\MTX_Config.ini" - - <stdin> <stdout>

Keep these keys OUT of FreeArc's own arc.ini — FreeArc rejects unknown keys in its sections. In arc.ini, pass --okcodes= / --toolMem= / … inside the MTX method's "default" line instead, as in Example 1.

Example 6 — Maximum throughput (MTX status on, tool line off)

Code:

MTX64.exe a -mzstd -c8m -t100p --progress=on --toolProgress=off - - <stdin> <stdout>

Keeps MTX's own live status but drops the wrapped tool's progress line — and with it the per-chunk pseudo-console that file-based tools need. For a fast compressor split into many small chunks that cost is measurable, so --toolProgress=off runs noticeably faster.

▌ DISTRIBUTION FILES

MTX64.exe — the tool (single exe, Win64)
MTX.ini — optional machine-global defaults (place next to the exe)
arc.ini — example FreeArc config with plain + MTX-wrapped methods (precomp, mpz, rz, nz, zstd, srep …)
README.txt — full documentation
WhatsNew.txt — version history

▌ NOTES

v0.3 archives use the MTX2 format; archives created by v0.1 must be repacked with the v0.3 build before migrating.
Compression and decompression machines never have to match in cores or RAM.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━

Feedback, bug reports, and edge cases are all welcome.

▌ Download

- Check the first post.

.