PMT - Parallel Multithreaded Encoder/Decoder

[78372]

PMT is a 4x4 like algorithm to make everything run parallelly. You can run any compressor/encoder that supports stdio/stdin/stdout or even nothing. The program can make a single threaded compressor multithreaded by using blocks, just like 4x4.

Run PMT.exe --h to see this help that describes a lot:

PMT - Parallel Multithreaded encoder/decoder
by 78372

Main Options:
PMT.exe e/d {Encoder} {Basic Options} Input Output
e/d represents encode/decoder
input/output can be specified as "-" for stdin/stdout
{Encoder} must be present both for encoding and decoding

Basic Options:
-t#: Number of threads to use(Default: number of threads you have)
-t#p: Percentage of threads to use
-b#: BlockSize(Encode only) (Default: 64m)

INI Options:
PMT.ini is required for encoder/decoder
The section name should be as the {Encoder}
The keys should be as below:
Encode = It should have the Encode Command Line
Decode = It should have the Decode Command Line
Infile = It should be the encode input file name for the {Encoder}. Default is p
mtinfile.tmp. It will be the output file for decoding
Outfile = It should be the encode output file name for the {Encoder}. Default is
pmtoutfile.tmp. It will be the input file for decoding
You can not encode using PMT and decode directly using {Encoder} and vice versa
Write <stdin> or <stdout> in Encode or Decode to specify stdin/stdout

Four tests:

Encode:

FreeArc 0.67 (September 13 2014) Creating archive: .\test1.arc using mpzapi
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,465,707 bytes. Ratio 1.09%
Compression time: cpu 0.88 sec/real 49.92 sec = 2%. Speed 8.17 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test2.arc using precomp
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 809,730,514 bytes. Ratio 198.46%
Compression time: cpu 1.31 sec/real 73.85 sec = 2%. Speed 5.52 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test3.arc using zlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 5,337,963 bytes. Ratio 1.31%
Compression time: cpu 0.94 sec/real 7.96 sec = 12%. Speed 51.26 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test4.arc using razor
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,923,279 bytes. Ratio 1.21%
Compression time: cpu 0.72 sec/real 20.47 sec = 4%. Speed 19.94 mB/s
All OK

Decode:

FreeArc 0.67 (September 13 2014) testing archive: test1.arc
Tested 3 files, 4,465,707 => 408,013,931 bytes. Ratio 1.09%
Testing time: cpu 0.80 sec/real 50.70 sec = 2%. Speed 8.05 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test2.arc
Tested 3 files, 809,730,514 => 408,013,931 bytes. Ratio 198.46%
Testing time: cpu 0.89 sec/real 35.82 sec = 2%. Speed 11.39 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test3.arc
Tested 3 files, 5,337,963 => 408,013,931 bytes. Ratio 1.31%
Testing time: cpu 0.70 sec/real 2.79 sec = 25%. Speed 146.12 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test4.arc
Tested 3 files, 4,923,279 => 408,013,931 bytes. Ratio 1.21%
Testing time: cpu 0.64 sec/real 5.52 sec = 12%. Speed 73.86 mB/s
All OK

PMT.ini Example:

[mpzapi]
Encode = mpzapi.exe c <stdin> <stdout>
Decode = mpzapi.exe d <stdin> <stdout>

[precomp]
Encode = precomp32 -cn -intense -opackedfile.pcf datafile
Decode = precomp32 -r -odatafile packedfile.pcf
Infile = datafile
Outfile = packedfile.pcf

[zlib]
Encode = zlib.exe c - pmtoutfile.tmp <stdin>
Decode = zlib d pmtoutfile.tmp - <stdout>

[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp

arc.ini Example:

[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd   = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>

When you use this, basically you don't need any other chunk based cls like cls-mpzmt, cls-rzmt or whatever.
You can make everything multithreaded with this.
As it's first release and I haven't tested it much, there can be bugs. Do report if you find one.
The program may seem too complex for beginners, but I can't make it simpler than this.

Good Luck and Merry Christmas.

Release 2 changelog:
1. The whole ini settings changed. See help again.
2. Minor bug fixed.

Source released on github