PMT - Parallel Multithreaded Encoder/Decoder

[78372]

PMT is a 4x4 like algorithm to make everything run parallelly. You can run any compressor/encoder that supports stdio/stdin/stdout or even nothing. The program can make a single threaded compressor multithreaded by using blocks, just like 4x4.

Run PMT.exe --h to see this help that describes a lot:

PMT - Parallel Multithreaded encoder/decoder
by 78372

Main Options:
PMT.exe e/d {Encoder} {Basic Options} Input Output
e/d represents encode/decoder
input/output can be specified as "-" for stdin/stdout
{Encoder} must be present both for encoding and decoding

Basic Options:
-t#: Number of threads to use(Default: number of threads you have)
-t#p: Percentage of threads to use
-b#: BlockSize(Encode only) (Default: 64m)

INI Options:
PMT.ini is required for encoder/decoder
The section name should be as the {Encoder}
The keys should be as below:
Encode = It should have the Encode Command Line
Decode = It should have the Decode Command Line
Infile = It should be the encode input file name for the {Encoder}. Default is p
mtinfile.tmp. It will be the output file for decoding
Outfile = It should be the encode output file name for the {Encoder}. Default is
pmtoutfile.tmp. It will be the input file for decoding
You can not encode using PMT and decode directly using {Encoder} and vice versa
Write <stdin> or <stdout> in Encode or Decode to specify stdin/stdout

Four tests:

Encode:

FreeArc 0.67 (September 13 2014) Creating archive: .\test1.arc using mpzapi
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,465,707 bytes. Ratio 1.09%
Compression time: cpu 0.88 sec/real 49.92 sec = 2%. Speed 8.17 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test2.arc using precomp
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 809,730,514 bytes. Ratio 198.46%
Compression time: cpu 1.31 sec/real 73.85 sec = 2%. Speed 5.52 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test3.arc using zlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 5,337,963 bytes. Ratio 1.31%
Compression time: cpu 0.94 sec/real 7.96 sec = 12%. Speed 51.26 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test4.arc using razor
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,923,279 bytes. Ratio 1.21%
Compression time: cpu 0.72 sec/real 20.47 sec = 4%. Speed 19.94 mB/s
All OK

Decode:

FreeArc 0.67 (September 13 2014) testing archive: test1.arc
Tested 3 files, 4,465,707 => 408,013,931 bytes. Ratio 1.09%
Testing time: cpu 0.80 sec/real 50.70 sec = 2%. Speed 8.05 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test2.arc
Tested 3 files, 809,730,514 => 408,013,931 bytes. Ratio 198.46%
Testing time: cpu 0.89 sec/real 35.82 sec = 2%. Speed 11.39 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test3.arc
Tested 3 files, 5,337,963 => 408,013,931 bytes. Ratio 1.31%
Testing time: cpu 0.70 sec/real 2.79 sec = 25%. Speed 146.12 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test4.arc
Tested 3 files, 4,923,279 => 408,013,931 bytes. Ratio 1.21%
Testing time: cpu 0.64 sec/real 5.52 sec = 12%. Speed 73.86 mB/s
All OK

PMT.ini Example:

[mpzapi]
Encode = mpzapi.exe c <stdin> <stdout>
Decode = mpzapi.exe d <stdin> <stdout>

[precomp]
Encode = precomp32 -cn -intense -opackedfile.pcf datafile
Decode = precomp32 -r -odatafile packedfile.pcf
Infile = datafile
Outfile = packedfile.pcf

[zlib]
Encode = zlib.exe c - pmtoutfile.tmp <stdin>
Decode = zlib d pmtoutfile.tmp - <stdout>

[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp

arc.ini Example:

[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd   = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>

When you use this, basically you don't need any other chunk based cls like cls-mpzmt, cls-rzmt or whatever.
You can make everything multithreaded with this.
As it's first release and I haven't tested it much, there can be bugs. Do report if you find one.
The program may seem too complex for beginners, but I can't make it simpler than this.

Good Luck and Merry Christmas.

Release 2 changelog:
1. The whole ini settings changed. See help again.
2. Minor bug fixed.

Source released on github

[Simorq]

Thanks 78372

[razor]
Encode = RZ a -d 32m packedfile.pcf datafile
Decode = RZ e packedfile.pcf datafile
InFileEnc = datafile
OutFileEnc = packedfile.pcf
InFileDec = packedfile.pcf
OutFileDec = datafile

[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd   = PMT.exe e {compressor} -b16M -t100p - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t4 - - <stdin> <stdout>

Compressed 8 files, 466,568,427 => 58,679,682 bytes. Ratio 12.58%
Compression time: cpu 0.38 sec/real 17.54 sec = 2%. Speed 26.60 mB/s
All OK

Tested 8 files, 58,679,682 => 466,568,427 bytes. Ratio 12.58%
Testing time: cpu 0.45 sec/real 1.55 sec = 29%. Speed 300.13 mB/s
All OK

It's great.

[78372]

^^.pcf isn't needed here, that was for precomp which gives
pcf as output. And -d 32m doesn't make sense with 16m blocks

[KaktoR]

Ohh, okay. This seems to be interresting. I will test it out next year if I have a bit more time

Thank you for all your stuff here until now!

[dixen]

Nice Work! Thanks.

[FitGirl]

Thanks for the Christmas gift, it's working as intended! Will do more tests.

[78372]

@Everyone.

Release 2 Uploaded. Changed the whole INI setup and made it as easy as fa. I wonder why I didn't make it before. Check first post for changes, settings and download.

[shazzla]

What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?

[78372]

Originally Posted by shazzla

What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?

If you have cls for both packing and unpacking then perhaps there is a way to use PMT.

[DiCaPrIo]

anyway to make bpk mt I tested it's not working

[FitGirl]

Originally Posted by DiCaPrIo

anyway to make bpk mt I tested it's not working

Of course it won't work, bpk can deal with complete bink files only, not trimmed by this tool.

[78372]

If you have a cls for both compression and decompression, you can use wrapcls.exe with PMT to use the cls. Here's an example of PMT.ini:

[lz4]
Encode = wrapcls.exe lz4 <stdin> <stdout>
Decode = wrapcls.exe d <stdin> <stdout>

Wrapcls is basically Compressor 2.2 by Bulat Ziganshin.So it can be used for other freearc algorithms too. I have just recompiled the program to rename cls-*.dll to arc-*.dll so that it doesn't create conflict while using with freearc. You need to rename your cls too to use this program.

[FitGirl]

Originally Posted by Simorq

@78372
Does not work with CLS-BPK.

C'mon, I have already answered to this. PMT is a BLOCK-BASED TRIMMER. IT SPLITS FILES INTO CHUNKS. BPK is a FILE-BASED ENCODER. It deals only with full files. PMT split bink files into chunks and they CANNOT BE COMPRESSED with BPK anymore. "L" for logic.

It's intended for codecs which rely on pure stream data - generic compressors, precompressors, MPZ (cause it deals with frames in MP3, not full files).

[78372]

@Simorq,
I think FitGirl has already answered your question. In addition, let me describe how PMT works.
PMT splits your input as blocksize you have specified. If your encoder doesn't support stdio, it makes temporary files and runs your encoder over it. It requires disk space for that. If your encoder supports stdio, then it uses memory for storing blocks. Basically the additional memory you need here is blocksize * threads + encoded size * threads. Same goes for non-stdio but it takes disk space and temp files then.
PMT can't work without extra memory for stdio as PMT itself supports stdio, and without using blocks, both stdio support for encoding+decoding is not possible. Also, even without stdio, running a lot of instances parallelly and writing to same output isn't possible. So either way it requires extra memory or temp files. As the program requires blocks, encoders/compressors that require full files(such as bpk or srep) won't work perfectly/won't work at all.

I think this answers and will answer to future questions regarding PMT.

[Entai44]

Is this working with packet compressor? I can compress with UltraARC, but decompression doesn't work. This is my configuration:

[External compressoracket]
packcmd = packetx64.exe a -v -r -mx -b512 -h4 $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = packetx64.exe x -v $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
datafile = $$arcdatafile$$.tmp
packedfile = $$arcpackedfile$$.tmp

it shows error when decompressing, like "unknown compression method", how to fix this problem?

Thanks btw for PMT