View Full Version : PMT - Parallel Multithreaded Encoder/Decoder
PMT is a 4x4 like algorithm to make everything run parallelly. You can run any compressor/encoder that supports stdio/stdin/stdout or even nothing. The program can make a single threaded compressor multithreaded by using blocks, just like 4x4.
Run PMT.exe --h to see this help that describes a lot:
PMT - Parallel Multithreaded encoder/decoder
by 78372
Main Options:
PMT.exe e/d {Encoder} {Basic Options} Input Output
e/d represents encode/decoder
input/output can be specified as "-" for stdin/stdout
{Encoder} must be present both for encoding and decoding
Basic Options:
-t#: Number of threads to use(Default: number of threads you have)
-t#p: Percentage of threads to use
-b#: BlockSize(Encode only) (Default: 64m)
INI Options:
PMT.ini is required for encoder/decoder
The section name should be as the {Encoder}
The keys should be as below:
Encode = It should have the Encode Command Line
Decode = It should have the Decode Command Line
Infile = It should be the encode input file name for the {Encoder}. Default is p
mtinfile.tmp. It will be the output file for decoding
Outfile = It should be the encode output file name for the {Encoder}. Default is
pmtoutfile.tmp. It will be the input file for decoding
You can not encode using PMT and decode directly using {Encoder} and vice versa
Write <stdin> or <stdout> in Encode or Decode to specify stdin/stdout
Four tests:
Encode:
FreeArc 0.67 (September 13 2014) Creating archive: .\test1.arc using mpzapi
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,465,707 bytes. Ratio 1.09%
Compression time: cpu 0.88 sec/real 49.92 sec = 2%. Speed 8.17 mB/s
All OK
FreeArc 0.67 (September 13 2014) Creating archive: .\test2.arc using precomp
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 809,730,514 bytes. Ratio 198.46%
Compression time: cpu 1.31 sec/real 73.85 sec = 2%. Speed 5.52 mB/s
All OK
FreeArc 0.67 (September 13 2014) Creating archive: .\test3.arc using zlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 5,337,963 bytes. Ratio 1.31%
Compression time: cpu 0.94 sec/real 7.96 sec = 12%. Speed 51.26 mB/s
All OK
FreeArc 0.67 (September 13 2014) Creating archive: .\test4.arc using razor
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,923,279 bytes. Ratio 1.21%
Compression time: cpu 0.72 sec/real 20.47 sec = 4%. Speed 19.94 mB/s
All OK
Decode:
FreeArc 0.67 (September 13 2014) testing archive: test1.arc
Tested 3 files, 4,465,707 => 408,013,931 bytes. Ratio 1.09%
Testing time: cpu 0.80 sec/real 50.70 sec = 2%. Speed 8.05 mB/s
All OK
FreeArc 0.67 (September 13 2014) testing archive: test2.arc
Tested 3 files, 809,730,514 => 408,013,931 bytes. Ratio 198.46%
Testing time: cpu 0.89 sec/real 35.82 sec = 2%. Speed 11.39 mB/s
All OK
FreeArc 0.67 (September 13 2014) testing archive: test3.arc
Tested 3 files, 5,337,963 => 408,013,931 bytes. Ratio 1.31%
Testing time: cpu 0.70 sec/real 2.79 sec = 25%. Speed 146.12 mB/s
All OK
FreeArc 0.67 (September 13 2014) testing archive: test4.arc
Tested 3 files, 4,923,279 => 408,013,931 bytes. Ratio 1.21%
Testing time: cpu 0.64 sec/real 5.52 sec = 12%. Speed 73.86 mB/s
All OK
PMT.ini Example:
[mpzapi]
Encode = mpzapi.exe c <stdin> <stdout>
Decode = mpzapi.exe d <stdin> <stdout>
[precomp]
Encode = precomp32 -cn -intense -opackedfile.pcf datafile
Decode = precomp32 -r -odatafile packedfile.pcf
Infile = datafile
Outfile = packedfile.pcf
[zlib]
Encode = zlib.exe c - pmtoutfile.tmp <stdin>
Decode = zlib d pmtoutfile.tmp - <stdout>
[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp
arc.ini Example:
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>
When you use this, basically you don't need any other chunk based cls like cls-mpzmt, cls-rzmt or whatever.
You can make everything multithreaded with this.
As it's first release and I haven't tested it much, there can be bugs. Do report if you find one.
The program may seem too complex for beginners, but I can't make it simpler than this.
Good Luck and Merry Christmas.
Release 2 changelog:
1. The whole ini settings changed. See help again.
2. Minor bug fixed.
Source released on github (https://github.com/RUDRA78372/PMT/)
Thanks 78372
[razor]
Encode = RZ a -d 32m packedfile.pcf datafile
Decode = RZ e packedfile.pcf datafile
InFileEnc = datafile
OutFileEnc = packedfile.pcf
InFileDec = packedfile.pcf
OutFileDec = datafile
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd = PMT.exe e {compressor} -b16M -t100p - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t4 - - <stdin> <stdout>
Compressed 8 files, 466,568,427 => 58,679,682 bytes. Ratio 12.58%
Compression time: cpu 0.38 sec/real 17.54 sec = 2%. Speed 26.60 mB/s
All OK
Tested 8 files, 58,679,682 => 466,568,427 bytes. Ratio 12.58%
Testing time: cpu 0.45 sec/real 1.55 sec = 29%. Speed 300.13 mB/s
All OK
It's great.:D
^^.pcf isn't needed here, that was for precomp which gives
pcf as output. And -d 32m doesn't make sense with 16m blocks
Ohh, okay. This seems to be interresting. I will test it out next year if I have a bit more time :D
Thank you for all your stuff here until now!
Thanks for the Christmas gift, it's working as intended! Will do more tests.
@Everyone.
Release 2 Uploaded. Changed the whole INI setup and made it as easy as fa. I wonder why I didn't make it before. Check first post for changes, settings and download.
What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?
What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?
If you have cls for both packing and unpacking then perhaps there is a way to use PMT.
DiCaPrIo
25-12-2019, 13:12
anyway to make bpk mt I tested it's not working
anyway to make bpk mt I tested it's not working
Of course it won't work, bpk can deal with complete bink files only, not trimmed by this tool.
If you have a cls for both compression and decompression, you can use wrapcls.exe with PMT to use the cls. Here's an example of PMT.ini:
[lz4]
Encode = wrapcls.exe lz4 <stdin> <stdout>
Decode = wrapcls.exe d <stdin> <stdout>
Wrapcls is basically Compressor 2.2 by Bulat Ziganshin.So it can be used for other freearc algorithms too. I have just recompiled the program to rename cls-*.dll to arc-*.dll so that it doesn't create conflict while using with freearc. You need to rename your cls too to use this program.
@78372
Does not work with CLS-BPK.
C'mon, I have already answered to this. PMT is a BLOCK-BASED TRIMMER. IT SPLITS FILES INTO CHUNKS. BPK is a FILE-BASED ENCODER. It deals only with full files. PMT split bink files into chunks and they CANNOT BE COMPRESSED with BPK anymore. "L" for logic.
It's intended for codecs which rely on pure stream data - generic compressors, precompressors, MPZ (cause it deals with frames in MP3, not full files).
@Simorq,
I think FitGirl has already answered your question. In addition, let me describe how PMT works.
PMT splits your input as blocksize you have specified. If your encoder doesn't support stdio, it makes temporary files and runs your encoder over it. It requires disk space for that. If your encoder supports stdio, then it uses memory for storing blocks. Basically the additional memory you need here is blocksize * threads + encoded size * threads. Same goes for non-stdio but it takes disk space and temp files then.
PMT can't work without extra memory for stdio as PMT itself supports stdio, and without using blocks, both stdio support for encoding+decoding is not possible. Also, even without stdio, running a lot of instances parallelly and writing to same output isn't possible. So either way it requires extra memory or temp files. As the program requires blocks, encoders/compressors that require full files(such as bpk or srep) won't work perfectly/won't work at all.
I think this answers and will answer to future questions regarding PMT.
Is this working with packet compressor? I can compress with UltraARC, but decompression doesn't work. This is my configuration:
[External compressor:packet]
packcmd = packetx64.exe a -v -r -mx -b512 -h4 $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = packetx64.exe x -v $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
datafile = $$arcdatafile$$.tmp
packedfile = $$arcpackedfile$$.tmp
it shows error when decompressing, like "unknown compression method", how to fix this problem?
Thanks btw for PMT
DiCaPrIo
28-12-2019, 09:25
put it in PMT.ini
[packet]
Encode = packetx64.exe a -v -r -mx -b512 -h4 packedfile.pct datafile
Decode = packetx64.exe x -v datafile packedfile.pct
Infile = datafile
Outfile = packedfile.pct
in arc.ini
[External compressor:packet]
header = 0
packcmd = PMT.exe e {compressor} -b32m -t75p - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t75p - - <stdin> <stdout>
[packet]
Encode = packetx64.exe a -v -r -mx -b512 -h4 packedfile.pct datafile
Decode = packetx64.exe x -v datafile packedfile.pct
Infile = datafile
Outfile = packedfile.pct
PMT save my life, but I must change it like this:
[packet2]
Encode = packetx64.exe a -v -r -mx -b512 -h4 packedfile.pct datafile
Decode = packetx64.exe x -v packedfile.pct datafile
Infile = datafile
Outfile = packedfile.pct
And this is the fix, but the temp files still there after decompressing I thought I must try it again
And BOOM!! After some changes the PMT temp is gone, my loved packet compressor is useful right know thanks anyway @DiCaPrIo
Compressed 1 file, 288,981,192 => 112,742,899 bytes. Ratio 39.01%
Compression time: cpu 0.22 sec/real 136.38 sec = 0%. Speed 2.12 mB/s
All OK
Testing 1 file, 288,981,192 bytes. Processed 92%
ERROR: general (de)compression error in razor
So what I doing wrong here?
Compressed 1 file, 288,981,192 => 112,742,899 bytes. Ratio 39.01%
Compression time: cpu 0.22 sec/real 136.38 sec = 0%. Speed 2.12 mB/s
All OK
Testing 1 file, 288,981,192 bytes. Processed 92%
ERROR: general (de)compression error in razor
So what I doing wrong here?
Post PMT.ini and arc.ini please
I just used the example settings (it was my first test run).
I just used the example settings (it was my first test run).
Did you run the sample bat packer as is or modified it?
Because default one works.
Nah, I just deleted test2.arc and test3.arc lines and changed test1.arc to -mrazor method.
That's all I changed.
Try to include srep first
@KaktoR, Can you give me a sample?
Also, try using other compressors, if that works, then probably it's a bug from razor itself
@KaktoR
test t2 for decode!
@KaktoR
test t2 for decode!
Thanks, this works!
Tested 1 file, 112,742,899 => 288,981,192 bytes. Ratio 39.01%
Testing time: cpu 0.20 sec/real 5.82 sec = 3%. Speed 49.69 mB/s
All OK
Source released on github
Check first post for source link.
Sergey3695
15-01-2020, 04:14
Source released on github
how to compile it?
https://d.radikal.ru/d18/2001/64/d07b30db8a79.jpg
if use ITask and compile, then the problem with "-mzlib" (write error etc.)
is correct source?
:confused:
If you had downloaded before yesterday, download it again, I have updated that yesterday.
Sergey3695
15-01-2020, 04:46
If you had downloaded before yesterday, download it again, I have updated that yesterday.
Utilities.pas Add files via upload 4 days ago
:)
ok)
---
upload pls source for PMT32 (1040 kb) . because i have problem with "-mzlib" if use PMT32 (1211 kb) (github exe) (general dec error).
darkwolves
16-01-2020, 06:44
the only thing i cannot get working is zlib it crashes no matter were in the command line i use it
i am particularly pleased with the multithreaded razor
Snake288
21-01-2020, 10:45
78372 Hello can I get your help ?
Issue : PMT.ini and arc.ini add multiple tools I sent you can you add tools ?
I tried so hard, I failed.
78372 Hello can I get your help ?
Issue : PMT.ini and arc.ini add multiple tools I sent you can you add tools ?
I tried so hard, I failed.
1.Never combine srep with PMT
2. PMT isn't FA. So {options} is not supported. packcmd and unpackcmd is not available, it should be Encode/Decode.
3. There's a typo in lolz section. Plus using lolz with pmt doesn't make sense as lolz is already multithreaded.
4. Invalid in+out file and parameter in uelr.
darkwolves
25-01-2020, 18:07
1.Never combine srep with PMT
2. PMT isn't FA. So {options} is not supported. packcmd and unpackcmd is not available, it should be Encode/Decode.
3. There's a typo in lolz section. Plus using lolz with pmt doesn't make sense as lolz is already multithreaded.
4. Invalid in+out file and parameter in uelr.
hmmm interesting i have used srep with pmt and it actually does work
i repacked a few games with it and had no problems
1.Never combine srep with PMT
Why? SREP+PMT fine works))
What about nanozip with PMT?
Srep+Pmt ? Why ?
Srep is lightning fast by default.
Why? SREP+PMT fine works))
What about nanozip with PMT?
Nanozip can be multithreaded using -p settings
When using the default parameters of "razor" in PMT.ini :
[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp
and in ARC.ini :
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>
when decompressing i got this: "unknown compression method"
the game i want to compress is "black mesa" ( i used "Dixen" Method).
can someone please show me a better parameters to avoid this error in both PMT.ini & ARC.ini , and the parameters of "rz" in ARC.ini.
thanks !
BLACKFIRE69
15-03-2020, 03:27
When using the default parameters of "razor" in PMT.ini :
[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp
and in ARC.ini :
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>
when decompressing i got this: "unknown compression method"
the game i want to compress is "black mesa" ( i used "Dixen" Method).
can someone please show me a better parameters to avoid this error in both PMT.ini & ARC.ini , and the parameters of "rz" in ARC.ini.
thanks !
get something,
I tried several times , in the end i got always the same error ! i'm using :
"PMT (razor) + srep:m3f " , i have no problem in compression but in decompression , it shows "unsupported compression method srep:m3f+razor"
can someone help me and show me an example for what i should add in :
Ultraarc 2.9.0.0 R6 Final :
config0.txt
ASIS.v7.0.8 :
Compressor.ini
arc.ini
Script.iss
thanks!:)
Kitsune1982
22-03-2020, 10:03
Hi guys, havent posted much in awhile so decided i would post a reply.
I saw the info about PMT and it looks great but i have a question.
Will this PMT work with installer scripts like C<ustom>I<nstaller>U<ltimate>
3.0.0.1.b4 by Cesar82 or is CIU's options diskspan and ultraarc only? Would love to be able to use this with CIU. let me know if its possible and if it is, what do I need to do to link the compressed files to my project?
thanks.
-Kitsune
Compressed 1 file, 288,981,192 => 112,742,899 bytes. Ratio 39.01%
Compression time: cpu 0.22 sec/real 136.38 sec = 0%. (https://www.igriceigre.net) Speed 2.12 mB/s
All OK
Testing 1 file, 288,981,192 bytes. Processed 92%
ERROR: general (de)compression error in razor
So what I doing wrong here?
Post PMT.ini and arc.ini please.
In some rare case,it happens with me,too.. (for example,Quantum break ,1xx GB version)
Maybe a bug in Razor ?!
In some rare case,it happens with me,too.. (for example,Quantum break ,1xx GB version)
Maybe a bug in Razor ?!
With Razor's patch, razor now supports stdio (What a pun :D )
Try using that one to remove temps and probably that will fix some stuff.
I used the STDIO patched version ,without PMT ! (RZ V1.03.7)
I have further tested with full games, it works like a charm and pretty fast :D
1:36:00 - Starting Archive Creation.
2:54:33 - Archive Creation Complete.
2:54:33 - Sorting Storage Directories Started.
2:54:34 - Sorting Storage Directories Complete.
2:54:34 - Starting Archive Creation.
4:11:32 - Archive Creation Complete.
4:11:32 - Sorting Storage Directories Started.
4:11:32 - Sorting Storage Directories Complete.
4:11:32 - Starting Archive Creation.
9:56:22 - Archive Creation Complete.
9:56:22 - Sorting Storage Directories Started.
9:56:22 - Sorting Storage Directories Complete.
10:28:36 - The Witcher Enhanced Edition Size: 13.67 GB.
10:28:36 - The Witcher 2 Size: 17.21 GB.
10:28:37 - The Witcher 3 Size: 42.99 GB.
10:28:37 - Full Input Size: 73.89 GB.
10:28:37 - Output Size: 45.63 GB.
Decode only possible with t2 or t1 for me, I don't know why :(
Tested with batch and CIU installer.
I used razor
Seems like some internal issues with razor. Did you test with other compressors? How do they behave? Probably the fast speed of razor is causing some I/O bottleneck or something with multiple threads. Single thread is enough for razor to be frank with such lightning decompression speed.
Not yet, but it's ok neverthless because rz is lightning fast in decompression :)
What other compressors do you suggest? What about zstd?
ZSTD has mt and way too fast to use with PMT.
Try with slow tools.
BCM, precomp, rzm etc. can work better.
We have.alternatives for them though.
Has anyone done any tests using PMT + pZLib3 (https://www.fileforums.com/showthread.php?t=98672)?
If yes please share the values for PMT.ini and Arc.ini.
Thanks!
Has anyone done any tests using PMT + pZLib3 (https://www.fileforums.com/showthread.php?t=98672)?
If yes please share the values for PMT.ini and Arc.ini.
Thanks!
What's the point? pZlib is already MT and fast. Moreover, you have xtool now.
@78372
Thanks for the information. I know little about compression methods.
So of the compressors used in CIU only RAZOR has benefits if used with PMT.
CIU supports these compressors in addition to some RAZOR tools for specific games.
SREP, ZTool, XTool(not v2019), pZLib3, UELR, AFR, OodleRec [4,5,6,7,8], MSC, MPZ, OGGRE, BPK, 7z, LOLZ, ZSTD, RAZOR
Does any of this list other than razor benefit if used with PMT?
@78372
Thanks for the information. I know little about compression methods.
So of the compressors used in CIU only RAZOR has benefits if used with PMT.
CIU supports these compressors in addition to some RAZOR tools for specific games.
SREP, ZTool, XTool(not v2019), pZLib3, UELR, AFR, OodleRec [4,5,6,7,8], MSC, MPZ, OGGRE, BPK, 7z, LOLZ, ZSTD, RAZOR
Does any of this list other than razor benefit if used with PMT?
mpz only I guess, at least for now.
mpz with PMT
Compressed 64 files, 209,434,083 => 180,728,551 bytes. Ratio 86.29%
Compression time: cpu 0.20 sec/real 38.46 sec = 1%. Speed 5.45 mB/s
Tested 64 files, 180,728,551 => 209,434,083 bytes. Ratio 86.29%
Testing time: cpu 0.19 sec/real 36.54 sec = 1%. Speed 5.73 mB/s
mpz only
Compressed 64 files, 209,434,083 => 180,670,089 bytes. Ratio 86.27%
Compression time: cpu 0.30 sec/real 103.93 sec = 0%. Speed 2.02 mB/s
Tested 64 files, 180,670,089 => 209,434,083 bytes. Ratio 86.27%
Testing time: cpu 0.36 sec/real 95.90 sec = 0%. Speed 2.18 mB/s
Tested with settings -b64m -t100p for encode/decode with a Ryzen 5 2600 (12 threads).
I just wonder why ratio is a little bit better without pmt :D
The same reason for everything, chunking.
DiCaPrIo
08-06-2020, 02:15
Hi. Is possible to add bpk compatibility? Thanks.
see at first page
L0v3craft
02-07-2020, 20:15
Is possible to add this?
https://github.com/encode84/bcm
it is BCM v1.5. If is not possible someone has a cls dll for BCM to make it multithreaded? Thanks.
Is possible to add this?
https://github.com/encode84/bcm
it is BCM v1.5. If is not possible someone has a cls dll for BCM to make it multithreaded? Thanks.
BCM 1.4 works with this. (https://fileforums.com/showthread.php?t=102794) I will make a stdio patch for 1.5 when I have time.
Thanks very much for this tool "78372"
If possible, I want the latest version from you "PMT_rel3"
Thanks very much for this tool "78372"
If possible, I want the latest version from you "PMT_rel3"
I don't think that the project requires an update, I haven't received such requests and I do not use it anyways.
vBulletin® v3.8.11, Copyright ©2000-2026, vBulletin Solutions Inc.