PMT - Parallel Multithreaded Encoder/Decoder
 

PMT is a 4x4 like algorithm to make everything run parallelly. You can run any compressor/encoder that supports stdio/stdin/stdout or even nothing. The program can make a single threaded compressor multithreaded by using blocks, just like 4x4.

Run PMT.exe --h to see this help that describes a lot:

Four tests:

Encode:
Decode:
PMT.ini Example:
arc.ini Example:
When you use this, basically you don't need any other chunk based cls like cls-mpzmt, cls-rzmt or whatever.
You can make everything multithreaded with this.
As it's first release and I haven't tested it much, there can be bugs. Do report if you find one.
The program may seem too complex for beginners, but I can't make it simpler than this.

Good Luck and Merry Christmas.

Release 2 changelog:
1. The whole ini settings changed. See help again.
2. Minor bug fixed.

Source released on github

Test
 

Thanks 78372
It's great.:D

^^.pcf isn't needed here, that was for precomp which gives
pcf as output. And -d 32m doesn't make sense with 16m blocks

Ohh, okay. This seems to be interresting. I will test it out next year if I have a bit more time :D

Thank you for all your stuff here until now!

Nice Work! Thanks.

Thanks for the Christmas gift, it's working as intended! Will do more tests.

@Everyone.

Release 2 Uploaded. Changed the whole INI setup and made it as easy as fa. I wonder why I didn't make it before. Check first post for changes, settings and download.

What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?

If you have cls for both packing and unpacking then perhaps there is a way to use PMT.

anyway to make bpk mt I tested it's not working

Of course it won't work, bpk can deal with complete bink files only, not trimmed by this tool.

If you have a cls for both compression and decompression, you can use wrapcls.exe with PMT to use the cls. Here's an example of PMT.ini:

Wrapcls is basically Compressor 2.2 by Bulat Ziganshin.So it can be used for other freearc algorithms too. I have just recompiled the program to rename cls-*.dll to arc-*.dll so that it doesn't create conflict while using with freearc. You need to rename your cls too to use this program.

C'mon, I have already answered to this. PMT is a BLOCK-BASED TRIMMER. IT SPLITS FILES INTO CHUNKS. BPK is a FILE-BASED ENCODER. It deals only with full files. PMT split bink files into chunks and they CANNOT BE COMPRESSED with BPK anymore. "L" for logic.

It's intended for codecs which rely on pure stream data - generic compressors, precompressors, MPZ (cause it deals with frames in MP3, not full files).

@Simorq,
I think FitGirl has already answered your question. In addition, let me describe how PMT works.
PMT splits your input as blocksize you have specified. If your encoder doesn't support stdio, it makes temporary files and runs your encoder over it. It requires disk space for that. If your encoder supports stdio, then it uses memory for storing blocks. Basically the additional memory you need here is blocksize * threads + encoded size * threads. Same goes for non-stdio but it takes disk space and temp files then.
PMT can't work without extra memory for stdio as PMT itself supports stdio, and without using blocks, both stdio support for encoding+decoding is not possible. Also, even without stdio, running a lot of instances parallelly and writing to same output isn't possible. So either way it requires extra memory or temp files. As the program requires blocks, encoders/compressors that require full files(such as bpk or srep) won't work perfectly/won't work at all.

I think this answers and will answer to future questions regarding PMT.

Is this working with packet compressor? I can compress with UltraARC, but decompression doesn't work. This is my configuration:

it shows error when decompressing, like "unknown compression method", how to fix this problem?

Thanks btw for PMT

put it in PMT.ini
[packet]
Encode = packetx64.exe a -v -r -mx -b512 -h4 packedfile.pct datafile
Decode = packetx64.exe x -v datafile packedfile.pct
Infile = datafile
Outfile = packedfile.pct

in arc.ini
[External compressor:packet]
header = 0
packcmd = PMT.exe e {compressor} -b32m -t75p - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t75p - - <stdin> <stdout>

PMT save my life, but I must change it like this:

And this is the fix, but the temp files still there after decompressing I thought I must try it again

And BOOM!! After some changes the PMT temp is gone, my loved packet compressor is useful right know thanks anyway @DiCaPrIo

So what I doing wrong here?

Post PMT.ini and arc.ini please

I just used the example settings (it was my first test run).

Did you run the sample bat packer as is or modified it?
Because default one works.

Nah, I just deleted test2.arc and test3.arc lines and changed test1.arc to -mrazor method.

That's all I changed.

Try to include srep first

@KaktoR, Can you give me a sample?
Also, try using other compressors, if that works, then probably it's a bug from razor itself

@KaktoR
test t2 for decode!

Thanks, this works!

Source released on github

Check first post for source link.

how to compile it?
https://d.radikal.ru/d18/2001/64/d07b30db8a79.jpg
if use ITask and compile, then the problem with "-mzlib" (write error etc.)
is correct source?
:confused:

If you had downloaded before yesterday, download it again, I have updated that yesterday.

Utilities.pas Add files via upload 4 days ago
:)
ok)
---
upload pls source for PMT32 (1040 kb) . because i have problem with "-mzlib" if use PMT32 (1211 kb) (github exe) (general dec error).

the only thing i cannot get working is zlib it crashes no matter were in the command line i use it
i am particularly pleased with the multithreaded razor

78372 Hello can I get your help ?
 

78372 Hello can I get your help ?

Issue : PMT.ini and arc.ini add multiple tools I sent you can you add tools ?
I tried so hard, I failed.

1.Never combine srep with PMT
2. PMT isn't FA. So {options} is not supported. packcmd and unpackcmd is not available, it should be Encode/Decode.
3. There's a typo in lolz section. Plus using lolz with pmt doesn't make sense as lolz is already multithreaded.
4. Invalid in+out file and parameter in uelr.

hmmm interesting i have used srep with pmt and it actually does work
i repacked a few games with it and had no problems

Why? SREP+PMT fine works))
What about nanozip with PMT?

Srep+Pmt ? Why ?
Srep is lightning fast by default.

Nanozip can be multithreaded using -p settings

try mcm or fp8

When using the default parameters of "razor" in PMT.ini :
[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp

and in ARC.ini :
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>

when decompressing i got this: "unknown compression method"
the game i want to compress is "black mesa" ( i used "Dixen" Method).

can someone please show me a better parameters to avoid this error in both PMT.ini & ARC.ini , and the parameters of "rz" in ARC.ini.

thanks !

get something,