Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #1  
Old 24-12-2019, 05:32
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
PMT - Parallel Multithreaded Encoder/Decoder

PMT is a 4x4 like algorithm to make everything run parallelly. You can run any compressor/encoder that supports stdio/stdin/stdout or even nothing. The program can make a single threaded compressor multithreaded by using blocks, just like 4x4.

Run PMT.exe --h to see this help that describes a lot:

Quote:
PMT - Parallel Multithreaded encoder/decoder
by 78372

Main Options:
PMT.exe e/d {Encoder} {Basic Options} Input Output
e/d represents encode/decoder
input/output can be specified as "-" for stdin/stdout
{Encoder} must be present both for encoding and decoding

Basic Options:
-t#: Number of threads to use(Default: number of threads you have)
-t#p: Percentage of threads to use
-b#: BlockSize(Encode only) (Default: 64m)

INI Options:
PMT.ini is required for encoder/decoder
The section name should be as the {Encoder}
The keys should be as below:
Encode = It should have the Encode Command Line
Decode = It should have the Decode Command Line
Infile = It should be the encode input file name for the {Encoder}. Default is p
mtinfile.tmp. It will be the output file for decoding
Outfile = It should be the encode output file name for the {Encoder}. Default is
pmtoutfile.tmp. It will be the input file for decoding
You can not encode using PMT and decode directly using {Encoder} and vice versa
Write <stdin> or <stdout> in Encode or Decode to specify stdin/stdout
Four tests:

Encode:
Quote:
FreeArc 0.67 (September 13 2014) Creating archive: .\test1.arc using mpzapi
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,465,707 bytes. Ratio 1.09%
Compression time: cpu 0.88 sec/real 49.92 sec = 2%. Speed 8.17 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test2.arc using precomp
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 809,730,514 bytes. Ratio 198.46%
Compression time: cpu 1.31 sec/real 73.85 sec = 2%. Speed 5.52 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test3.arc using zlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 5,337,963 bytes. Ratio 1.31%
Compression time: cpu 0.94 sec/real 7.96 sec = 12%. Speed 51.26 mB/s
All OK

FreeArc 0.67 (September 13 2014) Creating archive: .\test4.arc using razor
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 3 files, 408,013,931 => 4,923,279 bytes. Ratio 1.21%
Compression time: cpu 0.72 sec/real 20.47 sec = 4%. Speed 19.94 mB/s
All OK
Decode:
Quote:
FreeArc 0.67 (September 13 2014) testing archive: test1.arc
Tested 3 files, 4,465,707 => 408,013,931 bytes. Ratio 1.09%
Testing time: cpu 0.80 sec/real 50.70 sec = 2%. Speed 8.05 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test2.arc
Tested 3 files, 809,730,514 => 408,013,931 bytes. Ratio 198.46%
Testing time: cpu 0.89 sec/real 35.82 sec = 2%. Speed 11.39 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test3.arc
Tested 3 files, 5,337,963 => 408,013,931 bytes. Ratio 1.31%
Testing time: cpu 0.70 sec/real 2.79 sec = 25%. Speed 146.12 mB/s
All OK

FreeArc 0.67 (September 13 2014) testing archive: test4.arc
Tested 3 files, 4,923,279 => 408,013,931 bytes. Ratio 1.21%
Testing time: cpu 0.64 sec/real 5.52 sec = 12%. Speed 73.86 mB/s
All OK
PMT.ini Example:
Quote:
[mpzapi]
Encode = mpzapi.exe c <stdin> <stdout>
Decode = mpzapi.exe d <stdin> <stdout>

[precomp]
Encode = precomp32 -cn -intense -opackedfile.pcf datafile
Decode = precomp32 -r -odatafile packedfile.pcf
Infile = datafile
Outfile = packedfile.pcf

[zlib]
Encode = zlib.exe c - pmtoutfile.tmp <stdin>
Decode = zlib d pmtoutfile.tmp - <stdout>

[razor]
Encode = rz.exe a -d 32m pmtoutfile.tmp pmtinfile.tmp
Decode = rz.exe e -y pmtoutfile.tmp pmtinfile.tmp
arc.ini Example:
Code:
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd   = PMT.exe e {compressor} -b64M -t4 - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t100p - - <stdin> <stdout>
When you use this, basically you don't need any other chunk based cls like cls-mpzmt, cls-rzmt or whatever.
You can make everything multithreaded with this.
As it's first release and I haven't tested it much, there can be bugs. Do report if you find one.
The program may seem too complex for beginners, but I can't make it simpler than this.

Good Luck and Merry Christmas.

Release 2 changelog:
1. The whole ini settings changed. See help again.
2. Minor bug fixed.

Source released on github
Attached Files
File Type: rar PMT_Rel2.rar (7.59 MB, 758 views)
__________________
NOT AVAILABLE

Last edited by 78372; 17-01-2020 at 04:02.
Reply With Quote
The Following 26 Users Say Thank You to 78372 For This Useful Post:
Andu21 (02-01-2020), Balaji007 (05-10-2020), Behnam2018 (11-01-2020), BLACKFIRE69 (27-12-2019), Carldric Clement (24-12-2019), darkwolves (26-12-2019), DiCaPrIo (25-12-2019), dixen (25-12-2019), Entai44 (24-12-2019), ffmla (24-12-2019), FitGirl (25-12-2019), giothothan (29-07-2020), Harsh ojha (24-12-2019), KaktoR (24-12-2019), kassane (29-01-2020), kenzo34 (24-12-2019), L0v3craft (07-07-2020), Mortal Lord (18-07-2020), oltjon (11-03-2020), PsYcHo_RaGE (24-12-2019), rambohazard (27-12-2019), sanekbest1 (25-12-2019), Sergey3695 (10-01-2020), shazzla (24-12-2019), Simorq (24-12-2019), ZAZA4EVER (24-12-2019)
Sponsored Links
  #2  
Old 24-12-2019, 06:11
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,289 Times in 463 Posts
Simorq is on a distinguished road
Test

Thanks 78372
Code:
[razor]
Encode = RZ a -d 32m packedfile.pcf datafile
Decode = RZ e packedfile.pcf datafile
InFileEnc = datafile
OutFileEnc = packedfile.pcf
InFileDec = packedfile.pcf
OutFileDec = datafile
Code:
[External compressor:precomp,mpzapi,zlib,razor]
header = 0
packcmd   = PMT.exe e {compressor} -b16M -t100p - - <stdin> <stdout>
unpackcmd = PMT.exe d {compressor} -t4 - - <stdin> <stdout>
Code:
Compressed 8 files, 466,568,427 => 58,679,682 bytes. Ratio 12.58%
Compression time: cpu 0.38 sec/real 17.54 sec = 2%. Speed 26.60 mB/s
All OK
Code:
Tested 8 files, 58,679,682 => 466,568,427 bytes. Ratio 12.58%
Testing time: cpu 0.45 sec/real 1.55 sec = 29%. Speed 300.13 mB/s
All OK
It's great.
Reply With Quote
The Following 9 Users Say Thank You to Simorq For This Useful Post:
78372 (24-12-2019), Behnam2018 (11-01-2020), Carldric Clement (26-12-2019), ffmla (24-12-2019), KaktoR (24-12-2019), PsYcHo_RaGE (24-12-2019), sanekbest1 (25-12-2019), shazzla (24-12-2019), ZAZA4EVER (17-03-2020)
  #3  
Old 24-12-2019, 06:56
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
^^.pcf isn't needed here, that was for precomp which gives
pcf as output. And -d 32m doesn't make sense with 16m blocks

Last edited by 78372; 24-12-2019 at 07:28.
Reply With Quote
The Following 2 Users Say Thank You to 78372 For This Useful Post:
Behnam2018 (11-01-2020), Simorq (24-12-2019)
  #4  
Old 24-12-2019, 15:47
KaktoR's Avatar
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 3,551
Thanks: 932
Thanked 5,795 Times in 2,137 Posts
KaktoR is on a distinguished road
Ohh, okay. This seems to be interresting. I will test it out next year if I have a bit more time

Thank you for all your stuff here until now!
__________________
Haters gonna hate
Reply With Quote
The Following 2 Users Say Thank You to KaktoR For This Useful Post:
78372 (24-12-2019), Behnam2018 (11-01-2020)
  #5  
Old 24-12-2019, 23:57
dixen dixen is offline
Registered User
 
Join Date: Sep 2014
Location: Russia
Posts: 286
Thanks: 350
Thanked 327 Times in 148 Posts
dixen is on a distinguished road
Nice Work! Thanks.

Last edited by dixen; 25-12-2019 at 00:42.
Reply With Quote
The Following 2 Users Say Thank You to dixen For This Useful Post:
78372 (25-12-2019), Behnam2018 (17-01-2020)
  #6  
Old 25-12-2019, 07:30
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 189
Thanks: 16
Thanked 227 Times in 122 Posts
FitGirl is on a distinguished road
Thanks for the Christmas gift, it's working as intended! Will do more tests.
Reply With Quote
The Following 2 Users Say Thank You to FitGirl For This Useful Post:
78372 (25-12-2019), Behnam2018 (17-01-2020)
  #7  
Old 25-12-2019, 09:56
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
@Everyone.

Release 2 Uploaded. Changed the whole INI setup and made it as easy as fa. I wonder why I didn't make it before. Check first post for changes, settings and download.
Reply With Quote
The Following 2 Users Say Thank You to 78372 For This Useful Post:
DiCaPrIo (25-12-2019), Simorq (25-12-2019)
  #8  
Old 25-12-2019, 10:34
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 221
Thanks: 408
Thanked 86 Times in 63 Posts
shazzla is on a distinguished road
What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?
Reply With Quote
  #9  
Old 25-12-2019, 10:37
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
Quote:
Originally Posted by shazzla View Post
What if only CLS is available for unpacking? In this case how can i use PMT? Is it possible somehow?
If you have cls for both packing and unpacking then perhaps there is a way to use PMT.
Reply With Quote
The Following User Says Thank You to 78372 For This Useful Post:
shazzla (25-12-2019)
  #10  
Old 25-12-2019, 13:12
DiCaPrIo DiCaPrIo is offline
Registered User
 
Join Date: Apr 2017
Location: Don't Know
Posts: 50
Thanks: 90
Thanked 49 Times in 30 Posts
DiCaPrIo is on a distinguished road
anyway to make bpk mt I tested it's not working
Reply With Quote
  #11  
Old 25-12-2019, 18:04
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 189
Thanks: 16
Thanked 227 Times in 122 Posts
FitGirl is on a distinguished road
Quote:
Originally Posted by DiCaPrIo View Post
anyway to make bpk mt I tested it's not working
Of course it won't work, bpk can deal with complete bink files only, not trimmed by this tool.
Reply With Quote
The Following 4 Users Say Thank You to FitGirl For This Useful Post:
78372 (25-12-2019), DiCaPrIo (26-12-2019), L0v3craft (09-06-2020), shazzla (25-12-2019)
  #12  
Old 25-12-2019, 19:09
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
If you have a cls for both compression and decompression, you can use wrapcls.exe with PMT to use the cls. Here's an example of PMT.ini:

Quote:
[lz4]
Encode = wrapcls.exe lz4 <stdin> <stdout>
Decode = wrapcls.exe d <stdin> <stdout>
Wrapcls is basically Compressor 2.2 by Bulat Ziganshin.So it can be used for other freearc algorithms too. I have just recompiled the program to rename cls-*.dll to arc-*.dll so that it doesn't create conflict while using with freearc. You need to rename your cls too to use this program.
Attached Files
File Type: rar wrapcls_example.rar (549.3 KB, 92 views)
Reply With Quote
The Following 2 Users Say Thank You to 78372 For This Useful Post:
shazzla (25-12-2019), Simorq (26-12-2019)
  #13  
Old 26-12-2019, 05:21
FitGirl FitGirl is offline
Registered User
 
Join Date: Dec 2014
Location: Riga
Posts: 189
Thanks: 16
Thanked 227 Times in 122 Posts
FitGirl is on a distinguished road
Quote:
Originally Posted by Simorq View Post
@78372
Does not work with CLS-BPK.
C'mon, I have already answered to this. PMT is a BLOCK-BASED TRIMMER. IT SPLITS FILES INTO CHUNKS. BPK is a FILE-BASED ENCODER. It deals only with full files. PMT split bink files into chunks and they CANNOT BE COMPRESSED with BPK anymore. "L" for logic.

It's intended for codecs which rely on pure stream data - generic compressors, precompressors, MPZ (cause it deals with frames in MP3, not full files).

Last edited by FitGirl; 26-12-2019 at 06:36.
Reply With Quote
The Following 4 Users Say Thank You to FitGirl For This Useful Post:
78372 (26-12-2019), DiCaPrIo (26-12-2019), shazzla (26-12-2019), Simorq (26-12-2019)
  #14  
Old 26-12-2019, 06:40
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
@Simorq,
I think FitGirl has already answered your question. In addition, let me describe how PMT works.
PMT splits your input as blocksize you have specified. If your encoder doesn't support stdio, it makes temporary files and runs your encoder over it. It requires disk space for that. If your encoder supports stdio, then it uses memory for storing blocks. Basically the additional memory you need here is blocksize * threads + encoded size * threads. Same goes for non-stdio but it takes disk space and temp files then.
PMT can't work without extra memory for stdio as PMT itself supports stdio, and without using blocks, both stdio support for encoding+decoding is not possible. Also, even without stdio, running a lot of instances parallelly and writing to same output isn't possible. So either way it requires extra memory or temp files. As the program requires blocks, encoders/compressors that require full files(such as bpk or srep) won't work perfectly/won't work at all.

I think this answers and will answer to future questions regarding PMT.
Reply With Quote
The Following User Says Thank You to 78372 For This Useful Post:
Simorq (26-12-2019)
  #15  
Old 28-12-2019, 09:02
Entai44 Entai44 is offline
Registered User
 
Join Date: Nov 2018
Location: Indonesia
Posts: 49
Thanks: 72
Thanked 29 Times in 18 Posts
Entai44 is on a distinguished road
Is this working with packet compressor? I can compress with UltraARC, but decompression doesn't work. This is my configuration:

Quote:
[External compressoracket]
packcmd = packetx64.exe a -v -r -mx -b512 -h4 $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = packetx64.exe x -v $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
datafile = $$arcdatafile$$.tmp
packedfile = $$arcpackedfile$$.tmp
it shows error when decompressing, like "unknown compression method", how to fix this problem?

Thanks btw for PMT

Last edited by Entai44; 28-12-2019 at 09:05.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
plzip_modded - parallel 64bit lzma (de)compressor and a direct 4x4:lzma replacement elit Conversion Tutorials 17 26-03-2018 07:08
Driver Parallel Lines cdkiller PC Games - Protection List 0 03-07-2007 04:16
DRIVER Parallel Lines - savegame needed speedfreak General Gaming 0 30-06-2007 19:49
Driver: Parallel Lines (Multi lang to 1 lang) mondragon PC Games - CD/DVD Conversions 0 28-06-2007 10:03
Powerswitch thru USB or Parallel port EMPiRE Hardware 1 29-03-2006 04:38



All times are GMT -7. The time now is 16:30.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com