PMT - Parallel Multithreaded Encoder/Decoder - Page 4

78372 · #46 26-03-2020, 02:47

Quote:

Originally Posted by shazzla

In some rare case,it happens with me,too.. (for example,Quantum break ,1xx GB version)
Maybe a bug in Razor ?!

With Razor's patch, razor now supports stdio (What a pun

)
Try using that one to remove temps and probably that will fix some stuff.

shazzla · #47 26-03-2020, 03:33

I used the STDIO patched version ,without PMT ! (RZ V1.03.7)

KaktoR · #48 27-03-2020, 02:16

I have further tested with full games, it works like a charm and pretty fast

Code:

 1:36:00 - Starting Archive Creation. 
 2:54:33 - Archive Creation Complete. 
 2:54:33 - Sorting Storage Directories Started. 
 2:54:34 - Sorting Storage Directories Complete. 
 2:54:34 - Starting Archive Creation. 
 4:11:32 - Archive Creation Complete. 
 4:11:32 - Sorting Storage Directories Started. 
 4:11:32 - Sorting Storage Directories Complete. 
 4:11:32 - Starting Archive Creation. 
 9:56:22 - Archive Creation Complete. 
 9:56:22 - Sorting Storage Directories Started. 
 9:56:22 - Sorting Storage Directories Complete. 

10:28:36 - The Witcher Enhanced Edition Size: 13.67 GB. 
10:28:36 - The Witcher 2 Size: 17.21 GB. 
10:28:37 - The Witcher 3 Size: 42.99 GB. 
10:28:37 - Full Input Size: 73.89 GB. 
10:28:37 - Output Size: 45.63 GB.

Decode only possible with t2 or t1 for me, I don't know why

Tested with batch and CIU installer.

I used razor

78372 · #49 27-03-2020, 07:20

Seems like some internal issues with razor. Did you test with other compressors? How do they behave? Probably the fast speed of razor is causing some I/O bottleneck or something with multiple threads. Single thread is enough for razor to be frank with such lightning decompression speed.

KaktoR · #50 30-03-2020, 10:19

Not yet, but it's ok neverthless because rz is lightning fast in decompression

What other compressors do you suggest? What about zstd?

78372 · #51 30-03-2020, 11:13

ZSTD has mt and way too fast to use with PMT.
Try with slow tools.
BCM, precomp, rzm etc. can work better.
We have.alternatives for them though.

Cesar82 · #52 30-03-2020, 15:52

Has anyone done any tests using PMT + pZLib3?

If yes please share the values for PMT.ini and Arc.ini.
Thanks!

78372 · #53 30-03-2020, 22:38

Quote:

Originally Posted by Cesar82

Has anyone done any tests using PMT + pZLib3?

If yes please share the values for PMT.ini and Arc.ini.
Thanks!

What's the point? pZlib is already MT and fast. Moreover, you have xtool now.

Cesar82 · #54 30-03-2020, 23:15

@78372
Thanks for the information. I know little about compression methods.
So of the compressors used in CIU only RAZOR has benefits if used with PMT.

CIU supports these compressors in addition to some RAZOR tools for specific games.
SREP, ZTool, XTool(not v2019), pZLib3, UELR, AFR, OodleRec [4,5,6,7,8], MSC, MPZ, OGGRE, BPK, 7z, LOLZ, ZSTD, RAZOR

Does any of this list other than razor benefit if used with PMT?

78372 · #55 30-03-2020, 23:55

Quote:

Originally Posted by Cesar82

@78372
Thanks for the information. I know little about compression methods.
So of the compressors used in CIU only RAZOR has benefits if used with PMT.

CIU supports these compressors in addition to some RAZOR tools for specific games.
SREP, ZTool, XTool(not v2019), pZLib3, UELR, AFR, OodleRec [4,5,6,7,8], MSC, MPZ, OGGRE, BPK, 7z, LOLZ, ZSTD, RAZOR

Does any of this list other than razor benefit if used with PMT?

mpz only I guess, at least for now.

KaktoR · #56 31-03-2020, 02:32

Code:

mpz with PMT
Compressed 64 files, 209,434,083 => 180,728,551 bytes. Ratio 86.29%
Compression time: cpu 0.20 sec/real 38.46 sec = 1%. Speed 5.45 mB/s

Tested 64 files, 180,728,551 => 209,434,083 bytes. Ratio 86.29%
Testing time: cpu 0.19 sec/real 36.54 sec = 1%. Speed 5.73 mB/s

mpz only
Compressed 64 files, 209,434,083 => 180,670,089 bytes. Ratio 86.27%
Compression time: cpu 0.30 sec/real 103.93 sec = 0%. Speed 2.02 mB/s

Tested 64 files, 180,670,089 => 209,434,083 bytes. Ratio 86.27%
Testing time: cpu 0.36 sec/real 95.90 sec = 0%. Speed 2.18 mB/s

Tested with settings -b64m -t100p for encode/decode with a Ryzen 5 2600 (12 threads).

I just wonder why ratio is a little bit better without pmt

78372 · #57 31-03-2020, 20:57

The same reason for everything, chunking.

DiCaPrIo · #58 08-06-2020, 02:15

Quote:

Originally Posted by L0v3craft

Hi. Is possible to add bpk compatibility? Thanks.

see at first page

L0v3craft · #59 02-07-2020, 20:15

Is possible to add this?

Code:

https://github.com/encode84/bcm

it is BCM v1.5. If is not possible someone has a cls dll for BCM to make it multithreaded? Thanks.

78372 · #60 02-07-2020, 22:47

Quote:

Originally Posted by L0v3craft

Is possible to add this?

Code:

https://github.com/encode84/bcm

it is BCM v1.5. If is not possible someone has a cls dll for BCM to make it multithreaded? Thanks.

BCM 1.4 works with this. I will make a stdio patch for 1.5 when I have time.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
plzip_modded - parallel 64bit lzma (de)compressor and a direct 4x4:lzma replacement	elit	Conversion Tutorials	17	26-03-2018 07:08
Driver Parallel Lines	cdkiller	PC Games - Protection List	0	03-07-2007 04:16
DRIVER Parallel Lines - savegame needed	speedfreak	General Gaming	0	30-06-2007 19:49
Driver: Parallel Lines (Multi lang to 1 lang)	mondragon	PC Games - CD/DVD Conversions	0	28-06-2007 10:03
Powerswitch thru USB or Parallel port	EMPiRE	Hardware	1	29-03-2006 04:38

The Following User Says Thank You to KaktoR For This Useful Post:
78372 (27-03-2020)

The Following User Says Thank You to 78372 For This Useful Post:
KaktoR (30-03-2020)

The Following User Says Thank You to KaktoR For This Useful Post:
78372 (31-03-2020)

The Following User Says Thank You to 78372 For This Useful Post:
L0v3craft (02-07-2020)