CLS-PrecompMT (Multi tasking) [Archive]

Razor12911

10-06-2016, 06:03

Hello guys, again.

Made another CLS based on the one I did about a week ago.

Which you can find here:
http://fileforums.com/showthread.php?t=97938

What I did this time is made the cls launches parallel instances of precomp.exe to speed up the entire process.

Notes:
During compression, it launches the same number of instances as the number of threads your processor has and during decompression, it launches half that number.

cls name being cls-precompmt.dll, does not mean the executables must also be precompmt.exe and so on, I just renamed the cls but maintained its usage to precomp with the filenames precomp.exe and precomp64.exe for 64-bit.

precomp versions supported:
0.38-0.46

Enjoy

ChronoCross

10-06-2016, 06:38

Nice razor. Very clever.

Andrey167

10-06-2016, 06:56

work on 2 cores,and dont work in 4 threads(Hyper Threading)

Razor12911

10-06-2016, 07:04

Hmm weird, did you read the part that says during decompression, it uses half of what you have, unless you're saying with reference to compression step

Mini

10-06-2016, 07:39

thx! why it is impossible to choose how many streams to use? it would be convenient

coveiro

10-06-2016, 08:06

Very good, Thanks Razor12911

http://i.imgsafe.org/ad758a1f7c.png

felice2011

10-06-2016, 08:58

Great Speed ... Bro !!!

http://i20.photobucket.com/albums/b249/ElNinoSouthPole/BabeGifs/Selena%20Gomez/Thumbs-Up.gif

mikey26

10-06-2016, 09:54

nice work razor really awesome stuff. i think if guys have issues or problems dont say its not working.State your processor in detailed spec makes life easier for Razor to fix.

ok 8 threads working like a boss.Even with Reflate running :D
8 Temp folders created.

I7 4700MQ CPU Haswell
2.4GHZ turbo Boost to 3.2GHZ

felice2011

10-06-2016, 13:03

@mikey26 It works in parallel, it is as if at each thread would create a precomp execution, the overall size of the folders in mt is equal to the single temporary folder you created for the normal execution precomp.:)

Prince4

10-06-2016, 23:24

Pretty something that every one needed, Bravo Brother Razor
So now we can get MT Compressing so then can we also get MT Decompression? If possible then How?:)

mikey26

11-06-2016, 11:50

@felice2011 : i know that bro kinda hard not to miss or see that :P

@Prince4 : check wat razor mentioned in the start of the thread :
Notes:
During compression, it launches the same number of instances as the number of threads your processor has and during decompression, it launches half that number.

Mini

13-06-2016, 01:33

Razor12911, at a decompression it was possible to make so that 3\4 and 7\8 kernels were used, but not a half, what does you think? lay out please such version :)

Razor12911

01-11-2016, 08:35

Update uploaded

Added options for thread count and brute. (Not tested) I simply have no time, plus I'm not really sure why I'm doing this.

Mini

01-11-2016, 09:53

wow, big thanks Razor12911!!!

(Not tested)
everything well works

aswadd

01-11-2016, 10:21

Before Update :
everything is ok
After the update with brute 0 & threads works fine :
everything is ok

Sorry I forgot to put brute to zero :D now everything works fine thnx bro Razor & don't care about anyone who get your work & edit on it or use it without ur permission :) Just do it & make sure nothing you do will waste

LuzhinE

01-11-2016, 12:21

aswadd

because it changes from 1 to 0.
cls-precompmt.ini
[CLS]
Threads = 1
Brute = 1

if its value is 1 prederterminado

aswadd

01-11-2016, 13:30

aswadd

because it changes from 1 to 0.
cls-precompmt.ini
[CLS]
Threads = 1
Brute = 1

if its value is 1 prederterminado

who wants to use brute ?? it will talk all your live to complete .. I just Forgot to put it to Zero :D so it seemed to me like stuck

tanerjames

05-11-2016, 15:34

Hi,

Error with Unarc.dll:
Unarc.dll returned an error code: -1
ERROR: archive data corrupted (decomporession fails)

How to use inno setup and unarc.dll?

Thanks.

chum

28-11-2016, 22:33

When you say it support Precomp from 0.38 to 0.46 do you mean for compressing and decompression? I tried using 0.38 and 0.42 but none of them worked for compressing, only default 0.45 works.

LuzhinE

01-12-2016, 17:33

CAT8K

02-12-2016, 02:06

Tested in max payne 3 complete edition

precompmt+srep+lzma2
Game size = 32 GB
Size of Repack = 18.3 GB
Installation time 50 min

divided in 2 Setup.bin
*
Setup1.bin
The rest of the game folders

Setup2.bin
The movies folder is not compressed

For me the best option, since precomp 0.42 takes to install the game in 2 hours and compresses it up to 16 GB

i done from Max Payne 3 Complete Edition last ver 1.0.0.196

pzlib+srep+lzma

Game size = 32.5 GB
Size of Repack = 16.6 GB
Installation time 55 min on my cpu 8 Core amd

do this if you are interested in reducing Size

LuzhinE

02-12-2016, 05:07

I only have 4 threads, so for me it is double the installation with pzlib

CAT8K
Maybe if you use precompmt the installation for you will be 25 min thanks to its 8 threads

Simorq

14-12-2016, 09:31

Batman Arkham Knight 52.0 GB TO 31.0 GB precompmt+Srep+Lzma2:D
http://uupload.ir/files/w3gv_cp.png
http://uupload.ir/files/qxmn_dp.png

LuzhinE

14-12-2016, 12:36

Thanks for the test

Simorq

15-12-2016, 13:13

Hi Razor12911
cls-precompmt_update! Automatic threads your processor has and during decompression, it launches half that number?

Razor12911

16-12-2016, 06:18

Update2 uploaded

Changes

Automatic threads is half during compression and decompression
Added thread percentage option
Added optional work directory

Mini

16-12-2016, 06:43

Added optional work directory
{app} or how?

Razor12911

16-12-2016, 11:28

{app} is an inno constant, cls doesn't know what it is, so you have to use setinistring in a script to pass info to cls configuration just before you start unpacking archives.

aswadd

16-12-2016, 11:42

Now, that's the talk :D thnx Razor For the Update

vint56

19-12-2016, 22:52

Razor12911 I am sorry for my bad english
Installer crash when using CLS-PrecompMT when Cancel installation
I did so added to isdone 6 Finale

#define Precompmt
[Setup]

[Files]
#ifdef Precompmt
Source: Include\PrecompMt\cls-precompmt.dll; DestDir: {tmp}; Flags: dontcopy
Source: Include\PrecompMt\precomp.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\PrecompMt\precomp64.exe; DestDir: {tmp}; Flags: dontcopy
#endif

procedure CurStepChanged(CurStep: TSetupStep);
#ifdef Precompmt
SaveStringToFile(ExpandConstant('{tmp}\cls-precompmt.ini'), '[CLS]'+#13#10+'Threads=75%'+#13#10+'Brute=0'+#13#10+'Te mp='+ExpandConstant('{app}'), True);
ExtractTemporaryFile('cls-precompmt.dll');
ExtractTemporaryFile('precomp.exe');
ExtractTemporaryFile('precomp64.exe');
#endif

if not ISArcExtract ( 0, 0, ExpandConstant('{src}\data-1.bin'), ExpandConstant('{app}\'), '', false, '', ExpandConstant('{tmp}\cls-precompmt.ini'), ExpandConstant('{app}\'), notPCFonFLY{PCFonFLY}) then break;

http://s016.radikal.ru/i334/1612/ba/97f82ac9b10e.jpg (http://radikal.ru)

and if you do so then there is no error
if not ISExec( 0, 0, 0, ExpandConstant('{tmp}\unarc.exe'), ExpandConstant('x -o+ -cfg"{tmp}\cls-precompmt.ini" -w"{app}" -pPassword -dp"{app}" "{src}\data-1.bin"'), ExpandConstant('{tmp}'), '...',false) then break;

ZakirAhmad

26-02-2017, 09:16

please help me...
while unpacking i am getting unarc.dll has returned with error code 1.. arc data corrupted... I included precomp.exe, cls-precompmt.dll and cls-precompmt.ini in wpi script...

ZakirAhmad

27-02-2017, 02:15

The problem is only with scripts... cmd extractor works good...

vint56

27-02-2017, 04:27

example on ISDone0.6final

#define Precompmt
[Setup]

[Files]
#ifdef Precompmt
Source: Include\PrecompMt\cls-precompmt.dll; DestDir: {tmp}; Flags: dontcopy
Source: Include\PrecompMt\precomp.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\PrecompMt\precomp64.exe; DestDir: {tmp}; Flags: dontcopy
#endif

[Code]
var
i:integer;

procedure CancelButtonOnClick(Sender: TObject);
begin
SuspendProc;
if MsgBox(SetupMessage(msgExitSetupMessage), mbConfirmation, MB_YESNO) = IDYES then ISDoneCancel:=1;
#ifdef Precompmt
if not IsWin64 then begin
Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM Precomp.exe', '', SW_HIDE, ewWaitUntilTerminated, i);
end else begin
Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM Precomp64.exe', '', SW_HIDE, ewWaitUntilTerminated, i);
DelTree(ExpandConstant('{app}\'), True, True, True);
end;
#endif
ResumeProc;
end;

#ifdef Precompmt
SaveStringToFile(ExpandConstant('{tmp}\cls-precompmt.ini'), '[CLS]'+#13#10+'Threads=75%'+#13#10+'Brute=0'+#13#10+'Te mp='+ExpandConstant('{tmp}'), True);
ExtractTemporaryFile('cls-precompmt.dll');
ExtractTemporaryFile('precomp.exe');
ExtractTemporaryFile('precomp64.exe');
#endif

if not ISArcExtract ( 0, 0, ExpandConstant('{src}\data-1.arc'), ExpandConstant('{app}\'), '', false, 'pass', ExpandConstant('{tmp}\cls-precompmt.ini'), ExpandConstant('{app}'), notPCFonFLY{PCFonFLY}) then break;

Prince4

27-02-2017, 05:49

Braw why am i still getting

Could not get the file size of null00

with whatever version of pecomp i use and also the CLS this isn't working too..

Plz a little help?

There are about 3 games that i need to put in to DVD5s that have a lot of ZLibs in them :)

ZakirAhmad

27-02-2017, 05:57

please bro i have no experience in programming... can u please give me modified version of wpi which supports precompmt...

Simorq

27-02-2017, 06:21

example on isdone0.6final

#define precompmt
[setup]

[files]
#ifdef precompmt
source: Include\precompmt\cls-precompmt.dll; destdir: {tmp}; flags: Dontcopy
source: Include\precompmt\precomp.exe; destdir: {tmp}; flags: Dontcopy
source: Include\precompmt\precomp64.exe; destdir: {tmp}; flags: Dontcopy
#endif

[code]
var
il:integer;

procedure cancelbuttononclick(sender: Tobject);
begin
suspendproc;
if msgbox(setupmessage(msgexitsetupmessage), mbconfirmation, mb_yesno) = idyes then isdonecancel:=1;
#ifdef precompmt
if not iswin64 then begin
exec(expandconstant('{sys}\taskkill.exe'), ' /f /im precomp.exe', '', sw_hide, ewwaituntilterminated, i);
end else begin
exec(expandconstant('{sys}\taskkill.exe'), ' /f /im precomp64.exe', '', sw_hide, ewwaituntilterminated, i);
deltree(expandconstant('{app}\'), true, true, true);
end;
#endif
resumeproc;
end;

#ifdef precompmt
savestringtofile(expandconstant('{tmp}\cls-precompmt.ini'), '[cls]'+#13#10+'threads=75%'+#13#10+'brute=0'+#13#10+'te mp='+expandconstant('{app}'), true);
extracttemporaryfile('cls-precompmt.dll');
extracttemporaryfile('precomp.exe');
extracttemporaryfile('precomp64.exe');
#endif

if not isarcextract ( 0, 0, expandconstant('{src}\data-1.arc'), expandconstant('{app}\'), '', false, 'pass', expandconstant('{tmp}\arc.ini'), expandconstant('{app}'), notpcfonfly{pcfonfly}) then break;

ciu:(

ZakirAhmad

27-02-2017, 06:51

I said bro please provide wpi script which extracts precompmt. .. please please please...

vint56

27-02-2017, 09:16

ZakirAhmad

27-02-2017, 09:20

@vint56

thanks bro... is it working

ZakirAhmad

27-02-2017, 21:06

@vint56
This really helped but since i am using modified wpi script with archive.ini... i was not able to modify this using ur example script... please modify for me this script... thanks in advance...

ZakirAhmad

27-02-2017, 22:58

please any one reply...

vint56

27-02-2017, 23:06

ZakirAhmad

27-02-2017, 23:34

@vint56...
thanks bro,it worked... while compressing with precomp045 i am getting error arc.exe has stopped working on a 2gb file... while precomp044 works good... what am i doing wrong...

vint56

27-02-2017, 23:55

ZakirAhmad I do not know what kind of bug you but try to add it precomp -intense -cn -t-j
arc.ini
packcmd = precomp -intense -cn -t-j -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp

ZakirAhmad

28-02-2017, 01:24

Thanks... the modified script u uploaded for me has uninstall problemm... please correct... thanks alot...

-XCX-

28-02-2017, 01:30

Thanks... the modified script u uploaded for me has uninstall problemm... please correct... thanks alot...

add these files to your script
Source: Resources\Dark.png; DestDir: {app}\Uninstall; Attribs: hidden system;
Source: Resources\Light.png; DestDir: {app}\Uninstall; Attribs: hidden system;
Source: Graphics\Images\Setup1.jpg; DestDir: {app}\Uninstall; Attribs: hidden system;
Source: Resources\botva2.dll; DestDir: {app}\Uninstall; Attribs: hidden system;

ZakirAhmad

28-02-2017, 02:45

@-XCX

Before adding support for cls-precompmt.dll, everything was working great...

-XCX-

28-02-2017, 03:16

Dude Give Me Your Script I will solve your problem

ZakirAhmad

28-02-2017, 03:55

here it is

http://fileforums.com/attachment.php?attachmentid=17450&d=1488265586

-XCX-

28-02-2017, 05:26

Just Remove This Line
SaveStringToFile(ExpandConstant('{tmp}\cls-precompmt.ini'), '[CLS]'+#13#10+'Threads=75%'+#13#10+'Brute=0'+#13#10+'Te mp='+ExpandConstant('{tmp}'), True);
And It Will Work

-XCX-

28-02-2017, 05:29

if you get uninstallation problem in your script then add this files
Source: Resources\Dark.png; DestDir: {app}; Attribs: hidden system;
Source: Resources\Light.png; DestDir: {app}; Attribs: hidden system;
Source: Graphics\Images\Setup1.jpg; DestDir: {app}; Attribs: hidden system;
Source: Resources\botva2.dll; DestDir: {app}; Attribs: hidden system;

ZakirAhmad

28-02-2017, 07:02

THANKS FOR HELP @EVERYONE

uninstall problem solved...
but i am not able to extract files compressed with precompmt+pzlib+srep+lzma...

-XCX-

28-02-2017, 07:06

did you this files to your script
ExtractTemporaryFile('Pzlib.exe');
ExtractTemporaryFile('arc.ini');
ExtractTemporaryFile('hif2raw_dll.dll');
ExtractTemporaryFile('raw2hif_dll.dll');

ZakirAhmad

28-02-2017, 07:20

with out precommt everything worked... above in a post of mine is a file.... please take a look on it... thanks alot...

ZakirAhmad

28-02-2017, 07:56

with out precommt everything worked... above in a post of mine is a file.... please take a look on it... thanks alot...

ZakirAhmad

01-03-2017, 02:38

Can please any body post a compressor with working precompmt support... I tried everything to get through the error but I am still getting error arc.exe has stopped working... I tried it on 5gb folder files.... at about 4.5% i am getting the error...

Simorq

05-03-2017, 07:24

Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM pZLib.exe', '', SW_HIDE, ewNoWait, ISDoneCancel);
Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM Precomp.exe', '', SW_HIDE, ewWaitUntilTerminated, ISDoneCancel);
Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM Precomp64.exe', '', SW_HIDE, ewWaitUntilTerminated, ISDoneCancel);
DelTree(ExpandConstant('{app}\'), True, True, True);
:D

Chayan Manna

06-03-2017, 07:39

How can I get first release? I deleted it.

Chayan Manna

11-03-2017, 20:14

Please someone give me the first release. That's the full 7z file provided by razor12911.
Please someone give.......

L0v3craft

12-03-2017, 06:17

Hi guys, someone here can help me to get it working ?

I'm using:

- precomp/precomp64.exe (0.45 alpha version)

- cls-precompmt.dll + cls-precompmt.ini are in the same folder of precomp exe files

- the command that I'm trying is this(in my arc.ini file):

ultral24x4 = msc_frog+srep+delta+4x4:lzma:max/$void=srep+zstd/$deflate=precomp+srep+delta+4x4:lzma:max

[External compressor:precomp]
header = 0
packcmd = precomp64 -v -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = precomp64 -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

And in the task manager i can see only one precomp64.exe

Why it is not working in mt for me ? :(

ZakirAhmad

12-03-2017, 08:01

rinaldo

12-03-2017, 10:06

L0v3craft
pmt

L0v3craft

12-03-2017, 10:42

@L0v3craft
precompmt doesnt required arc.ini
include in compressor precomp64.exe precomp.exe cls-precompmt.dll cls-precompmt.ini
in precompmt.ini edit Temp=./ and
writr command for compression as -mprecompmt+srep+lzma...
also if source file is below 64mb only 1 thread will be used. if above64mb 2 and so on...

Thanks but doesn't work with the compressor that I'm using. Probably something is wrong in the arc.ini file.

pmt

Your compressor works but I want use my compressor.

I'm using this:

cut

can you make it working for me ? Thanks in advance.

rinaldo

12-03-2017, 10:48

L0v3craft

12-03-2017, 13:17

the mask has nothing to do with precompmt, just not to go out of the question ;)

to use mask with pmt, you just have to adapt the files for compression
metodo1,2,3,4,5 etc., then you have to adapt to your anthem scrip is simple but takes time for tests

Thanks but I solved my problem in a simple way:

I edited my arc.ini:

- I removed this part of the code:

[External compressor:precomp]
header = 0
packcmd = precomp64 -intense -v -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = precomp64 -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

- And changed this:

ultral24x4 = msc_frog+srep+delta+4x4:lzma:max/$void=srep+zstd/$deflate=precomp+srep+delta+4x4:lzma:max

with this

ultral24x4 = msc_frog+srep+delta+4x4:lzma:max/$void=srep+zstd/$deflate=precompmt+srep+delta+4x4:lzma:max

And all is working now :) But I don't understand the difference from "-mprecompmt" and "-precompmt". Is there some difference ?

rinaldo

12-03-2017, 13:30

o yes code error

[External compressor:precomp]
header = 0
packcmd = precomp64 -intense -v -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = precomp64 -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

correct is
[External compressor:precomp64]
header = 0
packcmd = precomp64 -intense -v -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = precomp64 -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

chum

03-04-2017, 07:59

Does precompmt ignore commands add in arc.ini? Or it only missing the option to disable MP3?

Schnaader say -t-3 disable MP3 compression, when i use it in Precomp045 that works fine but in precompmt it just ignoring this command. I need to disable MP3 compression because it cause error to my files.

Is there some solution Razor?

github

02-02-2021, 16:19

Ele

02-02-2021, 17:31

Unfortunately it doesn't take the -t-+ options

I guess BLACKFIRE69's cls-precomp.dll has this facility.
You can add your custom options into,

cls-precomp.ini >> [Precomp] >> Options= -cn -brute -t-+ ......

github

02-02-2021, 20:46

Thanks, that one's working however it will force creating all the output files to where the .dll or arc.exe is, including the compressed file, the temp file folders as well as the log file.

And it seems when several precomp running in parallel to split the task, it will expose a problem I found:

Some zip, 7z files , when only a few files going to be compressed, they will fail the decompress on CRC checks. When they are included in a larger set of files, they will pass the decompress.
And with a large set of files, precomp running in parallel , they will fail to decompress because of CRC errors. I guess the reason is, when those zip, 7z files mixed with large set of files, they work fine. When precomp running in parallel , the whole file set been splitted into multiple sections, and the section doesn't have enough files to save those kids.

The zip,7z files have problems are when you try to run this command

precomp.exe -cn -o1.pcf 121.zip

against them:

None of the given compression and memory levels could be used.
There will be no gain compressing the output file.

I'll try -cl or -cb with precomp's builtin compressing;;