Helping thread :)

[FitGirl]

Bulat, let me thank you for great tools you've created. Without them most repacks would be impossible, I guess

Now to topics you've mentioned.

1) Yes, now I'm aware of it. Could you please clarify if decompression threads number can be controlled by setting more dmem? Let's say I compress with 4x4:lzma256, what should I do, so users with 8-cores and GBs of RAM could decompress it in all their threads. I specifically put higher dictionary sizes here, cause smaller ones aren't good for compression ratio.

2) Yes, I've been talking about mode without blocks splitting and for sure I use it only on data with lots of already compressed data.

4) That would be cool, if it make decompression faster

Thank you again!

[Bulat]

Yesterday i had very productive debuggng seesion with MCWRX and he helped me to understand why srep+4x4:lzma is hard to master:

InsertTempfile procedure i already mentioned is fired in unarc.dll to adopt decompression algorithms to the real memory available. But on decompression stage, we don't have too much choices - eithe insert "tempfile" between methods and save intermedirte data to the tempfile or limit 4x4 to run less threads simultaneously

With built-in algos, it works fine (modulo bugs), but for external algos including srep there is no any API to limit their memory usage! so, many years ago i developed alternative approach specially to srep - it can be parameterized like srep:mem75%-600mb and this way it will alloc 75% of RAM minus 600 MB. It was supposed that you have compression line like srep+delta+lzma:512m, computed memory usage by remaining algorithms and make appropriate srep setting

It was fine for all built-in algos except for 4x4, whose decompression memory depends on the num_cores and thus cannot be predicted at compression time. So, we just don't have working instrument for limiting srep+4x4 by using only MY tools

Nevertheless, in MCWRX installer memory usage for srep is set by the installer script. I don't looked into details, but obviously community provided right instrument to deal with srep+4x4 case. Unfortunately, it needs some programming skills to manage - you need to find num_cores, compute memory usage of remaining algos, find RAM size, and then compute memory available for srep. And if it becomes too small (say, less than 500 MB) i recommend to limit number of 4x4 threads (by modifying compression string replacing 4x4 with 4x4:t4 or so) in order to prevent excessive disk swapping in srep

Yet another solution will be improvement of :mem language, adding support for smth like mem75%-100m-132m*t

And ideal solution will be implementation of srep as builtin method or just 64-bit unarc.dll. the second way looks much simpler so i will try

[Bulat]

Originally Posted by FitGirl

1) Yes, now I'm aware of it. Could you please clarify if decompression threads number can be controlled by setting more dmem? Let's say I compress with 4x4:lzma256, what should I do, so users with 8-cores and GBs of RAM could decompress it in all their threads. I specifically put higher dictionary sizes here, cause smaller ones aren't good for compression ratio.

as you can see in arcinfo/lt, each thread of 4x4:lzma:256m will require 516 MB of memory, 8 threads need more than 4 GB. the only way is to use external 4x4, i.e. fazip in stdin-to-stdout mode

please try it and write about results - positive or not

[ImmortalArc]

Can anyone help me out why all the time srep decompression fails and it says write error disk full?? Why??? Searched forum but nothing helps compressed whole 12 gb in 2 hour but happens again and again I am using -m3f with pzlib and lzma.

[Gupta]

Firstly, always remember for decompression u should have CLS-SREP.dll in the same folder where the arc.exe is.
If u got CLS-Srep.dll which is 86 Kb or 54 kb or something which is <100 kb then remember those only works during decompression if they are in the same folder where the arc.exe is during COMPRESSION then the arc simply throw disk full error or something other error
the above CLS-SREP.dll, works during compression too.

[ImmortalArc]

Thanks PrinceGupta will try it.
Do you know a good GUI version for BMS scripts to open game archives for rebuilding and referring data from them such as music and graphics...

[doofoo24]

try use
arc a -lc2000 -ed -ep1 -r -ma -md -mx -s; -w.\temp -mpzlib+srep+lzma:a1:mfbt4:d185m:fb273:mc10000:lc8 data.arc "game location"
and use srep 3.2 and freearc 0.67 date 15/3/2014 and in arc.ini use

[External compressor:srep64]
header = 0
packcmd = srep64 -m3f -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

[External compressorzlib]
header = 0
packcmd = pZLib e -m2 -c16m -b256k -t2 - -o - <stdin> <stdout>


and for decompress try to use cls-srep 3.3 by profargo with pzlip

and in arc.ini

[External compressorzlib]
header = 0
unpackcmd = pzlib d -t50p - -o - <stdin> <stdout>

try to use isdone original script with inno 5.5.9...

[ImmortalArc]

Srep 3.3 but I have 3.93 why older is better???

[doofoo24]

try ? form the original site on freearc by Bulat Ziganshin...
download freearc 0.67 alpha portable and srep 3.2
i found a couple of srep exe file infected.
use only the original file.
i found to be more reliable and virus free...

[doofoo24]

try to use in inno where ExtractTemporaryFile
ExtractTemporaryFile('unarc.dll');
ExtractTemporaryFile('arc.ini');
ExtractTemporaryFile('pzlib.exe');
ExtractTemporaryFile('cls-srep.dll');

and in ExpandConstan try

if not SrepInit(ExpandConstant('{app}\'),128,0) then break;
if not FileSearchInit(true) then break;
if not ISArcExtract ( 0, 100, ExpandConstant('{src}\file.arc'), ExpandConstant('{app}'), '', false, '', ExpandConstant('{tmp}\arc.ini'), ExpandConstant('{app}'), False) then break;
ISDoneError:=false;

[ImmortalArc]

But it says it is created by bulat.....
Or I am using wrong one

[doofoo24]

i am trying to help not to argue is easy to modify file attribute...
so please try first if it work great if not then ask why ?

[ImmortalArc]

Originally Posted by PrinceGupta2000

Firstly, always remember for decompression u should have CLS-SREP.dll in the same folder where the arc.exe is.
If u got CLS-Srep.dll which is 86 Kb or 54 kb or something which is <100 kb then remember those only works during decompression if they are in the same folder where the arc.exe is during COMPRESSION then the arc simply throw disk full error or something other error
the above CLS-SREP.dll, works during compression too.

I used cls-srep.dll by razor 12911 size is 979kb but when I decompress which is compressed with pzlib+srep+lzma it continuing decompress after 100% original data is about 2.8 gb (before compression) and it now continue extracting data more than 5 gb why ??

Edit: I compressed Twp folder data and update data have 1953 mb and update have 300 mb before compression and after it have 1.18 gb pzlib+srep+lzma

[Bulat]

look at http://freearc.org/Research.aspx

it links TWO srep pages, one with stable version (latest 3.2) and one with experimental version (latest 3.93). use 3.93 and check that your installers work

OTOH, you may use 3.2 if you wish. with maximum compression settings it's no different to 3.2 except for faster _compression_

but i don't provide DLLs, so the best approach imho is to use DLLs from trusted sources such as one by Razor

[doofoo24]

bulat great in time if you manage to answer me will be great.
is there a way to get constant speed in decompression when using lzma:a1:mfbt4:d185m:fb273:mc10000:lc8 without using fastzip or xz only builtin lzma and without 4x4...
i montor the disk usage is about 20 mb/s is there a better way to get 50 or 60 mb/s.
thank in advance...