Bulat, let me thank you for great tools you've created. Without them most repacks would be impossible, I guess :)

Now to topics you've mentioned.

1) Yes, now I'm aware of it. Could you please clarify if decompression threads number can be controlled by setting more dmem? Let's say I compress with 4x4:lzma256, what should I do, so users with 8-cores and GBs of RAM could decompress it in all their threads. I specifically put higher dictionary sizes here, cause smaller ones aren't good for compression ratio.

2) Yes, I've been talking about mode without blocks splitting and for sure I use it only on data with lots of already compressed data.

4) That would be cool, if it make decompression faster :)

Thank you again!

Yesterday i had very productive debuggng seesion with MCWRX and he helped me to understand why srep+4x4:lzma is hard to master:

InsertTempfile procedure i already mentioned is fired in unarc.dll to adopt decompression algorithms to the real memory available. But on decompression stage, we don't have too much choices - eithe insert "tempfile" between methods and save intermedirte data to the tempfile or limit 4x4 to run less threads simultaneously

With built-in algos, it works fine (modulo bugs), but for external algos including srep there is no any API to limit their memory usage! so, many years ago i developed alternative approach specially to srep - it can be parameterized like srep:mem75%-600mb and this way it will alloc 75% of RAM minus 600 MB. It was supposed that you have compression line like srep+delta+lzma:512m, computed memory usage by remaining algorithms and make appropriate srep setting

It was fine for all built-in algos except for 4x4, whose decompression memory depends on the num_cores and thus cannot be predicted at compression time. So, we just don't have working instrument for limiting srep+4x4 by using only MY tools

Nevertheless, in MCWRX installer memory usage for srep is set by the installer script. I don't looked into details, but obviously community provided right instrument to deal with srep+4x4 case. Unfortunately, it needs some programming skills to manage - you need to find num_cores, compute memory usage of remaining algos, find RAM size, and then compute memory available for srep. And if it becomes too small (say, less than 500 MB) i recommend to limit number of 4x4 threads (by modifying compression string replacing 4x4 with 4x4:t4 or so) in order to prevent excessive disk swapping in srep

Yet another solution will be improvement of :mem language, adding support for smth like mem75%-100m-132m*t

And ideal solution will be implementation of srep as builtin method or just 64-bit unarc.dll. the second way looks much simpler so i will try :)

as you can see in arcinfo/lt, each thread of 4x4:lzma:256m will require 516 MB of memory, 8 threads need more than 4 GB. the only way is to use external 4x4, i.e. fazip in stdin-to-stdout mode

please try it and write about results - positive or not

Can anyone help me out why all the time srep decompression fails and it says write error disk full?? Why??? Searched forum but nothing helps compressed whole 12 gb in 2 hour but happens again and again I am using -m3f with pzlib and lzma.

Firstly, always remember for decompression u should have CLS-SREP.dll in the same folder where the arc.exe is.
If u got CLS-Srep.dll which is 86 Kb or 54 kb or something which is <100 kb then remember those only works during decompression if they are in the same folder where the arc.exe is during COMPRESSION then the arc simply throw disk full error or something other error
the above CLS-SREP.dll, works during compression too.

Thanks PrinceGupta will try it.
Do you know a good GUI version for BMS scripts to open game archives for rebuilding and referring data from them such as music and graphics...

try use
arc a -lc2000 -ed -ep1 -r -ma -md -mx -s; -w.\temp -mpzlib+srep+lzma:a1:mfbt4:d185m:fb273:mc10000:lc8 data.arc "game location"
and use srep 3.2 and freearc 0.67 date 15/3/2014 and in arc.ini use

[External compressor:srep64]
header = 0
packcmd = srep64 -m3f -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

[External compressor:pzlib]
header = 0
packcmd = pZLib e -m2 -c16m -b256k -t2 - -o - <stdin> <stdout>


and for decompress try to use cls-srep 3.3 by profargo with pzlip

and in arc.ini

[External compressor:pzlib]
header = 0
unpackcmd = pzlib d -t50p - -o - <stdin> <stdout>

try to use isdone original script with inno 5.5.9...

Srep 3.3 but I have 3.93 why older is better???

try ? form the original site on freearc by Bulat Ziganshin...
download freearc 0.67 alpha portable and srep 3.2
i found a couple of srep exe file infected.
use only the original file.
i found to be more reliable and virus free...

try to use in inno where ExtractTemporaryFile
ExtractTemporaryFile('unarc.dll');
ExtractTemporaryFile('arc.ini');
ExtractTemporaryFile('pzlib.exe');
ExtractTemporaryFile('cls-srep.dll');

and in ExpandConstan try

if not SrepInit(ExpandConstant('{app}\'),128,0) then break;
if not FileSearchInit(true) then break;
if not ISArcExtract ( 0, 100, ExpandConstant('{src}\file.arc'), ExpandConstant('{app}'), '', false, '', ExpandConstant('{tmp}\arc.ini'), ExpandConstant('{app}'), False) then break;
ISDoneError:=false;

But it says it is created by bulat.....
Or I am using wrong one

i am trying to help not to argue is easy to modify file attribute...
so please try first if it work great if not then ask why ?

I used cls-srep.dll by razor 12911 size is 979kb but when I decompress which is compressed with pzlib+srep+lzma it continuing decompress after 100% original data is about 2.8 gb (before compression) and it now continue extracting data more than 5 gb why ??

Edit: I compressed Twp folder data and update data have 1953 mb and update have 300 mb before compression and after it have 1.18 gb pzlib+srep+lzma

look at http://freearc.org/Research.aspx

it links TWO srep pages, one with stable version (latest 3.2) and one with experimental version (latest 3.93). use 3.93 and check that your installers work

OTOH, you may use 3.2 if you wish. with maximum compression settings it's no different to 3.2 except for faster _compression_

but i don't provide DLLs, so the best approach imho is to use DLLs from trusted sources such as one by Razor

bulat great in time
 

bulat great in time if you manage to answer me will be great.
is there a way to get constant speed in decompression when using lzma:a1:mfbt4:d185m:fb273:mc10000:lc8 without using fastzip or xz only builtin lzma and without 4x4...
i montor the disk usage is about 20 mb/s is there a better way to get 50 or 60 mb/s.
thank in advance...

Yeah I have one what youre saying and cls from razor but the problem is it's extracting data more than original size and only in one file I got 8 Files but it's extracting only in 1 file and size is bigger than original. Why???

i truly try to help you man
try use srep witout cls in compression and with decompression use cls by profargo 0.33

@PrinceGupta I tried razor cls decompression failed because it's decompressing data more than actual but when I use cls which is 86 on deck pressing successfull with speed of 20 Mb/s

@doofoo24 OK now I had compressed the whole game with srep without cls if I use cls profeg one will it decompress successfully???

But why razor cls extracting more data than actual

ImmortalArc what the method you using

-m3f

not srep but in the batch file what srep + ?

default pzlib arc.bat
Method is -mpzlib+srep+lzma

But I don't know why cls makes decompress slower

ok now
are you using external lzma or fastzip ....
it do not matter
i use file in the folder
srep.exe
pzlib.exe
arc.ini
.bat file
arc.exe
and the folder you want to compress that all i don't use fastzip or other dll
if you have problem in decompression try it
only this file are necessary
try in bat file
arc a -lc4096 -ed -ep1 -r -ma -md -mx -s; -w.\temp -mpzlib+srep+lzma:a1:mfbt4:d185m:fb273:mc10000:lc8 31.arc "file\*"
and make sure their is no fastzip or other lzma in arc.ini only

[External compressor:pzlib]
header = 0
packcmd = pZLib e -m2 -c16m -b256k -t8 - -o - <stdin> <stdout>

[External compressor:srep]
header = 0
packcmd = srep -m3f -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

no. lzma is really that slow. lzma2 will be faster, since it just copies incomperssible data instead of trying to compress them

if you have problem, try to solve it in small steps: remove evrything else from your compression method and try with srep.exe, then with vaioyus dlls that you trust

bulat but in lzbench on github lzma or xz speed around 60 mb/ s,
so shouldn't lzma in freearc be the same ?

1. computers are different
2. speed depends on the data
3. we usually measure speed of decompressed stream i.e. if in 1 second 20 mb are decompressed into 60 mb, we say that the speed is 60 mb/s

try to compare with 7-zip/xz speed and you will find that arc is pretty close

I Would say it is an Internal Bug , i already reported it earlier
http://fileforums.com/showthread.php...684#post452684

yes i think its strange because cls by Profager is always manages to deliver correct data,
i think it is because the ClS-Srep by Profrager is kinda old and probably written using older sources of srep which doesn't have that bug, may be it can be possible
on the other hand CLS by Razor doesn't work on the ground level, It is simply like using arc.ini but with some more settings like use of x64 srep
I don't know how razor's CLS actually works its a conjecture nothing more

Bulat you right using 7z
[External compressor:lzma2]
header = 0
packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d128m:fb=273:mf=bt4:mc=10000:lc=4:lp=0 -mmt=8 -mx9 -si -so <stdin> <stdout>
unpackcmd = 7z x -txz -an -y -si -so <stdin> <stdout>

with freearc is almost the same as using lzma in freearc and the built-in lzma give better ratio 0.5% to 1%.
but i have other question why ZSTD IS WORST THAN 4X4:LZMA in decompression speed i have I7 6700K 4.7GH OVERCLOCK WITH SSD NVME
4x4:lzma decompress around 1gb/ sec but use more memory and cpu to the full while zstd only use 128mb of memory but limit to 400 to 500 mb /sec.
i find it odd ?

if lzma speed is 20 mb/s and 4x4:lzma speed is 1 gb/s, then you have at least 25 cpu cores :) start with measuring full decompression time from start to finish rather than looking into hdd read speeds

you have a good pc, only if u have what u says
but u should first read some Forums post and others to understand
How actually stuffs works?

@Bulat can you do something for this pls
http://fileforums.com/showthread.php...681#post452681
http://fileforums.com/showthread.php...684#post452684
http://fileforums.com/showthread.php...762#post452762

bulat
if you don't believe me try it for you self you the expert,
but i am not stupid...
i am not trying to argue just sharing what i encounter.
but try it you self...
the system i used is
I7 6700K
DDR4 2X8GB 3200MHZ TIMING 15-15-30
SSD SAMSUNG 950 PRO
THE HDD IS WD BLACK
AND THE EXTRNAL STORAGE IS Seagate STEA2000400.

your English is worse then mine

may be you get me wrong, i never said u r stupid nor i think like that, what i have said is that u need some reading first, nothing else

it is because
and LZMA is LZ77 algorithm. Moreover 4x4 can use more then one lzma instance to unpack files by making them work on seperate chunks since its very difficult for Lzma to do that alone and the most Important one, "IGOR don't want to implement it " or in his word he don't have time

Using Lzma the highest speed u can get is around 25-30 mb/s
but u r saying u r getting 175 mb/s with 4x4
this mean u r using 6 out of 8 threads of ur pc
but still saying 50% cpu usage , strange really strange

my frist post on the forum i said pardon me english :o
if someone have skylake or kabylake it is good to test with fast ram and try to montor cpu usage
first run test on normal hdd
the second run try on ssd
you will see different cpu usage
i will try soon agian just to be sure
i will try high clock like 4.5g and then
i will try to test by lower down the clock on the cpu to 3 ghz and see the difference
and if i have more time i will test by disable 2 core to will bee like i3 then i will test with 4 core but without hyper threading like i5
and report back...
but i can't promess so take it with grain of salt.

well, i said how to check the real speed. if you think that i'm wrong, i can't help youi

razor is correct - -sSIZE works only at compression stage. it allocs memory required to compress files up to this size. it's ignored at decompression stage. if atcompression you tries to process file larger than specified, it should just copy remaining data without further compression and correctly decompress it, but may i'm not sure whether it really works - try yourself with standalone srep.exe and report if it goes wrong

overall, if it goes wrong with srep+7z+shar, why you don't try srep alone?? do you expect that someone will solve yuor problems if you can't do simple things yourself?

the setting i used for zstd ver 1.1.2 IS

[External compressor:zstd]
header = 0
packcmd = zstd -22 --ultra <stdin> <stdout>

and for 4x4:lzma i used
arc a -ed -ep1 -lc2000 -r -ma -md -mx -s; -w.\temp -m4x4:lzma:a1:mfbt4:d185m:fb273:mc10000:lc8

only arc.exe witout srep or anything else.

if anyone have similar system I7 6700K with fast ddr4 and ssd nvme welcome to try and share the results..:confused:
i simple man trying to making sense of what i encounter

there article on pcper about how flash 3d nand accelerate data like burst write...
i don't know maybe this is the reason ?
if someone care to test and explain.
it does not matter to me like i said before i try to post what i experienced if someone find it useful great if not no need for hostile comments...

sir, i tried it many maany timeees, every times the same result
i compress those files with shar|srep|7z
since i am archiving with shar i know the correct input size so i give it to srep via -s while compressing
while decompressing i tried to decompress all the things in pipe but it doesn't work srep gives wrong data,
i tried doing every thing separately, first decompress with 7z, It says Every thing is correct, then srep it doesn't say anything, looking at logs and size fo resultant(decompressed) file from srep i concluded that srep is the culprit and it is not restoring the files correctly

if u say so i will run those tests again, i don't have those archives anymore but at morning, its 12AM here

that's great. so you can't describe what exactly happened nor have files. sorry, i can't help in that case