FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Helping thread :) (https://fileforums.com/showthread.php?t=98274)

doofoo24 25-01-2017 12:54

bulat
how the FreeArc Next development going ?
when are you planing to release it..

Bulat 25-01-2017 13:06

it's almost done. now i fix bugs and write docs

doofoo24 25-01-2017 13:10

great
are you plan to make it shareware like winrar or like 7z GNU source

Bulat 25-01-2017 13:12

i plan to release existing freearc as shareware so that those whio want to help fa'next development can do it by buying existing freearc

doofoo24 25-01-2017 13:15

so you planing to support freearc more add more feature like built in zstd-srep or just convert freearc 0.67 alpha to 1.0 stable

doofoo24 25-01-2017 13:29

anyway
i wish you best of luck on your journey
yours truly

Bulat 25-01-2017 13:57

existing freearc will be released as commercial product. fa'next will be developed as new beta product and finally released as freearc 2.0

doofoo24 25-01-2017 14:12

i done a couple of test using only 4x4:lzma with different setting '4x4:lzma:a1:mfbt4:d64m:fb273:mc10000:lc8' '4x4:lzma:a1:mfbt4:d128m:fb273:mc10000:lc8' '4x4:lzma:a1:mfbt4:d200m:fb273:mc10000:lc8'...
i found that using 4 core 8 threads any setting other than d:64 only 40% cpu utilized ?
using 4x4:lzma:a1:mfbt4:d64m:fb273:mc10000:lc8 it seem to me the best option in scaling cpu with 2 core only only use 384mb in decompression and use the 2 core to tha max...
using on 2 core with 4 threads and 4 core without hyperthreading use the same 756mb of memory and in 4 core 8 threads it use 1024mb and the cpu 100%...
when try to use 4x4:lzma:a1:mfbt4:d128m:fb273:mc10000:lc8 or 4x4:lzma:a1:mfbt4:d200m:fb273:mc10000:lc8 the cpu usage on 4 core 8 threads only 40% whille on 2 core 4 threads and 4 core without hyperthreading cpu usage is 85%? (in decompression not compression)
i am not an expert so no need for smart remarks.
so the best scaling in decompression (in compression on i7 it use around 57% of cpu with 1684mb of memory) is to use 4x4:lzma:a1:mfbt4:d64m:fb273:mc10000:lc8 it work ( i mean that it use the cpu 100% on all setting like pentium-i3-i5-i7)
if anyone find it useful great.
*using 4x4:t does not work for me in decompression i tried 4x4:t4 but when decompress it use 14% of i7 not 50% of i7 4 core 8 threads...

summary
if you use pzlib or precompmt use it with this
4x4:lzma:a1:mfbt4:d64m:fb273:mc10000:lc8 is super fast in decompression but it use the cpu to the max and require more memory but modern system use 64bit so no problem there, for old system and limited ram don't use 4x4 use lzma:a1:mfbt4:d185m:fb273:mc10000:lc8

if srep only use it with zstd the new version 1.1.2

add this line to arc.ini
[External compressor:zstd]
header = 0
packcmd = zstd -22 --ultra <stdin> <stdout>

in compression it use around limit core and only 769mb in compression
in decompression zstd alone use only 128mb and srep better to limit it in inno to 128mb so the total memory use is 256mb it super fast and less demanding on the cpu...

Gupta 27-01-2017 05:17

Quote:

so the best scaling is to use 4x4:lzma:a1:mfbt4:d64m:fb273:mc10000:lc8 it work ( i mean that it use the CPU 100% on all setting like pentium-i3-i5-i7)
I can't find any written document but judging from this
Quote:

else switch (*param) { // Параметры, содержащие значения
case 'b': p->BlockSize = parseMem (param+1, &error); continue;
case 't': p->NumThreads = parseInt (param+1, &error); continue;
case 'i': p->NumExtraBuffers = parseInt (param+1, &error); continue;
case 'r': p->MinOrder0Percents= parseDouble (param+1, &error); continue;
}
i think
Quote:

4x4:t%NUMBER_OF_PROCESSORS%:lzma:a1:mfbt4:d64m:fb2 73:mc10000:lc8
is betther idea

remember there is no constraint that you need the same number of cores to compress and decompress, so by using external compressor i.e. fazip you can fully control the CPU usage in both compression and decompression

Quote:

decompression it use only 128mb + srep you can limit the memory
still, it will write the required memory on hard drive making the process a little bit slower

doofoo24 27-01-2017 05:29

you the "expert" PrinceGupta2000...:o
in decompession i meant using zstd it use 128mb of memory and limt srep to 128mb not 512mb in inno will not make the proccess slower
i test large file (assassin's creed syndicate) with srep+zstd using '128mb' '256mb' and '512mb' in inno setup on normal hdd wd black 1tb write speed was around 176 mb/s max and the same time it took to install so using more than 128mb for srep on hdd is waste of memory usage...

Gupta 27-01-2017 05:46

Razor is the King he just taught me what are the pros and kons of using rep+srep.

doofoo24 29-01-2017 02:57

disk write when using 4x4:lzma on NVME SSD (CPU IS I7 6700K)
* note using srep+4x4:lzma:a1:mfbt4:d185m:fb273:mc10000:lc8
srep limited to 128mb in inno using isdone script

doofoo24 29-01-2017 03:15

disk write using zstd
* note using srep+zstd
setting of zstd 1.1.2
[External compressor:zstd]
header = 0
packcmd = zstd -22 --ultra <stdin> <stdout>

**srep limited to 128mb in inno using isdone script
the same data compress with two method

the ratio on zstd is worst but around 1 to 2% different

that's it i done testing.
if anyone find it useful great...
GREAT FORUM

shazzla 04-02-2017 01:58

1 Attachment(s)
hi all !

im in trouble.... i cant unpack my archives using the default isdone script,although it works fine with precomp/srep/lzma,etc.

if i use any extrenal compressors,it says :

Attachment 17068

My .ISS looks like this :



#define zstd
[Setup]

[Files]
#ifdef zstd
Source: Include\zstd.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\arc.ini; DestDir: {tmp}; Flags: dontcopy
#endif

[Code]
#ifdef zstd
ExtractTemporaryFile('zstd.exe');
ExtractTemporaryFile('arc.ini');
#endif

if not ISArcExtract ( 0, 0, ExpandConstant('{src}\data1.arc'), ExpandConstant('{app}\'), '', false, '', '', ExpandConstant('{app}'), notPCFonFLY {PCFonFLY}) then break;




end;


Upon execution ,setup unpacks the arc.ini ,zstd.exe ,everything ,but the result is the above picture.
I tried to change the unpack command to :

if not ISArcExtract ( 0, 0, ExpandConstant('{src}\data1.arc'), ExpandConstant('{app}\'), '', false, '', ExpandConstant('{tmp}\arc.ini'), ExpandConstant('{app}\'), notPCFonFLY{PCFonFLY}) then break;

Doesnt help.
The only working method for me is :

if not ISExec(0, 0, 0, ExpandConstant('{tmp}\arc.exe'), ExpandConstant('x -o+ -cfg"{tmp}\arc.ini" -w"{app}" -hp"mypw" -dp"{app}" "{src}\data1.arc"'), ExpandConstant('{tmp}'), '...', False) then break;

But its pretty ugly.
Is there any other way to extract my archives using ISArcExtract function ?

Arc.ini content :

[External compressor:zstd]
header = 0
unpackcmd = zstd -d -f <stdin> <stdout>

Thanks in advance !

Gupta 04-02-2017 02:14

try using this
http://fileforums.com/showthread.php?t=98121

shazzla 04-02-2017 02:51

1 Attachment(s)
CLS doesnt work,still the same error. :/

the file was compressed using this method : (FreeArc 0.67 ,March 15 2014)

arc.exe a -mzstd c:\x\chunk0.arc d:\chunk0.dat

arc.ini :
[External compressor:zstd]
header = 0
packcmd = zstd --ultra -v -22 $$arcdatafile$$.tmp -o $arcpackedfile$$.tmp

Tried with this ,too :
packcmd = zstd --ultra -v -22 <stdin> <stdout>

My script : Attachment 17069 (This is the basic ISDone script)

oh ,before i forget : i have the same issue with pzlib.....

maybe im failed at compression procedure... ?

Mini 04-02-2017 03:02

USE
if not ISArcExtract ( 0, 0, ExpandConstant('{src}\data.bin'), ExpandConstant('{app}\'), '', false, '123', ExpandConstant('{tmp}\arc.ini'), ExpandConstant('{app}\'), notPCFonFLY{PCFonFLY}) then break;

shazzla 04-02-2017 03:09

Nope,doesnt work. :/

doofoo24 04-02-2017 03:53

try using to
if ISDoneInit(ExpandConstant('{tmp}\records.inf'), $F777, Comps1,Comps2,Comps3, MainForm.Handle, 512, @ProgressCallback) then begin
repeat
ChangeLanguage('english');
if not SrepInit(ExpandConstant('{app}\'),128,0) then break;
if not FileSearchInit(true) then break;
if not ISArcExtract ( 0, 100, ExpandConstant('{src}\setup-1'), ExpandConstant('{app}'), '', false, '', ExpandConstant('{tmp}\arc.ini'), ExpandConstant('{app}'), False) then break;

shazzla 04-02-2017 04:27

still the original error.

doesnt matter.... im stay with the uglier unpack-method.
thanks to all for the help...!

doofoo24 04-02-2017 04:41

are you using zstd 32bit or 64bit in isdone ?
if you using zstd 64bit you can't decompress using zstd 32 you need to use zstd 64bit

vint56 04-02-2017 04:49

1 Attachment(s)
shazzla

shazzla 04-02-2017 05:37

Quote:

Originally Posted by doofoo24 (Post 455690)
are you using zstd 32bit or 64bit in isdone ?
if you using zstd 64bit you can't decompress using zstd 32 you need to use zstd 64bit


Successfully unpacked zstd 64 bit archives with x86 version.




Quote:

Originally Posted by vint56 (Post 455691)

File Type: rar ISDone0.6final zstd.rar (5.42 MB, 1 views)

Many-many thanks...
its working fine now !
I dont know what caused the error ,but its fine now!
im a little bit confused ,because the compression-syntax is pretty easy ,but...... :D
no matters. :)
thanks again !

ImmortalArc 18-02-2017 01:48

Where I can I get freearc .70 to download ?

Bulat 18-02-2017 03:02

correct question is "When". answer is never, i plan to release 1.0

ImmortalArc 18-02-2017 03:05

And when 1.0 ??

ImmortalArc 18-02-2017 03:09

And how can I optimize cpu and time both ?????

ImmortalArc 18-02-2017 03:18

But your site still shows .70 will arrive in Feb still ?? Hahha

ZakirAhmad 21-02-2017 06:56

How crysis 3 pak files can be decompressed using xdelta...

Viper645 23-02-2017 09:14

Any way to completely disable external compression in freearc? I mean making it impossible to enable by arc.ini or something else.

shazzla 24-02-2017 06:23

Quote:

Originally Posted by ZakirAhmad (Post 456207)
How crysis 3 pak files can be decompressed using xdelta...

Dont know ,but if u r trying to repack it.... You cant do that ,because it uses an asymmetric cypher. (AFAIK)

doofoo24 24-02-2017 14:07

so i tried to use fazip for lzma to get better ratio the limit in freearc lzma:a1:mfbt4:d185m:fb273:mc10000:lc8 using fazip i mange up lzma:a1:mfbt4:d756m:fb273:mc10000:lc8 work with ISDONE more than d756mb IT SAY NOT not enough memory...
but it is so slow compare to 4x4:lzma:64 or zstd
so ratio vs speed.
is there a way to get better ratio with zstd
[External compressor:zstd]
header = 0
packcmd = zstd -22 --ultra <stdin> <stdout>
*can i improve compression more than this setting with zstd ?
any ideas ??

shazzla 27-02-2017 10:28

Maybe delta can improve compression a little bit...Give it a shot :

arc.exe a -msrep:a8:m5f:l64+delta+zstd 1.arc 1.dat

Viper645 28-02-2017 20:29

Which is better?
Precomp 0.38 vs 0.4 vs 0.42

Gupta 28-02-2017 20:49

Which is better?
Precomp 0.38 vs 0.4 vs 0.42

Precomp 0.45, and one which is in currently in development is even better, uses only 2 temp file per stream
https://github.com/schnaader/precomp-cpp

Viper645 28-02-2017 20:55

I don't know why, but freearc mx compression is better than precomp04+srep+lzma for some games, such as GTA San Andreas. precomp04+srep+lzma=1.27GB, freearc mx=1.2GB

Gupta 28-02-2017 20:58

if Game doesn't get deflate stream and < 4GB then even 7zip's -mx will beat both
precomp04+srep+lzma
and freearc's -mx
in compression , compression time and decompression time.

MCWRX 01-03-2017 06:04

So, I see recent posts about 4x4. Let me tell you my experience with it. When users were installing my repacks some of them, got "Not enough memory" error. Srep was given 256mb memory in script

Code:

if not SrepInit(ExpandConstant('{app}\'),256,0) then break;
Do you think if I would have given it 128mb, no one had this errors? I just left 4x4 and moved onto lzma2.

And have any of you had such error?

Viper645 01-03-2017 10:27

Quote:

Originally Posted by PrinceGupta2000 (Post 456485)
Which is better?
Precomp 0.38 vs 0.4 vs 0.42

Precomp 0.45, and one which is in currently in development is even better, uses only 2 temp file per stream
https://github.com/schnaader/precomp-cpp

Well then which precomp is the fastest while compressing? (.38/.4/.42)

Viper645 01-03-2017 10:32

Another question, PPMonstr compressor shows not enough memory while having more than enough memory in windows 10. Why? Any way to fix it?


All times are GMT -7. The time now is 02:08.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com