FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   CLS - Fast LZMA2 (https://fileforums.com/showthread.php?t=103052)

78372 26-09-2019 01:39

CLS - Fast LZMA2
 
3 Attachment(s)
Hi, here's another cls for fast lzma2.
I haven't yet able to convert the streaming functions to delphi, so this cls uses 64MB chunks.

Here is the conversion

Here's the cls anyways.

If you are able to convert the whole c header file to delphi, please send me a copy.

[Update]
Added FPC Build. More optimized and uses less memory in some cases.

PSP17 09-11-2019 10:12

How to use it with freearc ? what should be add in arc.ini and temp files of installer ?

78372 10-11-2019 03:28

Nothing. Just keep cls-fl2.dll and fast-lzma2.dll

PSP17 16-11-2019 16:23

Hi, did test on "INSOMNIA The Ark" 10 GB file, and got nothing :(

packed with cls | 1133.47 sec = 7%. Speed 9.17 mB/s
unpack with cls | 141.39 sec = 48%. Speed 73.48 mB/s

packed with no cls | 1125.90 sec = 8%. Speed 9.23 mB/s
unpack with no cls | 140.49 sec = 55%. Speed 73.95 mB/s

Maybe I am doing something wrong ?

78372 17-11-2019 07:22

What do you mean by packed without cls? What was your compression method? It should not be lzma, it should be fl2.

PSP17 17-11-2019 11:39

Quote:

What do you mean by packed without cls?
It mean not using cls to see if it is any difference in pack/unpack speed.

Quote:

What was your compression method?
here is the whole chain:
batch file
Code:

arc.exe a -r --dirs -s; -ep1 -i1 -lc- -w.\_tmp -msrep+lzma2 "data.bin" "pack\*"
arc.ini
Code:

[External compressor:lzma2]
header = 0
default = :d25:fb=273:mf=bt4:mc=1000000:lc=4:lp=0
packcmd = "7-Zip" a -txz -an -mcrc=0 -m1=lzma2{:option}:mf=bt4 -mmt=on -mx9 -si -so <stdin> <stdout>


78372 17-11-2019 19:20

It should be -msrep+fl2 to use the cls.

Gupta 19-11-2019 08:34

You should try shared memory. I did implement that for zstd and this and it does provide speed boosts but that have deadlocks (like 1 out of 10) and I didn't find enough motivation to debug. Also, I did implement that from the ground up. You can use some library or something to make it work XD

Anyway nice work lately.

darkwolves 19-11-2019 09:25

it packs just fine for me but it will not let me unpack gives me a -2 error

78372 20-11-2019 06:36

64 bit FL2 - 256MB Chunks
 
1 Attachment(s)
Quote:

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 943,130,025 => 125,282,928 bytes. Ratio 13.28%
Compression time: cpu 8.30 sec/real 57.45 sec = 14%. Speed 16.42 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: data.arc
Tested 1 file, 125,282,928 => 943,130,025 bytes. Ratio 13.28%
Testing time: cpu 2.28 sec/real 8.42 sec = 27%. Speed 112.01 mB/s
All OK
64bit FL2 Compile, about 2MB sized exe :p

Quote:

[External compressor:fl2]
header = 0
packcmd = fl2 c - - <stdin> <stdout>
unpackcmd = fl2 d - - <stdin> <stdout>

darkwolves 20-11-2019 08:23

Quote:

Originally Posted by 78372 (Post 483594)
64bit FL2 Compile, about 2MB sized exe :p

i will try this i rather enjoyed the compression speed

darkwolves 20-11-2019 17:25

oh i like this i was even able to put in my lzma2 commands behind it and they worked
thank you kindly

doofoo24 20-11-2019 22:39

@78372 do you know if there a way to use 7-Zip-zstd in <stdin> <stdout> with (Fast LZMA2, Zstandard, Brotli...)...
https://github.com/mcmilk/7-Zip-zstd/releases

78372 20-11-2019 22:59

ZSTD and Brotli already supports stdio afaik.
And lz4/brotli aren't suitable for game compression

doofoo24 20-11-2019 23:11

sorry i ask the wrong question, :o
do you know the cmd for (Zstd-Brotli) in 7-Zip-zstd to use with freearc like normal 7z...
like "packcmd = "7-Zip" a -txz -an -mcrc=0 -m1=lzma2{:option}:mf=bt4 -mmt=on -mx9 -si -so <stdin> <stdout>"

78372 21-11-2019 06:16

-m1=lzma2
Here change lzma2 to your favourite compressor.
and -txz to-t7z.
Note that it won't support stdio

Carldric Clement 21-11-2019 11:02

Nice one. But I didn't understand at all. What different fast lzma2 and the 7z [lzma2]? Can you explain how they can compress between the size of huge files? Just ask. :D

Also I like how you create with those codes from GitHub to cls. I can understand the code was almost same like delphi. :)

78372 21-11-2019 11:08

Fast lzma2 uses a different matchfinder and faster than normal lzma2. It can use multiple threads more effeciently.

darkwolves 22-11-2019 14:08

i got a very negative ratio using it with oorecm (oodle mermaid)

FitGirl 26-11-2019 08:26

Because Oodle Mermaid is better than fast LZMA2

github 01-02-2021 23:36

Thanks. This gives nice good compression ratio at super fast speed

dixen 02-02-2021 10:51

Small test

SREP+4x4:lzma

Quote:

Compressed 1 file, 1,444,079,652 => 666,703,063 bytes. Ratio 46.17%
Compression time: cpu 622.41 sec/real 206.39 sec = 302%. Speed 7.00 mB/s
All OK
SREP+FL2

Quote:

Compressed 1 file, 1,444,079,652 => 679,032,090 bytes. Ratio 47.02%
Compression time: cpu 299.06 sec/real 103.33 sec = 289%. Speed 13.98 mB/s
All OK

biotec2k8 02-02-2021 11:48

I Use Reflate+Srep+Delta+nz as kompression. The Speed is slow in both Ways. I will test the FL2

github 02-02-2021 14:38

the fl2.zip (569.0 KB ) found in #10 seems to produce almost the same size (0.01% smaller) result but is 3% slower than fl2_fpc.zip (137.7 KB )

dixen 08-02-2021 04:26

Unpack test

xt20+srep+4x4:lzma

Quote:

Tested 5 files, 6,139,044,914 => 7,799,289,310 bytes. Ratio 78.71%
Testing time: cpu 212.50 sec/real 222.03 sec = 96%. Speed 35.13 mB/s
xt20+srep+fl2

Quote:

Tested 5 files, 5,912,090,557 => 7,799,289,310 bytes. Ratio 75.80%
Testing time: cpu 343.20 sec/real 478.89 sec = 72%. Speed 16.29 mB/s
All OK

github 08-02-2021 23:53

does this fl2 (cls dll/exe) support pass in compressing levels or any other parameters?


All times are GMT -7. The time now is 20:41.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com