CLS - Fast LZMA2

[78372]

Hi, here's another cls for fast lzma2.
I haven't yet able to convert the streaming functions to delphi, so this cls uses 64MB chunks.

Here is the conversion (https://github.com/RUDRA78372/fast-lzma2-delphi)

Here's the cls anyways.

If you are able to convert the whole c header file to delphi, please send me a copy.

[Update]
Added FPC Build. More optimized and uses less memory in some cases.

[PSP17]

How to use it with freearc ? what should be add in arc.ini and temp files of installer ?

[78372]

Nothing. Just keep cls-fl2.dll and fast-lzma2.dll

[PSP17]

Hi, did test on "INSOMNIA The Ark" 10 GB file, and got nothing :(

packed with cls | 1133.47 sec = 7%. Speed 9.17 mB/s
unpack with cls | 141.39 sec = 48%. Speed 73.48 mB/s

packed with no cls | 1125.90 sec = 8%. Speed 9.23 mB/s
unpack with no cls | 140.49 sec = 55%. Speed 73.95 mB/s

Maybe I am doing something wrong ?

[78372]

What do you mean by packed without cls? What was your compression method? It should not be lzma, it should be fl2.

[PSP17]

What do you mean by packed without cls?
It mean not using cls to see if it is any difference in pack/unpack speed.

What was your compression method?
here is the whole chain:
batch file
arc.exe a -r --dirs -s; -ep1 -i1 -lc- -w.\_tmp -msrep+lzma2 "data.bin" "pack\*"
arc.ini
[External compressor:lzma2]
header = 0
default = :d25:fb=273:mf=bt4:mc=1000000:lc=4:lp=0
packcmd = "7-Zip" a -txz -an -mcrc=0 -m1=lzma2{:option}:mf=bt4 -mmt=on -mx9 -si -so <stdin> <stdout>

[78372]

It should be -msrep+fl2 to use the cls.

[Gupta]

You should try shared memory. I did implement that for zstd and this and it does provide speed boosts but that have deadlocks (like 1 out of 10) and I didn't find enough motivation to debug. Also, I did implement that from the ground up. You can use some library or something to make it work XD

Anyway nice work lately.

[darkwolves]

it packs just fine for me but it will not let me unpack gives me a -2 error

[78372]

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 943,130,025 => 125,282,928 bytes. Ratio 13.28%
Compression time: cpu 8.30 sec/real 57.45 sec = 14%. Speed 16.42 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: data.arc
Tested 1 file, 125,282,928 => 943,130,025 bytes. Ratio 13.28%
Testing time: cpu 2.28 sec/real 8.42 sec = 27%. Speed 112.01 mB/s
All OK

64bit FL2 Compile, about 2MB sized exe :p

[External compressor:fl2]
header = 0
packcmd = fl2 c - - <stdin> <stdout>
unpackcmd = fl2 d - - <stdin> <stdout>

[darkwolves]

64bit FL2 Compile, about 2MB sized exe :p

i will try this i rather enjoyed the compression speed

[darkwolves]

oh i like this i was even able to put in my lzma2 commands behind it and they worked
thank you kindly

[doofoo24]

@78372 do you know if there a way to use 7-Zip-zstd in <stdin> <stdout> with (Fast LZMA2, Zstandard, Brotli...)...
https://github.com/mcmilk/7-Zip-zstd/releases

[78372]

ZSTD and Brotli already supports stdio afaik.
And lz4/brotli aren't suitable for game compression

[doofoo24]

sorry i ask the wrong question, :o
do you know the cmd for (Zstd-Brotli) in 7-Zip-zstd to use with freearc like normal 7z...
like "packcmd = "7-Zip" a -txz -an -mcrc=0 -m1=lzma2{:option}:mf=bt4 -mmt=on -mx9 -si -so <stdin> <stdout>"

[78372]

-m1=lzma2
Here change lzma2 to your favourite compressor.
and -txz to-t7z.
Note that it won't support stdio

[Carldric Clement]

Nice one. But I didn't understand at all. What different fast lzma2 and the 7z [lzma2]? Can you explain how they can compress between the size of huge files? Just ask. :D

Also I like how you create with those codes from GitHub to cls. I can understand the code was almost same like delphi. :)

[78372]

Fast lzma2 uses a different matchfinder and faster than normal lzma2. It can use multiple threads more effeciently.

[darkwolves]

i got a very negative ratio using it with oorecm (oodle mermaid)

[FitGirl]

Because Oodle Mermaid is better than fast LZMA2

[github]

Thanks. This gives nice good compression ratio at super fast speed

[dixen]

Small test

SREP+4x4:lzma

Compressed 1 file, 1,444,079,652 => 666,703,063 bytes. Ratio 46.17%
Compression time: cpu 622.41 sec/real 206.39 sec = 302%. Speed 7.00 mB/s
All OK

SREP+FL2

Compressed 1 file, 1,444,079,652 => 679,032,090 bytes. Ratio 47.02%
Compression time: cpu 299.06 sec/real 103.33 sec = 289%. Speed 13.98 mB/s
All OK

[biotec2k8]

I Use Reflate+Srep+Delta+nz as kompression. The Speed is slow in both Ways. I will test the FL2

[github]

the fl2.zip (569.0 KB ) found in #10 seems to produce almost the same size (0.01% smaller) result but is 3% slower than fl2_fpc.zip (137.7 KB )

[dixen]

Unpack test

xt20+srep+4x4:lzma

Tested 5 files, 6,139,044,914 => 7,799,289,310 bytes. Ratio 78.71%
Testing time: cpu 212.50 sec/real 222.03 sec = 96%. Speed 35.13 mB/s

xt20+srep+fl2

Tested 5 files, 5,912,090,557 => 7,799,289,310 bytes. Ratio 75.80%
Testing time: cpu 343.20 sec/real 478.89 sec = 72%. Speed 16.29 mB/s
All OK

[github]

does this fl2 (cls dll/exe) support pass in compressing levels or any other parameters?