CLS - Fast LZMA2

[78372]

Hi, here's another cls for fast lzma2.
I haven't yet able to convert the streaming functions to delphi, so this cls uses 64MB chunks.

Here is the conversion

Here's the cls anyways.

If you are able to convert the whole c header file to delphi, please send me a copy.

[Update]
Added FPC Build. More optimized and uses less memory in some cases.

[PSP17]

How to use it with freearc ? what should be add in arc.ini and temp files of installer ?

[78372]

Nothing. Just keep cls-fl2.dll and fast-lzma2.dll

[PSP17]

Hi, did test on "INSOMNIA The Ark" 10 GB file, and got nothing

packed with cls | 1133.47 sec = 7%. Speed 9.17 mB/s
unpack with cls | 141.39 sec = 48%. Speed 73.48 mB/s

packed with no cls | 1125.90 sec = 8%. Speed 9.23 mB/s
unpack with no cls | 140.49 sec = 55%. Speed 73.95 mB/s

Maybe I am doing something wrong ?

[78372]

What do you mean by packed without cls? What was your compression method? It should not be lzma, it should be fl2.

[PSP17]

What do you mean by packed without cls?

It mean not using cls to see if it is any difference in pack/unpack speed.

What was your compression method?

here is the whole chain:
batch file

arc.exe a -r --dirs -s; -ep1 -i1 -lc- -w.\_tmp -msrep+lzma2 "data.bin" "pack\*"

arc.ini

[External compressor:lzma2]
header = 0
default = :d25:fb=273:mf=bt4:mc=1000000:lc=4:lp=0
packcmd = "7-Zip" a -txz -an -mcrc=0 -m1=lzma2{:option}:mf=bt4 -mmt=on -mx9 -si -so <stdin> <stdout>

[78372]

It should be -msrep+fl2 to use the cls.

[Gupta]

You should try shared memory. I did implement that for zstd and this and it does provide speed boosts but that have deadlocks (like 1 out of 10) and I didn't find enough motivation to debug. Also, I did implement that from the ground up. You can use some library or something to make it work XD

Anyway nice work lately.

[darkwolves]

it packs just fine for me but it will not let me unpack gives me a -2 error

[78372]

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 943,130,025 => 125,282,928 bytes. Ratio 13.28%
Compression time: cpu 8.30 sec/real 57.45 sec = 14%. Speed 16.42 mB/s
All OK

FreeArc 0.67 (March 15 2014) testing archive: data.arc
Tested 1 file, 125,282,928 => 943,130,025 bytes. Ratio 13.28%
Testing time: cpu 2.28 sec/real 8.42 sec = 27%. Speed 112.01 mB/s
All OK

64bit FL2 Compile, about 2MB sized exe

[External compressor:fl2]
header = 0
packcmd = fl2 c - - <stdin> <stdout>
unpackcmd = fl2 d - - <stdin> <stdout>

[darkwolves]

Originally Posted by 78372

64bit FL2 Compile, about 2MB sized exe

i will try this i rather enjoyed the compression speed

[darkwolves]

oh i like this i was even able to put in my lzma2 commands behind it and they worked
thank you kindly

[doofoo24]

@78372 do you know if there a way to use 7-Zip-zstd in <stdin> <stdout> with (Fast LZMA2, Zstandard, Brotli...)...
https://github.com/mcmilk/7-Zip-zstd/releases

[78372]

ZSTD and Brotli already supports stdio afaik.
And lz4/brotli aren't suitable for game compression

[doofoo24]

sorry i ask the wrong question,
do you know the cmd for (Zstd-Brotli) in 7-Zip-zstd to use with freearc like normal 7z...
like "packcmd = "7-Zip" a -txz -an -mcrc=0 -m1=lzma2{ption}:mf=bt4 -mmt=on -mx9 -si -so <stdin> <stdout>"