Cls-lzham

[Gupta]

No chunking, No Temps, No external executable


LZHAM is a lossless data compression codec written in C/C++ (specifically C++03), with a compression ratio similar to LZMA but with 1.5x-8x faster decompression speed. It officially supports Linux x86/x64, Windows x86/x64, OSX, and iOS, with Android support on the way.


Current Version: 1092018

[doofoo24]

test it on small file size decompress work will with inno with the setting that you gave lzham:b4:w24:s4:l10 decompression speed around 87 mb/s on ssd but in compression it slower than lzma...
is there other setting to make better ratio or this is the max setting for lzham,
the cls work great so far no error...

[Gupta]

L10 is not that useful try l9, and increase window size until u get error

Increase buffer size to 16 may be, lzham really like big inputs at once

[DiCaPrIo]

got error with 64ws

[Gupta]

Increase it slowly slowly, there is no limit but is limited by the hardware, or maybe something i am unware of,,,,,,, lzham initialization fails for too big win size

So increments it by 1 and see how far u can get, decompression remains unaffected from Window size

[Razor12911]

Limited by memory probably, cls is x86 therefore 2GB memory limit

[Gupta]

Cpp compilers provides tricks to use int64_t

Or maybe u r right haven't checked lzham sources may be changing int type for x86 build will run tests tomorrow

[DiCaPrIo]

got error on 27ws
worked 1 to 26 ws

[Razor12911]

Cpp compilers provides tricks to use int64_t

Ir maybe u r right will run tests tomorrow

I think that's a 64-bit integer, nothing to do with architecture. That's just an integer that's constructed using 64 bits in terms of binary.

[doofoo24]

@Razor12911 please can you update the cls-zstd to the newer ver 1.3.3 (there new setting like ultra and Long Range Mode)

[Gupta]

If u can't represent more than 4 gb in int how will u address memory space more then that

[Razor12911]

If u can't represent more than 4 gb in int how will u address memory space more then that

You can represent 4GB> in int. Remember, 64-bit converts to 8 bytes. Int64 takes up just 8 bytes in memory, so really you can store that much data in the integer, but when it comes to addressing memory, this is not possible. At least from my understanding.

Edit:
At least in pascal, we are told that memory addresses can be recalled by pointers. These pointers in 32-bit environment are an equivalence to a 32-bit integer because they are 4 bytes while in 64-bit, they are 8 bytes.
@Razor12911 please can you update the cls-zstd to the newer ver 1.3.3 (there new setting like ultra and Long Range Mode)

I'll assign 78372 to do it :)

[doofoo24]

shouldn't lzham:b512mb:w24:s4:l9 use more memory like 5gb???
it using 1gb and the ratio is worst than lzham:b4:w24:s4:l10...

[Gupta]

@razor12911
For memory addressing processor or os prefers native integer types, 32 bit os performs bit level arithmetic to adress memory address more tha limit, and compilers stay away from this arithmetic overhead in x86 build unles told to do soo

[Gupta]

Don't go too much high, lzham may flush data on high ram uses and can't use the redundancy and patterns from previously processed chunks

[78372]

@Razor12911 please can you update the cls-zstd to the newer ver 1.3.3 (there new setting like ultra and Long Range Mode)

Afaik zstd already supports stdio, so why you need cls?

[doofoo24]

i konw it supports stdio, but i asked for cls-zstd without the exe for decompression like cls-afr...
if it difficult to do then no need:D

[78372]

Not difficult, but still it will need an extra dll without cls

[doofoo24]

but it will be built-in setup without external cmd exe...

[Gupta]

Update Available

+ Using different set of APIs this time
+ Support Multithreading(not hurt compression)(no chunking)

[shazzla]

Is this supports stdio ?

[Gupta]

Not difficult, but still it will need an extra dll without cls

long distance match and other advance options are the part of Advance API,
you can't use them without static linking... aka they will not work with dll

Is this supports stdio ?
just put Cls-Lzham where arc.exe is, it will do both compression and dll, see test.bat for options

[78372]

long distance match and other advance options are the part of Advance API,
you can't use them without static linking... aka they will not work with dll

phir to geya XD

Then I am leaving the idea, no cls.

[doofoo24]

the ratio of lzham worst than 4x4:lzma:a1:mfbt4:d1m:fb273:mc10000:lc8...
i used the new cls-lzham_rev1092018 with the setting lzham:d26:l4:u8:t0...

[ZakirAhmad]

is their any precompressor available for lzham. the titanfall files r lzahm compressed.

[shazzla]

it doesnt worth the time to precompress. lzham is pretty slow @compression.
imho there is no precompressor for lzham.

[Gupta]

lzham_precompressor: http://nishi.dreamhosters.com/u/lzhamdet_v2-alpha.rar