NEW LOLZ V22c4

[doofoo24]

BIG THANK TO ProFrager from krinkels
updated lolz test22c4b

[shazzla]

do u have any changelog ?

nevermind. got it :

implemented ldmf (long distance match finder), which looks for matches outside the range of the main matchfinder dictionary. It was developed as an alternative to srep, with its pluses and minuses. Disable / Enabled with the option -ldmf [0..1]. The ratio of the compression ratio to the required memory for decompression is set with the option -ldc [0..9] at 0, the decompression memory control is disabled, maximizing the compression ratio. The size of the minimum search length is specified by the option -ldl [5..12]. This option affects compression (the smaller, the better compression), the used memory for compression (the smaller, the more memory according to the formula ldmf_compr_mem = src_size / ((2 ^ [ldl]) / 16), i.e. ldl5 will add another half of the size of the input file to the memory already used by lolz, the used memory for unpacking (the smaller, the more memory will be needed) and the compression speed (the smaller, the slower). Added the option -lde [0..2] specifying the level of parsing ldmf matches. 2nd mode is the slowest, 0 is the fastest, but there is a minimum difference in compression. 1st mode is almost as fast as 2nd mode, but compression is usually worse than even the 0th mode. Therefore, at the moment I recommend using either -lde0 or -lde2. The default is -lde0;
added for anpak console and cls.ini parameters -ldmfTempPath and -ldmfMaxMemoryUsage, specifying the path to the ldmf swap files and the size of the memory for ldmf, respectively. The ldmfMaxMemoryUsage parameter when calculating memory includes the size of the main dictionary and even a penny on the model (but the latter is considered by eye, from the balda), for example, if -ldmfMaxMemoryUsage = 64m, one stream and the size of the 16 MB dictionary will be allocated for ldmf memory 16MB, the rest will be discarded in swap. The parameter -ldmfDeleteTmp [0..1] disables / enables deletion of the swap file after unpacking. Disabling is only necessary for debugging purposes and controlling the size of this file. Default = 1;
remade the synchronization function of the thread of the optimal parser and matchfighters. The efficiency of the use of cores in compression with detection has increased, respectively, the overall compression rate has increased;
knocked out the use of large pages, no difference anyway;
added the option -ac [0..1], which includes transferring pricing coding of matches in the matchfinder thread, which speeds up compression with -tt is greater than 4, with less -tt the speed is either the same or worse. Compression may differ slightly from the -ac0 mode, both in plus and minus. Also in this mode, you need a little more memory for compression, which depends on the -tt parameter (add_mem = 200k + 450k * tt);
reduced the size of the statistics of the basic models by limiting some parameters -b * that do not change in practice;
changed the coding model dxt color idx and alpha idx. Now, better compression and faster unpacking of dxt textures;
fixed small jambs and added additional contexts for the raw graphics compression model. The size of the statistics of the graphical model increased from 700kb to 2mb, but compression also improved (in a 24-bit picture from 5'544'705 to 5'406'975).
repaired the -fbb option that was inoperable from some point, but at least the sense of it;

[doofoo24]

lolz v22c4 [Dec 29 2018 17:22:52]

Settings:
----------------------------- detection settings -----------------------------
detection is enabled(-dt)
use probs propagate(-dtp[0..1]): 1 use brute detect(-dtb[0..1]): 0
o1 detect(-dto[0..1]): 1
raw graphic mm detect(-dtm[0..1]): 1 width detect algo(-dtw[0..1]): 1
dxt detect(-dtd[0..1]): 1
--------------------------- multithreading settings --------------------------
mt type(-mtt[0..1]): 0
mt threads(-mt[1..16]): 1 mt block size MB(-mtb[2..512]): 8m
---------------------------- lz & parser settings ----------------------------
dict size in MB(-d[0..2047]): 16m num parser arrivals(-tt[1..256]): 4
parser hard lim(-oh[8..14]): 12 parser soft lim(-os[0..hard_lim]): 8
fb type A(-fba[0..4096,0=off]): 256 fb type B(-fbb[0..4096,0=off]): 0
calc all literals(-al[0..1]): 1 slow parser mode (mlr)(-x[0..2]): 0
move some of calculations to lz matchfinder thread(-ac[0..1]): 0
---------------------------- matchfinder settings ----------------------------
use only lz matchfinder (-rt[0..2]: 0)
lz mf cycles(-mc[2..1023]): 128
use long distance mf(-ldmf[0..1]): 1
ldmf min len exp(-ldl[5..12]): 8 ldmf compr/mem coeff (-ldc[0..9]): 1
ldmf efficient lvl (-lde[0..2]): 0
------------------------------- models settings ------------------------------
general models :
use simple cm(-cm[0..1]): 1
context mixer o1 ctx (-bc[0..6]): 4
use only o1 nibble literal model (-lm[0..4]): 0
literal hi o1 ctx (-blo[0..8]): 8 literal lo o1 ctx (-bll[0..8]): 8
literal hi rep0lit ctx (-blr[0..6]): 4
match flag rep0lit ctx (-bm[0..6]): 4
all -p* settings are ignored due to using detect
dxt models :
dxt color(X) and alpha(Y) model (-dmXY[X[0..3], Y[0..4]]: 34)
raw graphic models :
color mdl options (-gmXY[X[0..2], Y[0..1]]): 00

[doofoo24]

*note
ldmf does not work with -mtt1.
ldmf option more like srep just use it for testing , very unlikely to help improve ratio
setting for ldmf -ldl -ldc -lde.
ac1 help speed up compression when tt larger than 4...
also fbb option work...

[felice2011]

@doofoo24 ... I try some quick tests to realize the real news.

Test On Enwik9: 1GB

Old Lolz_x64

Mthd Default :tt1

Comp/Time = 14m20s

Ratio = 21.83%

Used CPU % = (20% - 23%)

Used RAM = (733MB) Max Peak

New Lolz_x64

Mthd Default :tt1

Comp/Time = 15m25s

Ratio = 21.72%

Used CPU(%) = (20% - 27%)

Used RAM(MB) = (798MB) Max Peak

Improves the compression ratio, but with about 1 min more time.

Extraction Time: cpu 3.25 sec/real 10.05 sec = 32%. Speed 99.50 mB/s

I Try :

-mlolz_x64:ldmf1:ldl5:mt6:mc1023

-mlolz_x64:ldmf0:ldl12:mt6:mc1023

I did not see any difference same time and compression ratio...any tip or fix?

[doofoo24]

"mlolz_x64:ldmf0:ldl12:mt6:mc1023"

why ldl with ldmf set to 0
ldl -ldc -lde options when ldmf are enabled (set to 1)

[felice2011]

Originally Posted by doofoo24

"mlolz_x64:ldmf0:ldl12:mt6:mc1023"

why ldl with ldmf set to 0

logically I'm trying to understand the difference in performance, speed and compression with the switches enabled or without.

Originally Posted by doofoo24

ldl -ldc -lde options when ldmf are enabled (set to 1)

ok I'll try with different options.

[KaktoR]

Can't get unpack to work.

"Archive data corrupted"

[felice2011]

yes you need new *.dll files in unpacked.

[KaktoR]

I have.

It seems that archives created with previous lolz version are not compatible anymore (I have compressed a game with previous lolz version).

I will test this with a small file and report back.

[doofoo24]

fix for unpacking error or something

[yasitha]

I can help just tell me what to do

[oltjon]

file size 219.585.545 srep+lolz 135.655.003

file size 219.585.545 lolz:ldmf1 135.602.872

file size 219.585.545 srep+lolz:ldmf1 135.658.543

[doofoo24]

the issue for me using ldmf ram usage on large games in decompression
you need to limit idmf in cls.ini

[lolz]
Bufsize=512k
transfer_ReadBufSize=512k
transfer_WriteBufSize=512k
MaxThreadsUsage=50%
MaxMemoryUsage=50%
ldmfTempPath={app}
ldmfMaxMemoryUsage=1024m
ldmfDeleteTmp=0
*ldmfMaxMemoryUsage you need to specify ram, percentage option does not work..

[doofoo24]

Originally Posted by oltjon

file size 219.585.545 srep+lolz 135.655.003

file size 219.585.545 lolz:ldmf1 135.602.872

file size 219.585.545 srep+lolz:ldmf1 135.658.543

@oltjon

can you test ldmf1 with ldl5 and ldc0