New lolz
 

BIG THANK TO ProFrager from krinkels :)
updated lolz test22c4b

do u have any changelog ?

nevermind. got it :

implemented ldmf (long distance match finder), which looks for matches outside the range of the main matchfinder dictionary. It was developed as an alternative to srep, with its pluses and minuses. Disable / Enabled with the option -ldmf [0..1]. The ratio of the compression ratio to the required memory for decompression is set with the option -ldc [0..9] at 0, the decompression memory control is disabled, maximizing the compression ratio. The size of the minimum search length is specified by the option -ldl [5..12]. This option affects compression (the smaller, the better compression), the used memory for compression (the smaller, the more memory according to the formula ldmf_compr_mem = src_size / ((2 ^ [ldl]) / 16), i.e. ldl5 will add another half of the size of the input file to the memory already used by lolz, the used memory for unpacking (the smaller, the more memory will be needed) and the compression speed (the smaller, the slower). Added the option -lde [0..2] specifying the level of parsing ldmf matches. 2nd mode is the slowest, 0 is the fastest, but there is a minimum difference in compression. 1st mode is almost as fast as 2nd mode, but compression is usually worse than even the 0th mode. Therefore, at the moment I recommend using either -lde0 or -lde2. The default is -lde0;
added for anpak console and cls.ini parameters -ldmfTempPath and -ldmfMaxMemoryUsage, specifying the path to the ldmf swap files and the size of the memory for ldmf, respectively. The ldmfMaxMemoryUsage parameter when calculating memory includes the size of the main dictionary and even a penny on the model (but the latter is considered by eye, from the balda), for example, if -ldmfMaxMemoryUsage = 64m, one stream and the size of the 16 MB dictionary will be allocated for ldmf memory 16MB, the rest will be discarded in swap. The parameter -ldmfDeleteTmp [0..1] disables / enables deletion of the swap file after unpacking. Disabling is only necessary for debugging purposes and controlling the size of this file. Default = 1;
remade the synchronization function of the thread of the optimal parser and matchfighters. The efficiency of the use of cores in compression with detection has increased, respectively, the overall compression rate has increased;
knocked out the use of large pages, no difference anyway;
added the option -ac [0..1], which includes transferring pricing coding of matches in the matchfinder thread, which speeds up compression with -tt is greater than 4, with less -tt the speed is either the same or worse. Compression may differ slightly from the -ac0 mode, both in plus and minus. Also in this mode, you need a little more memory for compression, which depends on the -tt parameter (add_mem = 200k + 450k * tt);
reduced the size of the statistics of the basic models by limiting some parameters -b * that do not change in practice;
changed the coding model dxt color idx and alpha idx. Now, better compression and faster unpacking of dxt textures;
fixed small jambs and added additional contexts for the raw graphics compression model. The size of the statistics of the graphical model increased from 700kb to 2mb, but compression also improved (in a 24-bit picture from 5'544'705 to 5'406'975).
repaired the -fbb option that was inoperable from some point, but at least the sense of it;

lolz v22c4 [Dec 29 2018 17:22:52]

Settings:
----------------------------- detection settings -----------------------------
detection is enabled(-dt)
use probs propagate(-dtp[0..1]): 1 use brute detect(-dtb[0..1]): 0
o1 detect(-dto[0..1]): 1
raw graphic mm detect(-dtm[0..1]): 1 width detect algo(-dtw[0..1]): 1
dxt detect(-dtd[0..1]): 1
--------------------------- multithreading settings --------------------------
mt type(-mtt[0..1]): 0
mt threads(-mt[1..16]): 1 mt block size MB(-mtb[2..512]): 8m
---------------------------- lz & parser settings ----------------------------
dict size in MB(-d[0..2047]): 16m num parser arrivals(-tt[1..256]): 4
parser hard lim(-oh[8..14]): 12 parser soft lim(-os[0..hard_lim]): 8
fb type A(-fba[0..4096,0=off]): 256 fb type B(-fbb[0..4096,0=off]): 0
calc all literals(-al[0..1]): 1 slow parser mode (mlr)(-x[0..2]): 0
move some of calculations to lz matchfinder thread(-ac[0..1]): 0
---------------------------- matchfinder settings ----------------------------
use only lz matchfinder (-rt[0..2]: 0)
lz mf cycles(-mc[2..1023]): 128
use long distance mf(-ldmf[0..1]): 1
ldmf min len exp(-ldl[5..12]): 8 ldmf compr/mem coeff (-ldc[0..9]): 1
ldmf efficient lvl (-lde[0..2]): 0
------------------------------- models settings ------------------------------
general models :
use simple cm(-cm[0..1]): 1
context mixer o1 ctx (-bc[0..6]): 4
use only o1 nibble literal model (-lm[0..4]): 0
literal hi o1 ctx (-blo[0..8]): 8 literal lo o1 ctx (-bll[0..8]): 8
literal hi rep0lit ctx (-blr[0..6]): 4
match flag rep0lit ctx (-bm[0..6]): 4
all -p* settings are ignored due to using detect
dxt models :
dxt color(X) and alpha(Y) model (-dmXY[X[0..3], Y[0..4]]: 34)
raw graphic models :
color mdl options (-gmXY[X[0..2], Y[0..1]]): 00

*note
ldmf does not work with -mtt1.
ldmf option more like srep just use it for testing , very unlikely to help improve ratio
setting for ldmf -ldl -ldc -lde.
ac1 help speed up compression when tt larger than 4...
also fbb option work...

@doofoo24 ... I try some quick tests to realize the real news.

Test On Enwik9: 1GB

Improves the compression ratio, but with about 1 min more time.

I Try :

I did not see any difference same time and compression ratio...any tip or fix?

"mlolz_x64:ldmf0:ldl12:mt6:mc1023"
:confused:
why ldl with ldmf set to 0
ldl -ldc -lde options when ldmf are enabled (set to 1)

logically I'm trying to understand the difference in performance, speed and compression with the switches enabled or without.:confused::rolleyes:

ok I'll try with different options.:)

Can't get unpack to work.

"Archive data corrupted"

yes you need new *.dll files in unpacked.

I have.

It seems that archives created with previous lolz version are not compatible anymore (I have compressed a game with previous lolz version).

I will test this with a small file and report back.

new lolz test22c4b
 

fix for unpacking error or something

I can help just tell me what to do :)

file size 219.585.545 srep+lolz 135.655.003

file size 219.585.545 lolz:ldmf1 135.602.872

file size 219.585.545 srep+lolz:ldmf1 135.658.543

the issue for me using ldmf ram usage on large games in decompression
you need to limit idmf in cls.ini

[lolz]
Bufsize=512k
transfer_ReadBufSize=512k
transfer_WriteBufSize=512k
MaxThreadsUsage=50%
MaxMemoryUsage=50%
ldmfTempPath={app}
ldmfMaxMemoryUsage=1024m
ldmfDeleteTmp=0
*ldmfMaxMemoryUsage you need to specify ram, percentage option does not work..

@oltjon

can you test ldmf1 with ldl5 and ldc0

I like to help so tell me what should I do? :)

file size 219.585.545 lolz:ldmf1:ldl5:ldc0 135.602.872

Hy bro, doofoo24
How do i change Srep decompressing
Virtual Memory.tmp size?

How do i keep it under 5GB?
srep settings is (l512/c512 a4)
This is the default Srep.. I use default settings.

ldmf only work mt1:(

@Simorq can you please check the inbox.. and reply me back... It's about Srep virtual memory issue :( help please..

@Simorq: nope ,it works for me with d256mb:mt2:ldmf1

@yashita : patience man. :)


V 3.93x has additional precompressing parameters. It is -d .
It can decrease well the decomp memory req. :)
But v3.93 has bugs,too:
For me,it creates crc errors @decompression if i use my pc @compress-stage(browsing,etc).

Examples for the -d parameter:

Srep64 -a8 -m5f -l512 -d1g input.dat output.srep
-d1g can be modified until out of RAM. (D2g ,d4g,d512mb,d1536mb,etc) :D

And now,here goes my question to the real professionals,especially @Bulat : :)

V393x, the -d (dict. compression) parameter has additional settings. h,l and c. Yes,dont mess with them,but sometimes i must. :)
Srep -l512 -d8g:h2g input.dat output.srep
It creates srepfile with 2.5 gb decomp. meamreq. Without the -d its 8.3 gb.
I pass those to arc.exe with the following ,but it IGNORES:
arc.exe -msrep:l512:d8g:dh2g output.arc input.dat
Resulting files decomp memory req still 8.3gb ,while the created with srep is 2.5 gb.

What am i did wrong?!

@mods : sorry,wrong thread(?). Please move it to the correct place. Thank you very much !

@shazzal can you help me?
Im stuck with this question almost 2 days, try different kind of things... But still not fixed..

:( problem is. Srep create big virtual memory tmp file while decompressing..

i used srep:m3f:l128:c128:a4 some times l512:c512:a4

VM file create almost 15GB on some big games ot create 35GB Tmp file why. How do i get rid of it :(
Help me bro please..

Try srep v393 and use its -d parameter.

Arc.exe -msrep:a8:m5f:l512:d1g archive.arc yourdata.dat

As u see above...
But imho write here : https://www.fileforums.com/showthrea...=102501&page=3

Thanks but i use 3.92 cause 93 is buggy..

35GB srep-virtual-memory.tmp:confused:
What is this game? Its size is more than 100 gigabytes?

Yes 103GB Middle Earth Shadow of War

any idea what should i do?
I have pack mad max to 1DVD5 4.23GB
while decompressing that.
VMT File Creating... 15 - 17 GB Why :(

How can i keep it under 5GB or 10GB...? :(

Or how do i fix this?
Do i have to pack all game again?
Or can I fix this with setup.exe?

I've try many ways... Nothing works....
That's why i ask you to help me.
No matter what i do,
it will create VMT File bigger :(

Please help me... :(

Well, your hand is not to create step-virtual-memory.tmp.
The decompression operation does not take place.
Unless you have 64GB of RAM.

You can from
Srep: m1f
Srep: m2f
use.
But the Ratio may increase. (About 200 mb to 1 gb depends on the size of the game)

-------------------------
MAD MAX >> srep:m1f:l512 >> 2GB ram or srep-virtual-memory.tmp
MAD MAX >> srep:m3f:l512 >> 12GB ram or srep-virtual-memory.tmp

I have pack all my games with this settings..
srep:m3f:l128:c128:a4 / srep:m3f:l512:c512:a4

os is there any other way to keep this low?
While decompressing?

Unfortunately no
But you can increase the memory. To use all your RAM.

CLS.ini
[SREP]
Memory=95%

Ohh.. That means, if memory use more...
Then VMTF will low... Right?

The thing is my C drive space is always low.
otherwise i don't care how much gb crate for temp...

1. Yes

2. :confused:

TmpDir To APPDIR

My script doesn't have CLS.ini

Only arc.ini it's smaller one.
From CLS-DISKSPAN R2 IS_Example scripts

That's what i use...


One last question about this,

This cause by M3F Or L512?

i have used on Mad Max
srep:l128 and it takes 16GB for decompressing

Full code srep:l128:c128:a4

So why am i missing here.. who cause this VMT Big file trouble :confused:

Srep's decomp.mem.req depends on the data.
Sometimes you can dedupe 20 gb to its half with 100mb memory,other times it needs 3 gb.

so Virtual Memory.tmp decide what to do with Data.
it's out of our hands? Right?

VMTF Size depends on the game data? so its normal
Thigh i messed up :D m3f:l128 / l512 is fine with a4 right

What is this a4 mean speed accelerate for what decompressing?

This parameters tested from members?? (found from lolz_x64.exe file)

and lm1/4 switches?? (default: lm0)

lm4 --> ...fcm binary model
lm2 --> ...o1 nibble lit model and fcm binary model
lm0 --> o1 nibble lit model

Note: lm0 its work and make decompresseable archive. lm1-4 get better ratio and decompression error! Not yet implemented lm1-4 settings from decompressor?? (cls-lolz_x86/64.exe files) Use from testing only!

some testings from my datasets:

dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:tt32:fb a4096:mc1023 --> 32669kB (its works)
dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:tt32:gm 01:oh14:fba4096:mc1023 --> 32620kB (its best and working version)

Better compression, and not decompresseable archives:

dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:tt32:oh 14:gm01:lm4:fba4096:mc1023 --> 32067kB (gain 2%, big dataset (10+ GB) more than ~4.5% gain!!)
dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:tt32:oh 14:gm01:lm2:fba4096:mc1023 --> 31361kB! (gain 4%, big dataset estimated gain ~8-12%!)

Not decompresseable archive posted in the topic post

Note: precompressors switches before lolz --> xZLib+srep:m3f+a0

-rd and -rmc testing now compare than 32620/32669kB's archive size.

Note2: Try use -rt1 or -rt2 switch to enable rolz matchfinder. And crashed in now! Not implemented the rolz mode?? USe any combinations of switches and dont work. Hmm.

Added three mini sample from decompression testing. Don't need arc/cls.ini! Pure LOLZ archives.

This files compressed use simple switch: lolz_x64.exe -lmX file.ext lmX.lolz archive.lolz (X = 0 or 2 or 4)
Decompress: cls-lolz_x86/64.exe lmX.lolz file.ext

Maybe this helps you

https://krinkels-org.translate.goog/...o=ajax,nv,elem

KaktoR: The newest LOLZ versions in future time, fixed by ProFrager the some rt/lm switches, algorithms in future?? In last v22c4b released in ~3 years ago.

The lm2 or lm4 switches, use from testing only, from demonstrate better compression ratio than working ones. A couple of little mini-tests on this would be nice. Its compression differences.

This compression use lm1/4 its works. Decompression fails. Will hope for wait fix.

Good evening everybody. As far as I know, one of the best settings to have a very high compression is: dtb1:d512m:mtt1:mc1023. Is there any setting for higher compression? Maybe it's up to the files I want to compress, but are there any setting for higher compression? Thank you!!!

Can someone give me good config for faster compression?