NEW LOLZ V22c4

[doofoo24]

BIG THANK TO ProFrager from krinkels :)
updated lolz test22c4b

[shazzla]

do u have any changelog ?

nevermind. got it :

implemented ldmf (long distance match finder), which looks for matches outside the range of the main matchfinder dictionary. It was developed as an alternative to srep, with its pluses and minuses. Disable / Enabled with the option -ldmf [0..1]. The ratio of the compression ratio to the required memory for decompression is set with the option -ldc [0..9] at 0, the decompression memory control is disabled, maximizing the compression ratio. The size of the minimum search length is specified by the option -ldl [5..12]. This option affects compression (the smaller, the better compression), the used memory for compression (the smaller, the more memory according to the formula ldmf_compr_mem = src_size / ((2 ^ [ldl]) / 16), i.e. ldl5 will add another half of the size of the input file to the memory already used by lolz, the used memory for unpacking (the smaller, the more memory will be needed) and the compression speed (the smaller, the slower). Added the option -lde [0..2] specifying the level of parsing ldmf matches. 2nd mode is the slowest, 0 is the fastest, but there is a minimum difference in compression. 1st mode is almost as fast as 2nd mode, but compression is usually worse than even the 0th mode. Therefore, at the moment I recommend using either -lde0 or -lde2. The default is -lde0;
added for anpak console and cls.ini parameters -ldmfTempPath and -ldmfMaxMemoryUsage, specifying the path to the ldmf swap files and the size of the memory for ldmf, respectively. The ldmfMaxMemoryUsage parameter when calculating memory includes the size of the main dictionary and even a penny on the model (but the latter is considered by eye, from the balda), for example, if -ldmfMaxMemoryUsage = 64m, one stream and the size of the 16 MB dictionary will be allocated for ldmf memory 16MB, the rest will be discarded in swap. The parameter -ldmfDeleteTmp [0..1] disables / enables deletion of the swap file after unpacking. Disabling is only necessary for debugging purposes and controlling the size of this file. Default = 1;
remade the synchronization function of the thread of the optimal parser and matchfighters. The efficiency of the use of cores in compression with detection has increased, respectively, the overall compression rate has increased;
knocked out the use of large pages, no difference anyway;
added the option -ac [0..1], which includes transferring pricing coding of matches in the matchfinder thread, which speeds up compression with -tt is greater than 4, with less -tt the speed is either the same or worse. Compression may differ slightly from the -ac0 mode, both in plus and minus. Also in this mode, you need a little more memory for compression, which depends on the -tt parameter (add_mem = 200k + 450k * tt);
reduced the size of the statistics of the basic models by limiting some parameters -b * that do not change in practice;
changed the coding model dxt color idx and alpha idx. Now, better compression and faster unpacking of dxt textures;
fixed small jambs and added additional contexts for the raw graphics compression model. The size of the statistics of the graphical model increased from 700kb to 2mb, but compression also improved (in a 24-bit picture from 5'544'705 to 5'406'975).
repaired the -fbb option that was inoperable from some point, but at least the sense of it;

[doofoo24]

lolz v22c4 [Dec 29 2018 17:22:52]

Settings:
----------------------------- detection settings -----------------------------
detection is enabled(-dt)
use probs propagate(-dtp[0..1]): 1 use brute detect(-dtb[0..1]): 0
o1 detect(-dto[0..1]): 1
raw graphic mm detect(-dtm[0..1]): 1 width detect algo(-dtw[0..1]): 1
dxt detect(-dtd[0..1]): 1
--------------------------- multithreading settings --------------------------
mt type(-mtt[0..1]): 0
mt threads(-mt[1..16]): 1 mt block size MB(-mtb[2..512]): 8m
---------------------------- lz & parser settings ----------------------------
dict size in MB(-d[0..2047]): 16m num parser arrivals(-tt[1..256]): 4
parser hard lim(-oh[8..14]): 12 parser soft lim(-os[0..hard_lim]): 8
fb type A(-fba[0..4096,0=off]): 256 fb type B(-fbb[0..4096,0=off]): 0
calc all literals(-al[0..1]): 1 slow parser mode (mlr)(-x[0..2]): 0
move some of calculations to lz matchfinder thread(-ac[0..1]): 0
---------------------------- matchfinder settings ----------------------------
use only lz matchfinder (-rt[0..2]: 0)
lz mf cycles(-mc[2..1023]): 128
use long distance mf(-ldmf[0..1]): 1
ldmf min len exp(-ldl[5..12]): 8 ldmf compr/mem coeff (-ldc[0..9]): 1
ldmf efficient lvl (-lde[0..2]): 0
------------------------------- models settings ------------------------------
general models :
use simple cm(-cm[0..1]): 1
context mixer o1 ctx (-bc[0..6]): 4
use only o1 nibble literal model (-lm[0..4]): 0
literal hi o1 ctx (-blo[0..8]): 8 literal lo o1 ctx (-bll[0..8]): 8
literal hi rep0lit ctx (-blr[0..6]): 4
match flag rep0lit ctx (-bm[0..6]): 4
all -p* settings are ignored due to using detect
dxt models :
dxt color(X) and alpha(Y) model (-dmXY[X[0..3], Y[0..4]]: 34)
raw graphic models :
color mdl options (-gmXY[X[0..2], Y[0..1]]): 00

[doofoo24]

*note
ldmf does not work with -mtt1.
ldmf option more like srep just use it for testing , very unlikely to help improve ratio
setting for ldmf -ldl -ldc -lde.
ac1 help speed up compression when tt larger than 4...
also fbb option work...

[felice2011]

@doofoo24 ... I try some quick tests to realize the real news.

Test On Enwik9: 1GB

Old Lolz_x64

Mthd Default :tt1

Comp/Time = 14m20s

Ratio = 21.83%

Used CPU % = (20% - 23%)

Used RAM = (733MB) Max Peak



New Lolz_x64

Mthd Default :tt1

Comp/Time = 15m25s

Ratio = 21.72%

Used CPU(%) = (20% - 27%)

Used RAM(MB) = (798MB) Max Peak


Improves the compression ratio, but with about 1 min more time.


Extraction Time: cpu 3.25 sec/real 10.05 sec = 32%. Speed 99.50 mB/s


I Try :


-mlolz_x64:ldmf1:ldl5:mt6:mc1023

-mlolz_x64:ldmf0:ldl12:mt6:mc1023


I did not see any difference same time and compression ratio...any tip or fix?

[doofoo24]

"mlolz_x64:ldmf0:ldl12:mt6:mc1023"
:confused:
why ldl with ldmf set to 0
ldl -ldc -lde options when ldmf are enabled (set to 1)

[felice2011]

"mlolz_x64:ldmf0:ldl12:mt6:mc1023"
:confused:
why ldl with ldmf set to 0

logically I'm trying to understand the difference in performance, speed and compression with the switches enabled or without.:confused::rolleyes:

ldl -ldc -lde options when ldmf are enabled (set to 1)

ok I'll try with different options.:)

[KaktoR]

Can't get unpack to work.

"Archive data corrupted"

[felice2011]

yes you need new *.dll files in unpacked.

[KaktoR]

I have.

It seems that archives created with previous lolz version are not compatible anymore (I have compressed a game with previous lolz version).

I will test this with a small file and report back.

[doofoo24]

fix for unpacking error or something

[yasitha]

I can help just tell me what to do :)

[oltjon]

file size 219.585.545 srep+lolz 135.655.003

file size 219.585.545 lolz:ldmf1 135.602.872

file size 219.585.545 srep+lolz:ldmf1 135.658.543

[doofoo24]

the issue for me using ldmf ram usage on large games in decompression
you need to limit idmf in cls.ini

[lolz]
Bufsize=512k
transfer_ReadBufSize=512k
transfer_WriteBufSize=512k
MaxThreadsUsage=50%
MaxMemoryUsage=50%
ldmfTempPath={app}
ldmfMaxMemoryUsage=1024m
ldmfDeleteTmp=0
*ldmfMaxMemoryUsage you need to specify ram, percentage option does not work..

[doofoo24]

file size 219.585.545 srep+lolz 135.655.003

file size 219.585.545 lolz:ldmf1 135.602.872

file size 219.585.545 srep+lolz:ldmf1 135.658.543

@oltjon

can you test ldmf1 with ldl5 and ldc0

[yasitha]

@oltjon

can you test ldmf1 with ldl5 and ldc0

I like to help so tell me what should I do? :)

[oltjon]

@oltjon

can you test ldmf1 with ldl5 and ldc0

file size 219.585.545 lolz:ldmf1:ldl5:ldc0 135.602.872

[Simorq]

Dead Island Definitive Collection *.rpack; 9.31 GB

Creating archive: Test.Bin using lolz:d96m:mtb48mb:mt8:ldmf1:ldl5:mc1023
Compressed 85 files, 9,998,947,300 => 3,573,858,833 bytes. Ratio 35.74%
Compression time: cpu 49.33 sec/real 6819.44 sec = 1%. Speed 1.47 mB/s
All OK

Creating archive: Test.Bin using srep:m3f:l256+lolz:d96m:mtt1:mt8:mc1023
Compressed 85 files, 9,998,947,300 => 2,428,264,000 bytes. Ratio 24.29%
Compression time: cpu 34.17 sec/real 3551.16 sec = 1%. Speed 2.82 mB/s
All OK

[yasitha]

Hy bro, doofoo24
How do i change Srep decompressing
Virtual Memory.tmp size?

How do i keep it under 5GB?
srep settings is (l512/c512 a4)
This is the default Srep.. I use default settings.

[Simorq]

Dead Island Definitive Collection *.rpack; 9.31 GB

Creating archive: Test.Bin using lolz:d96m:mtb48mb:mt8:ldmf1:ldl5:mc1023
Compressed 85 files, 9,998,947,300 => 3,573,858,833 bytes. Ratio 35.74%
Compression time: cpu 49.33 sec/real 6819.44 sec = 1%. Speed 1.47 mB/s
All OK

Creating archive: Test.Bin using srep:m3f:l256+lolz:d96m:mtt1:mt8:mc1023
Compressed 85 files, 9,998,947,300 => 2,428,264,000 bytes. Ratio 24.29%
Compression time: cpu 34.17 sec/real 3551.16 sec = 1%. Speed 2.82 mB/s
All OK

Creating archive: Test.Bin using lolz:dtb1:d192:mtb96:ldl5:mc128
Compressed 85 files, 9,998,947,300 => 2,411,589,568 bytes. Ratio 24.12%
Compression time: cpu 46.08 sec/real 20318.99 sec = 0%. Speed 0.49 mB/s
All OK

ldmf only work mt1:(

[yasitha]

@Simorq can you please check the inbox.. and reply me back... It's about Srep virtual memory issue :( help please..

[shazzla]

@Simorq: nope ,it works for me with d256mb:mt2:ldmf1

@yashita : patience man. :)


V 3.93x has additional precompressing parameters. It is -d .
It can decrease well the decomp memory req. :)
But v3.93 has bugs,too:
For me,it creates crc errors @decompression if i use my pc @compress-stage(browsing,etc).

Examples for the -d parameter:

Srep64 -a8 -m5f -l512 -d1g input.dat output.srep
-d1g can be modified until out of RAM. (D2g ,d4g,d512mb,d1536mb,etc) :D

And now,here goes my question to the real professionals,especially @Bulat : :)

V393x, the -d (dict. compression) parameter has additional settings. h,l and c. Yes,dont mess with them,but sometimes i must. :)
Srep -l512 -d8g:h2g input.dat output.srep
It creates srepfile with 2.5 gb decomp. meamreq. Without the -d its 8.3 gb.
I pass those to arc.exe with the following ,but it IGNORES:
arc.exe -msrep:l512:d8g:dh2g output.arc input.dat
Resulting files decomp memory req still 8.3gb ,while the created with srep is 2.5 gb.

What am i did wrong?!

@mods : sorry,wrong thread(?). Please move it to the correct place. Thank you very much !

[yasitha]

@shazzal can you help me?
Im stuck with this question almost 2 days, try different kind of things... But still not fixed..

:( problem is. Srep create big virtual memory tmp file while decompressing..

i used srep:m3f:l128:c128:a4 some times l512:c512:a4

VM file create almost 15GB on some big games ot create 35GB Tmp file why. How do i get rid of it :(
Help me bro please..

[shazzla]

Try srep v393 and use its -d parameter.

Arc.exe -msrep:a8:m5f:l512:d1g archive.arc yourdata.dat

As u see above...
But imho write here : https://www.fileforums.com/showthread.php?t=102501&page=3

[yasitha]

Try srep v393 and use its -d parameter.

Arc.exe -msrep:a8:m5f:l512:d1g archive.arc yourdata.dat

As u see above...
But imho write here : https://www.fileforums.com/showthread.php?t=102501&page=3
Thanks but i use 3.92 cause 93 is buggy..

[Simorq]

@shazzal can you help me?
Im stuck with this question almost 2 days, try different kind of things... But still not fixed..

:( problem is. Srep create big virtual memory tmp file while decompressing..

i used srep:m3f:l128:c128:a4 some times l512:c512:a4

VM file create almost 15GB on some big games ot create 35GB Tmp file why. How do i get rid of it :(
Help me bro please..

35GB srep-virtual-memory.tmp:confused:
What is this game? Its size is more than 100 gigabytes?

[yasitha]

35GB srep-virtual-memory.tmp:confused:
What is this game? Its size is more than 100 gigabytes?

Yes 103GB Middle Earth Shadow of War

any idea what should i do?
I have pack mad max to 1DVD5 4.23GB
while decompressing that.
VMT File Creating... 15 - 17 GB Why :(

How can i keep it under 5GB or 10GB...? :(

Or how do i fix this?
Do i have to pack all game again?
Or can I fix this with setup.exe?

I've try many ways... Nothing works....
That's why i ask you to help me.
No matter what i do,
it will create VMT File bigger :(

Please help me... :(

[Simorq]

Well, your hand is not to create step-virtual-memory.tmp.
The decompression operation does not take place.
Unless you have 64GB of RAM.

You can from
Srep: m1f
Srep: m2f
use.
But the Ratio may increase. (About 200 mb to 1 gb depends on the size of the game)

-------------------------
MAD MAX >> srep:m1f:l512 >> 2GB ram or srep-virtual-memory.tmp
MAD MAX >> srep:m3f:l512 >> 12GB ram or srep-virtual-memory.tmp

[yasitha]

Well, your hand is not to create step-virtual-memory.tmp.
The decompression operation does not take place.
Unless you have 64GB of RAM.

You can from
Srep: m1f
Srep: m2f
use.
But the Ratio may increase. (About 200 mb to 1 gb depends on the size of the game)

-------------------------
MAD MAX >> srep:m1f:l512 >> 2GB ram or srep-virtual-memory.tmp
MAD MAX >> srep:m3f:l512 >> 12GB ram or srep-virtual-memory.tmp

I have pack all my games with this settings..
srep:m3f:l128:c128:a4 / srep:m3f:l512:c512:a4

os is there any other way to keep this low?
While decompressing?

[Simorq]

I have pack all my games with this settings..
srep:m3f:l128:c128:a4 / srep:m3f:l512:c512:a4

os is there any other way to keep this low?
While decompressing?

Unfortunately no
But you can increase the memory. To use all your RAM.

CLS.ini
[SREP]
Memory=95%

[yasitha]

Unfortunately no
But you can increase the memory. To use all your RAM.

CLS.ini
[SREP]
Memory=95%

Ohh.. That means, if memory use more...
Then VMTF will low... Right?

The thing is my C drive space is always low.
otherwise i don't care how much gb crate for temp...

[Simorq]

Ohh.. That means, if memory use more...
Then VMTF will low... Right?

The thing is my C drive space is always low.
otherwise i don't care how much gb crate for temp...

1. Yes

2. :confused:

TmpDir To APPDIR

Source: Include\CLS.ini; DestDir: {tmp}; Flags: dontcopy

//if not SrepInit(ExpandConstant('{app}\'), 512, 0) then MyError := True;
ExtractTemporaryFile('CLS.ini');
SetIniString('Srep', 'TempPath', ExpandConstant('{app}'), ExpandConstant('{tmp}\CLS.ini'));

[yasitha]

1. Yes

2. :confused:

TmpDir To APPDIR

Source: Include\CLS.ini; DestDir: {tmp}; Flags: dontcopy

//if not SrepInit(ExpandConstant('{app}\'), 512, 0) then MyError := True;
ExtractTemporaryFile('CLS.ini');
SetIniString('Srep', 'TempPath', ExpandConstant('{app}'), ExpandConstant('{tmp}\CLS.ini'));


My script doesn't have CLS.ini

Only arc.ini it's smaller one.
From CLS-DISKSPAN R2 IS_Example scripts

That's what i use...


One last question about this,

This cause by M3F Or L512?

i have used on Mad Max
srep:l128 and it takes 16GB for decompressing

Full code srep:l128:c128:a4

So why am i missing here.. who cause this VMT Big file trouble :confused:

[shazzla]

Srep's decomp.mem.req depends on the data.
Sometimes you can dedupe 20 gb to its half with 100mb memory,other times it needs 3 gb.

[yasitha]

Srep's decomp.mem.req depends on the data.
Sometimes you can dedupe 20 gb to its half with 100mb memory,other times it needs 3 gb.

so Virtual Memory.tmp decide what to do with Data.
it's out of our hands? Right?

VMTF Size depends on the game data? so its normal
Thigh i messed up :D m3f:l128 / l512 is fine with a4 right

What is this a4 mean speed accelerate for what decompressing?

[kj911]

This parameters tested from members?? (found from lolz_x64.exe file)

rolz idx size in MB(-rd[2..512]): %um rolz mf cycles(-rmc[2..1023]): %u

and lm1/4 switches?? (default: lm0)

lm4 --> ...fcm binary model
lm2 --> ...o1 nibble lit model and fcm binary model
lm0 --> o1 nibble lit model

Note: lm0 its work and make decompresseable archive. lm1-4 get better ratio and decompression error! Not yet implemented lm1-4 settings from decompressor?? (cls-lolz_x86/64.exe files) Use from testing only!

some testings from my datasets:

dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:tt32:fb a4096:mc1023 --> 32669kB (its works)
dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:tt32:gm 01:oh14:fba4096:mc1023 --> 32620kB (its best and working version)

Better compression, and not decompresseable archives:

dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:tt32:oh 14:gm01:lm4:fba4096:mc1023 --> 32067kB (gain 2%, big dataset (10+ GB) more than ~4.5% gain!!)
dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:tt32:oh 14:gm01:lm2:fba4096:mc1023 --> 31361kB! (gain 4%, big dataset estimated gain ~8-12%!)

Not decompresseable archive posted in the topic post (https://fileforums.com/showpost.php?p=494894&postcount=18)

Note: precompressors switches before lolz --> xZLib+srep:m3f+a0

-rd and -rmc testing now compare than 32620/32669kB's archive size.

Note2: Try use -rt1 or -rt2 switch to enable rolz matchfinder. And crashed in now! Not implemented the rolz mode?? USe any combinations of switches and dont work. Hmm.

Added three mini sample from decompression testing. Don't need arc/cls.ini! Pure LOLZ archives.

This files compressed use simple switch: lolz_x64.exe -lmX file.ext lmX.lolz archive.lolz (X = 0 or 2 or 4)
Decompress: cls-lolz_x86/64.exe lmX.lolz file.ext

[KaktoR]

Maybe this helps you

https://krinkels-org.translate.goog/resources/lolz.264/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=nl&_x_tr_pto=ajax,nv,elem

[kj911]

KaktoR: The newest LOLZ versions in future time, fixed by ProFrager the some rt/lm switches, algorithms in future?? In last v22c4b released in ~3 years ago.

The lm2 or lm4 switches, use from testing only, from demonstrate better compression ratio than working ones. A couple of little mini-tests on this would be nice. Its compression differences.

This compression use lm1/4 its works. Decompression fails. Will hope for wait fix.

[Dragonis40]

Good evening everybody. As far as I know, one of the best settings to have a very high compression is: dtb1:d512m:mtt1:mc1023. Is there any setting for higher compression? Maybe it's up to the files I want to compress, but are there any setting for higher compression? Thank you!!!

[Spotless]

Can someone give me good config for faster compression?

[L33THAK0R]

Can someone give me good config for faster compression?

Depends on the data/title you're compressing, but for general improvements srep + lolz is your best bet for something quick and dirty.

[Spotless]

Depends on the data/title you're compressing, but for general improvements srep + lolz is your best bet for something quick and dirty.

i need lolz options for best compression and fast speed

[KaktoR]

There is no such option.

[Spotless]

There is no such option.

How?

i mean lolz options like dict size and mtt and etc.

[KaktoR]

You want "best compression" and "fast speed". Both together aren't possible.

Default lolz settings are well balanced in most cases. So try lolz:mt# (# = number cpu threads)

Data detection options:
-dt [0..1] - enables / disables the pos_ctx / dxt / raw detection. no headers, everything is detected based on the analysis of data statistics. Default: -dt1;
-dtp [0..1] - enables / disables transmission of statistics from previous blocks to subsequent blocks in the detector. Default: -dtp1;
-dtb [0..1] - enables / disables enumeration of all options regardless of heuristics. Default: dtb0;
-dto [0..1] - enables / disables detection of the best positional o1 context. Default: -dto1;
-dtm [0..1] - enables / disables raw graphics multimedia detection. Default: -dtm1;
-dtw [0..1] - enables / disables width detection for raw graphics and dxt textures;
-dtd [0..1] - enables / disables detection of dxt textures;

Multithreading options:
-mtt [0..1] - for multithreading, specifies the operating mode to be used. When set to 0, the dictionary size must be at least 2 times the block size. In this mode, data for each stream will be loaded alternately in block size. In this mode, in most cases, you can achieve better compression than in the second, however, decompression requires as many streams as in compression. When 1, each block is compressed separately, without dependencies on neighboring data, respectively, the compression here is usually worse than in the first mode, but you can specify any number of streams for unpacking. It is for this mode that the options from cls.ini MaxThreadsUsage and MaxMemoryUsage are used. Default: -mtt0;
-mt [1..16] - sets the number of threads to process. With -mt1 and -mtt0, normal sequential lossless compression is obtained by dividing the stream into blocks. Default: -mt1;
-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, respectively, the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;

Main options:
-d [16..2032] - dictionary size in MB. Default: -d64;
-tt [1..256] - the number of paths to consider in the optimal parser. It greatly affects the speed and compression ratio, but not unpacking. You shouldn't ask more than 16, I assure you it's not worth it. Default: -tt4;
-oh [8..14] - sets the maximum number of bytes that the optimal parser will process at a time (2 ^ X). Default: -oh12;
-os [0 .. -oh] - sets the minimum number of bytes that the parser will process at a time (2 ^ X). Default: -os8;
-fba [0..4096] - sets the size of the minimum match, at which the parser will not bother much in the calculations. Compression is decently accelerated (2 times) with a small loss of compression. At 0, these simplifications are disabled. Default: -fba256;
-fbb [0..4096] - THIS OPTION DOES NOT WORK AT THE MOMENT. Asked for even greater simplifications;
-al [0..1] - enables / disables the calculation of the literal price even if rep0 matches. Default: -al1;
-x [0..2] - turns on the slow modes of the parser with the calculation of (almost) all lengths of the found match, as well as the options match + lit + rep0match. Very slow and merciless. And the benefit is very small. It's easier to add -tt; Default: -x0;

Matchfinder options:
-rt [0..2] - THIS OPTION DOES NOT WORK AT THE MOMENT. I set the type of matchfinder - lz, rolz or hybrid mode, but rolz did not meet expectations and I made all the changes without taking it into account, so it does not work now; Default: -rt0;
-mc [2..1023] - Specifies the maximum number of traversals of the binary tree of matches, after which matches for the given position are no longer searched; Default: -mc128;

Model options:
-cm [0..1] - enables / disables a simple context mixer in some critical places, which mixes a couple of models in each place. When enabled, it improves compression, but slows down decompression. Default: -cm1;
-bc [0..8] - sets the level of influence of the previous byte on the mixer; Default: -bc4;
-lm [0..4] - THIS OPTION DOES NOT WORK AT THE MOMENT. Specifies the type of an "elementary" literal. Complex high-order models with cm did not perform well, so I abandoned them, as did rolz. Default: -lm0;
-blo [0..8] - sets the degree of influence of the previous byte on the encoding of the upper part of the literal. Default: -blo8;
-bll [0..8] - sets the degree of influence of the previous byte on the encoding of the lower part of the literal. Default: -bll8;
-blr [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the upper part of the literal. Default: -blr4;
-bm [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the match type flag. Default: -bm4;
-pc [0..4] - sets the positional context for all encoding operations. It is automatically ignored when the detector is on. Default: -pc2;
-dmXY(X [0..3], Y [0..4]) - sets the model for encoding color pairs (X) and alpha channel (Y) pairs. At the maximum value of each parameter, adaptive switching between models is used, while the decompression speed is reduced, but the compression is better in most cases. Default: -dm34;
-gmXY(X [0..2], Y [0..1]) - X - sets the model for encoding raw graphics. at the maximum value, adaptive switching between models is enabled. However, in this case, it is rare to see a gain over the adaptive mode. Generally, mode 0 is in the lead, but its unpacking is 2 times slower than mode 1. Y - enables updating of statistics of models when they were not used (for example, there was a long match). For X0 and X1, it usually gives a small gain in compression, but the speed drops by a factor of 2 (it all depends on the data). In general, the most appropriate is -gm00, he - and the default mode;

[Spotless]

You want "best compression" and "fast speed". Both together aren't possible.

Default lolz settings are well balanced in most cases. So try lolz:mt# (# = number cpu threads)

Data detection options:
-dt [0..1] - enables / disables the pos_ctx / dxt / raw detection. no headers, everything is detected based on the analysis of data statistics. Default: -dt1;
-dtp [0..1] - enables / disables transmission of statistics from previous blocks to subsequent blocks in the detector. Default: -dtp1;
-dtb [0..1] - enables / disables enumeration of all options regardless of heuristics. Default: dtb0;
-dto [0..1] - enables / disables detection of the best positional o1 context. Default: -dto1;
-dtm [0..1] - enables / disables raw graphics multimedia detection. Default: -dtm1;
-dtw [0..1] - enables / disables width detection for raw graphics and dxt textures;
-dtd [0..1] - enables / disables detection of dxt textures;

Multithreading options:
-mtt [0..1] - for multithreading, specifies the operating mode to be used. When set to 0, the dictionary size must be at least 2 times the block size. In this mode, data for each stream will be loaded alternately in block size. In this mode, in most cases, you can achieve better compression than in the second, however, decompression requires as many streams as in compression. When 1, each block is compressed separately, without dependencies on neighboring data, respectively, the compression here is usually worse than in the first mode, but you can specify any number of streams for unpacking. It is for this mode that the options from cls.ini MaxThreadsUsage and MaxMemoryUsage are used. Default: -mtt0;
-mt [1..16] - sets the number of threads to process. With -mt1 and -mtt0, normal sequential lossless compression is obtained by dividing the stream into blocks. Default: -mt1;
-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, respectively, the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;

Main options:
-d [16..2032] - dictionary size in MB. Default: -d64;
-tt [1..256] - the number of paths to consider in the optimal parser. It greatly affects the speed and compression ratio, but not unpacking. You shouldn't ask more than 16, I assure you it's not worth it. Default: -tt4;
-oh [8..14] - sets the maximum number of bytes that the optimal parser will process at a time (2 ^ X). Default: -oh12;
-os [0 .. -oh] - sets the minimum number of bytes that the parser will process at a time (2 ^ X). Default: -os8;
-fba [0..4096] - sets the size of the minimum match, at which the parser will not bother much in the calculations. Compression is decently accelerated (2 times) with a small loss of compression. At 0, these simplifications are disabled. Default: -fba256;
-fbb [0..4096] - THIS OPTION DOES NOT WORK AT THE MOMENT. Asked for even greater simplifications;
-al [0..1] - enables / disables the calculation of the literal price even if rep0 matches. Default: -al1;
-x [0..2] - turns on the slow modes of the parser with the calculation of (almost) all lengths of the found match, as well as the options match + lit + rep0match. Very slow and merciless. And the benefit is very small. It's easier to add -tt; Default: -x0;

Matchfinder options:
-rt [0..2] - THIS OPTION DOES NOT WORK AT THE MOMENT. I set the type of matchfinder - lz, rolz or hybrid mode, but rolz did not meet expectations and I made all the changes without taking it into account, so it does not work now; Default: -rt0;
-mc [2..1023] - Specifies the maximum number of traversals of the binary tree of matches, after which matches for the given position are no longer searched; Default: -mc128;

Model options:
-cm [0..1] - enables / disables a simple context mixer in some critical places, which mixes a couple of models in each place. When enabled, it improves compression, but slows down decompression. Default: -cm1;
-bc [0..8] - sets the level of influence of the previous byte on the mixer; Default: -bc4;
-lm [0..4] - THIS OPTION DOES NOT WORK AT THE MOMENT. Specifies the type of an "elementary" literal. Complex high-order models with cm did not perform well, so I abandoned them, as did rolz. Default: -lm0;
-blo [0..8] - sets the degree of influence of the previous byte on the encoding of the upper part of the literal. Default: -blo8;
-bll [0..8] - sets the degree of influence of the previous byte on the encoding of the lower part of the literal. Default: -bll8;
-blr [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the upper part of the literal. Default: -blr4;
-bm [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the match type flag. Default: -bm4;
-pc [0..4] - sets the positional context for all encoding operations. It is automatically ignored when the detector is on. Default: -pc2;
-dmXY(X [0..3], Y [0..4]) - sets the model for encoding color pairs (X) and alpha channel (Y) pairs. At the maximum value of each parameter, adaptive switching between models is used, while the decompression speed is reduced, but the compression is better in most cases. Default: -dm34;
-gmXY(X [0..2], Y [0..1]) - X - sets the model for encoding raw graphics. at the maximum value, adaptive switching between models is enabled. However, in this case, it is rare to see a gain over the adaptive mode. Generally, mode 0 is in the lead, but its unpacking is 2 times slower than mode 1. Y - enables updating of statistics of models when they were not used (for example, there was a long match). For X0 and X1, it usually gives a small gain in compression, but the speed drops by a factor of 2 (it all depends on the data). In general, the most appropriate is -gm00, he - and the default mode;

i have readed this but it didnt go well with me

can u send one for fast compression and no matter for size
and one for slow compression but best size
like dodi and fitgirl

[KaktoR]

Fast but worse ratio
lolz:dt0:mtt1:mt#:d64:oh14:os14:fba4096

Slow but good ratio
lolz:dtb1:d256m:mt#:mc1023

PS: All depends of input data ofc

[Spotless]

Fast but worse ratio
lolz:dt0:mtt1:mt#:d64:oh14:os14:fba4096

Slow but good ratio
lolz:dtb1:d256m:mt#:mc1023

PS: All depends of input data ofc

OK Thanks i will give it a try
Do you have any social media link for u?

+ the input data perform well with xtool:mermaid+srep+lolz but the problem is when xtool stucks a lot means too slow i literally waited 6 hrs for 22gb and reached 84% for xtool only

[yasitha]

hey guys, anyone know how to compress,
Monster.Hunter.World.v15.11.01
(105,712,123,904 bytes)

i tried Oodle Kraken, as the previous version but won't work.
I tried all oodle core 4 - 7 and
oo2core_7_win64.dll this one seems working fine but it won't compress.

For example.

There's a file 170MB and it will become 898mb once i use oo2core_7_win64.dll but it won't do any effect on lolz or z stranded. Won't compress.
if i try lolz it will become 168.5mb,
but with z standard it will become 186mb much worse..

If anyone know how to compress this
Please let me know..

Thank you 😇

[KaktoR]

hey guys, anyone know how to compress,
Monster.Hunter.World.v15.11.01
(105,712,123,904 bytes)

i tried Oodle Kraken, as the previous version but won't work.
I tried all oodle core 4 - 7 and
oo2core_7_win64.dll this one seems working fine but it won't compress.

For example.

There's a file 170MB and it will become 898mb once i use oo2core_7_win64.dll but it won't do any effect on lolz or z stranded. Won't compress.
if i try lolz it will become 168.5mb,
but with z standard it will become 186mb much worse..

If anyone know how to compress this
Please let me know..

Thank you 😇

It could be one of this games
https://fileforums.com/showpost.php?p=496455&postcount=2926

[yasitha]

It could be one of this games
https://fileforums.com/showpost.php?p=496455&postcount=2926

So there's nothing we can do about it?

[yasitha]

hey guys i need best lolz configuration .

compressing slow is ok, i need best ratio configuration please write down.....

[kj911]

Can someone shed some light on what kind of data types the previous old versions of LOLZ compress better than the last two public versions? I read about this in several places, but a specific description, about the specific data types, could not be found in this regard. The same can be said for v21a7 vs. also for v22c4b versions. Are there specific examples of this?

What is certain in the case of a game, when tested, is that the current version compresses ~3% better and ~10% slower than the previous public version. But what about even older versions? It would be nice to test it. Many public compressors its tested, the specific dataset. (Not widely available on public.)

[shazzla]

Hi guys!

When i use the ldmf1 parameter,many times ,the resulted files cannot be decompressed! (Freearc)
Somewhere in the middle it stop with an error.
Sadly i cant write the exact message.
I dont use the cls-ini file,because i have much more than enough RAM...
RAM modules are good,no faults,lot of free space on my drives...

Anyone faced this ?
Ideas?

Ups,forget : default LOLZ settings used all the time,only dictionary size modified,up to 2 Gb,but it doesnt matter.

[kj911]

Does anyone have the lolly v19j version binaries handy, in cls format? (two x86/64 exe and dll files. And working properly from WinXP.)

Original firstly publicated cls-based exe's, without dll file, dont't work, crashing now with decompression testing big 3+ GB size archive. The x86 exe, totally not working from WinXP. Win7 SP1 its ok, without arc usage methods decompressing.* (Newer version its have better comp. ratio in my testings.)

*Directly packed any data via lolz, ex: lolz_x64.exe <switches> data.dat packed.lolz
Decomprssing via cls-lolz_x86/64.exe packed.lolz data.dat

[L0v3craft]

Hi guys. These are the parameters that usually I use:

-d256mb -dtb1 -mtt1 -tt4 -mt10 -mc1023

today I tried to add two parameters: -fba4096 -gm21

and the decompression doesn't work, doesn't start and I get unarc error.

Someone knows the reason?

[Lord.Freddy]

Hi guys. These are the parameters that usually I use:

-d256mb -dtb1 -mtt1 -tt4 -mt10 -mc1023

today I tried to add two parameters: -fba4096 -gm21

and the decompression doesn't work, doesn't start and I get unarc error.

Someone knows the reason?

well, the (-gm21) parameter doesn't work, use (-gm20) instead.

[kj911]

I wonder who would be able to compress the contents of the archive linked below (extracted into 100MB a *.arc file!) to a size smaller than 30MB or 30000KB... 25MB or less would not be bad. (Although, would it be of any practical use, in addition to benchmarking, to the given compressor/archiver if it is not capable of decompressing at a speed of 5-10+ MB/s.)

Compression (last v22c4b): lolz_x64.exe -dt -d64 -fba4096 -mc1023

Whatever you can use, you can start with the PAQ family.*

Link: https://www.upload.ee/files/16721310/TEST0.lolz.html

Decompressing use: cls-lolz_x64/86.exe TEST0.lolz TEST0.arc

*Note: FP8_v6 or LPAQ or MCM not better than LOLZ, 5-10% ratio less.

Question: Is there an asymmetric CM compressor?

[Masquerade]

^^
You'd be better off posting this on encode.su - it's not really suitable here. Nobody uses PAQ based compressors for PC games.

[kj911]

>> You'd be better posting this site from many results, publicly or private...

This encode.su focused on any not game related data, ex: enwik8-10, calgary and more datasets... 5-6 years ago in released last LOLZ/Razor versions.

What I would like to know:

Obviously, there have been various compressors/archivers under development in recent years, which are not yet public. Well, in connection with these, I would like to know whether, in the case of the data set above, we can still hope for a size reduction of at least another 20-30%. (Just as, to any extent, with other types of data.)

So, anyone who still has an idea/trick up their sleeve can test it, even with programs that have never been seen before. (rough example: Razor 1.6 developer version, LOLZ2, etc... Provided that you don't have to wait long until the public release of v2.0.) But, previous LOLZ versions can also be considered, up to the December 2016 release retroactively. Or anything else. If the given result is not public, it can also come privately.

Compare lists:

LOLZ v22c4b: 33 492 307 byte
RAZOR 1.03.7 with -d 64M: 37 446 869 byte (~331 sec) / -d 128M: 37 319 110 byte (~326 sec)
SCM v0.0.1d: 34 578 565 byte (used profile file. 10-25* slower than LOLZ.)
WinRK 3.12: 33 326 819 byte (~98 minutes!)
Paq8px_v208fix1 with level9: 30 505 692 byte (21 h 2 min!)
EMMA v0.1.25 x64 Fast/1024MB/order6/128MB/None/Low: 39 855 061 byte (~1056 sec)
EMMA v0.1.25 x64 (aka v1.25) modded Audio model profile: 32 522 027 byte (less than 1 hour)*
EMMA v0.1.25 x64 used preconfigured Audio (Max) profile: 31 835 173 byte (11298,3 sec)**

Note: Barely harder (slower) LOLZ settings given minimally better (~150-200kB) results than WinRK.

*Note2: No fast long matches search and Ludicrous complexity mode off. (Options: 1024MB/order9)
**Note3: Try it "Executables (Max)" or "Best" preconfigured settings, probably get in minimally better results. (estimately 200-500kB smaller size. Compress times around 6 to 9 hours.)

[KaktoR]

Why even bother for some kilobytes better result?

[kj911]

Thanks from reply Kaktor!

I would rather measure in MBs, that extra care, for a better result.

I hope that there will be some progress (perhaps there already is) in the field of compression in the near future.

From the above 1-2 results, it is clear that what was only available and approachable in 2011-2012 with slow CM compressors, LOLZ has brought to 2018, within the LZ family, together with fast unpacking. (WinRK: ~90-100 minutes vs. LOLZ less than 2-3 seconds.)

GDCC'23 also brought one or two interesting results, which may or may not have been useful for game textures and compression. I had a dataset of JPG images that contained a larger raw image and smaller size/quality "copies" of it, similar to textures found in games.
In terms of size, it was 940+ MB, even with precomp and other tricks it could not be reduced below 700MB, while the winning compressor was able to compress it by more than half, not much above the 300MB size. Losslessly. (PAQ8px_v208fix1 from T1 Dataset in has 670 million bytes.)

https://encode.su/threads/4048-GDCC2023-Task1-(-Jpeg-lossless-recompression-

UPD: Previous post updated and probably finished more tests...