FileForums - NEW LOLZ V22c4

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - NEW LOLZ V22c4 (https://fileforums.com/showthread.php?t=102509)

Quote:

Originally Posted by Simorq (Post 478537)

Unfortunately no
But you can increase the memory. To use all your RAM.

CLS.ini
[SREP]
Memory=95%

Ohh.. That means, if memory use more...
Then VMTF will low... Right?

The thing is my C drive space is always low.
otherwise i don't care how much gb crate for temp...

Quote:

Originally Posted by yasitha (Post 478539)

Ohh.. That means, if memory use more...
Then VMTF will low... Right?

The thing is my C drive space is always low.
otherwise i don't care how much gb crate for temp...

1. Yes

2. :confused:

TmpDir To APPDIR

Code:

Source: Include\CLS.ini; DestDir: {tmp}; Flags: dontcopy



//if not SrepInit(ExpandConstant('{app}\'), 512, 0) then MyError := True;

    ExtractTemporaryFile('CLS.ini');

    SetIniString('Srep', 'TempPath', ExpandConstant('{app}'), ExpandConstant('{tmp}\CLS.ini'));

Quote:

Originally Posted by Simorq (Post 478541)

1. Yes

2. :confused:

TmpDir To APPDIR

Code:

Source: Include\CLS.ini; DestDir: {tmp}; Flags: dontcopy



//if not SrepInit(ExpandConstant('{app}\'), 512, 0) then MyError := True;

    ExtractTemporaryFile('CLS.ini');

    SetIniString('Srep', 'TempPath', ExpandConstant('{app}'), ExpandConstant('{tmp}\CLS.ini'));

My script doesn't have CLS.ini

Only arc.ini it's smaller one.
From CLS-DISKSPAN R2 IS_Example scripts

That's what i use...

One last question about this,

This cause by M3F Or L512?

i have used on Mad Max
srep:l128 and it takes 16GB for decompressing

Full code srep:l128:c128:a4

So why am i missing here.. who cause this VMT Big file trouble :confused:

Srep's decomp.mem.req depends on the data.
Sometimes you can dedupe 20 gb to its half with 100mb memory,other times it needs 3 gb.

Quote:

Originally Posted by shazzla (Post 478546)

Srep's decomp.mem.req depends on the data.
Sometimes you can dedupe 20 gb to its half with 100mb memory,other times it needs 3 gb.

so Virtual Memory.tmp decide what to do with Data.
it's out of our hands? Right?

VMTF Size depends on the game data? so its normal
Thigh i messed up :D m3f:l128 / l512 is fine with a4 right

What is this a4 mean speed accelerate for what decompressing?

This parameters tested from members?? (found from lolz_x64.exe file)

Code:

rolz idx size in MB(-rd[2..512]): %um rolz mf cycles(-rmc[2..1023]): %u

and lm1/4 switches?? (default: lm0)

lm4 --> ...fcm binary model
lm2 --> ...o1 nibble lit model and fcm binary model
lm0 --> o1 nibble lit model

Note: lm0 its work and make decompresseable archive. lm1-4 get better ratio and decompression error! Not yet implemented lm1-4 settings from decompressor?? (cls-lolz_x86/64.exe files) Use from testing only!

some testings from my datasets:

dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:tt32:fb a4096:mc1023 --> 32669kB (its works)
dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:tt32:gm 01:oh14:fba4096:mc1023 --> 32620kB (its best and working version)

Better compression, and not decompresseable archives:

dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:tt32:oh 14:gm01:lm4:fba4096:mc1023 --> 32067kB (gain 2%, big dataset (10+ GB) more than ~4.5% gain!!)
dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:tt32:oh 14:gm01:lm2:fba4096:mc1023 --> 31361kB! (gain 4%, big dataset estimated gain ~8-12%!)

Not decompresseable archive posted in the topic post

Note: precompressors switches before lolz --> xZLib+srep:m3f+a0

-rd and -rmc testing now compare than 32620/32669kB's archive size.

Note2: Try use -rt1 or -rt2 switch to enable rolz matchfinder. And crashed in now! Not implemented the rolz mode?? USe any combinations of switches and dont work. Hmm.

Added three mini sample from decompression testing. Don't need arc/cls.ini! Pure LOLZ archives.

This files compressed use simple switch: lolz_x64.exe -lmX file.ext lmX.lolz archive.lolz (X = 0 or 2 or 4)
Decompress: cls-lolz_x86/64.exe lmX.lolz file.ext

KaktoR: The newest LOLZ versions in future time, fixed by ProFrager the some rt/lm switches, algorithms in future?? In last v22c4b released in ~3 years ago.

The lm2 or lm4 switches, use from testing only, from demonstrate better compression ratio than working ones. A couple of little mini-tests on this would be nice. Its compression differences.

This compression use lm1/4 its works. Decompression fails. Will hope for wait fix.

Good evening everybody. As far as I know, one of the best settings to have a very high compression is: dtb1:d512m:mtt1:mc1023. Is there any setting for higher compression? Maybe it's up to the files I want to compress, but are there any setting for higher compression? Thank you!!!

Can someone give me good config for faster compression?

Quote:

Originally Posted by Spotless (Post 496713)

Can someone give me good config for faster compression?

Depends on the data/title you're compressing, but for general improvements srep + lolz is your best bet for something quick and dirty.

Quote:

Originally Posted by L33THAK0R (Post 496714)

Depends on the data/title you're compressing, but for general improvements srep + lolz is your best bet for something quick and dirty.

i need lolz options for best compression and fast speed

There is no such option.

Quote:

Originally Posted by KaktoR (Post 496733)

There is no such option.

How?

i mean lolz options like dict size and mtt and etc.

You want "best compression" and "fast speed". Both together aren't possible.

Default lolz settings are well balanced in most cases. So try lolz:mt# (# = number cpu threads)

Code:

Data detection options:

-dt  [0..1] - enables / disables the pos_ctx / dxt / raw detection. no headers, everything is detected based on the analysis of data statistics. Default: -dt1;

-dtp [0..1] - enables / disables transmission of statistics from previous blocks to subsequent blocks in the detector. Default: -dtp1;

-dtb [0..1] - enables / disables enumeration of all options regardless of heuristics. Default: dtb0;

-dto [0..1] - enables / disables detection of the best positional o1 context. Default: -dto1;

-dtm [0..1] - enables / disables raw graphics multimedia detection. Default: -dtm1;

-dtw [0..1] - enables / disables width detection for raw graphics and dxt textures;

-dtd [0..1] - enables / disables detection of dxt textures;



Multithreading options:

-mtt [0..1]   - for multithreading, specifies the operating mode to be used. When set to 0, the dictionary size must be at least 2 times the block size. In this mode, data for each stream will be loaded alternately in block size. In this mode, in most cases, you can achieve better compression than in the second, however, decompression requires as many streams as in compression. When 1, each block is compressed separately, without dependencies on neighboring data, respectively, the compression here is usually worse than in the first mode, but you can specify any number of streams for unpacking. It is for this mode that the options from cls.ini MaxThreadsUsage and MaxMemoryUsage are used. Default: -mtt0;

-mt  [1..16]  - sets the number of threads to process. With -mt1 and -mtt0, normal sequential lossless compression is obtained by dividing the stream into blocks. Default: -mt1;

-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, respectively, the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;



Main options:

-d   [16..2032] - dictionary size in MB. Default: -d64;

-tt  [1..256]   - the number of paths to consider in the optimal parser. It greatly affects the speed and compression ratio, but not unpacking. You shouldn't ask more than 16, I assure you it's not worth it. Default: -tt4;

-oh  [8..14]    - sets the maximum number of bytes that the optimal parser will process at a time (2 ^ X). Default: -oh12;

-os  [0 .. -oh] - sets the minimum number of bytes that the parser will process at a time (2 ^ X). Default: -os8;

-fba [0..4096]  - sets the size of the minimum match, at which the parser will not bother much in the calculations. Compression is decently accelerated (2 times) with a small loss of compression. At 0, these simplifications are disabled. Default: -fba256;

-fbb [0..4096]  - THIS OPTION DOES NOT WORK AT THE MOMENT. Asked for even greater simplifications;

-al  [0..1]     - enables / disables the calculation of the literal price even if rep0 matches. Default: -al1;

-x   [0..2]     - turns on the slow modes of the parser with the calculation of (almost) all lengths of the found match, as well as the options match + lit + rep0match. Very slow and merciless. And the benefit is very small. It's easier to add -tt; Default: -x0;



Matchfinder options:

-rt [0..2]    - THIS OPTION DOES NOT WORK AT THE MOMENT. I set the type of matchfinder - lz, rolz or hybrid mode, but rolz did not meet expectations and I made all the changes without taking it into account, so it does not work now; Default: -rt0;

-mc [2..1023] - Specifies the maximum number of traversals of the binary tree of matches, after which matches for the given position are no longer searched; Default: -mc128;



Model options:

-cm  [0..1] - enables / disables a simple context mixer in some critical places, which mixes a couple of models in each place. When enabled, it improves compression, but slows down decompression. Default: -cm1;

-bc  [0..8] - sets the level of influence of the previous byte on the mixer; Default: -bc4;

-lm  [0..4] - THIS OPTION DOES NOT WORK AT THE MOMENT. Specifies the type of an "elementary" literal. Complex high-order models with cm did not perform well, so I abandoned them, as did rolz. Default: -lm0;

-blo [0..8] - sets the degree of influence of the previous byte on the encoding of the upper part of the literal. Default: -blo8;

-bll [0..8] - sets the degree of influence of the previous byte on the encoding of the lower part of the literal. Default: -bll8;

-blr [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the upper part of the literal. Default: -blr4;

-bm  [0..8] - sets the degree of influence of the rep0lit byte on the encoding of the match type flag. Default: -bm4;

-pc  [0..4] - sets the positional context for all encoding operations. It is automatically ignored when the detector is on. Default: -pc2;

-dmXY(X [0..3], Y [0..4]) - sets the model for encoding color pairs (X) and alpha channel (Y) pairs. At the maximum value of each parameter, adaptive switching between models is used, while the decompression speed is reduced, but the compression is better in most cases. Default: -dm34;

-gmXY(X [0..2], Y [0..1]) - X - sets the model for encoding raw graphics. at the maximum value, adaptive switching between models is enabled. However, in this case, it is rare to see a gain over the adaptive mode. Generally, mode 0 is in the lead, but its unpacking is 2 times slower than mode 1. Y - enables updating of statistics of models when they were not used (for example, there was a long match). For X0 and X1, it usually gives a small gain in compression, but the speed drops by a factor of 2 (it all depends on the data). In general, the most appropriate is -gm00, he - and the default mode;