Unreal Engine 3/4 and lolzx Compressor

[danswano]

What is this compressor "lolzx"?
Is it available for public or it's just for personal use.
Because i'm seeing a very good compression ratio when used with Unreal Engine 3 or 4 games especially the textures.
Can you provide a detailed info about it please?
Thanks.

[Simorq]

What is this compressor "lolzx"?
Is it available for public or it's just for personal use.
Because i'm seeing a very good compression ratio when used with Unreal Engine 3 or 4 games especially the textures.
Can you provide a detailed info about it please?
Thanks.

private compressor

The better ratio is due to the use of a precompressor (Pzlib)

LZMA is better

http://krinkels.org/threads/what-is-lolz.3509/

[danswano]

LZMA didn't give me the same ratio.

[Simorq]

LZMA didn't give me the same ratio.

You can use Lzma Ultra settings, but you need 16 to 32 GB of RAM

[FitGirl]

LZMA is better
No, it's not ;)

[Jiva newstone]

Yes . lzma2(7z) is best and fast

[danswano]

@JIvisPC
give us the parameters for lzma2 to get a better ratio or at least the same as of lolz

[78372]

Afaik, lolz is slower than lzma when compressing, but very fast when decompressing. It supersedes lzma by decompression speed and ratio.

[danswano]

Afaik, lolz is slower than lzma when compressing, but very fast when decompressing. It supersedes lzma by decompression speed and ratio.

Do you have the compressor?

[Simorq]

No, it's not ;)


Are you a lolz developer?

[78372]

Guys, this is a private compressor by Profager. He may gifted it to fitgirl, or may fitgirl bought it from Profager. But it's not a public compressor afaik.


Do you have the compressor?

Ofc not, and if I had I would have never used it.

[danswano]

Guys, this is a private compressor by Profager. He may gifted it to fitgirl, or may fitgirl bought it from Profager. But it's not a public compressor afaik.




Ofc not, and if I had I would have never used it.

Then how you knew it's slower? did you just read that somewhere or there some benchmark tests?
There is no single compressor on the internet can compress unreal engine games 3/4 in such ratio better than lolz with such a very good decompression speed.

[78372]

Well actually you don't have to 'have' the compressor to know. I learned fron soneone I can trust. lolz has a better compression ratio, I appreciate that. But I don't like to wait hours and days to compress a gane

[danswano]

Well actually you don't have to 'have' the compressor to know. I learned fron soneone I can trust. lolz has a better compression ratio, I appreciate that. But I don't like to wait hours and days to compress a gane

Sometimes it's worth the waiting, some games might end up with LZMA for 4.38 G.B and it's really sucks that you can't fir that game on a DVD5 just for those extra megabytes, and here comes the role of compressors like lolz to make a better ratio when needed.
I hope i can get my hands on it or something similar soon.

[78372]

You can use paq8 to do that, if it's really worth waiting

[Simorq]

You can use paq8 to do that, if it's really worth waiting

Paq8 low compression speed is the speed of extraction?

[EzzEldin16]

Hey @everyone how can pzlib be used instead of lolzx for lzo compressed files ?
aren't pzlib for deflate and zlib detected streams ?
and lolzx for lzo detected streams ?

[danswano]

lolzx is a misleading name.

[EzzEldin16]

what is it for then ?

[FitGirl]

lolzx is a misleading name.
It's not misleading. lolz have nothing with lzo.

lolz, lolzi, lolzx, lol... are simply my namings for each generation and for saving backwards compatibility

[danswano]

It's not misleading. lolz have nothing with lzo.

lolz, lolzi, lolzx, lol... are simply my namings for each generation and for saving backwards compatibility

Can you point me to a precompressor/compressor that can detect the streams in the .uassets files of unreal engine 3/4 to get a better ratio other than your private tool?

[FitGirl]

Can you point me to a precompressor/compressor that can detect the streams in the .uassets files of unreal engine 3/4 to get a better ratio other than your private tool?
There are no such public tools.

[danswano]

Awesome :(

[EzzEldin16]

Can you point me to a precompressor/compressor that can detect the streams in the .uassets files of unreal engine 3/4 to get a better ratio other than your private tool?

@Fitgirl who uses that kind of tool are they blackbox or you ?

[FitGirl]

Other groups (excluding Revenants) can only decompress UE3 files, making them not bit-perfect. But in many games it leads to game bugs, including inability to play them through.

[PsYcHo_RaGE]

Afaik, lolz is slower than lzma when compressing, but very fast when decompressing. It supersedes lzma by decompression speed and ratio.

Yes lolz is stupid turtle but gives better compression ratio on many file formats in terms of lzma/lzma2 here are my 101tests on it i didn't write too much still but i hope this will be helpful for beginners;)

Real Example for Beginners

By Compressing Unreal Engine Files for e.g .tfc, .upk, .umap etc etc
I have found that by using uelr+srep64+lolz given me the compression rate was slightly strong than using uelr+srep+lolz
Note: srep options were -m3f -ia- -l512 compressed 1289kb-->848kb
srep64 options were -m3f -ia- -l512 compressed 1289kb-->846kb
So my opinion is to use uelr+srep64( -m3f -ia- -l512 )+lolz on these unreal engine files

On Compressing sound .pck files compression rate was slightly great using srep64+lolz than using srep+lolz
Note: srep options were -m3f -ia- -l512 56.5mb-->48.6mb(1kb increased as compared to srep64)
srep64 options were -m3f -ia- -l512 56.5mb-->48.6mb
srep64 options were -m5f -ia- -l512 56.5mb-->48.5mb
srep64 options( without -ia- ) were -m5f -l512 56.5mb--> 48.6(9kb decreased as compared to srep64(48.6mb)
So my opinion is to use srep64( -m5f -ia- -l512 )+lolz on these .pck sound files

On compressing sound .pck files compression was awesomely fast by using srep64+lzma:ultra(but 2mb increase in file that i used to compress) than using srep64+lolz(slow compression), i think lzma is doing better in .pck files

On compressing .forge files with afr+srep64+lolz taken much time(headache) than using afr+srep64+lzma:ultra(fast compression but about 187mb in increase in file that i used to compress
NOTE: My file size was 1.91GB

As far as i know now i have tested a 35,264kb .forge file and the compression test was this:
by using afr+srep(options)+lolz Based on srep(options) there was slight changing in compression
by using afr+srep64(options)+lolz Based on srep64(options) there was also slight changing in compression
{options of srep} using srep(-m3f -ia- -l512) compression was from 35,264kb-->10,397kb
using srep(-m5f -ia- -l512) compression was from 35,264kb-->10,463kb
but if we use srep(-m3f -l512) compression will be from 35, 264kb-->10,398kb (1kb increased without using -ia-)
but if we use srep(-m5f -l512) compression will be from 35, 264kb-->10,464kb (1kb increased without using -ia-)
{options of srep64} using srep64(-m3f -ia- -l512) compression was from 35,264kb-->10,400kb
using srep64(-m5f -ia- -l512) compression was from 35,264kb-->10,458kb
but if we use srep64(-m3f -l512) compression will be from 35, 264kb-->10,401kb (1kb increased without using -ia-)
but if we use srep64(-m5f -l512) compression will be from 35, 264kb-->10,459kb (1kb increased without using -ia-)
At last i think for .forge files it's good to use srep(-m3f -ia- l512) but it may differ from time to time:D

[PsYcHo_RaGE]

Well actually you don't have to 'have' the compressor to know. I learned fron soneone I can trust. lolz has a better compression ratio, I appreciate that. But I don't like to wait hours and days to compress a gane


so what's your opinion which compressor shuld i use instead of lolz(turtle)?;)

[78372]

Ya better use paq8 XD, 8h to decompress a 100m file :p
lolz is just too slow for me, 167k/s(Yeah I have a noob pc), not worth that speed, so I prefer dlz which also allows better dds compression like lolz.

[elit]

Ya better use paq8 XD, 8h to decompress a 100m file :p
lolz is just too slow for me, 167k/s(Yeah I have a noob pc), not worth that speed, so I prefer dlz which also allows better dds compression like lolz.

Try -mc8 -tt1 -dto0 -dm00 and you will see the difference. Use only on textures, e.g. .tfc, .dds, .tex,... while lzma on rest. I just repacked almost 60g xcom 2 wotch like that, game compressed in ~2h how about that. Size = same as FG, not counting extra 6g gain from recoding vids.

[masen485]

Unreal Engine 3 games engine Frontlines Fuel of War file extention .upk

upk compression
https://youtu.be/IctB0BqQgJY
problem in loading
https://youtu.be/iT9mH_sdsK8

I get an error when loading continuously

i think i tried everything

I'm trying to drop a single dvd in this game

[shazzla]

Imho you cant compress it to a single dvd.
Its simple too big.
Some years ago i repacked it with srep+lzma2 (1gb) : 2 full dvds. :)

[78372]

Try -mc8 -tt1 -dto0 -dm00 and you will see the difference. Use only on textures, e.g. .tfc, .dds, .tex,... while lzma on rest. I just repacked almost 60g xcom 2 wotch like that, game compressed in ~2h how about that. Size = same as FG, not counting extra 6g gain from recoding vids.

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

[elit]

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

You are using only 1 thread and probably weak cpu so it *will* be slow. I guess that 33h would be ~6h max on my 4690k 4.2ghz with 4t. But I doubt dlz is going to be any faster on same cmp ratio. In my tests it was worse in both ratio and speed although I didn't gave it enough time. Dlz is IMO pretty much old and obsolete version of lolz, they both combine lzma with cm and thats that.

Here is what you need to know:
-lolz seem to be heavily affected by -mc as much as -tt, I think they are chained together so mc in lolz will slow down more significantly than on pure lzma.
-lolz however seems more stable in speed on bigger dictionaries and block size. Lzma is much faster with 64m/64m than say 128m/256m, but lolz seems to be almost same always therefore it does get closer to lzma on bigger dict and blocks.
-mc have much lower effect on textures, 8 should be more than enough
-other than those settings, only -dto0 -dm00 is worth it. -dto0 will gain about 30-40% in speed and -dm00 speedup scanning and I found it sometimes even better than default autodetect. Dm00 is already used on raw images as default. These 2 options affect cmp only minimally, from what I seen.

With above settings, I can get ~4m/s cmp with lolz. With lzma and similarly big dict/block about ~5-6m/s. Cmp ratio suffer minimally, about ~1-3% vs full settings, but oh boy 20h vs 4h! And when you offload non-textures to lzma(which you should because low mc will make lolz compress non-textures worse in such case) then you can further fasten usually to 2-3h, like 20+gb game. Again today I compressed Ni no Kuni 2, got almost same size as FG but it took under 2h to compress 41gb of game data.

[78372]

Here is what happened :
There was 34.6G of data
First I used dlz
Speed was 180kbps
I think dlz has entropy check because sometime it was 7mbps (copy) and sometime as slow as 180kbps(lowest), 5%/h.
Then I tried your lolz settings with 128m blocks on some normal data just to test.
It was better than dlz, far better than lzma and 380kb/s speed.
But also remember, my data size is 36.4G, so it asked for 33h because sometimes speed slowed down and sometimes increased.
Then I turned almost everything off in lolz and used the lightest settings.
The ratio was even worse than lzma.
Then it asked for 14hours.

Well at last I understand that my *CELERON* pc isn't going to handle these things at a decent speed, I must upgrade.

[PsYcHo_RaGE]

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

you might want to try this compression ratio increasing and faster speed

lolz -dto0 -tt1 -dm00 -mc1023

[78372]

mc1023 decreases speed, not increase.

However, probably I will use lzt -32 for faster speed and a bit worse ratio than lzma(5%)

[elit]

Here is what happened :
I think dlz has entropy check because sometime it was 7mbps (copy) and sometime as slow as 180kbps(lowest), 5%/h.

That is almost certainly because of bursting. It happen also for example with xnlz <stdio> through FA, it load data of block size to memory = during this time speed in FA GUI show 100+m/s, then it start deceasing as it actually process the block and then process repeat again. Dlz is likely simply copying next data to block for processing and show burst speed to output.

Also it could be speed of scanning rather than processing, lolz btw also vary between 2-7mbps.

Finally, 7mbps cannot be a speed of raw copy, that would be too slow for such thing.

Try it on smaller size like 1gb and compare *final* compression time between dlz and lolz(but also ratio). Then you get best picture. If you get similar ratio but visibly better time on dlz then you may be right, but this is highly unlikely and I would be surprised. If anything, I would expect lolz to give you better both speed and ratio - with settings I provided.

[78372]

Your settings is good, and my pc is noob XD
DLZ shoes that it's copying data in the console window, then I saw process hacker which says it's 7mbps, probably it includes scanning speed too.
Anyways, lolz is still slow (as I said, even my settings was lower than you) and in the lowest settings it asks for 14h, obviously because of my celeron processor :p
However, I finished packing the game, only rpf files(except x64d.rpf and mp dlcs), pzlib:c128mb+srep:m2f+lzt:32, result = 20.7G.
For a small test your settings gave 2x speed than dlz and better ratio