Unreal Engine 3/4 and lolzx Compressor
 

What is this compressor "lolzx"?
Is it available for public or it's just for personal use.
Because i'm seeing a very good compression ratio when used with Unreal Engine 3 or 4 games especially the textures.
Can you provide a detailed info about it please?
Thanks.

private compressor

The better ratio is due to the use of a precompressor (Pzlib)

LZMA is better

http://krinkels.org/threads/what-is-lolz.3509/

LZMA didn't give me the same ratio.

You can use Lzma Ultra settings, but you need 16 to 32 GB of RAM

No, it's not ;)

Yes . lzma2(7z) is best and fast

@JIvisPC
give us the parameters for lzma2 to get a better ratio or at least the same as of lolz

Afaik, lolz is slower than lzma when compressing, but very fast when decompressing. It supersedes lzma by decompression speed and ratio.

Do you have the compressor?

Are you a lolz developer?

Guys, this is a private compressor by Profager. He may gifted it to fitgirl, or may fitgirl bought it from Profager. But it's not a public compressor afaik.


Ofc not, and if I had I would have never used it.

Then how you knew it's slower? did you just read that somewhere or there some benchmark tests?
There is no single compressor on the internet can compress unreal engine games 3/4 in such ratio better than lolz with such a very good decompression speed.

Well actually you don't have to 'have' the compressor to know. I learned fron soneone I can trust. lolz has a better compression ratio, I appreciate that. But I don't like to wait hours and days to compress a gane

Sometimes it's worth the waiting, some games might end up with LZMA for 4.38 G.B and it's really sucks that you can't fir that game on a DVD5 just for those extra megabytes, and here comes the role of compressors like lolz to make a better ratio when needed.
I hope i can get my hands on it or something similar soon.

You can use paq8 to do that, if it's really worth waiting

Paq8 low compression speed is the speed of extraction?

Hey @everyone how can pzlib be used instead of lolzx for lzo compressed files ?
aren't pzlib for deflate and zlib detected streams ?
and lolzx for lzo detected streams ?

lolzx is a misleading name.

what is it for then ?

It's not misleading. lolz have nothing with lzo.

lolz, lolzi, lolzx, lol... are simply my namings for each generation and for saving backwards compatibility

Can you point me to a precompressor/compressor that can detect the streams in the .uassets files of unreal engine 3/4 to get a better ratio other than your private tool?

There are no such public tools.

Awesome :(

@Fitgirl who uses that kind of tool are they blackbox or you ?

Other groups (excluding Revenants) can only decompress UE3 files, making them not bit-perfect. But in many games it leads to game bugs, including inability to play them through.

Turtle(Slow & Steady wins the Race)
 

Yes lolz is stupid turtle but gives better compression ratio on many file formats in terms of lzma/lzma2 here are my 101tests on it i didn't write too much still but i hope this will be helpful for beginners;)

Real Example for Beginners

:D

so what's your opinion which compressor shuld i use instead of lolz(turtle)?;)

Ya better use paq8 XD, 8h to decompress a 100m file :p
lolz is just too slow for me, 167k/s(Yeah I have a noob pc), not worth that speed, so I prefer dlz which also allows better dds compression like lolz.

Try -mc8 -tt1 -dto0 -dm00 and you will see the difference. Use only on textures, e.g. .tfc, .dds, .tex,... while lzma on rest. I just repacked almost 60g xcom 2 wotch like that, game compressed in ~2h how about that. Size = same as FG, not counting extra 6g gain from recoding vids.

Unreal Engine 3 games engine Frontlines Fuel of War file extention .upk

upk compression
https://youtu.be/IctB0BqQgJY
problem in loading
https://youtu.be/iT9mH_sdsK8

I get an error when loading continuously

i think i tried everything

I'm trying to drop a single dvd in this game

Imho you cant compress it to a single dvd.
Its simple too big.
Some years ago i repacked it with srep+lzma2 (1gb) : 2 full dvds. :)

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

You are using only 1 thread and probably weak cpu so it *will* be slow. I guess that 33h would be ~6h max on my 4690k 4.2ghz with 4t. But I doubt dlz is going to be any faster on same cmp ratio. In my tests it was worse in both ratio and speed although I didn't gave it enough time. Dlz is IMO pretty much old and obsolete version of lolz, they both combine lzma with cm and thats that.

Here is what you need to know:
-lolz seem to be heavily affected by -mc as much as -tt, I think they are chained together so mc in lolz will slow down more significantly than on pure lzma.
-lolz however seems more stable in speed on bigger dictionaries and block size. Lzma is much faster with 64m/64m than say 128m/256m, but lolz seems to be almost same always therefore it does get closer to lzma on bigger dict and blocks.
-mc have much lower effect on textures, 8 should be more than enough
-other than those settings, only -dto0 -dm00 is worth it. -dto0 will gain about 30-40% in speed and -dm00 speedup scanning and I found it sometimes even better than default autodetect. Dm00 is already used on raw images as default. These 2 options affect cmp only minimally, from what I seen.

With above settings, I can get ~4m/s cmp with lolz. With lzma and similarly big dict/block about ~5-6m/s. Cmp ratio suffer minimally, about ~1-3% vs full settings, but oh boy 20h vs 4h! And when you offload non-textures to lzma(which you should because low mc will make lolz compress non-textures worse in such case) then you can further fasten usually to 2-3h, like 20+gb game. Again today I compressed Ni no Kuni 2, got almost same size as FG but it took under 2h to compress 41gb of game data.

Here is what happened :
There was 34.6G of data
First I used dlz
Speed was 180kbps
I think dlz has entropy check because sometime it was 7mbps (copy) and sometime as slow as 180kbps(lowest), 5%/h.
Then I tried your lolz settings with 128m blocks on some normal data just to test.
It was better than dlz, far better than lzma and 380kb/s speed.
But also remember, my data size is 36.4G, so it asked for 33h because sometimes speed slowed down and sometimes increased.
Then I turned almost everything off in lolz and used the lightest settings.
The ratio was even worse than lzma.
Then it asked for 14hours.

Well at last I understand that my *CELERON* pc isn't going to handle these things at a decent speed, I must upgrade.

you might want to try this compression ratio increasing and faster speed

lolz -dto0 -tt1 -dm00 -mc1023

mc1023 decreases speed, not increase.

However, probably I will use lzt -32 for faster speed and a bit worse ratio than lzma(5%)

That is almost certainly because of bursting. It happen also for example with xnlz <stdio> through FA, it load data of block size to memory = during this time speed in FA GUI show 100+m/s, then it start deceasing as it actually process the block and then process repeat again. Dlz is likely simply copying next data to block for processing and show burst speed to output.

Also it could be speed of scanning rather than processing, lolz btw also vary between 2-7mbps.

Finally, 7mbps cannot be a speed of raw copy, that would be too slow for such thing.

Try it on smaller size like 1gb and compare *final* compression time between dlz and lolz(but also ratio). Then you get best picture. If you get similar ratio but visibly better time on dlz then you may be right, but this is highly unlikely and I would be surprised. If anything, I would expect lolz to give you better both speed and ratio - with settings I provided.

Your settings is good, and my pc is noob XD
DLZ shoes that it's copying data in the console window, then I saw process hacker which says it's 7mbps, probably it includes scanning speed too.
Anyways, lolz is still slow (as I said, even my settings was lower than you) and in the lowest settings it asks for 14h, obviously because of my celeron processor :p
However, I finished packing the game, only rpf files(except x64d.rpf and mp dlcs), pzlib:c128mb+srep:m2f+lzt:32, result = 20.7G.
For a small test your settings gave 2x speed than dlz and better ratio