Unreal Engine 3/4 and lolzx Compressor

[shazzla]

Imho you cant compress it to a single dvd.
Its simple too big.
Some years ago i repacked it with srep+lzma2 (1gb) : 2 full dvds.

[78372]

Originally Posted by elit

Try -mc8 -tt1 -dto0 -dm00 and you will see the difference. Use only on textures, e.g. .tfc, .dds, .tex,... while lzma on rest. I just repacked almost 60g xcom 2 wotch like that, game compressed in ~2h how about that. Size = same as FG, not counting extra 6g gain from recoding vids.

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

[elit]

Originally Posted by 78372

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

You are using only 1 thread and probably weak cpu so it *will* be slow. I guess that 33h would be ~6h max on my 4690k 4.2ghz with 4t. But I doubt dlz is going to be any faster on same cmp ratio. In my tests it was worse in both ratio and speed although I didn't gave it enough time. Dlz is IMO pretty much old and obsolete version of lolz, they both combine lzma with cm and thats that.

Here is what you need to know:
-lolz seem to be heavily affected by -mc as much as -tt, I think they are chained together so mc in lolz will slow down more significantly than on pure lzma.
-lolz however seems more stable in speed on bigger dictionaries and block size. Lzma is much faster with 64m/64m than say 128m/256m, but lolz seems to be almost same always therefore it does get closer to lzma on bigger dict and blocks.
-mc have much lower effect on textures, 8 should be more than enough
-other than those settings, only -dto0 -dm00 is worth it. -dto0 will gain about 30-40% in speed and -dm00 speedup scanning and I found it sometimes even better than default autodetect. Dm00 is already used on raw images as default. These 2 options affect cmp only minimally, from what I seen.

With above settings, I can get ~4m/s cmp with lolz. With lzma and similarly big dict/block about ~5-6m/s. Cmp ratio suffer minimally, about ~1-3% vs full settings, but oh boy 20h vs 4h! And when you offload non-textures to lzma(which you should because low mc will make lolz compress non-textures worse in such case) then you can further fasten usually to 2-3h, like 20+gb game. Again today I compressed Ni no Kuni 2, got almost same size as FG but it took under 2h to compress 41gb of game data.

[78372]

Here is what happened :
There was 34.6G of data
First I used dlz
Speed was 180kbps
I think dlz has entropy check because sometime it was 7mbps (copy) and sometime as slow as 180kbps(lowest), 5%/h.
Then I tried your lolz settings with 128m blocks on some normal data just to test.
It was better than dlz, far better than lzma and 380kb/s speed.
But also remember, my data size is 36.4G, so it asked for 33h because sometimes speed slowed down and sometimes increased.
Then I turned almost everything off in lolz and used the lightest settings.
The ratio was even worse than lzma.
Then it asked for 14hours.

Well at last I understand that my *CELERON* pc isn't going to handle these things at a decent speed, I must upgrade.

[PsYcHo_RaGE]

Originally Posted by 78372

Thanks. I will be applying on GTA V rpf files, everything is textures except few, should I go for this settings?

lolz -mc8 -tt1 -mt1 -mtb512 -d128 -fba4096 -dto0 -dm00

Edit: Still it's asking for 33hours, I think I should use a more simpler settings

Edit2: Ok, I need an upgrade :P

you might want to try this compression ratio increasing and faster speed

lolz -dto0 -tt1 -dm00 -mc1023

[78372]

mc1023 decreases speed, not increase.

However, probably I will use lzt -32 for faster speed and a bit worse ratio than lzma(5%)

[elit]

Originally Posted by 78372

Here is what happened :
I think dlz has entropy check because sometime it was 7mbps (copy) and sometime as slow as 180kbps(lowest), 5%/h.

That is almost certainly because of bursting. It happen also for example with xnlz <stdio> through FA, it load data of block size to memory = during this time speed in FA GUI show 100+m/s, then it start deceasing as it actually process the block and then process repeat again. Dlz is likely simply copying next data to block for processing and show burst speed to output.

Also it could be speed of scanning rather than processing, lolz btw also vary between 2-7mbps.

Finally, 7mbps cannot be a speed of raw copy, that would be too slow for such thing.

Try it on smaller size like 1gb and compare *final* compression time between dlz and lolz(but also ratio). Then you get best picture. If you get similar ratio but visibly better time on dlz then you may be right, but this is highly unlikely and I would be surprised. If anything, I would expect lolz to give you better both speed and ratio - with settings I provided.

[78372]

Your settings is good, and my pc is noob XD
DLZ shoes that it's copying data in the console window, then I saw process hacker which says it's 7mbps, probably it includes scanning speed too.
Anyways, lolz is still slow (as I said, even my settings was lower than you) and in the lowest settings it asks for 14h, obviously because of my celeron processor
However, I finished packing the game, only rpf files(except x64d.rpf and mp dlcs), pzlib:c128mb+srep:m2f+lzt:32, result = 20.7G.
For a small test your settings gave 2x speed than dlz and better ratio