Cost vs return - aka worthiness debate

[elit]

That was indeed idea that appealed to me, thats why I tried that 64bit xz/7z in the past, instead of internal LZMA. I loved 64bit idea. But, it wasnt useful, it was slower, used more memory and did not gave me much better ratio. I think internal LZMA of FA is better than people give it credit for. I think they underestimate it because its only 32bit but I would not be surprised if Bulat applied his own extra tricks on it.

Btw believe or not but not long ago I srep+fa:m5 one ~360mb game. Then I used only -m5 with internal rep but everything else same. For some reason, the one with srep got smaller. Not by much but still, and it wasnt even m5f, I only used m3f. Go figure...

As for masked compressor, I know about it I saw the page before. Looks great but I dont see much point, is it replicating what FA with GUI already have? Or is there anything it can do that FA cant in terms of better compression?(btw I dont like game installers, I like to archive into single archive format)

PS:(I was PM'd again regarding RAZOR Compressor. I know about it and tried it before. Back then when I tried it was extremely slow and only marginal gain but, later I will update post here to add it to that benchmark for completion, so stay sharp ^_^.)

[1234567890123]

Originally Posted by elit

I was PM'd again regarding RAZOR Compressor. I know about it and tried it before. Back then when I tried it was extremely slow and only marginal gain but, later I will update post here to add it to that benchmark for completion, so stay sharp ^_^.

the reason of the use razor is installation time.it's very fast there with better ratio
waiting your benchmarks

[elit]

Greetings again everyone, so I just finished RAZOR compressor results, but I also threw in zpaq on -m4 and -m5 settings for comparison. I always had certain sympathy for zpaq and its family compressors, but I never fount it worth the time wasted. There is a significant difference in ratio(and of course, time) between -m4 vs -m5 but on that later. So lets start.

RAZOR Compressor: 264.55mb That is 2.95% or: 3% better ratio(vs 272.59mb FA -m5)
3% is like, around ~2gb better savings on 60gb game. Sure 2gb less is nice but in the grand scheme of such big data... not really. Or rather, it would be if it wasnt for... time:
FA -m5: 41s
razor: 10min(pffff)

Thats almost 15x slower, so if you compress 60gb game(and remember with precompressing it will actually be around 90+gb), if FA -m5 took ~2h which is typical, you are going to need around 30h(more than full day) with razor. Now, since razor is not multithreaded(used about 33% on 4 core cpu so like 1 1/2), maybe if you 4x4 bind it on FA if possible then that is more reasonable. In that case you could probably get around ~8h - still 4x slower but at least more reasonable, you could let this over night and be fine next morning.

At this point its up to your own judgment if 3% is worth it. For me its not but if other type of data can get better ratio AND if you can 4x4 it(which I am almost sure you cant yet), then maybe. I think its not possible to 4x4 it because razor only work on files not stdin/out and i think thats needed for 4x4. Again, if somebody know better please let me know.

Now to Zpaq:
zpaq -m4: 284.51mb = sucks, 4x slower and worse than old FA -m5 lzma lol, but
zpaq -m5: 246.81mb

Thats ~10% better vs FA -m5! Now we are talking, not with those single digit garbage differences! And hell even that is for my spoiled brain not much but at least 2 digits difference! For this particular file, zpaq -m5 beat razor compressor but keep in mind this:
- zpaq already utilized all 4 cores so no more speedup and...
- it took 12min and...
- it will take same amount of time to decompress, which mean you decide today to play 60gb game you will play it tomorrow at soonest = no thanks

Conclusion from this:
Zpaq is out of game despite being best winner with visibly better ratio margin(for this particular file), no need to talk about paq family in general anymore.
FreeArc -m5 is still the best overall, the only potential replacement worth bragging about could be razor compressor, but not in current state if you cannot 4x4 it. And if you can, let me know but even then its a question of whether ~3[-5?]% gain is worth 4x more compression time. For current single threaded 15x s-l-o-w-n-e-s-s certainly not - not to me at least.

TL;DR:
Keep using FreeArc -m5 with srep64+ztool and stop wasting your time. You have life to live. Thats it.

And so I am done for now. Take this thread for what you will, I hope it was somewhat helpful to you guys despite its limitations. Do feel free to continue discussion though, one never know what can be discovered down the road ^_^.

[elit]

Today I tried well acclaimed zcm from encode.ru at max settings: -m8 -s -t1. It resulted in ~100mb bigger archive than FA -m5 lol. I also tried to tune zpaq(with x, s options that provide more possibilities) to see if I can get ti to more reasonable speed, while keeping good ratio. I was never able to even match FA -m5 with custom setting, anything other than -m5 was no go.


Anyway, since lzma seems like a best option, I decided to test its various settings to see if I can squeeze more from it while maintaining speed. I was inspired by one user in this forum. For this test I used only 16mb dictionary because I wanted avoid FA throttling when it reach memory limit(it wont 4x4 in such cases). But once worthy parameters are found I test them with default rest against previous posts references. I will also refer compression times.

LZMA have multiple parameters, I tested most important ones: mc, lc, bt4, fb. Later also dictionary for comparison. Fb is also known as "word size" in 7z and it is the one that almost everyone set to max 273. In FA it is only referred as a number without letter in lzma parameters. Ok lets start.

Default lzma16mb:
lzma d16mb: 274.18mb - 27s (rest default: mc32,non-bt4, 32[fb], lc3)

Testing mc:
lzma d16mb mc128: 273.89mb - 58s
lzma d16mb mc1000: 273.78mb - 4.53min

^mc have a huge(multi fold) impact on speed while not giving better ratio by almost nothing - aka 0.1%!

Next I tried fb aka "word":
lzma d16mb 64: 273.97mb - 33s
lzma d16mb 128: 273.85mb - 42s
lzma d16mb 273: 274.07mb - 1min

^between default 32 and 128 is only 0.12% difference while compression time increased 55%. With 273 ratio actually worsened. However, this 273 is also affected by dictionary size, for comparison lzma:273:128mb resulted in 272.03mb while lzma:32:128mb resulted 272.39mb. Still gain is almost nothing but, one took 40s while another 2min!

Testing bt4:
lzma d16mb bt4: 273.36mb - 36s

^this parameter gained 0.3% while slowing packing down by 33%. Meh.

Dictionary 96mb(vs 16mb reference above):
lzma 96mb: 272.59mb - 42s

^0.6% gain, 55% time increase and a memory hog.

Finally, lc:
lzma d16mb lc8: 268.22mb - 28s

^yep, ~2.2% gain for F.R.E.E.


So to conclude, the only option worth changing from default was lc. Now lets use that with default settings and compare with previous posts results(-mc4x4/4x4:lzma:lc8):

lzma lc8: 266.57mb - 42s

^but something funny is here, FA defaults to 64mb dictionary, but in -m5 it use 96mb. Lets enforce it:
lzma 96mb lc8: 266.77mb - 37s

^dont ask me why bigger dictionary resulted in slightly bigger file and lower compression time, I just dont...

Conclusion:
The only parameter worth bragging about was lc, which can give you nice 2.2% for free. Rest I would recommend to leave default. The cost/return seems not worth it but maybe you encounter different scenario. With additional 2.2% gain, FA -m5 now not only canceled scary 2 digit(10%) gain of zpaq(back to single digit), it almost matched razor compressor! Which gave 264.55mb or ~3% gain. But at what cost! Razor is now only ~0.75% better while being 15x slower. FitGirl you are c-r-a-z-y .

And so, default FA -m5, now with lc8 extra tunning is even more appealing to me. Hopefully other people will find this helpful too.

[elit]

bcm -b64mb: 323.64mb - 1.30min
bcm -b512mb: 324.26mb - 1.40min

[elit]

RAR5:
rar -m5 -ma5 -md96m: 328.11mb - 32s
rar -m5: 315.53mb - 47s

[artag]

since we are talking about decent space gains vs reasonable compression time:

Using 7zip in replace of internal compression, gives me a good decompression time, because
how it handles uncompressable blocks. Also, it gives me some times better compression than
internal LZMA (because it does not add overhead to uncompressable blocks, try with mkv/mp4 file
and you'll see).

To all the experienced users: what switches do you use for improving compression
time to sane levels (and lose some mb but stay practicall).

I'm talking about internal LZMA and 7zip LZMA.

Also, any advice for a (stable and tested) substitute to lzma(2)?

oodle?
the new lolz?


thanks

[elit]

Originally Posted by artag

Using 7zip in replace of internal compression, gives me a good decompression time, because
how it handles uncompressable blocks. Also, it gives me some times better compression than
internal LZMA (because it does not add overhead to uncompressable blocks, try with mkv/mp4 file
and you'll see).

Unless something changed, its exactly opposite in my experiences. 7zip never looked like skipping uncompressable data(but maybe it changed now?) to me and was much slower than FA -m5, only FA -mx seemed like to not skip. FA -m5 is the quickest LZMA I know, skipping everything that is compressed - the jumps it make during compression are very, very visible. Thats what arc.groups file is also for btw.

And for decompression, 7zip cannot even compare, it is using only single thread while FA can use 4 threads(if you compressed with -m5). Good luck beating that with 7zip.

[78372]

7z LZMA2 skips incompressible data and can do multithreading better.
In my case I don't use multithreading and because of my poor pc, I just can't use bigger dictionary so I use srep and afterwords xz(lzma2) with preconfigured options(9x). It is not that slow but ratio is good.

[artag]

Ok, it seems there is a missunderstanding here.

I don't use the builting methods of FA. Those are masked (group) compression and some other pre-built stuff. If it works for you, be happy.

I'm talking about something else.

I have mi own compression chains. Usually, people use somethin like

pzlib+srep+lzma for game assets compression (or 3d scenes/models in my case)

lzma on FA only use two cores and has some speed issues during decompression.

7zip LZMA2 can use more than 2 cores and it has increased speed during compression,

also during decompression thanks to skipped uncompressable blocks.

It gives me faster compression and decompresion and is a good substitute for internal

LZMA, specially if time is a concern. Even if you use less agressive settings (because of time) it its a better feature - vs - size compared to internal lzma.

I know about 4x4 lzma, but the memory scales too much during compression if i use a larger
block, with 7zip i can use 2 or 3 threads and save memory if i want larger blocks.


My question to experienced users (or repackers?) is if there is a better candidate than
7zip. (Even if i sacrifice some MB, better compression / decompression speed is desirable).


you can use 7zip as external like this:

[External compressor:7zip]
header = 0
packcmd   = 7za a -txz -an -mcrc=0 -mf=off -myx=0 -mmt=4 -m0=lzma2{:option} -si -so <stdin> <stdout>
unpackcmd = 7za x -txz -an -y -si -so <stdin> <stdout>

Set -mmt=4 to 3 or 2 if it's eating more memory that you have or you wat larger blocks.


invoke it like this

7zip:d=512m or 7zip:d=64m:fb=16 or 7zip:d=64m:fb=16:a=0

pzlib+srep+7zip ,etc

etc
less d=, less memory, or reduce mmt.

use task manager to monitor the exe

<stdin> <stdout> works ok on my end, even with inno setup

good luck

[78372]

Here is the XZ I typically use instead of 7z xz. It also supports stdio, and you have some predefined levels of compression. It is only ST.

[Gupta]

In repacking we compress once and decompress many times ~ the answer for all ur arguments

RZ takes more time for compression but decompress faster then lzma with compression benefit ofc same goes for lolz srep etc

[elit]

Originally Posted by PrinceGupta2000

In repacking we compress once and decompress many times ~ the answer for all ur arguments
RZ takes more time for compression but decompress faster then lzma with compression benefit ofc same goes for lolz srep etc

RZ take *way* too long and results are diminishing, I tried it again few days ago this time on ~12+gb game(with decompiled/decompressed resources). Srep even hurt it but also without, gain was like nothing. RZ gave me 6.4gb without srep and 6.5gb with srep, while FA -m5+srep gave me around 6.6-6.7gb. FA packed it in few minutes while RZ took night. Frankly I dont get all the fuss about RZ and the like, its a slow garbage with diminishing returns, srep + non-aggressive lzma can come so close to it for under 3% loss most of the time.

He also said "Even if i sacrifice some MB, better compression / decompression speed is desirable" so no it does not answer his arguments. Furthermore, while I agree that decompressing speed is more important, I dont agree about its exclusivity. Unless you are repacker who upload your repacks to the world, chances are you are likely only going to decompress on occasion. But wasting day and night compressing for marginal gain is crazy.

Originally Posted by artag

Ok, it seems there is a missunderstanding here. I don't use the builting methods of FA.
lzma on FA only use two cores and has some speed issues during decompression.
7zip LZMA2 can use more than 2 cores and it has increased speed during compression,
also during decompression thanks to skipped uncompressable blocks.
I know about 4x4 lzma, but the memory scales too much during compression if i use a larger
block, with 7zip i can use 2 or 3 threads and save memory if i want larger blocks.

you can use 7zip as external like this:

[CODE][External compressor:7zip]

I see, first thank you for config snippets, when I have time I will definitely give it try. With that said, when you say lzma on FA only use 2 cores its again flag dependent. -mx use only 2, -m5 use 4. Or more precisely, 7z use lzma2 while FA use 4xlzma1(or 2xlzma1 as one can use 2 cores I believe). For decompression again I cant see 7zip beating FA because regardless how you compress with 7z it only use 1 core during decompression from what I know. Sure its better now if it can skip through uncompresseable data but so can do FA and with 4 cores in use. Also, I cant see how FA use "too much memory" in 4x4 mode when it is known to have a hardcoded limitation(its only 32bit after all). FA use around same much memory in -mx and -m5, in -mx it allocate 176mb dictionary+2 cores, while in -m5 96mb dict+4x4.

I think what you are trying to say is that in same 2core compression mode, 7z is better than FA. Which I think is true, but it was your decision not to use -m5(or any build in methods as you mentioned). I have made many tests outside this topic, I never saw once srep+FA -mx having any significant gain over srep+FA -m5. Without srep that would be different though. I will try again srep+7z with your settings though.

Originally Posted by 78372

7z LZMA2 skips incompressible data and can do multithreading better.

Will try it how well it "skips", though multithreading as mentioned already is only good for compression. There are people here willing to waste days on marginal compression, saying decompression speed matter most, yet many of them have no problem with limiting single threaded decompression of 7z. Go figure. Me it bothered a lot when I used 7z in the past, before I discovered FA.

[Gupta]

LoL{no offense} answer will be here when i have time In the mean time can you edit ur post with the srep flags u use

I don't waste days and on decompression... Have my own wrapper for multithreaded decompression { hint: if you don't know programming u can use an inbuilt program for multithreaded decompression}

I don't think it "skips"

[elit]

Originally Posted by PrinceGupta2000

In the mean time can you edit ur post with the srep flags u use

Have my own wrapper for multithreaded decompression

I don't think it "skips"

- I just used normal -m3f on the latest srep(which have m1f-m5f range)

- RZ doesnt even support stdio and its not an open source. Are you telling me you can "stream" from multiple parallel RZ's into single archive output??

-Indeed it doesnt, I just tested, will be part of my next big "data" comment. I dont know where are these guys getting such a false info.