Cost vs return - aka worthiness debate

[elit]

So, I have made my own repacks "couple" of times before. And one thing I repeatedly kept stumbling upon was how certain(often propagated) extreme compression methods were just not worth it. And I always wondered if I missed something or what, because in all my cases extreme compression methods that took like half day gained at most 1% or less than those that could finish job in 1-2h.

For example, packing Project CARS 2 with (freearc)srep:m5f+lzma:mx took I think ~5h and rendered about ~25.4gb from +-40gb. With srep:m3f+lzma:m5 it took about 1+1/2h and I got 25.68gb. And a lot of that time was probably disk bottleneck. There is a huge speed difference on FreeArc between -mx and -m5 as -m5 utilize all 4 cores and other speedups. I understand LZMA is more effective with 1-2 cores instead of breaking data to blocks, but I just dont see much benefits of it - at least if used with srep.

Another example like above was Ghost Recon Wildlands, with similar ridiculous ratio. In fact every single time I decided to "ok lets try on this one again", -mx option simply wasnt worth it. I saw bigger difference only if I did not used srep, then sometimes it really helped more(but not groundbreaking more either), but with anything chained from srep it just didnt do much better. In fact for one or 2 cases(I dont remember which games anymore), srep did the job and lzma gained 0%, you could pretty much just srep them and be done with it.

And if that wasnt enough, I tried to replace default FA's lzma, I tried srep+7z, then srep+(horribly-slow-at-max-settings)xz, and it wasnt worth it at all. Like, say 1.58gb with max xz vs 1.61gb with FA -m5. Thats nothing and one took at least hour+ to do it while other only around ~5min.
Oh and sometimes, FA's -mx gave me few kb bigger archive than -m5!
Hell I even tried zpaq -m4, that one gave better ratio when alone only(vs lzma without srep) but not better when with srep+zpaq or vs srep+lzma.

But I keep reading everywhere people like to chain srep with xz or 7z or whatever and I wonder... why?
With srep at least, even -m5f if fine and quick enough, but this extreme lzma BSDM fetish puzzle me.

------------------------------------------------------------------------------------------------------------------------------

So with that said I would like to know, if you have different experience and you tried that FA's -m5 to compare, can you tell me where you got that significant difference, at which game and how much better and with what parameters? What game can you confirm to see better ratio worth bragging about with srep+lzma:mx(xz,7z) vs srep+lzma:m5? Remember only with srep combined.

Thanks for discussion .

[ChronoCross]

I think you've grown, and you're an adult.
Analyzing these things makes you realize how much time you've lost on things like this. LOL.
I tell you something, before going to work, I turn on the computer and start doing my backups - repack and when I go home they are ready.
I have an AMD and as it is known the cores work independently of each other as opposed to intel that reinforces their cores with each other. so I only use one core to compress. It takes a lot of time but good is the best option.
Finally the difference between m5 or mx on my PC results vary from 500mb to 800mb in games eg: 8gb> m5 = compressed in 2500mb ---- and ---- mx = compressed in 1800mb I have a big difference, but it always depends on the data to be compressed and the experience of each of the backups you have done, like wich tech or algo i need to compress this game.
so i choose mx instead m5.
I do not mind the compression time.
but the time of extraction if I care.
i like compression slow and faster unpack.
Good luck in the ilumination path.

[Razor12911]

Project CARS 2 is a bad example because the game is encrypted, what do you think all compression methods will do to high entropy data? Absolutely nothing, which is why there isn't much difference between weak and strong compression method but all in all you do have a point.

Edit:
Best thing you can run comparison is on data that's uncompressed because also Ghost Recon Wildlands, from what I can read, you compressed already compressed data, try precompressing game first then run test once more.

[felice2011]

If you want to save time, with fast scans of files in general to know the level of entropy, before you test for hours the methods sometimes ineffective on some files, http://www.fileforums.com/showthread.php?t=99070 & http://www.fileforums.com/showthread.php?t=99136. As the "INDEX - Conversion Tutorial Index" has not been updated ... For those who want something magical there is always this http://www.fileforums.com/showthread.php?t=99080, for sure they will be satisfied.

[Chayan Manna]

Originally Posted by felice2011

For those who want something magical there is always this http://www.fileforums.com/showthread.php?t=99080, for sure they will be satisfied.

Hehe, It's really a magical tool

[elit]

Originally Posted by ChronoCross

Finally the difference between m5 or mx on my PC results vary from 500mb to 800mb in games eg: 8gb> m5 = compressed in 2500mb ---- and ---- mx = compressed in 1800mb I have a big difference

Thanks a lot, but was that with srep combined?

Originally Posted by Razor12911

Project CARS 2 is a bad example because the game is encrypted... Best thing you can run comparison is on data that's uncompressed...

Yes that is good advice, but remember original size was over 40+gb so even without that you can still compress it to almost half size - without decrypting. In that regards shouldnt -mx vs -m5 still show a visible difference? Its still a lot of gigs that were compressed after all.

Originally Posted by Chayan Manna

Hehe, It's really a magical tool

Yeah, he have me for stupid

[elit]

Originally Posted by PrinceGupta2000

compress a game with a64 and see yourself

A64 is an accelerator flag and if anything, higher number should provide worse compression(although in latest versions its about same as a1), but more specifically, it doesnt have anything to do with compression ratio it is to speed up process.

With that said, I dont have problem with srep at max at all its quick enough, it is lzma that makes me question.

[elit]

One thing i was considering to try is uharc(for games). I know it have a 2gb file limit though and its single core only. But I was thinking "maybe if I pipe it in freearc through <stdin><stdout>" and bind it into 4x4, then FA would run 4 instances on separate blocks(just like it does 4x4:lzma on -m5), would that work?

Something like:
-mc:4x4/4x4:64mb:uharc

packcmd=uharc {cmd} - - <stdin><stdout>
unpackcmd=uharc {cmd} - - <stdin><stdout>


Or like that, I wonder if that would give visibly better ratio than lzma since uharc is a multimedia cmp...?

[Andu21]

Interesting topic, curiosly enough i find myself in that same situation. You could try Uharc's cls made by Razor12911 as well http://fileforums.com/showthread.php?t=98005 iirc it overcomes the filesize limit. Regarding compression ratio i still have to find a game in which uharc beats lzma, maybe a precompressor is needed in order to do that or maybe a test in older games.

[elit]

Originally Posted by Andu21

You could try Uharc's cls made by Razor12911...

Thank you! I completely missed this one, man that guy is full of good surprises
Interestingly he use 0.6a in his package, I am pretty sure latest version was 0.6b. Will try to replace exe with newer it should work.

[elit]

Alright since you people may be interested in this, I tried uharc0.6b at various settings and also FA with -m5 and -mx and additionally 7zip:ultra for comparison. This was done on random preprocessed 660mb .forge file(from its original 448mb, using ztool). This is of course only single small test and doesnt have enough data to prove, but it does reflect with my other tests and show qualities and flaws well enough IMO.

On uharc these additional options were used: -md32768 -mm+
Then I tested 3 its main methods each: mx, mz, m3.

Results:
original: 660mb
uharc -mz: 317.22mb
uharc -mx: 290.79mb
uharc -m3: 284.27mb
7z -ultra: 271.37mb
FA -mx: 271.46mb
FA -m5: 272.59mb

FA -m5 was absolutely quickest by significant margin to any other test, while resulting in only slightly bigger archive - specifically 0.4145% difference from FA -mx. Halo? Anyone?
Ehm, anyway, for uharc except -mz all others were very slow, binding them to 4x4(if possible) would help but my guess is they would only match FA -mx speed then. FA -mx was still quicker than uharc(except -mz) but its true that FA use 2 cores while uharc used only 1.

7zip was great no doubt !but!, first it is slow as hell despite using all 4 cores and second, it used over 3gb+ memory for this single 660mb file(!!!). FA was much more rational with memory usage on both -m5 and -mx(1gb at peak and even shrinked during compression). Uharc had best memory usage, in -mx mode it used only 50mb(!), this would put him as a good candidate to chain it with 4x4 and srep could then pipe directly to 4xuha:ppm for only ~200mb usage without waiting for itself to finish 1st cycle.

Anyway, between uharc -mx and FA -m5 was still 6.67% difference in favor of FA -m5!

In conclusion, this small test once again confirmed my thoughts, i.e. dont bother with anything higher than FA -m5, especially with ztool/srep, its really good enough and you are likely wasting your time with anything higher. Bulat has it figured out for us and gave us best and most optimized compression tool ever made. FA -m5 rule.

EDIT: I just tried MCM compressor with -m9 option that I was PM'd, 271.08mb + taking ages = not worth it

[78372]

May I ask, why are you looking for internal compression methods while we can use srep+lzma which is better. For me, I just use srep:m3f+lzma which us actually good who wants to spend less time and resources while compressing. You want good difference between mx/m5 vs srep+lzma, then you can try compressing a big game without compressed streams like far cry 4

[Joe Forster/STA]

Originally Posted by ChronoCross

I think you've grown, and you're an adult. Analyzing these things makes you realize how much time you've lost on things like this.

Very wisely said, I cannot agree more! However, if I may add something, this time was not completely lost if you learnt something new, interesting during those experiments, detours.

[elit]

Originally Posted by 78372

May I ask, why are you looking for internal compression methods while we can use srep+lzma which is better. For me, I just use srep:m3f+lzma which us actually good who wants to spend less time and resources while compressing. You want good difference between mx/m5 vs srep+lzma, then you can try compressing a big game without compressed streams like far cry 4

But lzma is also internal compression of FreeArc. What I was questioning is people going extra efforts to replacing internal LZMA with external LZMA's which, at least in my limited tests did not proved to be any significantly better and in fact were pretty much always(significantly) slower.

Now, in case you meant why to use freearc at all and not just piped external command tools(like srep+xz in command like for example), I was thinking about it in the past and really like the idea of own, clean tool chain on cmd, but freearc still offer other things, like groups, arc universal format regardless of replaced tools, sorting, UI, specific optimizations like exe, bmp and wav, skipping of compressed data and so on. Would really simple srep+lzma be better than srep+freearc with its many other advantages? Or am I missing here something?

[78372]

As you say of external, a major benefit of using external is removing 32bit limitation. srep+lzma is always better than srep+mx or soo, if you are not using old srep versions as srep 1.5. Yesterday I tested this on a 513mb sample file. "513mb" so rep can on work on it instead of srep. First I applied mx and got 454MB on that file. Then I used srep+lzma, which didn't took much time and gave me 452mb. These 513mb files got some executables, some texts and some other files and clearly fa's grouping didn't benefited me anyways. srep+lzma is not always good, if you want multimedia and other detection and use algorithms as groups, you must use masked data compression. You can find it here