Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #1  
Old 27-09-2017, 12:39
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Cost vs return - aka worthiness debate

So, I have made my own repacks "couple" of times before. And one thing I repeatedly kept stumbling upon was how certain(often propagated) extreme compression methods were just not worth it. And I always wondered if I missed something or what, because in all my cases extreme compression methods that took like half day gained at most 1% or less than those that could finish job in 1-2h.

For example, packing Project CARS 2 with (freearc)srep:m5f+lzma:mx took I think ~5h and rendered about ~25.4gb from +-40gb. With srep:m3f+lzma:m5 it took about 1+1/2h and I got 25.68gb. And a lot of that time was probably disk bottleneck. There is a huge speed difference on FreeArc between -mx and -m5 as -m5 utilize all 4 cores and other speedups. I understand LZMA is more effective with 1-2 cores instead of breaking data to blocks, but I just dont see much benefits of it - at least if used with srep.

Another example like above was Ghost Recon Wildlands, with similar ridiculous ratio. In fact every single time I decided to "ok lets try on this one again", -mx option simply wasnt worth it. I saw bigger difference only if I did not used srep, then sometimes it really helped more(but not groundbreaking more either), but with anything chained from srep it just didnt do much better. In fact for one or 2 cases(I dont remember which games anymore), srep did the job and lzma gained 0%, you could pretty much just srep them and be done with it.

And if that wasnt enough, I tried to replace default FA's lzma, I tried srep+7z, then srep+(horribly-slow-at-max-settings)xz, and it wasnt worth it at all. Like, say 1.58gb with max xz vs 1.61gb with FA -m5. Thats nothing and one took at least hour+ to do it while other only around ~5min.
Oh and sometimes, FA's -mx gave me few kb bigger archive than -m5!
Hell I even tried zpaq -m4, that one gave better ratio when alone only(vs lzma without srep) but not better when with srep+zpaq or vs srep+lzma.

But I keep reading everywhere people like to chain srep with xz or 7z or whatever and I wonder... why?
With srep at least, even -m5f if fine and quick enough, but this extreme lzma BSDM fetish puzzle me.

------------------------------------------------------------------------------------------------------------------------------

So with that said I would like to know, if you have different experience and you tried that FA's -m5 to compare, can you tell me where you got that significant difference, at which game and how much better and with what parameters? What game can you confirm to see better ratio worth bragging about with srep+lzma:mx(xz,7z) vs srep+lzma:m5? Remember only with srep combined.

Thanks for discussion .
Reply With Quote
The Following 10 Users Say Thank You to elit For This Useful Post:
1234567890123 (28-09-2017), 78372 (28-09-2017), Andu21 (29-09-2017), COPyCAT (24-01-2018), devil777 (06-08-2020), EzzEldin16 (01-01-2018), mubbii (19-03-2019), oltjon (27-09-2017), Razor12911 (27-09-2017), shazzla (27-09-2017)
Sponsored Links
  #2  
Old 27-09-2017, 19:54
ChronoCross's Avatar
ChronoCross ChronoCross is offline
Registered User
 
Join Date: Sep 2014
Location: Acacia Dragons
Posts: 254
Thanks: 145
Thanked 199 Times in 115 Posts
ChronoCross is on a distinguished road
I think you've grown, and you're an adult.
Analyzing these things makes you realize how much time you've lost on things like this. LOL.
I tell you something, before going to work, I turn on the computer and start doing my backups - repack and when I go home they are ready.
I have an AMD and as it is known the cores work independently of each other as opposed to intel that reinforces their cores with each other. so I only use one core to compress. It takes a lot of time but good is the best option.
Finally the difference between m5 or mx on my PC results vary from 500mb to 800mb in games eg: 8gb> m5 = compressed in 2500mb ---- and ---- mx = compressed in 1800mb I have a big difference, but it always depends on the data to be compressed and the experience of each of the backups you have done, like wich tech or algo i need to compress this game.
so i choose mx instead m5.
I do not mind the compression time.
but the time of extraction if I care.
i like compression slow and faster unpack.
Good luck in the ilumination path.
Reply With Quote
The Following 2 Users Say Thank You to ChronoCross For This Useful Post:
1234567890123 (28-09-2017), elit (28-09-2017)
  #3  
Old 27-09-2017, 20:16
Razor12911's Avatar
Razor12911 Razor12911 is offline
Programmer
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,628
Thanks: 2,003
Thanked 10,315 Times in 2,197 Posts
Razor12911 is on a distinguished road
Project CARS 2 is a bad example because the game is encrypted, what do you think all compression methods will do to high entropy data? Absolutely nothing, which is why there isn't much difference between weak and strong compression method but all in all you do have a point.

Edit:
Best thing you can run comparison is on data that's uncompressed because also Ghost Recon Wildlands, from what I can read, you compressed already compressed data, try precompressing game first then run test once more.

Last edited by Razor12911; 27-09-2017 at 20:37.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
COPyCAT (24-01-2018), elit (28-09-2017)
  #4  
Old 27-09-2017, 23:23
felice2011's Avatar
felice2011 felice2011 is offline
Registered User
 
Join Date: Feb 2011
Location: italy
Posts: 834
Thanks: 348
Thanked 1,142 Times in 386 Posts
felice2011 is on a distinguished road
If you want to save time, with fast scans of files in general to know the level of entropy, before you test for hours the methods sometimes ineffective on some files, http://www.fileforums.com/showthread.php?t=99070 & http://www.fileforums.com/showthread.php?t=99136. As the "INDEX - Conversion Tutorial Index" has not been updated ... For those who want something magical there is always this http://www.fileforums.com/showthread.php?t=99080, for sure they will be satisfied.
__________________
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
I Mediocri Imitano, I Geni Copiano, Dio Crea & Distrugge (Io Ridefinisco & Perfeziono le Loro Opere Rendendole Uniche)
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
Mediocrities Imitate, Genius Copy, God Creates & Destroys (I Reconsider & Improve Their Works, Rending Them One And Only)

Last edited by felice2011; 27-09-2017 at 23:30.
Reply With Quote
The Following 3 Users Say Thank You to felice2011 For This Useful Post:
COPyCAT (24-01-2018), elit (28-09-2017), Razor12911 (28-09-2017)
  #5  
Old 27-09-2017, 23:46
Chayan Manna's Avatar
Chayan Manna Chayan Manna is offline
Registered User
 
Join Date: Jun 2016
Location: India
Posts: 82
Thanks: 89
Thanked 111 Times in 37 Posts
Chayan Manna is on a distinguished road
Quote:
Originally Posted by felice2011 View Post
For those who want something magical there is always this http://www.fileforums.com/showthread.php?t=99080, for sure they will be satisfied.
Hehe, It's really a magical tool
Reply With Quote
The Following User Says Thank You to Chayan Manna For This Useful Post:
Razor12911 (28-09-2017)
  #6  
Old 28-09-2017, 04:39
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Quote:
Originally Posted by ChronoCross View Post
Finally the difference between m5 or mx on my PC results vary from 500mb to 800mb in games eg: 8gb> m5 = compressed in 2500mb ---- and ---- mx = compressed in 1800mb I have a big difference
Thanks a lot, but was that with srep combined?

Quote:
Originally Posted by Razor12911 View Post
Project CARS 2 is a bad example because the game is encrypted... Best thing you can run comparison is on data that's uncompressed...
Yes that is good advice, but remember original size was over 40+gb so even without that you can still compress it to almost half size - without decrypting. In that regards shouldnt -mx vs -m5 still show a visible difference? Its still a lot of gigs that were compressed after all.

Quote:
Originally Posted by Chayan Manna View Post
Hehe, It's really a magical tool
Yeah, he have me for stupid
Reply With Quote
  #7  
Old 28-09-2017, 04:48
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Quote:
Originally Posted by PrinceGupta2000 View Post
compress a game with a64 and see yourself
A64 is an accelerator flag and if anything, higher number should provide worse compression(although in latest versions its about same as a1), but more specifically, it doesnt have anything to do with compression ratio it is to speed up process.

With that said, I dont have problem with srep at max at all its quick enough, it is lzma that makes me question.
Reply With Quote
  #8  
Old 28-09-2017, 04:57
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
One thing i was considering to try is uharc(for games). I know it have a 2gb file limit though and its single core only. But I was thinking "maybe if I pipe it in freearc through <stdin><stdout>" and bind it into 4x4, then FA would run 4 instances on separate blocks(just like it does 4x4:lzma on -m5), would that work?

Something like:
-mc:4x4/4x4:64mb:uharc

packcmd=uharc {cmd} - - <stdin><stdout>
unpackcmd=uharc {cmd} - - <stdin><stdout>


Or like that, I wonder if that would give visibly better ratio than lzma since uharc is a multimedia cmp...?
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
EzzEldin16 (04-10-2017)
  #9  
Old 28-09-2017, 06:17
Andu21 Andu21 is offline
Registered User
 
Join Date: Feb 2016
Location: Impossible Planet
Posts: 114
Thanks: 179
Thanked 48 Times in 38 Posts
Andu21 is on a distinguished road
Interesting topic, curiosly enough i find myself in that same situation. You could try Uharc's cls made by Razor12911 as well http://fileforums.com/showthread.php?t=98005 iirc it overcomes the filesize limit. Regarding compression ratio i still have to find a game in which uharc beats lzma, maybe a precompressor is needed in order to do that or maybe a test in older games.

Last edited by Andu21; 28-09-2017 at 06:23.
Reply With Quote
The Following 2 Users Say Thank You to Andu21 For This Useful Post:
elit (28-09-2017), EzzEldin16 (04-10-2017)
  #10  
Old 28-09-2017, 08:32
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Quote:
Originally Posted by Andu21 View Post
You could try Uharc's cls made by Razor12911...
Thank you! I completely missed this one, man that guy is full of good surprises
Interestingly he use 0.6a in his package, I am pretty sure latest version was 0.6b. Will try to replace exe with newer it should work.
Reply With Quote
  #11  
Old 28-09-2017, 10:15
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Alright since you people may be interested in this, I tried uharc0.6b at various settings and also FA with -m5 and -mx and additionally 7zip:ultra for comparison. This was done on random preprocessed 660mb .forge file(from its original 448mb, using ztool). This is of course only single small test and doesnt have enough data to prove, but it does reflect with my other tests and show qualities and flaws well enough IMO.

On uharc these additional options were used: -md32768 -mm+
Then I tested 3 its main methods each: mx, mz, m3.

Results:
original: 660mb
uharc -mz: 317.22mb
uharc -mx: 290.79mb
uharc -m3: 284.27mb
7z -ultra: 271.37mb
FA -mx: 271.46mb
FA -m5: 272.59mb

FA -m5 was absolutely quickest by significant margin to any other test, while resulting in only slightly bigger archive - specifically 0.4145% difference from FA -mx. Halo? Anyone?
Ehm, anyway, for uharc except -mz all others were very slow, binding them to 4x4(if possible) would help but my guess is they would only match FA -mx speed then. FA -mx was still quicker than uharc(except -mz) but its true that FA use 2 cores while uharc used only 1.

7zip was great no doubt !but!, first it is slow as hell despite using all 4 cores and second, it used over 3gb+ memory for this single 660mb file(!!!). FA was much more rational with memory usage on both -m5 and -mx(1gb at peak and even shrinked during compression). Uharc had best memory usage, in -mx mode it used only 50mb(!), this would put him as a good candidate to chain it with 4x4 and srep could then pipe directly to 4xuha:ppm for only ~200mb usage without waiting for itself to finish 1st cycle.

Anyway, between uharc -mx and FA -m5 was still 6.67% difference in favor of FA -m5!

In conclusion, this small test once again confirmed my thoughts, i.e. dont bother with anything higher than FA -m5, especially with ztool/srep, its really good enough and you are likely wasting your time with anything higher. Bulat has it figured out for us and gave us best and most optimized compression tool ever made. FA -m5 rule.

EDIT: I just tried MCM compressor with -m9 option that I was PM'd, 271.08mb + taking ages = not worth it

Last edited by elit; 28-09-2017 at 14:09.
Reply With Quote
  #12  
Old 28-09-2017, 10:58
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
May I ask, why are you looking for internal compression methods while we can use srep+lzma which is better. For me, I just use srep:m3f+lzma which us actually good who wants to spend less time and resources while compressing. You want good difference between mx/m5 vs srep+lzma, then you can try compressing a big game without compressed streams like far cry 4
__________________
NOT AVAILABLE

Last edited by 78372; 28-09-2017 at 11:02.
Reply With Quote
  #13  
Old 28-09-2017, 11:28
Joe Forster/STA's Avatar
Joe Forster/STA Joe Forster/STA is offline
Senior forum member
 
Join Date: Nov 2000
Location: Hungary
Posts: 9,706
Thanks: 10
Thanked 297 Times in 188 Posts
Joe Forster/STA is on a distinguished road
Quote:
Originally Posted by ChronoCross View Post
I think you've grown, and you're an adult. Analyzing these things makes you realize how much time you've lost on things like this.
Very wisely said, I cannot agree more! However, if I may add something, this time was not completely lost if you learnt something new, interesting during those experiments, detours.
__________________
Joe Forster/STA
For more information, see the FileForums forum rules and the PC Games forum FAQ!
Don't contact me via E-mail or PM to ask for help with anything other than patches (or software in general) done by me, otherwise your request may be deleted without any reply!
Homepage: http://sta.c64.org, E-mail: [email protected]; for attachments, send compressed (ZIP or RAR) files only, otherwise your E-mail will bounce back!
Reply With Quote
  #14  
Old 28-09-2017, 13:05
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 224
Thanks: 163
Thanked 287 Times in 99 Posts
elit is on a distinguished road
Quote:
Originally Posted by 78372 View Post
May I ask, why are you looking for internal compression methods while we can use srep+lzma which is better. For me, I just use srep:m3f+lzma which us actually good who wants to spend less time and resources while compressing. You want good difference between mx/m5 vs srep+lzma, then you can try compressing a big game without compressed streams like far cry 4
But lzma is also internal compression of FreeArc. What I was questioning is people going extra efforts to replacing internal LZMA with external LZMA's which, at least in my limited tests did not proved to be any significantly better and in fact were pretty much always(significantly) slower.

Now, in case you meant why to use freearc at all and not just piped external command tools(like srep+xz in command like for example), I was thinking about it in the past and really like the idea of own, clean tool chain on cmd, but freearc still offer other things, like groups, arc universal format regardless of replaced tools, sorting, UI, specific optimizations like exe, bmp and wav, skipping of compressed data and so on. Would really simple srep+lzma be better than srep+freearc with its many other advantages? Or am I missing here something?
Reply With Quote
  #15  
Old 28-09-2017, 20:08
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 649
Thanks: 772
Thanked 935 Times in 308 Posts
78372 is on a distinguished road
As you say of external, a major benefit of using external is removing 32bit limitation. srep+lzma is always better than srep+mx or soo, if you are not using old srep versions as srep 1.5. Yesterday I tested this on a 513mb sample file. "513mb" so rep can on work on it instead of srep. First I applied mx and got 454MB on that file. Then I used srep+lzma, which didn't took much time and gave me 452mb. These 513mb files got some executables, some texts and some other files and clearly fa's grouping didn't benefited me anyways. srep+lzma is not always good, if you want multimedia and other detection and use algorithms as groups, you must use masked data compression. You can find it here
__________________
NOT AVAILABLE
Reply With Quote
The Following User Says Thank You to 78372 For This Useful Post:
mubbii (15-11-2018)
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Return To Castle Wolfenstein info the_fsr PC Games 0 01-04-2004 18:20



All times are GMT -7. The time now is 20:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com