Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #1  
Old 17-04-2018, 09:40
joekkerr joekkerr is offline
Registered User
 
Join Date: Apr 2018
Location: Panama
Posts: 10
Thanks: 4
Thanked 8 Times in 6 Posts
joekkerr is on a distinguished road
Smile Current compressing techniques guidance

Hi, im JoeKkerr

I was actively participating in making repacks from 2010-2014.

i'm not completely new to repacking, but i remember my lack of knowledge in my end days where i wasnt able to compress or keep up with others size.

what i did was what others were doing take files > Freearc > batch file unpacking.
or some files needed precomp or a combination of precomp + srep but problem started arising when some files just werent getting reduced, even after using precomp they werent 'precompressing' or as i'd call it deflating. while groups like blackbox and kaos were able to do it. (ik it applies to only some files that can actually be deflated)

i even tried studying thier repacks and their exe's to replicate result and hence i gave up knowing more people were doing better job than me and my internet was 1MbPS with 100kb download and 25 upload.

I'm thinking of getting back into data compression. whats the current method of compressing data ? i know different games , different methods but most of them are same. i assume freearc is still the king, 0.50 version compressed better than 0.666 version back in the day.

Can anyone guide me? i was inspired by skullptura btw

Last edited by Grumpy; 18-04-2018 at 18:50. Reason: Removed Warez site link!
Reply With Quote
The Following User Says Thank You to joekkerr For This Useful Post:
Spexxos (18-04-2018)
Sponsored Links
  #2  
Old 17-04-2018, 10:59
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 589
Thanks: 3,266
Thanked 993 Times in 383 Posts
Simorq is on a distinguished road
FreeArc 6.7
Reply With Quote
The Following User Says Thank You to Simorq For This Useful Post:
Spexxos (18-04-2018)
  #3  
Old 17-04-2018, 11:16
joekkerr joekkerr is offline
Registered User
 
Join Date: Apr 2018
Location: Panama
Posts: 10
Thanks: 4
Thanked 8 Times in 6 Posts
joekkerr is on a distinguished road
Thanks but i was asking for an elaborate explanation.
Reply With Quote
The Following User Says Thank You to joekkerr For This Useful Post:
Spexxos (18-04-2018)
  #4  
Old 17-04-2018, 11:47
ffmla ffmla is offline
Registered User
 
Join Date: Nov 2014
Location: Digital world
Posts: 71
Thanks: 171
Thanked 21 Times in 19 Posts
ffmla is on a distinguished road
Quote:
Originally Posted by joekkerr View Post
Hi, im JoeKkerr

I was actively participating in making repacks from 2010-2014.

i'm not completely new to repacking, but i remember my lack of knowledge in my end days where i wasnt able to compress or keep up with others size.

what i did was what others were doing take files > Freearc > batch file unpacking.
or some files needed precomp or a combination of precomp + srep but problem started arising when some files just werent getting reduced, even after using precomp they werent 'precompressing' or as i'd call it deflating. while groups like blackbox and kaos were able to do it. (ik it applies to only some files that can actually be deflated)

i even tried studying thier repacks and their exe's to replicate result and hence i gave up knowing more people were doing better job than me and my internet was 1MbPS with 100kb download and 25 upload.

I'm thinking of getting back into data compression. whats the current method of compressing data ? i know different games , different methods but most of them are same. i assume freearc is still the king, 0.50 version compressed better than 0.666 version back in the day.

Can anyone guide me? i was inspired by skullptura btw
You may edit the repack groups names from your post.Otherwise the forum MOD warn you..!
At first i am also got the warning.
Reply With Quote
The Following User Says Thank You to ffmla For This Useful Post:
Spexxos (18-04-2018)
  #5  
Old 17-04-2018, 16:42
Spexxos Spexxos is offline
Registered User
 
Join Date: Apr 2017
Location: Germany
Posts: 12
Thanks: 21
Thanked 1 Time in 1 Post
Spexxos is on a distinguished road
I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, and since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers
Reply With Quote
The Following User Says Thank You to Spexxos For This Useful Post:
EzzEldin16 (18-04-2018)
  #6  
Old 18-04-2018, 00:06
joekkerr joekkerr is offline
Registered User
 
Join Date: Apr 2018
Location: Panama
Posts: 10
Thanks: 4
Thanked 8 Times in 6 Posts
joekkerr is on a distinguished road
Quote:
Originally Posted by Spexxos View Post
I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, and since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers
precomp is used when there is a file already compressed (i dont know the technical word) . But a file that is , if you try freearc you wont get a good result because its already compressed with low level compression, this makes high level compression lose its effectiveness. Hence, precomp..deflates the file, i.e removes the low level compression and allows it to be compressed by high level compression freearc offers.

an example would be, try RAR ing a normal text file with medium compression, then 7zip that file, you wont get any good result.
now try taht with using precomp first or on raw text file without winrar.

last time i checked (which was a long time ago) LZMA (7zip) was better than freearc for text files.
That brings us to your next question, why freearc you ask ?
Some files get better compression ratios with Freearc , some with 7zip. Mostly its freearc hands down, but some game files like Assassins creed files get better result with 7zip. Depends on game
Reply With Quote
The Following 3 Users Say Thank You to joekkerr For This Useful Post:
EzzEldin16 (18-04-2018), flaek1337 (03-06-2018), Spexxos (18-04-2018)
  #7  
Old 18-04-2018, 02:05
78372's Avatar
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 540
Thanks: 611
Thanked 639 Times in 237 Posts
78372 is on a distinguished road
Quote:
Originally Posted by Spexxos View Post
I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers
1. DELTA is a preprocessor for executables and wav audio files. You get better ratio by using delta+lzma rather than just lzma for those files.
2. Know your data, use Razor12911's GFS for testing what to use. If your data contains zlib, use pzlib from ztool/xtool/normal pZlib v3, if contains lz4, use plz4 from ztool, if it has .forge archives from AC games, use AFR, if game is from unreal engine, use uelr and etc. etc. Use msc before anything if your data contains multimedia data.
3. Yep, external compressors, you can't use them with rar and you need to learn coding for using them with 7zip, and with FreeArc you can use them just by editing arc.ini
4. Yeah, xz is a container for LZMA2 compression which has better decompression speed and multithreading which doesn't exist in lzma
5. Yes, you get more speed, but they aren't much matured yet.
6. Check the forum, you may find more.



Joekkerr and others, if you want to know something, ask questions, don't say that I want to know everything, be specific, it will be help for others to answer.
Reply With Quote
The Following 3 Users Say Thank You to 78372 For This Useful Post:
EzzEldin16 (20-04-2018), flaek1337 (03-06-2018), Spexxos (18-04-2018)
  #8  
Old 18-04-2018, 06:13
joekkerr joekkerr is offline
Registered User
 
Join Date: Apr 2018
Location: Panama
Posts: 10
Thanks: 4
Thanked 8 Times in 6 Posts
joekkerr is on a distinguished road
Ok i'll be as specific as possible
I remember using precomp + srep and freearc combo on gta 5, but it failed, failed as in compression ratio was barely 70% i.e only 30% got reduced.
IIRC precomp didnt deflate much of the files.

second qn. I looked at the 'specific games technique thread' , i can understand the commands for an executable (-mx, -m2500 ,etc) but i saw some new programs for example here this line
Code:
pzlib:m3:max:c128m:t100p+srep:m2f:a2:lolz:d32 without audio file size 16.8gb.
i googled LOLZ, from my understanding, its must be similar to Freearc.
Pzlib is like precomp (correct me if im wrong)
i know srep.
So what is lolz, & why is it preferred over freearc?
am i correct about pzlib ?

Code:
Know your data, use Razor12911's GFS for testing what to use. If your data contains zlib, use pzlib from ztool/xtool/normal pZlib v3, if contains lz4, use plz4 from ztool, if it has .forge archives from AC games, use AFR, if game is from unreal engine, use uelr and etc. etc. Use msc before anything if your data contains multimedia data.
are the programs you mentioned are preprocessors like precomp ?

whats an 'msc'?

from my understanding, freearc+precomp+srep was the combo to go, whats the current ones? (i understand different for specific games, but in general ?)

Code:
and with FreeArc you can use them just by editing arc.ini
i assume you're talking about using other programs just like we do in a 'batch' file

Code:
Yeah, xz is a container for LZMA2 compression which has better decompression speed and multithreading which doesn't exist in lzma
i'll tell you what i udnerstood and then you correct me.
LZMA type of compression is from 7zip, i suppose this xz is a program (which you refer to as container) has lzma2 with decomp speed and multithread.

Code:
 wav audio files
I think using lame.exe for temporary converting them into mp3 then unpacking them into wav would be much efficient right ?

Code:
but they aren't much matured yet
as in ? unstable ?
Reply With Quote
The Following User Says Thank You to joekkerr For This Useful Post:
Spexxos (18-04-2018)
  #9  
Old 18-04-2018, 06:23
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 115
Thanks: 184
Thanked 47 Times in 27 Posts
shazzla is on a distinguished road
LOLZ,MSC : google for it (krinkels.org)
Reply With Quote
The Following User Says Thank You to shazzla For This Useful Post:
Spexxos (18-04-2018)
  #10  
Old 18-04-2018, 06:23
78372's Avatar
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 540
Thanks: 611
Thanked 639 Times in 237 Posts
78372 is on a distinguished road
GTA V contains headerless zlib streams and precomp can't detect them all. You need to use pZlib/reflate.

pZlib is like precomp.

LOLZ is like lzma, but compression gain and decompression speed is far better than lzma.

In general pzlib+srep+lolz is best for games like Mad Max, GTA V etc. with zlib/deflate streams.

About xz, that's external lzma2 implementation.

MSC is better, it's a multimedia compressor for wav/bmp/mp3 etc.

Yeah, fa011 is not that stable.

Yeah, I mentioned those ztool/xtool/pzlib/plz4 etc. are like precomp.

Now tell me, do you know how to configure freearc to use external compressors?
Reply With Quote
The Following 2 Users Say Thank You to 78372 For This Useful Post:
flaek1337 (03-06-2018), Spexxos (18-04-2018)
  #11  
Old 18-04-2018, 06:33
78372's Avatar
78372 78372 is offline
Registered User
 
Join Date: Dec 2016
Location: Bangladesh
Posts: 540
Thanks: 611
Thanked 639 Times in 237 Posts
78372 is on a distinguished road
Wink All you need to know about precompression

Most of the games these days use to compress their games before publishing. Generally we know that a game contains audio, texture, video and etc. But we don't see well known files of these types in every games. That is because they make a compressed archive with all the files required for the game. A file that is compressed can't be compressed again. But if we unpack game files and recompress them again, in most of the cases we lost CRC and in some cases the game refuses to run. So to keep the files CRC-Perfect and achieve better compression, we need to precompress them. Precompressors decompresses contents of the game and restores them with perfect condition and 100% CRC match.

In the old days, most of the games were compressed using zlib. Zlib is a well known free and open source data compressor. Zlib uses huffman, that means if we don't precompress, we are not able to achieve a bit of compression of zlibbed files. PRECOMP was well known and the first zlib precompressor to do this stuff. In 2011, reflate - universal zlib precompressor was born in encode.ru but still precomp was mostly used because of reflate's slow speed during recompression and instability. In 2015, GTA V, a game with zlib streams without zlib headers (known as deflate method) was released, precomp was unable to precompress this game. Then, the repackers need to use reflate, which worked very well on GTA V and was able to compress it with 50% ratio. But reflate was slow and so GTA V, Mad Max and other games using reflate used to decompress very slowly. In 2016, Razor12911 released pZlib which was better than precomp in terms of speed and gave same ratio. In december 2016, we received pZlib v3(and v2 before a month, but that was buggy) which was very powerful and fast as hell. Now we have xtool which is even faster than pZlib and gives better ratio. Use it and leave precomp, it will be better.

Now games uses other methods too, they are not using zlib only, so you can't only be happy with pzlib/reflate/precomp. Like Far Cry series uses lzo and lz4, Assassin's Creed series uses lzo and oodle, Need for Speed uses lz4 and oodle, mass effect Andromeda used zstd, unreal engine games use lzo and unity uses lz4 etc. To precompress them, we need to use various precompressors according to game, not a single universal one.
Reply With Quote
The Following 5 Users Say Thank You to 78372 For This Useful Post:
Bulat (20-05-2018), EzzEldin16 (20-04-2018), flaek1337 (03-06-2018), Spexxos (18-04-2018), TheGeekyGamer (28-06-2018)
  #12  
Old 18-04-2018, 08:00
KaktoR KaktoR is offline
Lame User
 
Join Date: Jan 2012
Location: From outer space
Posts: 2,620
Thanks: 660
Thanked 3,792 Times in 1,367 Posts
KaktoR is on a distinguished road
You forgot to mention that GFS is buggy. Use only max 4 cores and 16mb chache, and not all opotions on same scan, use only 1 at same time.
__________________
Haters gonna hate
RIP GOD
Reply With Quote
The Following 2 Users Say Thank You to KaktoR For This Useful Post:
78372 (18-04-2018), Spexxos (18-04-2018)
  #13  
Old 18-04-2018, 08:03
joekkerr joekkerr is offline
Registered User
 
Join Date: Apr 2018
Location: Panama
Posts: 10
Thanks: 4
Thanked 8 Times in 6 Posts
joekkerr is on a distinguished road
Quote:
Originally Posted by shazzla View Post
LOLZ,MSC : google for it (krinkels.org)
Even after using google translate, its hard to read. Not an native english speaker
Reply With Quote
The Following User Says Thank You to joekkerr For This Useful Post:
Spexxos (18-04-2018)
  #14  
Old 18-04-2018, 08:04
joekkerr joekkerr is offline
Registered User
 
Join Date: Apr 2018
Location: Panama
Posts: 10
Thanks: 4
Thanked 8 Times in 6 Posts
joekkerr is on a distinguished road
Holy shit, thank you so so so so much man! you were so informative.

Quote:
Now tell me, do you know how to configure freearc to use external compressors
I cant understand the meaning of this.
Is it the case now that freearc can now run as an 'batch file' and process all the commands for external compressors in itself ?

Quote:
Most of the games these days use to compress their games before publishing. Generally we know that a game contains audio, texture, video and etc. But we don't see well known files of these types in every games. That is because they make a compressed archive with all the files required for the game. A file that is compressed can't be compressed again. But if we unpack game files and recompress them again, in most of the cases we lost CRC and in some cases the game refuses to run. So to keep the files CRC-Perfect and achieve better compression, we need to precompress them. Precompressors decompresses contents of the game and restores them with perfect condition and 100% CRC match.
Holy shit i remember back when crysis was released it had packed files which can be easily recreated with winzip or rar, and i made each one separate during unpacking. basically ship it packed, then while unpacking used rar comand line to pack it again, each one of it LOL.

Quote:
Now games uses other methods too, they are not using zlib only, so you can't only be happy with pzlib/reflate/precomp. Like Far Cry series uses lzo and lz4, Assassin's Creed series uses lzo and oodle, Need for Speed uses lz4 and oodle, mass effect Andromeda used zstd, unreal engine games use lzo and unity uses lz4 etc. To precompress them, we need to use various precompressors according to game, not a single universal one.
You mentioned these methods, by how do you identify which game has which.
Reply With Quote
The Following User Says Thank You to joekkerr For This Useful Post:
Spexxos (18-04-2018)
  #15  
Old 18-04-2018, 08:09
Spexxos Spexxos is offline
Registered User
 
Join Date: Apr 2017
Location: Germany
Posts: 12
Thanks: 21
Thanked 1 Time in 1 Post
Spexxos is on a distinguished road
thats a whole lot of new information. thank you guys for that.

So the general approach for data of unkown compression would be to run a scanner like GFS over it. (Are there any other scanners avaiable besides GFS?) Then figure out what you have to do afterwards but here in lies my problem.

@78372 About xtool vs precomp. Precompv046 gave me better results than xtool on my testfiles (Witcher1Savegames) I probably should scan the files so see what they are made of but for now I am wondering if my xtool settings are wrong. To be honest I have no clue how to properly configure xtool/arc.ini
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Simple Tutorial of Compressing GTAV Razor12911 Conversion Tutorials 26 29-01-2019 01:31
Tutorial about compressing games faster using UltraArc 78372 Conversion Tutorials 2 29-04-2017 17:25
KaOs Krew compressing method PanosBabo PC Games - CD/DVD Conversions 4 29-05-2014 06:33
NASCAR The Game 2013 (inno setup script) sentinelks PC Games - CD/DVD Conversions 28 08-09-2013 20:30
problem with compressing .bik game video files -sbx- General Gaming 6 23-07-2010 12:20



All times are GMT -7. The time now is 07:55.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Copyright 2000-2018, FileForums @ https://fileforums.com