Current compressing techniques guidance
 

Hi, im JoeKkerr

I was actively participating in making repacks from 2010-2014.

i'm not completely new to repacking, but i remember my lack of knowledge in my end days where i wasnt able to compress or keep up with others size.

what i did was what others were doing take files > Freearc > batch file unpacking.
or some files needed precomp or a combination of precomp + srep but problem started arising when some files just werent getting reduced, even after using precomp they werent 'precompressing' or as i'd call it deflating. while groups like blackbox and kaos were able to do it. (ik it applies to only some files that can actually be deflated)

i even tried studying thier repacks and their exe's to replicate result and hence i gave up knowing more people were doing better job than me and my internet was 1MbPS with 100kb download and 25 upload.

I'm thinking of getting back into data compression. whats the current method of compressing data ? i know different games , different methods but most of them are same. i assume freearc is still the king, 0.50 version compressed better than 0.666 version back in the day.

Can anyone guide me? i was inspired by skullptura btw

FreeArc 6.7

Thanks but i was asking for an elaborate explanation.

You may edit the repack groups names from your post.Otherwise the forum MOD warn you..!
At first i am also got the warning.:)

I am also trying to get into the whole compression thing. In my case its primarily for compressing savegames. For that matter I would be interested in the current state of things aswell. I think a might be a bit ahead of you in my knowledge, JoeKkerr, and since I dont want to open up another thread with the same topic I am going to add my questions here. I am sure we both can benefit from this. Here we go:

1. What is DELTA and when to use it?
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?

Cheers

precomp is used when there is a file already compressed (i dont know the technical word) . But a file that is , if you try freearc you wont get a good result because its already compressed with low level compression, this makes high level compression lose its effectiveness. Hence, precomp..deflates the file, i.e removes the low level compression and allows it to be compressed by high level compression freearc offers.

an example would be, try RAR ing a normal text file with medium compression, then 7zip that file, you wont get any good result.
now try taht with using precomp first or on raw text file without winrar.

last time i checked (which was a long time ago) LZMA (7zip) was better than freearc for text files.
That brings us to your next question, why freearc you ask ?
Some files get better compression ratios with Freearc , some with 7zip. Mostly its freearc hands down, but some game files like Assassins creed files get better result with 7zip. Depends on game

1. DELTA is a preprocessor for executables and wav audio files. You get better ratio by using delta+lzma rather than just lzma for those files.
2. Know your data, use Razor12911's GFS for testing what to use. If your data contains zlib, use pzlib from ztool/xtool/normal pZlib v3, if contains lz4, use plz4 from ztool, if it has .forge archives from AC games, use AFR, if game is from unreal engine, use uelr and etc. etc. Use msc before anything if your data contains multimedia data.
3. Yep, external compressors, you can't use them with rar and you need to learn coding for using them with 7zip, and with FreeArc you can use them just by editing arc.ini
4. Yeah, xz is a container for LZMA2 compression which has better decompression speed and multithreading which doesn't exist in lzma
5. Yes, you get more speed, but they aren't much matured yet.
6. Check the forum, you may find more.



Joekkerr and others, if you want to know something, ask questions, don't say that I want to know everything, be specific, it will be help for others to answer.

Ok i'll be as specific as possible
I remember using precomp + srep and freearc combo on gta 5, but it failed, failed as in compression ratio was barely 70% i.e only 30% got reduced.
IIRC precomp didnt deflate much of the files.

second qn. I looked at the 'specific games technique thread' , i can understand the commands for an executable (-mx, -m2500 ,etc) but i saw some new programs for example here this line i googled LOLZ, from my understanding, its must be similar to Freearc.
Pzlib is like precomp (correct me if im wrong)
i know srep.
So what is lolz, & why is it preferred over freearc?
am i correct about pzlib ?

are the programs you mentioned are preprocessors like precomp ?

whats an 'msc'?

from my understanding, freearc+precomp+srep was the combo to go, whats the current ones? (i understand different for specific games, but in general ?)

i assume you're talking about using other programs just like we do in a 'batch' file

i'll tell you what i udnerstood and then you correct me.
LZMA type of compression is from 7zip, i suppose this xz is a program (which you refer to as container) has lzma2 with decomp speed and multithread.

I think using lame.exe for temporary converting them into mp3 then unpacking them into wav would be much efficient right ?

as in ? unstable ?

LOLZ,MSC : google for it (krinkels.org) :)

GTA V contains headerless zlib streams and precomp can't detect them all. You need to use pZlib/reflate.

pZlib is like precomp.

LOLZ is like lzma, but compression gain and decompression speed is far better than lzma.

In general pzlib+srep+lolz is best for games like Mad Max, GTA V etc. with zlib/deflate streams.

About xz, that's external lzma2 implementation.

MSC is better, it's a multimedia compressor for wav/bmp/mp3 etc.

Yeah, fa011 is not that stable.

Yeah, I mentioned those ztool/xtool/pzlib/plz4 etc. are like precomp.

Now tell me, do you know how to configure freearc to use external compressors?

All you need to know about precompression
 

Most of the games these days use to compress their games before publishing. Generally we know that a game contains audio, texture, video and etc. But we don't see well known files of these types in every games. That is because they make a compressed archive with all the files required for the game. A file that is compressed can't be compressed again. But if we unpack game files and recompress them again, in most of the cases we lost CRC and in some cases the game refuses to run. So to keep the files CRC-Perfect and achieve better compression, we need to precompress them. Precompressors decompresses contents of the game and restores them with perfect condition and 100% CRC match.

In the old days, most of the games were compressed using zlib. Zlib is a well known free and open source data compressor. Zlib uses huffman, that means if we don't precompress, we are not able to achieve a bit of compression of zlibbed files. PRECOMP was well known and the first zlib precompressor to do this stuff. In 2011, reflate - universal zlib precompressor was born in encode.ru but still precomp was mostly used because of reflate's slow speed during recompression and instability. In 2015, GTA V, a game with zlib streams without zlib headers (known as deflate method) was released, precomp was unable to precompress this game. Then, the repackers need to use reflate, which worked very well on GTA V and was able to compress it with 50% ratio. But reflate was slow and so GTA V, Mad Max and other games using reflate used to decompress very slowly. In 2016, Razor12911 released pZlib which was better than precomp in terms of speed and gave same ratio. In december 2016, we received pZlib v3(and v2 before a month, but that was buggy) which was very powerful and fast as hell. Now we have xtool which is even faster than pZlib and gives better ratio. Use it and leave precomp, it will be better.

Now games uses other methods too, they are not using zlib only, so you can't only be happy with pzlib/reflate/precomp. Like Far Cry series uses lzo and lz4, Assassin's Creed series uses lzo and oodle, Need for Speed uses lz4 and oodle, mass effect Andromeda used zstd, unreal engine games use lzo and unity uses lz4 etc. To precompress them, we need to use various precompressors according to game, not a single universal one.

You forgot to mention that GFS is buggy. Use only max 4 cores and 16mb chache, and not all opotions on same scan, use only 1 at same time.

Even after using google translate, its hard to read. Not an native english speaker

Holy shit, thank you so so so so much man! you were so informative.

I cant understand the meaning of this.
Is it the case now that freearc can now run as an 'batch file' and process all the commands for external compressors in itself ?

Holy shit i remember back when crysis was released it had packed files which can be easily recreated with winzip or rar, and i made each one separate during unpacking. basically ship it packed, then while unpacking used rar comand line to pack it again, each one of it LOL.

You mentioned these methods, by how do you identify which game has which.

thats a whole lot of new information. thank you guys for that.

So the general approach for data of unkown compression would be to run a scanner like GFS over it. (Are there any other scanners avaiable besides GFS?) Then figure out what you have to do afterwards but here in lies my problem.

@78372 About xtool vs precomp. Precompv046 gave me better results than xtool on my testfiles (Witcher1Savegames) I probably should scan the files so see what they are made of but for now I am wondering if my xtool settings are wrong. To be honest I have no clue how to properly configure xtool/arc.ini

:)

From precomp 046, it's a complete compressor, not only a precompressor, so it will give better result on small files and when you have compression on. To use xtool, you must use xtool+srep+lzma(or xtool+srep+lolz) to get better ratio, else it will just inflate files, not compress, plus xtool isn't complete yet, it's a beta tool, it has bugs.

My bad I should have been more detailed. I used precompv46 only as a precompressor, so with these parameters ("precomp -cn -intense").
The file that precomp delivered was about %10 bigger than the one from xtoo. Which means precomp found more to decompress?

Aferwards I put both the testfiles from precomp and from xtool into srep393a (-mx), then I proceeded to compress them with 7z(7za.exe a -t7z -m0=LZMA2 -mmt=1 -mx9 -md=1536m -mfb=273 -ms=on -mqs=on)

Result is the precomp testfile is half the size of the xtool one. That makes me think I have to configure xtool differently.

May be xtool didn't work good on that, it's a beta tool after all.
Try pZlib v3 instead

what settings do you suggest for pzlib?


EDIT: pzlibv3, christmas edition, gives the same result as precomp v046 (pZLib3a.exe e -c128m -t75p). It is also noticeably faster. Is there any sense in checking out older versions of pzlib? I have seen older versions have more switches/options. Is that because the needed more manual input from the user to give the best result, and pzlib3 does that automatically or is there an other reason for that?


EDIT2: After Precomp/pzlib what defines the order in which you apply further tools like DELTA and Srep, msc etc? Which one comes first, and why? Or does the order not matter at all?

MSC should be used *ONLY* for multimedia data and delta should be used *ONLY* for exe and wav files, applying delta for normal game files will cause worse ratio, and without multimedia data, msc can't do anything.
MSC should come first, then pzlib, then srep and at last lzma/lolz

Thank you. The thing about DELTA is that, that on my witchersave testfile I got better compression with precompv46+srep393a+DELTA+lzma2 compared to precompv46+srep393a+lzma2. Any Idea why? Or is that the exception that confirms the rules?

That's a save file, not a game file and probably delta works better for it. Not everywhere delta works and using them on executables is purely safe and that's why I mentioned about that. But in general, delta doesn't work better on game files(such as textures).

I see. My main interest for now is compressing my savegames. 500gb unpacked/ 190gb packed. They got quite large recently. I think its time to change that. Anyway. Thank you very much for your help.

Its a shame that there is no "Best Compression Methods for 'Specific' SaveGames INDEX" Maybe its time to do something about that:cool:


EDIT:

ok, two more questions popped up: Where can I find lolz, the only thread I found was Lolzx which is apparently a private compressor?
And second whats the matter with pzlib3 according to razor there is this command "-x" which is supposed to give the best output. Cant find it anywhere else mentioned though, not even in the instructions for pzlib. What does this command do? Also what is the use case for pzlibs "-r" deflate. Its different decompression mode I assume?

Hard to tell, sometimes delta before srep improves compression, same with msc.

http://fileforums.com/showthread.php?t=99953

According to this page [SPOILER]https://translate.googleusercontent....ygj-rpHOhRlC3g[/SPOILER]

these setting should provide good compression without sacrificing it for a faster compression. Am I right? Results are ~7% better than with lzma2 in my testing. Is there room for any improvement? lolz has a lot of options I havent even touched yet.

"lolz -mmt0 -mt1 -mtb64 -d2032 input output"


How do you acutally unpack lolz archives?

So, I have been testing alot the last two days. Compressors like lolz, lz4,5, mcm, rzm nz etc. I found that Nanozip gives me the best compression ratio. Do you guys think that Nanozip is reliable enough for a longterm use?

Here is how it work today:

You install the game to get its individual files in folder. First, nowadays we have many specialized compressors, we have bpk for compressing bik/bk2 (bink) videos, we have mpz for mp3 files, we have oggre for ogg files, we have packjpg for jpg files and some others but these count most.

So you use their respective compressors for those. Next we replaced internal lzma of FA for 64bit variant(to allow bigger dictionaries etc). XZ/7zip is one choice, I made xnlz by modifying plzip to get as close to FA:xlzma experience as possible in terms of speed and features, however you can also use FA's default its not that big difference.

Bigger difference is from lolz which can pull ~10% better to lzma. Lolz is a general purpose compressor but specialized on textures using lzma with additional cm modeling for textures/images, whose can detect within files, so its good for games. Its slower so you may combine it with lzma for different files. Its not really better outside of textures than lzma thats why.

As you may noticed, I spoke lot about specialization and individual files, thats the game now. If game files are in packs and compressed, then(and only then) you need tools like ztool, xtool,punity or precomp. In fact for precomp I dont see much purpose anymore, it doesnt support stdio and is generally slower, really tools from Razor is all about. Also, often you may need to go deeper and extract/repack game files using quickbms scripts and other tools, for example you can only compress XCOM2 WotCh that way as it use oodle compression.

I forgot, there are also specialized precomps for certain engines, uelr for Unreal's lzo, afr for anvil(Ubisoft's .forge) lzo, xtool for CRI, punity for Unity's lz4. Ztool is for older lzo/zstd/lz4 but work always for pzlib/deflate.

To scan files you can use GFS scanner or my lz_scan, GFS have nice UI but is buggy and unstable in anything other than zlib/deflate. lz_scan also support afr/uelr scan.

Srep you use latest 393a 64bit, FA 0.67 as it fixed *lot* of bugs and added features, dont even think about anything lower.

Finally, you generally use default internal delta of FA on all but images/textures. As for xdelta, this has to do with recomposing CRC matched files back. For example, before xtool you would have to use quickbms with cpk.bms to extract .cpk files. When you repack them back they were not same because of header data change so you would use xdelta for that final touch. Same goes for recompressing .wem files. Other than that I dont see much point for xdelta compression itself, its all about canceling duplicates and srep is already doing that AFAIK(FitGirl you may want to step here to explain better if I am wrong).


1. What is DELTA and when to use it?
A: delta type compression for canceling duplicates and get diffs, use always except on textures, or never, usually no big deal
2. When to use what precompressor or any of the programs following? (precomp, srep, pzlib, ztool, xtool, PrecompMT, msc )
I have seen different versions of precompressors delivering different results. Sometimes older versions worked better than newer ones?
A: first forget precomp(MT),pzlib,msc and all that crap, just use:
ztool for deflate/pzlib,lzo,lz4,zstd(last 3 will likely not work on newest games)
xtool for deltate/pzlib,CRI
punity for *some* unity games
uelr for most unreal games(.tfc, .upk...)
afr for anvil games
srep always, its for data deduplication not a recompressor
3. Why is FreeArc alsways the recommended tool? Depending on my files I got better results with also 7z (lzma2) and rar? Is it because of the ease of use for external compressors?
A: Because FA is so much more that that. I dont think you can use bpk+oggre+mpz+packjpg+uelr+srep+lzma in one go, on files sorted with one of many advanced ways best fitting its game files structure, in anything else other than FA or based on FA, can you? You can use 7zip as external LZMA within FA if you like.
4. Whats are the benefits of xz? (In my unterstanding its just a container, am i right?)
A: Same as 7zip or xnlz, 64bit support
5. Is there any benefit in using FreeArcs successor (FreeArcNext(?)?)
A: Its still early alpha/beta and lack feature I believe. Its also not (supposed) to be free once finished. My personal opinion is that Bulat pretty much killed his own legacy, almost nobody will use it and he is wasting his time.
6. Most importantly: Are there any already written guides that I have missed that have answers to my questions?
A: Start reading threads on all mentioned tools upside down both here and on encode.ru. In fact, read this whole site(and encode). Then learn Russian and read whole Krinkels.org website. Then (and only then), you will become Jedi.


So thats about it. As you see its not anymore old times when you could just precomp+(srep)+FA and thought you done. Today its much more complicated and often exotic practices are required. Good luck.

Can i have your lz_scan??

Unfortunately i am too here a newbie i mean newbie to all these things that but i have been reading every post here since 3 months, dude only 3 months

i am new to all these things only 3 months, i was just wandering around looking for games on repack sites and i did't know what was repacking and what r repacks my only target was to find the game download it and play, this was my life after that i was curious whats it different sites different game sizes i feel very curious about what it was and then after searching a lot of stuff on internet i found these forums joined em and start reading posts, using discord chat with new friends and now its still going, now i can repack most of games on my PC, still i am noob i want to be a jedi as @elit mentioned