Okay. I try'd to use the DiskPan, but i have a small problem. It's a bit weird. When i normally create a compressed file, i cant put it in some program. Or can I? Can you please help me?

^^
What do you mean?

Hello. I was looking at Inno Setup Compiler. I dont know why, but when i do some wizard and install it, the file of the game dont want to open. Maybe some decompress error. But can you please help me with it?

Just catching up
 

HI not sure if this is the right area..
Im old school back in the days of MYTH, Paradigm etc... So I know now, how they got those files down to size. ripping. Back then is was crucial due to the 2400b - 1440b.. now a days with fiber optics no biggy. So I still like to compress all my games. I don't rip them anymore. but I found this forum and here is what I use:

I do all this in a bat. keep in mind I've used this for some time, so forgive me if it is ancient

This was done with skyrim with all the lang taken out but english. saved about 10gb. Leaving a size of approx 20gb

compress: (all the user input var and other features including escape codes for color and locations have been omited to show just compression)
yes this is line-by-line, I like it that way).
Result: 13gb


Another compress from here (chain)
surprisingly 13gb actually 40mb smaller

Now this works great with just using a .bat. the iss are cool but I love bat stuff. And yes this is just a basic run of the mill I use and does a really good job for the most part.

so getting up to date, are there newer methods that are standard for .bat file compression? Or are these still the tried and true holy trinity :-), I'm old so be gentle on the corrections. I am always looking for new ways of shrinking without ripping.

I'd say check out xtool for potentially better pre-compression, can't hurt to experiment with other final compressors like razor or lolz, other than that it's difficult for general advice as the level of research and experimentation for a given title varies greatly I've found.

Hi, thanks for the reply!
I’ve tried a few different tools, and in my experience, when it comes to compressing (not ripping), there’s usually only a minor reduction in size. Using methods like Precomp and SREP with either ARC or 7z, the results typically fall within a 10–20% range, depending on the game and file structure. Of course, larger games might show more variance, but that’s not always the case.

I haven’t used "lolz" yet—I actually thought it was an emoji at first, haha! I’ll definitely have to check that one out.

Lolz is an archiver designed specifically for game texture data such as DDS (DXT#) images. It's slower than lzma but is multithreaded and will yield smaller ratios. Out of all tools made by ProFrager, lolz is the most stable.

You may see some overlap between the effectiveness of precomp vs xtool on games compressed with deflate, as both tools support this algorithm. XTool is superior due to having multithreading and a diverse range of supported algorithms. By default there's zlib/deflate, oodle, lz4 and zstd but you can add further codecs and add plug in other algorithms. Over the years I've plugged in doboz and yaz0, for example.

It's all about finding the right algo. If you try to compress a unity lz4 game without the unity lz4 XTool plugin, you'll get an severely larger result, the same result if you used precomp.

Hello, I am yet another newbie prostrating myself before the alter of knowledge :p
Anyway, I've been doing my research, getting my feet wet and I think I have a reasonable understanding of the processes as far as a newbie can be.

So as it stands, I understand that some of the games I may want to repack, require unpacking and to do this, I need to find out what form of compression was initially used.

In one particular case, I found that a game uses some form of ARC compression but it seems to be custom on that particular game. Is there a way of overcoming forms of custom compression in order to unpack files from a .bin? Am I sniffing up the wrong tree?

Thanks in advance for any clarity on this.

What makes you think that the particular game uses "ARC" compression?

You can always share some small samples here if possible so experienced users can trake a look.

I read that it was suggested to check for "magic numbers" via a hex editor as this should give a fairly strong indication as to what compression may be used.

41 72 43 01 00 00 06 07 41 72 43 01 02 73 74 6F
72 69 6E 67 00 10 10 D7 C8 DE A3 45 45 C1 4E 00
00 00 00 00 00 10 00 E9 FE 04 00 01 F2 DB 00 00

Above is the first three lines of a .bin file from the game I am looking at. It's 41 72 43 that outputs text of "ArC".

If I am sniffing up the wrong tree, happy to be pointed in the right direction and undertake more learning.

Thanks again for your time.

This is a standard FreeArc archive.
https://i.imgur.com/AR9liWW.png

You should be able to extract it with FreeArc as long as it is not encrypted. Other then that there are no tools you can use to precompress it.

PS: The "storing" has nothing to do with the actual compression settings used.

What game is this?

Appreciate the clarity, thank you :)

This is "Graviteam Tactics: Mius Front". I literally just took a game from my collection at random to try and "cut my teeth on" and put what I read in to actionable experience.

Could you share a sample file in question from this game?

My games are from certain groups if you catch my meaning.
The .bin in question is 1.49gb in size.

Happy to share, not sure on the best method to do so as I'll assume that sharing links of that nature would be against forum rules?

I presume that the games being from certain groups wouldn't add any complexity to the process?

It would; please, don't continue in this direction.

Oh ok, so this is actually warez (we aren't allowed to talk about warez, nor sharing it here apparently).

In this case I missunderstood you. I thought the bin files are actually part of the game itself. I looked on steamdb depots for this game and there aren't any file types in this case so I was confused. Now I understand.

Ok so, then this bin/arc archive should be a part of an installer I guess, which contains the actual game files just compressed (like we do if we creating repacks).

The problem now is, that you can't compress this already compressed files further to achive a better ratio. First you have to extract the archive (the game files) and then apply better compression on them to gain a better ratio, simple said.

No worries, appreciate the clarity and the information.
I will figure this out in time :)

Hi all!

In SREP 4.90 WIP there is the -M5R parameter.
Anybody knows what is it?
When and how recommended its usage?
Did some tests ,but i have no valuable results.

Hi Guys,

I really am sorry for the noob question, but I am an absolute beginner concerning compression.

I am backing up my Steam Library and thus far have used the Inno setup wizard to create the Installers. What bothers me most is that max limit of roughly 2.2 GB per file.

I would love to know how to create for example one 40 GB *.bin file instead of 17 - 19 of them (like Fit girl's installers).

I think that Freearc is involved, but apart from that i have no idea how to go about it.

Do you compress the Game Directory with free arc and then somehow integrate it into Inno-setup?

I have seen all these custom installing scripts on here but at first glance nothing that specifically helps me with my problem.

I tried the search function but i could not really find anything . But again it could be that i do not know what to search/look for in all of this data.

Thanks in advance and sorry again

Yes we use freearc here as a pipe for other compressors.

If you really just want to get rid of the 2GB limitation of those bin files, there is also the possibility to use just 7zip and make a SFX archive, but then without inno setup.

If you want to have inno setup as an installer, use DSG and use 7zip lzma2 compression or go with something other methods if you want to achieve better ratios. Then get a installer script which is compatible with DSG, like this https://www.fileforums.com/showthread.php?t=103408

Hey, thanks for the quick answer.

Well to be honest, if I could, I would love to get a whole game into a bin, like most of the fit girls installers I tried out (apart from the optional languages/Redistributable and extras bin).

I will take a look at DSG and the script and try to find tutorials or experiment a bit with it.

Thank you very much for the help!

Hello everyone,

I'm currently using mini compressor tool to compress my games into SFX.exe files. I'm looking for an easy way to quickly identify a good compression method for my games. I'm not looking for maximum compression, just a good balance of compression and decompression/extraction speed that can handedly beat just using 7zip.

is there any kind of tool that can scan a game folder and give you a report of file types found?
is there a common method that will be good enough to just throw at most games?
mini compressor has presets for example:
mxc = xprecomp+lolz1/$bitmapz=bmpz/$bink=bink/$void=void/$photo=jpg/$ogg=ogg/$wavz=wvp/$mp3=mpz

any help would be appreciated. i just want a simple way to store my games and save storage space.

wiztree gives you each file types in great detail, i use it myself. there are also plenty of examples in these threads for methods:
https://fileforums.com/showthread.php?t=99554
https://fileforums.com/showthread.php?t=101639

its quite easy to learn the best methods for unity or unreal games, for example. i also suggest that you switch to DiskSpan GUI: https://fileforums.com/showthread.php?t=104507

i used both mini compressor and DiskSpan, and i can tell you that DiskSpan will be much better once you learn how to use it.

Friends. I have a great interest in repacking and am always fascinated by how blatantly you can compress some games. I can program, but here in the forum you are really overwhelmed with tools and programs.

Theoretically it works like this:

1) Precomp the data with xtool (there are already so many plugins here...)
a. Here it already fails. If I want to read in an entire folder, I always get 0 / 0 streams and the precomp file is exactly the same size as the source (that shouldn't be the case?)
b. In addition, there are countless possibilities and every game reacts differently to the combination of precomp algorithms. Can't I read in a game folder and it automatically returns the best combination? Somehow I'm at a loss here.

c. The precomp files must then be compressed (ideally with FreeArc?).
- If I try
the data.arc is 1.5x too large as the source file?

I would love your help - somehow I'm at a loss.

Many thanks!

I will take care of the unpacking/installing afterwards. That should be easier...

You have to know what compression a game is using.

Your archive is larger because you did not deduplicated the data (srep or xtool:dd1..5) and you did not compress the data at the end. However if you precompress the data with xtool zlib and the archive is larger than the original input, you have possibly found out that the input is zlib compressed.

And how do you do that? Or is that exactly what the precompress is for?

To understand: Do I need to know what compression the games already have so that
a. to remove/optimize the compression during precompress and (better) compress again later or
b. so that I can pack it even better (over it)?

I'm (still) at a loss here.

Experience, knowledge, repetitions.

I would say you can go with game engine repetitions most of the time.

I can tell you that nearly 99.9% of all unreal engine games using either zlib or oodle compression. Older engine versions like UE3 used lzo algo mainly back then.

Most of the unity engine games (if they have bundle files) using lz4hc compression with level 9.

RE_Engine (Resident Evil and all these games) were compressed with either zlib or zstd, but to be honest I didn't worked with this engine since years so things might have changed.

Precompression just means that already compressed streams will be decompressed, that's why the output will be larger than the input, so you can compress the decompressed streams with a better compression than the original used compression. Like if you compress some random folder on your pc with 7z. The compressed files (7z archive) will be smaller then the original input. If you extract the compressed 7z archive, the output will be larger again. That's how precompression works to keep it simple.

I think a and b are the same, or I am really confused.

To keep it simple, these steps in this order:
Precompress -> remove duplicated data -> apply stronger compression

Thanks for the explanation!

Let's take this as an example:


With this command
I arrive at 6.21 GB. I don't understand that.
Since it is not compromised according to the quote, does precompress not apply?

Apparently I still don't quite get it.

xZLib belongs to an old XTool version. If you use current XTool version (0.8.9 I guess) use xtool:zlib instead (maybe reflate is a possible option aswell, I don't have the game).

srep is there for deduplication. Actually you just need srep.exe (64bit version) and then use xtool option :dd1..5 (where 1 is fastest and 5 is slowest but might get a few more duplicates. Default is 3 and in my humble opinion this is peak most of the time).
exe and delta are freearc built-in compressors (honestly I don't really know what they do actually).
lzma2 is from 7z archiver so you need 7z.exe and 7z.dll files aswell.

You can also go with freearc built-in lzma in addition with 4x4 instead of lzma2 to use up to 16 cpu threads.

I downloaded that game and investigated it real quick. Here are my results
https://i.imgur.com/saojnEi.png

The *.bundle files contain WAVE audio. You can use xtool flac plugin or msc on them.
https://i.imgur.com/PRvRCMB.png

The movie files are Bink1 videos. You can use xtool bink plugin on them.
https://i.imgur.com/UsjPcPO.png

Thank you!

Where i can find xtool flac and bink plugin?

FLAC is built-in, bink and the majority if not all plugins can be found here

Confused on how to repack games
 

Hello everyone, yesterday i tried to repack one of my favourite games, i tried looking for some guides but i wasn't lucky at all. Here are the things i did:
1. Got my game folder and packed it into a .arc file (without compression)
2. Ran the file in precomp and got a .pcf file

After that, i wanted to make an installer, but i really can't find anyway to do it.
Could someone help me? Thanks :D

https://imgur.com/a/Wt2o3LZ

Lossless H264 Recompression
 

Has anyone managed to write or create a working lossless h264 recompressor in recent years from existing losslessh264 and or avrecode sources?? (Not in all games, but the presence of x264 videos, next to/instead of Bink2, is becoming more common.) Compression ratio avg ~15% better than 7z/any ~1-3% size reduction. Leaked executables used from DropBox??

Shelwien preserved few years ago halfly working H264 recompressor from encode.su site. The file, not available.

Hi, after reading this thread I think I have some knowledge but I wanna be sure of what I know :

1. Deduplication is the process of deleting identical files ?
2. Precompression is for decompressing already compressed files to compress them better afterwards ?

Now I have some questions about things I don't understand :

3. What are streams ?
4. What do changing chunks, dictionary size and depth do ?
5. If DiskSpanGUI is a compressor why can't I directly extract files with WinRar for example ?

dedup it's not only files, but stucture of file
for example, a .zip file full of GIFs.
Each GIF file will start the same (header), and dedup will also take this into account to reduce the final size.

Precompression => yes decompress file and some kind of file (gif, png ,mp3) to let them compressed later with a better compression system

Chunks/dictionary and depth :
They will impact performances (better speed, better compression, better detection)
these values are a compromise between performance and speed.
values that are too high can also lead to CRC errors.
like dictionary, it's a way to reduce a file size if inside they is several time the same words, same structure, or same hex data
depth is to find stream deeper inside file
like if you have a zip file with inside a movie with a wav file. depth 0 => zip file , depth1 =>zip file with something inside=>movie , depth3 => zip file with something inside=>movie => wav
but with a high depth, soft may need to read file several times => more time to compress/uncompress
also going deep may add some errors and you will have CRC error during decompression
again => these values are a compromise between performance and speed.

streams...
They are in files
For example, to put it simply, a large game file can contain images, sound, texture, text...
during the analysis process, software can indentify a sound stream and decode/decompress/recompress

diskspangui is a compressor, but to be fair, is a collection of diffèrent programs
in the same file you can have several type of data

like if you mix oil and vinegar to make a salad sauce
then you want to put back oil and vinegar in their bottle
you wll need a tool to separate them and work on them each on their own.

diskspangui it's a GUI for several freearc function.
you can open files with freearc (not to unpack, just read them, if you want to unpack, you will need to add soft/config in freearc software)

Thank you very much for your answer, do you mind sharing config files (arc.ini I guess ?) or softwares (if that's what you mean by "soft") or even threads about those :]

I have been having great success using DiskSpan GUI and the Installer Script example that comes with it here https://www.fileforums.com/showthread.php?t=104507

However, I have recently come across a game that has an extremely long path name that almost exceeds 260 characters in length (245), which heavily limits where it can be installed, because the installer errors out when trying to write to a path that exceeds 260 characters in length. It will install without error when the installation path is less than 15 characters (260-245)

Does anyone know what steps I should take to avoid the problem? I am very new to this, thanks!