Newbie Question Thread
 

To all new members, welcome to FileForums!

This thread is for newbies to ask questions regarding learning how to compress their own games. The idea behind this thread is to have an informal pool of knowledge where new members can ask questions and more experienced people can answer without polluting the forum with single question topics.

Anything goes, just please be respectful and follow the Forum Rules.

If you have a question about compressing a specific game, please ask here.

1. What are usually the steps for a repacker to start working on repacking a game? (Assuming all gamefiles are not already packed or encrypted and are "open")
2. What is an easy Inno Setup "template" can you suggest for beginners (and has enough documentation to get started easily) and tutorials for making new ones?
3. Is AutoIt scripts a suitable replacement for batch files?
4. How do I adopt my repacking setup to be suitable and convenient for making xdelta patches?

Thank you.

^^
1. If there's no compression or encryption to look for, then jump straight to compressing your archive. Tools like XTool ([1], [2]) can act as precompressors for certain algorithms so you don't have to manually decompress the data when you find a compressed game. XTool handles it whilst making your archive.

2. Try this. All the info you need is there and it is equipped with the required libraries to unpack FreeArc archives.

3. Never used it, batch files are just easier and don't trigger AntiVirus.

4. Making patches is a bit outside the scope of our forum since this is for compressing game backups. However, there are two AIO utilities I recommend to make patches - Anadius' tool and ISXPM. Anadius' tool is easier to use but offers only XDelta whereas ISXPM offers more customisation and choice of patch engine (XDelta, HDiffZ or JDiff).

Thank you for the reply.
In a previous post you have recommended DiskSpan_GUI for compression, and unfortunately not enough documentation is provided on how to split into several bin files, and how to seperate decompression method for that particular bin files. It is a useful tool, but not much information for now.
I was essentially looking for templates that work with DiskSpan_GUI, but SIS is excellent and simple.

As for precomp, is it necessary to first go through one type of files on the game, running a command/script to compress all of them, then run another command, for example, for the soundtracks, etc? And after being done with all that, what would be the best compression (for .bin files) post-precomping?

Is it preferable to put the decomp scripts and utilities on the bin files, and after extraction, run the scripts to decompress the assets, or some other way is preferred?

I also do not understand the compression method command thingy, an example is this:
Thank you again.

I wrote a quick guide here: https://fileforums.com/showpost.php?...6&postcount=13
In the next days I will write even more informations about how to properly set-up all the settings.

Thank you very much for writing the guide and also for choosing a game that I was working on as well! :D

Just a smol issue (literally :p) some of the screenshots are too small to be able to zoom-in in browser and see clearly, so a little fix pls

But other than that, it is very informational.

I have added direct links to imgur.com above the images. They are in the original measurements there.

Alright, now that the files are compressed, its now time to create the actual setup.exe program. How does one go about that? I noticed that DiskSpan_GUI makes a Setup.dll
which I assume holds all of the decomp program for installing (?) .
However, I might be wrong, so how DO you use the Setup.dll file?
Thanks.

Setup.dll is a freearc archive actually. It contains all files needed to unpack the compressed files.

You can use one of the supported installers.
https://www.fileforums.com/showpost....87&postcount=3

However you can also use DiskSpan_GUI.exe for installing. Just copy DiskSpan_GUI.exe to the same folder where Setup.dll is and run it.

I see. It is quite useful; however I would like to use the supported installers, which, I don't know what to use and how to use them with Setup.dll, etc.

A few pointers would be appreciated :)

Thank you!

For all the installers, you have just to make the settings you want, like setting the game name, size and so on, then compile the script. After that you got a setup.exe file which you copy to the same folder where setup.dll and the compressed archives are located and then run setup.exe and install. Simple :)

Another question:
What if I convert all the mp3 files in a game to ogg beforehand, and then attach a ogg to mp3 converter and a batch script to revert them back to mp3? Will it improve compression ratio, even a little bit depending on how many mp3 files are there?

Why would you do this anyways? You have some tools to compress mp3 files. Another thing is, that in most cases you wouldn't be able to convert the ogg files back to mp3 with the correct hashsum, which means you have to create some diff files to patch the converted files back to original.

precomp and wav files
 

I have several games that rely on wav files. what would be the best way to go about this. precomp that I know of cannot handle wav files.

thanks!!

Example, I like RIPPING. Skyrim has gigs of wav files that i can lame down to about 10%-25%, but I need to get them back into the bsa. How would I do this?

For WAV audio, use MSC with Tak plugin. You can rebuild the BSA archives using BSAOpt.
https://www.nexusmods.com/skyrim/mods/247/

Please note if you recode the wavs to mp3 with lame, the game might not recognise them anymore.

Masquerade: The recompressed BSA archives, betterly compressibble or not?? Or better ways recompressing in store mode (supported?) and packing with any better methods? (ex: msc+srep+lolz)

Hi, thanks for the fast reply!
I have BSAOPT I'll have to take another look at it. I used to do this with other games that use wav and use lame to crush them down. so far no issues, except one game. I had a wav compressor at one time, however, my main issue was getting the files BACK INTO the bsa file LOL

Currently I did the usual precomp+srep basic and got 27GB skyrim to 11GB, but I am sure if I can at least get the wav files down to size, I should be able to get it under 10GB.

why are you converting mp3 to ogg?
It appears that you do not have the LAME encoder based on your post. Instead, it seems that you have a decOGG compressor. If this is correct, I recommend replacing it with LAME, which can be easily found online. I'm not sure if I'm allowed to share a link, but you can look for it in various places. LAME will effectively compress your MP3 files, so there's no need to use OGG in this scenario.

BSAopt is not suitable for repacking since every time you make an archive with it, the CRC is different.

I'm not up to scratch with BSA format, but I don't think the media files are compressed. Beyond that, I believe LZ4 compression is used.

SkyrimSE using lz4f with level 10 if I remember correct. And if I am not mistaken you have to use lz4 library 1.8.0 or something like that to get a decent amount of streams.

confirmed, they are not compressed. I forgot about the CRC. Does - skyrim(and/or bethesda) make use of that.

Howdy! Apologies for my grammer and thanks in advance for your help.

How would i add ztool to diskspan please...

OK, I am from the old BBS days (Myth, Paradigm etc..) and have been fascinated by extreme compression. These days I learn it as a hobby. So wanted to thank you all for this site.

Sorry I have a noob question.

I know this is part of an arc bat file (I love working with batch), however what is this part

-mprecomp+srep+lzma:a1:mfbt4:d256m:fb128:mc1000:lc8

Because when I run it,
arc a -lc8 -ep1 -ed -r -w.\ game-1.bin -mprecomp+srep+lzma:a1:mfbt4:d256m:fb128:mc1000:lc8 "D:\GAMES\my_game_file\*"

gives me an error
recent one was general (de)compression error in precomp
other times it looks like it skips both precomp and srep and just finishes susccessfully.

How am I using this incorrectly?

Thanks!

^^

The -m parameter in FreeArc specifies the compression method to be used when making your archives.

Methods can be internal methods built into FreeArc or they can be defined in arc.ini as external compressors.

In your case, it looks like precomp and srep are external compressors whereas you are using FreeArc's internal lzma compressor.

Secondly, when you have methods with colons after their name, this is a way of defining parameters for that compressor at runtime.

For example, the oggre compressor by ProFrager, arc.ini would look like this:

FreeArc has many internal constants, two of which I have shown above.

{compressor} instructs FA to look for oggre.exe. It simply fills the {compressor} with the name defined at the start of the entry.

{options} is where we pass our arguments at runtime. For example, if I used -moggre:s3, FA would translate the s3 to -s3 and use that parameter for oggre.exe.

@Masquerade

thanks.. easy to follow.
I noticed when reading the "specific game" section they put up a series of commands. Using an example where in the command line is that inserted?

Is there a thread here that goes into more detail. I would like to explore this more in depth.

Thanks!

hI
OK, One final Noob question

regarding the "specific game" section ??

What criteria is needed to determine which method to use.

Lets say for example I have a ton of wav files in a game, or lots of DDS files. What would I need to look for?

Thanks!

Please may you give an example of what you mean?
for wavs, use msc with tak plugin. For DDS, use srep then lolz.

^^^
Example: Yes
Here is a good page I think covers the information I am looking for

https://fileforums.com/showpost.php?...1&postcount=11

Which goes hand-in-hand with my question for the following you suggested:
How are these and what you suggested implemented?

Please keep in mind, that I am from the old BBS days where we ripped games in order to get them to make for a decent download on 9,600 - 14.4bps modems. So this method with arc is a little new to me after some 30yrs :-) (ok, Im an old fart)

^
You cannot use more then one -m option in command line for a single archive. So either combine them or make two archives.

As for the TAK plugin for MSC, you just have to place "pack_TAK.exe" in the same folder where "msc.exe" is located for compression and use in arc.ini like this

I caught that right as I sent that. I am working with too many bat files, so cleaned it up. I'll take another look again

Thanks!

Best compression method for text files?

Razor,ppmd.

LOLZ speed vs. Windows frameworks...
 

Special question: This minimal software requirement, prerequisities have installed on freshly installed Win7 x64 machine???

Tested few compressors in pure Win7 x64 installations and get very slow compression speed's. Not installed VC++, NETFX and more...

LOLZ in get around 100-150kB/s speed and not utilize more core/threads..
Chechked in LZTurbo v1.2 its slow.
Checked Razor 1.03.7 its slooow.
Checked Nanozip 0.09 x64 its finely goodly packing via optimum1 mode utilize 8 threads and get avg. 13MB/s compression speed! optimum2 or cc mode its slow. (Tested in highly compressibble data sample, compression ratio higher than 20:1)
Tested WinRAR 5.60 x64 and packing in very goodly avg. 25MB/s speed use, Best/Solid mode and 1GB dictionary. CPU usage 30-60% percentage, probably bottlenecked via 17 years old HDD.

Installed VC++ 2012 packages taken from Catalyst 15.7.1 package and LOLZ compress its faster! Any idea?

Any members I have Xeon CPU??

My PC in has Xeon E5-2665 CPU + 16GB RAM. Will hope compression speed via 1 threads around probably really getting avg. 400-800kB/s or higher. (LDMF mode) Or higher than ~2MB/s via MTT mode and 8+ threads.

lolz don't have dependencies.
It's compression speed for 1 thread is slow, you can use faster params or turn on one of two MT modes.

On your Xeon you can utilize all 16 threads, it's also the maximum lolz can use anyway.

Also, ldmf mode is very slow compared to non-ldmf, while giving not much better results than srep+lolz

Hello. I reviewed literally everything on this forum which is considered as a "guide", and I still can't understand what I should do and how should I apply all that compression methods and scripts on different kind of files (audio, movies etc.) and moreover how should I determine what is what and which files should I compress with which method?
I reviewed best compression methods index for the game I wanted to compress (gta4) and I have no clue what should I do with that information:
gta iv complete edition 12.9GB
audio file srep+lolz...
you can use msc but i didn't see improve ratio...
movie bkp...
rest of the file zlib:max:c32m+srep:m3f:a2+lolz...
I downloaded DiskSpanGUI and all I understood for now is how to select the input game folder. How should I learn about game compression if there is no comprehensible clear guide for begginers, only some general info like "use XTool and compress your games", but not a single example about how to apply all that scary configs and scripts for a certain files.
Please, if I missed something on this forum what potentially could've helped me - point me on that.

There is literally not a single comprehensive step-by-step instruction for beginners on this forum, all I see as a beginner is methods and troubleshooting discussions. Sorry for that, but as a beginner who spent 2 days on this forum I understood nothing, even though I really wanted. Thank you.

@sailwhist

I will take your example for GTA4.

Sometimes you already see what a game file could include just by file name or folder name.

For example the following file
"sfx" just means "sound effects" in the audio terminology. This means, you already know what is inside this file or what the file will contain mostly - namely sound/audio files of some sorts.

I don't have the game so I have to check steamdb depot for this game and sort the file list by type. What I can see is that literally all *.rpf files contain sound/audio files, just by looking at the file name and/or folder where the files are located.

Another example for this type is
Just looking at all the file names beginning with radio_ I am pretty sure all these files contain all the music for the radio stations in game, just because I know the word "radio" and the name of the radio station in the file name.

But because I don't have the game nor any of the files, I can just guess that the files contain riff-wave files, mp3 files or raw files or whatever else, because I know msc can handle them mostly. And the post creator you have quoted said "you can use msc but i didn't see improve ratio...", so I think my guess is correct here lol.

Next one: movie files
For bink movies (BIK or BK2) you can mostly use BinkPack (bpk), which will compress this files lossless. Just note that sometimes it will not work, either because the bink files using some sort of alpha channel, using a codec which bpk cannot handle correctly, or - which applies to BK2 files - have a newer codec which BinkPack cannot handle at all. If you have Bink video files, you can do the following:

1. Use HxD and see how the beginning of the file looks like. If the file has a "KB2n" (image) or anything greater then the "n" letter, BinkPack will fail on these files (this is mostly the case in new games, especially games which are build with unreal engine 4/5).
2. Copy all the Bink video files to a test folder and use bpk on them, testing compression and decompression. Or use bpk file-by-file instead of all at once. This way you can sort out faulty ones of the list and note them somewhere, so you know what files making trouble to exclude them from bpk compression.

Next one: zlib etc.
Again back to steamdb depot for the game GTA4, I also see "*img" files, which are 6.91 GiB in size. I would guess that these files contain all sorts of other files and textures. So now, the post you quoted sais "rest of the file zlib:max:c32m+srep:m3f:a2+lolz... ", which is including this *.img files. In other words, it seems that these *.img files are compressed with either zlib or deflate I would say. Devs simply do this to save space. To be sure, you can follow this little guide (to be fair this is for oodle compression, but just ignoring oodle option and using zlib option isn't that complicated).

We always want to extract all the data/streams in the game files if possible, in other words we want to decompress them (xtool/other tools) to have fully possibility of deduplication (srep) and compression (lolz/lzma/whatever else) in order to compress them even better than the original.

Thank you for your assistance, but still I have no clue how to operate with all of these in DiskSpanGUI and how to apply all these methods, select files by files and make everything combined work altogether in a final compressed game.
Talking about audio/movies - I'm beginning to understand where they stored, but I'm not sure how to find every single audio/movie file in game folder to make sure I compressed all of these files with the right method.
Also, I don't understand how to choose exact settings for that method for gta4 (zlib and etc.). In diskspangui it only shows me presets for that but without numbers like in those guy method. There are a lot of settings in that software which scares me.
I literally don't know everything and I would like to see a detailed guide with an example on some game about how to apply all of these so me and others could use that as a complete FAQ on compressing games.
Thank you.

Ah, I forget to explain a small but important detail about all the numbers in the method "those guys" sometimes share.

First, you have to keep in mind that tools are changing over time. Old tools like "zlib:max:c32m" (which should be this tool or maybe this) are not used anymore nowadays, because some tools are "evolving" over time, new tools are comming and old ones are obsolete.

So, the guy you quoted posted this method on 14-03-2018, which was over 5 years ago. On this time there was no "xtool". Means, methods and tools can (and most likely will) change over time. The method "zlib:max:c32m" belongs to an old tool. No one is using this tool anymore. "zlib" is the codec, ":max" is some other option I don't remember (I guess it has something to do with stream detection) and ":c32m" is the chunk size used for this tool.

Basically you have to "transfer old methods into new methods". So the old method "zlib:max:c32m" is something like "xtool:c32mb:mzlib" in diskspangui.

Edit:
Again I forgot to tell you "how to use this method on GTA4 for your needs" lol, sorry.

Personally I do all of this configuration stuff through the INI file "Resources\DSG_Settings.ini", just because I find it more easier.

So for your example of GTA4 it could look like this:
Data1 only includes *.rpf (audio) files, so I would suggest msc over srep+lolz, just because for the fact that lolz is not meant to used with audio files at all. lolz is meant to be used especially for texture files. So I would use msc, which is faster and should have near the same ratio. Sometimes it is a good idea to use even msc+srep, but I cannot say for sure here in this case because I don't have this game to test it.
Data2 includes only *.img files, which probably using some compression like zlib or deflate, so you have to use xtool here. If xtool:mzlib doesn't work, try with another codec like xtool:mpreflate or xtool:mreflate.
Data3 includes only *.bik files. I guess it ok to use bpk method only.
Data4 includes all other files, excluding *.img, *.rpf and *.bik, so you can just use srep+lolz if you want.

|0 indicates that this method belongs to freearc, while |1 is 7-zip (.7z) and |2 is RAR (.rar) (freearc methods cannot be used with |1 or |2). I guess using |1 or |2 are special and absolutely rare cases of some sorts.
InclText and ExclText telling diskspangui/freearc, that only given file types/files with paths should be included in given archive, or exclude them of the given archive. In the example above we have 3 archives which includes specific file types. So we need a last archive which includes all other files except the 3 previous archives, otherwise we would missing all other files on installation which are not exclusively included in other archives. Makes sense, does it? PS: In InclText and ExclText you can (or have to) separate file types/files with paths with a | character.
<game><data> are just placeholders or variables. <game> is actually [Game#Settings] (it takes the number # from [Game1Settings] section, so 1), <data> is actually Data#, which takes the number # from Data#. However you can name the archives to whatever you want. Just make sure to not use any super special characters.

FitGirl: Thanks from reply!

Two MT mode? Or MTT only?? LDMF its supporting multithreaded compression, with lower ratio! Special commands trick to usage.

https://i.kek.sh/bNhofYWbNzw.png
https://i.kek.sh/2udWoqik8Ut.png

Xeon 16 threads with 16GB RAM from LOLZ --> Use 128MB dictionary size. Memory its fully loaded.

srep:m3f:a0:l512+LOLZ:mtt1:mt2/4+ worser ratio than srep:m3f:a0:l512+LOLZ:ldmf1 mode.