Preprocessing / Optimization / Re compression For Better Compression of Games

[Gupta]

[DEFLATE/ZLIB]
1.Precomp: Precomp is a command line pre-compressor. You can use it to achieve better compression on some file-/streamtypes (works on files and streams that are compressed with Zlib or the Deflate compression method, bZip2, GIF, JPG, and MP3). Precomp tries to decompress the streams, and if they can be decompressed and "re-"compressed so that they are bit-to-bit-identical with the original stream, the decompressed stream can be used instead of the compressed one.
Links:
:Gitter Chat
:Github Repository
:Github Releases
:Universal CLS-Precomp By RAZOR12911
:CLS-PRECOMPMT By RAZOR12911
Downloads:
:PrecompInside0.31.7z

2.PZLIB V3: A fast and really very fastest re-compressor for Deflate and Zlib, developed by RAZOR12911 but has some bugs when restoring several files via stdin stdout. Hence advised to be used separately on files
Links:
:Official Thread

3.PZLIB V1: The predecessor of PZLIB V3, fast and good for games containing zlib streams compressed using default and maximum level
which have headers example GTA IV, Dragon Age etc.
Links:
:Official Thread

4.Reflate: universal deflate re-compressor
Links
:CLS-Reflate by RAZOR12911
Download:
:Reflate V1l1 (GCC Compiled)

[Stream/Specific Compressors]
1.Media Stream Compressor (MSC): bmp Packer/preprocessor for multimedia data. It currently supports the following data types: dds (raw, dxtBMP, Wav (PCM), mp3. Because This preprocessor is sharpened by working with freearc, the data processing is carried out with a single input file. The output is just one file of its own format.
Suggestion: Should be used be before rep or basically before any "compressor"
-mmsc+srep+lzma or -mprecomp+msc+srep+lzma -> correct
-mprecomp+srep+msc or -msrep+msc -> incorrect, will ruin compression
Download:
:msc0.064.7z (Contains CLS)

2.MPZ: Best Mp3 compressor Developed by Eugene Shelwein. It usually reduces file size by 15 % but on some files, it can go par 60%
Download:
:mpzapi_v1a.rar

3.PackJPG: A compression program for further compressing JPEG image files Github:

4.PackARC: packARC is an archiver program specially designed for lossless further compression of media files in the JPEG, MP3, BMP and PNM formats. Github:

5.PackMP3: A compression program for further compressing MP3 audio files Github

6.PackPNM: A compression program for further compressing PNM, BMP and Radiance HDR files Github

7.unpackJPG: Unpack only, PackJPG Github

8.Lepton: Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
Links
:Github repository
:Github releases
:Official Blog Post

9.Png optimize suite(Win32) (Ver 2012-06-15): set of console compressor for Pngs
Download
:pngopt-120615.7z
:Blog Post

10.OptimFrog:OptimFROG is a lossless audio compression program. Its main goal is to reduce at maximum the size of audio files while permitting bit identical restoration for all input. It is similar with the ZIP compression, but it is highly specialized to compress audio data.
Downloads:
OptimFROG_Win_x86_5100.zip
OptimFROG_Win_x64_5100.zip

11.TAK: Tom's lossless Audio Kompressor (TAK) is a lossless audio compressor which promises compression performance similar to Monkey's Audio “High” and decompression speed similar to FLAC.
Downloads:
:TAK_2.3.0.zip
Links
:HomePage

[Prerocessors]

Delta: This preprocesses data improving their further compression. It detects tables of binary records and
1) substracts successive values in columns,
2) reorder columns trying to maximize results of further compression.
should be used on raw data i.e. not processed with srep, any deflate compressor, or something other

Usage:

fazip delta infile outfile

Rep : A LZ77 compressor detects repetition in 1GB Block
Usage:

fazip rep:<dictionary size>mb infile outfile

Super-rep : Successor of rep, supports dictionaries larger than RAM available.Compression speed is 100 mb/s and decompression runs at 250 mb/s on i3-2100.
:Description of The Algo
:srep393a.7z
CLS:SrepInside0.33.7z

[Loossy Compressors]

OggEnc: A very good lossy Compressor for Waves
Links
Official Site
Downloads:
oggenc2.88-1.3.5-generic.zip
oggenc2.88-1.3.5-x64.zip

Binks Lossy Compressor: Binks Reencoding script written by PANKER1992.
Link:
Official Discussion
Downloads
Bink_Enc.7z

[Prince4]

Some knowledge collaborative like Razor's!!

Keep it up!

[Gupta]

ppx2: ppx2 allows parallel processing of arguments
Changes by me:

setting priorities of the processes
start process with new console window
verbose mode
and others i don't remembering

Here it is
ppx2.7z

For help:
ppx2 --help

The source can be found here:
https://github.com/guptaprince/ppx2

[EzzEldin16]

i have a question
i compressed a game with zlib streams found alot of them
i compressed with precompressor it packed well but unpacking didn't
is the problem with the .iss script or what i'm using wpi script by razor

[Gupta]

Are u talking about pzlib?
Then it is the pre-compressor itself.
there is no way to fix that

[ZakirAhmad]

Xdelta

How xdelta can be used to decompress Compressed files... I have seen it in crysis 3 repack of Fitgirl... The files r md5 after decompression are orginal...

[Joe Forster/STA]

Xdelta is used for patches. (You know, game X version 1.0 to game X version 1.1: version 1.0 files are the input, version 1.1 are the output.) It does compress the patch file, though. See .RTPatch for a very similar, but commercial, software.

[EzzEldin16]

@joe foster/STA can you explain how it is used precisely ??

[Joe Forster/STA]

Imagine that you're the developer of game X. You've already released version 1.0 (CD/DVD, Steam etc.) but found some bugs since (wow!). You fixed (some of) them, calling it version 1.1, but you can't re-release the whole game; you rather collect the changes into a patch. This patch can be a package of the full files that have been changed between version 1.0 and 1.1 but that can be really big if it contains many files and/or some of the files are big.

A smarter idea is collecting the actual differences between the files. (Obviously, if those files exist in both versions.) Say, you have a big data file that has an internal structure and it contains many little files. (E.g. as simple as a ZIP archive.) If you add one internal file then only that part of the big data file changes, the rest stays the same. If you fix some code and recompile the program file (executable) then, again, most of the file stays the same, only some parts of it (added code, added data, modified jump offsets, modified data pointers/references) will change.

Xdelta and .RTPatch are programs that can do a binary comparison between a bunch of "old" files (those of version 1.0) and "new" files (version 1.1) and create a patch. If your customer downloads the patch and has the "old" files, xdelta can use ("apply") the patch to convert the "old" files into the "new" ones.

It doesn't matter if the "old" and "new" files are big, if the actual differences are small then the patch will be small, too. You can create a single patch file out of the differences of multiple files, including creating a new file (that didn't exist in version 1.0 yet), deleting a file (that doesn't exist in version 1.1 anymore) or modifying a file. I'm not sure about xdelta but with .RTPatch you can also create multi-version patches (version 1.0 or 1.1 or 1.2 -> 1.3) that can modify many versions of "old" files into the "new" ones.

I've seen xdelta used in the game called "The Thing" (from 2002 or something). Most developers use .RTPatch because it probably has more features but it's extremely expensive (thousands of dollars) and closed source, while xdelta is free and open source.

[ImmortalArc]

Originally Posted by Joe Forster/STA

.RTPatch it's extremely expensive (thousands of dollars) and closed source, while xdelta is free and open source.

Whoaaa

[Gupta]

UPDATED

- New Lossy Section
- Add some Other Compressor

Guys, Please please help me further improve this thread
Thanks.

[78372]

May I say, you forgot AntiZ