Go Back   FileForums > Off Topic > Software

Reply
 
Thread Tools Display Modes
  #1  
Old 11-02-2017, 00:20
Gupta Gupta is offline
Banned
 
Join Date: Aug 2016
Location: https://t.me/pump_upp
Posts: 399
Thanks: 139
Thanked 714 Times in 231 Posts
Gupta is on a distinguished road
Send a message via ICQ to Gupta Send a message via AIM to Gupta Send a message via Yahoo to Gupta
Preprocessing / Optimization / Re compression For Better Compression of Games

[DEFLATE/ZLIB]
1.Precomp: Precomp is a command line pre-compressor. You can use it to achieve better compression on some file-/streamtypes (works on files and streams that are compressed with Zlib or the Deflate compression method, bZip2, GIF, JPG, and MP3). Precomp tries to decompress the streams, and if they can be decompressed and "re-"compressed so that they are bit-to-bit-identical with the original stream, the decompressed stream can be used instead of the compressed one.
Links:
:Gitter Chat
:Github Repository
:Github Releases
:Universal CLS-Precomp By RAZOR12911
:CLS-PRECOMPMT By RAZOR12911
Downloads:
:PrecompInside0.31.7z

2.PZLIB V3: A fast and really very fastest re-compressor for Deflate and Zlib, developed by RAZOR12911 but has some bugs when restoring several files via stdin stdout. Hence advised to be used separately on files
Links:
:Official Thread

3.PZLIB V1: The predecessor of PZLIB V3, fast and good for games containing zlib streams compressed using default and maximum level
which have headers example GTA IV, Dragon Age etc.
Links:
:Official Thread

4.Reflate: universal deflate re-compressor
Links
:CLS-Reflate by RAZOR12911
Download:
:Reflate V1l1 (GCC Compiled)

[Stream/Specific Compressors]
1.Media Stream Compressor (MSC): bmp Packer/preprocessor for multimedia data. It currently supports the following data types: dds (raw, dxtBMP, Wav (PCM), mp3. Because This preprocessor is sharpened by working with freearc, the data processing is carried out with a single input file. The output is just one file of its own format.
Suggestion: Should be used be before rep or basically before any "compressor"
-mmsc+srep+lzma or -mprecomp+msc+srep+lzma -> correct
-mprecomp+srep+msc or -msrep+msc -> incorrect, will ruin compression
Download:
:msc0.064.7z (Contains CLS)

2.MPZ: Best Mp3 compressor Developed by Eugene Shelwein. It usually reduces file size by 15 % but on some files, it can go par 60%
Download:
:mpzapi_v1a.rar

3.PackJPG: A compression program for further compressing JPEG image files Github:

4.PackARC: packARC is an archiver program specially designed for lossless further compression of media files in the JPEG, MP3, BMP and PNM formats. Github:

5.PackMP3: A compression program for further compressing MP3 audio files Github

6.PackPNM: A compression program for further compressing PNM, BMP and Radiance HDR files Github

7.unpackJPG: Unpack only, PackJPG Github

8.Lepton: Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
Links
:Github repository
:Github releases
:Official Blog Post

9.Png optimize suite(Win32) (Ver 2012-06-15): set of console compressor for Pngs
Download
:pngopt-120615.7z
:Blog Post

10.OptimFrog:OptimFROG is a lossless audio compression program. Its main goal is to reduce at maximum the size of audio files while permitting bit identical restoration for all input. It is similar with the ZIP compression, but it is highly specialized to compress audio data.
Downloads:
OptimFROG_Win_x86_5100.zip
OptimFROG_Win_x64_5100.zip

11.TAK: Tom's lossless Audio Kompressor (TAK) is a lossless audio compressor which promises compression performance similar to Monkey's Audio “High” and decompression speed similar to FLAC.
Downloads:
:TAK_2.3.0.zip
Links
:HomePage

[Prerocessors]

Delta: This preprocesses data improving their further compression. It detects tables of binary records and
1) substracts successive values in columns,
2) reorder columns trying to maximize results of further compression.
should be used on raw data i.e. not processed with srep, any deflate compressor, or something other

Usage:
Code:
fazip delta infile outfile
Rep : A LZ77 compressor detects repetition in 1GB Block
Usage:
Code:
fazip rep:<dictionary size>mb infile outfile
Super-rep : Successor of rep, supports dictionaries larger than RAM available.Compression speed is 100 mb/s and decompression runs at 250 mb/s on i3-2100.
:Description of The Algo
:srep393a.7z
CLS:SrepInside0.33.7z

[Loossy Compressors]

OggEnc: A very good lossy Compressor for Waves
Links
Official Site
Downloads:
oggenc2.88-1.3.5-generic.zip
oggenc2.88-1.3.5-x64.zip

Binks Lossy Compressor: Binks Reencoding script written by PANKER1992.
Link:
Official Discussion
Downloads
Bink_Enc.7z

Last edited by Gupta; 03-04-2017 at 21:48.
Reply With Quote
The Following 18 Users Say Thank You to Gupta For This Useful Post:
1234567890123 (25-05-2017), 78372 (01-04-2017), BLACKFIRE69 (11-06-2019), elit (16-02-2018), EzzEldin16 (14-07-2017), Gehrman (07-03-2022), Harsh ojha (28-11-2021), kassane (15-02-2017), oltjon (07-04-2017), oo7 (30-12-2021), Prince4 (13-02-2017), Razor12911 (25-02-2017), romulus_ut3 (31-03-2017), shazzla (11-06-2020), Simorq (06-03-2017), teusma (14-10-2017), TheGeekyGamer (28-06-2018), Vladislav989 (13-09-2022)
Sponsored Links
  #2  
Old 13-02-2017, 05:48
Prince4 Prince4 is offline
Registered User
 
Join Date: Mar 2015
Location: India->Delhi->Rohini
Posts: 121
Thanks: 109
Thanked 19 Times in 15 Posts
Prince4 is on a distinguished road
Smile

Some knowledge collaborative like Razor's!!

Keep it up!
__________________
Who's Got Time To Play Games? Huh??
Reply With Quote
  #3  
Old 14-02-2017, 23:23
Gupta Gupta is offline
Banned
 
Join Date: Aug 2016
Location: https://t.me/pump_upp
Posts: 399
Thanks: 139
Thanked 714 Times in 231 Posts
Gupta is on a distinguished road
Send a message via ICQ to Gupta Send a message via AIM to Gupta Send a message via Yahoo to Gupta
ppx2: ppx2 allows parallel processing of arguments
Changes by me:
Quote:
setting priorities of the processes
start process with new console window
verbose mode
and others i don't remembering
Here it is
ppx2.7z

For help:
ppx2 --help

The source can be found here:
https://github.com/guptaprince/ppx2

Last edited by Gupta; 27-02-2017 at 08:01.
Reply With Quote
The Following User Says Thank You to Gupta For This Useful Post:
Simorq (27-02-2018)
  #4  
Old 15-02-2017, 00:06
EzzEldin16's Avatar
EzzEldin16 EzzEldin16 is offline
Registered User
 
Join Date: Sep 2016
Location: USA
Posts: 112
Thanks: 343
Thanked 47 Times in 23 Posts
EzzEldin16 is on a distinguished road
i have a question
i compressed a game with zlib streams found alot of them
i compressed with precompressor it packed well but unpacking didn't
is the problem with the .iss script or what i'm using wpi script by razor
__________________
learn it
Reply With Quote
  #5  
Old 15-02-2017, 00:16
Gupta Gupta is offline
Banned
 
Join Date: Aug 2016
Location: https://t.me/pump_upp
Posts: 399
Thanks: 139
Thanked 714 Times in 231 Posts
Gupta is on a distinguished road
Send a message via ICQ to Gupta Send a message via AIM to Gupta Send a message via Yahoo to Gupta
Are u talking about pzlib?
Then it is the pre-compressor itself.
there is no way to fix that
Reply With Quote
The Following User Says Thank You to Gupta For This Useful Post:
EzzEldin16 (14-07-2017)
  #6  
Old 23-02-2017, 09:28
ZakirAhmad ZakirAhmad is offline
Registered User
 
Join Date: Sep 2015
Location: Kmr
Posts: 290
Thanks: 56
Thanked 56 Times in 37 Posts
ZakirAhmad is on a distinguished road
Xdelta

How xdelta can be used to decompress Compressed files... I have seen it in crysis 3 repack of Fitgirl... The files r md5 after decompression are orginal...
Reply With Quote
  #7  
Old 24-02-2017, 00:13
Joe Forster/STA's Avatar
Joe Forster/STA Joe Forster/STA is offline
Senior forum member
 
Join Date: Nov 2000
Location: Hungary
Posts: 9,790
Thanks: 16
Thanked 327 Times in 212 Posts
Joe Forster/STA is on a distinguished road
Xdelta is used for patches. (You know, game X version 1.0 to game X version 1.1: version 1.0 files are the input, version 1.1 are the output.) It does compress the patch file, though. See .RTPatch for a very similar, but commercial, software.
__________________
Joe Forster/STA
For more information, see the FileForums forum rules and the PC Games forum FAQ!
Don't contact me via E-mail or PM to ask for help with anything other than patches (or software in general) done by me, otherwise your request may be deleted without any reply!
Homepage: http://sta.c64.org, E-mail: [email protected]; for attachments, send compressed (ZIP or RAR) files only, otherwise your E-mail will bounce back!
Reply With Quote
The Following User Says Thank You to Joe Forster/STA For This Useful Post:
Simorq (06-03-2017)
  #8  
Old 24-02-2017, 03:39
EzzEldin16's Avatar
EzzEldin16 EzzEldin16 is offline
Registered User
 
Join Date: Sep 2016
Location: USA
Posts: 112
Thanks: 343
Thanked 47 Times in 23 Posts
EzzEldin16 is on a distinguished road
@joe foster/STA can you explain how it is used precisely ??
__________________
learn it
Reply With Quote
  #9  
Old 24-02-2017, 07:46
Joe Forster/STA's Avatar
Joe Forster/STA Joe Forster/STA is offline
Senior forum member
 
Join Date: Nov 2000
Location: Hungary
Posts: 9,790
Thanks: 16
Thanked 327 Times in 212 Posts
Joe Forster/STA is on a distinguished road
Imagine that you're the developer of game X. You've already released version 1.0 (CD/DVD, Steam etc.) but found some bugs since (wow!). You fixed (some of) them, calling it version 1.1, but you can't re-release the whole game; you rather collect the changes into a patch. This patch can be a package of the full files that have been changed between version 1.0 and 1.1 but that can be really big if it contains many files and/or some of the files are big.

A smarter idea is collecting the actual differences between the files. (Obviously, if those files exist in both versions.) Say, you have a big data file that has an internal structure and it contains many little files. (E.g. as simple as a ZIP archive.) If you add one internal file then only that part of the big data file changes, the rest stays the same. If you fix some code and recompile the program file (executable) then, again, most of the file stays the same, only some parts of it (added code, added data, modified jump offsets, modified data pointers/references) will change.

Xdelta and .RTPatch are programs that can do a binary comparison between a bunch of "old" files (those of version 1.0) and "new" files (version 1.1) and create a patch. If your customer downloads the patch and has the "old" files, xdelta can use ("apply") the patch to convert the "old" files into the "new" ones.

It doesn't matter if the "old" and "new" files are big, if the actual differences are small then the patch will be small, too. You can create a single patch file out of the differences of multiple files, including creating a new file (that didn't exist in version 1.0 yet), deleting a file (that doesn't exist in version 1.1 anymore) or modifying a file. I'm not sure about xdelta but with .RTPatch you can also create multi-version patches (version 1.0 or 1.1 or 1.2 -> 1.3) that can modify many versions of "old" files into the "new" ones.

I've seen xdelta used in the game called "The Thing" (from 2002 or something). Most developers use .RTPatch because it probably has more features but it's extremely expensive (thousands of dollars) and closed source, while xdelta is free and open source.
__________________
Joe Forster/STA
For more information, see the FileForums forum rules and the PC Games forum FAQ!
Don't contact me via E-mail or PM to ask for help with anything other than patches (or software in general) done by me, otherwise your request may be deleted without any reply!
Homepage: http://sta.c64.org, E-mail: [email protected]; for attachments, send compressed (ZIP or RAR) files only, otherwise your E-mail will bounce back!
Reply With Quote
The Following 2 Users Say Thank You to Joe Forster/STA For This Useful Post:
78372 (30-08-2017), EzzEldin16 (01-04-2017)
  #10  
Old 13-03-2017, 23:38
ImmortalArc's Avatar
ImmortalArc ImmortalArc is offline
Registered User
 
Join Date: Nov 2016
Location: Santiago
Posts: 72
Thanks: 44
Thanked 9 Times in 4 Posts
ImmortalArc is on a distinguished road
Quote:
Originally Posted by Joe Forster/STA View Post
.RTPatch it's extremely expensive (thousands of dollars) and closed source, while xdelta is free and open source.
Whoaaa
Attached Images
File Type: jpg IMG_20170314_130618_022.JPG (116.2 KB, 1876 views)
__________________
Sharing is caring : immortal words..
Reply With Quote
  #11  
Old 31-03-2017, 21:37
Gupta Gupta is offline
Banned
 
Join Date: Aug 2016
Location: https://t.me/pump_upp
Posts: 399
Thanks: 139
Thanked 714 Times in 231 Posts
Gupta is on a distinguished road
Send a message via ICQ to Gupta Send a message via AIM to Gupta Send a message via Yahoo to Gupta
UPDATED

- New Lossy Section
- Add some Other Compressor

Guys, Please please help me further improve this thread
Thanks.
Reply With Quote
The Following User Says Thank You to Gupta For This Useful Post:
romulus_ut3 (20-04-2017)
  #12  
Old 25-11-2021, 08:24
kj911 kj911 is offline
Registered User
 
Join Date: Apr 2010
Location: world
Posts: 179
Thanks: 130
Thanked 61 Times in 42 Posts
kj911 is on a distinguished road
QOI Lossless Image Format

Introducing QOI — the Quite OK Image format. It losslessly compresses RGB and RGBA images to a similar size of PNG, while offering a 20x-50x speedup in compression and 3x-4x speedup in decompression. All single-threaded, no SIMD. It's also stupidly simple.

https://phoboslab.org/log/2021/11/qo...ge-compression

Nice preprocessors from visually losslessly recompress PNG files smaller than ~30 to 50% percentage to original ones. (Hint: compress encoded QOI files use CM/LZ* or any algo.)

Compressed PNG files to QOI format, its bigger or smaller than original PNG file and fairly compressibble than orig. PNG files. Decoded PNG files its smaller/bigger than original PNG files. Not bitperfect lossless compression. Visually lossless only.

Example:

Original test.png file: 213 745 bytes
QOI file: 398 930 byte
Decompressed PNG file to QOI file: 336 428 byte
Compressed QOI file with RAR: 96 430 byte (internal size in archive)

Example2:

pg-coral.png 265K
pg-coral.png.balz 266K // balz cannot improve png!
pg-coral.qoi 371K // seems bad? but not if further compressed
pg-coral.qoi.zip 250K // better than PNG!
pg-coral.qoi.balz 222KB // even better! total-code size (Balz + qoi + strip comments&headers) is maybe 500 lines of code.

I think if qoi used my suggestions (tiling+delta), we can get the file much smaller, maybe 200K.

Further tests with XZ:

pg-coral.qoi.xz 182KB

Again, if qoi used tile+delta, maybe we can get the file down to 150KB? (using xz)

Compiled Win XP x86 compatible EXE attached.
Attached Files
File Type: rar qoiconv_x86_winxp_20211125.rar (40.5 KB, 9 views)
Reply With Quote
The Following User Says Thank You to kj911 For This Useful Post:
Gehrman (07-03-2022)
  #13  
Old 30-11-2021, 05:44
Dev_0016 Dev_0016 is offline
Registered User
 
Join Date: Nov 2021
Location: India
Posts: 5
Thanks: 0
Thanked 1 Time in 1 Post
Dev_0016 is on a distinguished road
Hey I need your help how fitgirls and other repacker like ( elamigos, rg ...)..
how they were compress thier files ( highly compress ) which type of tools they were use
to compress the files and make repack of them..??? will you tell me please ..
Reply With Quote
  #14  
Old 18-02-2022, 03:12
kj911 kj911 is offline
Registered User
 
Join Date: Apr 2010
Location: world
Posts: 179
Thanks: 130
Thanked 61 Times in 42 Posts
kj911 is on a distinguished road
rANS data compression in patented!!!

ALERT! S...ing MS!!!

Microsoft last month received a US patent covering modifications to a data-encoding technique called rANS, one of several variants in the Asymmetric Numeral System (ANS) family that support data compression schemes used by leading technology companies and open source projects....

https://www.theregister.com/2022/02/...ft_ans_patent/

https://patents.google.com/patent/US11234023B2/en
Reply With Quote
  #15  
Old 19-10-2022, 03:25
kj911 kj911 is offline
Registered User
 
Join Date: Apr 2010
Location: world
Posts: 179
Thanks: 130
Thanked 61 Times in 42 Posts
kj911 is on a distinguished road
Qouting from krinkels posts, these last Optimfrog EXE file modification says its to Edison007:

4. Last but probably the most interesting. The seek parameter is the number of samples in the compressed block. The original values ​​are as follows:

Code:
default options number of samples: 
fast = 44100 * 10 
normal = 44100 * 20 
slow = 44100 * 40 
min = 44100 * 120 

mod1 
fast = 88200 * 40 
normal = 88200 * 80 
slow = 88200 * 120 
min = 88200 * 160 

mod2 
fast = 352800 * 60 
normal = 352800 * 80 
slow = 352800 * 120 
min = 352800 * 160
Actually, you can patch ofr.exe by changing them in any direction (but with an increase, the consumed RAM also grows), this can affect both for the better and for the worse. Whoever knows how to use IDA can find and change where it is, who can't wait for either a patcher or patched executables from me, I'll think about how simpler it is.


These any values increase --> its have better compression ratio?? How to its modding the EXE file?? I would replace the 2nd number with 320 instead of 160.

Also: how to methods from upgrading CLS-MSC.dll files with newest TAK/OFR/BMF libraries??
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Games Are For Kids JoyBoy Chit Chat 83 16-11-2021 14:28
Multiple Games to DVD Grumpy PC Games - CD/DVD Conversions 6 22-12-2005 15:47
Frequently Asked Questions Joe Forster/STA PC Games - Frequently Asked Questions 0 29-11-2005 09:48
i got the email in the mail about copying ps2 games EVOEvo PS2 Games 7 21-11-2003 07:56
WARNING: New games use new compression for movies files! Flavangel DC Games 0 26-06-2001 14:35



All times are GMT -7. The time now is 01:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com