FileForums - Masked Compression (Ultimate compression in one go)

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Masked Compression (Ultimate compression in one go) (https://fileforums.com/showthread.php?t=97530)

The only thing you must figure out is how the header is structured.

Normally the the precomp header gives the version. It must tell you where the processed streams are located then after finding then, it makes it very easy to make the unpacker.
Another tip, if using handles, never write anything on cmd window that isn't part of the data. Meaning all that Precomp v0.4.4 - UNSTABLE bla bla has to go else you'll end up with errors unnecessarily.

thanks, dont worry i know where to find you in case i find trouble :P

i hope you will be determined to help then as well :)

Panker, one question, what is the required ram for decompressing this data which is compressed using Masked Compression?

a lot of :D i expect you know that you can pack with x64 program and unpack with 32 bit program

if you use srep64 for decomp you can see 4 gigs of ram easily

and if you use nanozip it uses the same amount of ram as for compression

if you use my xz method which is ultra lz77 it uses total 500~800mb ram :P

Hi Panker, using the methods with the final mask configured in this way

Code:

"msc+srep64+delta+exe2+xz/$jpgsolid=jpg+zstd/$void=srep64+zstd/$text=nz/$precomp=precomp+msc+srep64+delta+lz77"

the compression is successful only with the "Precomp 0.38" with all other versions I get this error:

http://i67.tinypic.com/ogbcaq.jpg

Is there any solution, or for the time being, the insertion of the algorithm Precomp in final mask is recommended only with the use of precom0.38.:rolleyes:

well i have studied a lot of precomp source code lately,

i can tell some facts that has come to my attention

precomp 0.38 was a masterpiece of software, all other precomps are pretty unstable the way i see it, precomp 0.40 and above when fail to see zlib header come with this error in 90% of the cases

second fact , i havent uploaded nor updated my Masked Compression to use precomp 0.44 or even precomp 0.45 which is unstable :P not even previous precomps

my private messages are flooded by people begging to implement precomp 0.44 or even other precomps

i will not yet update the masked compression to support, because it spawn errors like these!!!

today i emailed Christian Shneider, critical error was fixed with memory leak, and i sent him library with tak, next precomp will most likely support wav precompression
and it already takes support for mp3 precompression

in the meantime precomp 0.44 offers nothing, nothing new is inside precomp 0.44 that will make precomp 0.38 obsolete

i literally beg people to not put precomp 0.44 to official public type tools for massive usage, and i havent released precomp 0.45 for this reason as well because i have the exe file but i keep it under heavy testing

therefore Felice i conclude that precomp 0.44 MUST be withdrawn anbd not yet be massively used under any circumstance especially on critical Data

Please Bare with me, for any changes on precomp i will post myself
it's not like i keep it for myslelf or something

sorry for this long post :)

OK masks works very well, with the configuration in the last mask "/$precomp=" with Precomp added, should only be used with precomp 0.38, other versions Precomp can be used safely without entering Precomp in the last mask.

Last Mask : "/$precomp=precomp+msc+srep64+...etc" (Only precomp0.38)

Last Mask : "/$precomp=msc+srep64+delta+exe2...etc" (All Precomp Version)

Panker you're doing a great job.;)

i will tell you a secret,

i have implemented it in my last v2.1 update

$precomp= ...etc etc is a mask to filter data that lz77, xz, fazip and generally any lz77 variant can compress only 5~8%, this is why you put precomp here because using precomp before lz77 can increase ratio by about 45~60% max

you can use precomp filter mask to select out uncompresable data and store using

$precomp=msc+srep+zstd this mask will allow you to select uncompressable data

for store purposes .... i have used it in my sonido mask (sonido = speed of sound )
my ultra fast mode in my tools :)

and yes

use can use any precomp if you put it in the beginning
for example
ultra = precomp+msc+srep64+delta+exe2+lz77/$jpgsolid=jpg+zstd/$void=srep64+zstd/$text=nz

but that will also increase time so a dual shaped sword is more like it :)

Hey @panker1992! First of all you're doing a great work!

I'm getting this error while decompressing!
http://i.imgur.com/g0NhwmM.png
Game's MAD MAX
used -mrefl8 to compress!
Input Size = 1.29 GB (one of the splitted folder)
Output Size = 355 MB

But when trying to decompress with WPI for masked compression! it shows the above error! I've checked the setup file, its not corrupted!
Please Help

The reflate razor made cannot be used in inno
It errors every time you need to have manual reflate for mad max
This is known issue :( I am sorry I cannot help

Reflate is beta tool for now

Masked Compression v 2.2

To show you my Gratitude for your continued support, i decided to roll Masked Compression v2.2

i already posted everything in the main post
so to change climate i will post some photos of my tools under usage :)

if you happen to test them :) give me your honest opinion

A) Would you like for me to add a new feature ??

B) Do you think i missed something important ??

anything you want i will test and add if i see potential

3 pics below: bik encoding using my script, success on decompressing steam folder after using on it all my tools simultaneously , and my new mask on usage offering 18% ratio up from 11%

Enjoy :)

Do not worry I do not update beta versions tester on my utility..;)
You run too much with your masks...:D, you have added the mask for the recompression data in MP3, not hear any difference in the optimization of compression threshold.

Tests executed on a data folder mp3 approximately 234MB

Masked method used for Test: "precomp+msc+srep64+delta+exe2+lz77/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz"

Your update: v 2.2..:rolleyes:

http://i66.tinypic.com/2ekhy79.png

M_S_Kompressor with optimized profile based multi mask By Panker..:)

http://i66.tinypic.com/9gf4mb.png

http://i63.tinypic.com/k4gebm.png

Personally, I may be wrong, I not have so much experience in data compression, but exceed on a number of algorithms containing in the method multi-maschera, involves a loss of a compression stream, increasing the compression time, and decreasing the compression threshold.

Hmmm use the mask directly .... test this maskless

and remove msc :P

both msr and msc can compress mp3 files ...either -mp3=0 or remove msc entirely from the test

to see if my mask really works :P else i need to find what it goes wrong :/

I had already tried...
The compression level of an audio file (*. mp3) and higher using a profile with MSC.
The same level of audio compression the lowers considerably by using MSR.
I'm sorry but I prefer to MSC

MSR(precomp0.45) is masked for only mp3 that are found on the folder,
if you pack your music library or photo library you will see difference,

MSR(precomp0.45) as it is now it doesnt replace msc in any way :) but the jpg part of it is worth it and of course its smaller size :P

I had noticed the Precomp0.45 disguised as MSR...:D also the warning that you wrote me in arc.ini...:p

ahahaha :D I like your work !!

i put so much effort not to pock you but your Italian and i cant resist :P
no hard feelings about arc.ini :) i respect you very much only wanted to pock you a little

precomp.45 offers a bit of mp3 recompression, along with bug fixes on a long forgotten packjpg.exe 2.5j edition, regarding memory leaks

this is the main reason that i switched to jpg compression with precomp0.45 :P

also jpg with precomp.045 is a little more optimized at least for me

Panker you did a mistake for the Ram Optimization, you didn't added mp3 method in the arc.ini

EDIT - but i copied from the arc.ini orginal from the masked compression 2.2

Quote:

Originally Posted by panker1992 (Post 446343)

i put so much effort not to pock you but your Italian and i cant resist :P
no hard feelings about arc.ini :) i respect you very much only wanted to pock you a little

Mi sa che tu un italiano doc non l'hai mai conosciuto....:D
se non ti calmi con tutte queste maschere, e riesci a compilare qualcosa di veramente concreto, a carnevale prendo l'aereo, in un paio d'ore sto da te in grecia...mi offri un pranzo con specialità greche..e poi ce ne andiamo a donne...ne ho conosciute di tutte le razze ma con una ragazza greca non ci sono mai stato..:p:p:p
Uses the online translation....:D

Quote:

Originally Posted by felice2011 (Post 446346)

Even, i read this message and understood that LOL

Felice you are welcome anytime :P

what i give is concrete :P everything i have ever made was because of experimentation

if you do not fail before you achieve something :P then its not funny at all

Basically i attend a university in Patrai :P it has ship for Italy so i can come instead :P

Hey Panker...you are also welcome, if you are in Campania near Naples let me know.:)

Pratasso seems like a nice place to vacation sea and hills, ideal.:D

Doing something concrete was just a joke, we are here for fun and hobbies, no one forces us to to do anything, and no one pays us here.;)

I will send you something when I get home I am outside at the moment

I also post a nasty bug I found :)

http://i.imgur.com/an0jXw5.jpg

I get this no matter what method I use to compress the archive, upon extraction, everytime I see this, and DLG's extractor tells me my disk is full (likely needs memory parameter tweaking) -_-

I need to post, a very nasty bug i fixed yesterday but it has nothing to do with decompression

it's high likely that you have a leak somewhere please post what you compress
which method and what parameters, any modification ....

Here's my "Advanced_Data_Comp.bat"

Code:

@echo off

color 0c

Title Compressor - by Panker1992

echo.   **************************************************************************

echo.   *    External Options for  Compressor  By Panker1992                     *

echo.   * Modes: Options stored in arc.ini see below for best results            *

echo.   * 1) fast =super fast, optimal =best w/o precomp, max =optimal+precomp *  

echo.   * smartmax =masked precomp, smartlzt =smartmax + lzt, legacy =oldmethod  *

echo.   * devastate = heavy duty comp, will take so much time you will freak     *

echo.   * 2) I added txt, void and jpeg as external for custom made bat files    *

echo.   **************************************************************************

pause

@echo Recompressing Data...

arc a -lc1024 -ld1024 -ep1 -ed -r -hp -pYourpassword -w.\ Data\data.bin -mstrong  "G:\Sniper Elite V2\*"

::examples, I recommend optimal and smartmax for ultimate results 

::devastate has ultimate compression but its so slow that its best only when you store long term

::arc a -lc1024 -ld1024 -ep1 -ed -r -hp -pYourpassword -w.\ Data\data1.bin -mfast  "D:\Games\Steam\*"

::arc a -lc1024 -ld1024 -ep1 -ed -r -hp -pYourpassword -w.\ Data\data1.bin -mstrong  "D:\Games\Steam\*"

::arc a -lc1024 -ld1024 -ep1 -ed -r -hp -pYourpassword -w.\ Data\data1.bin -mdevastate  "D:\Games\Steam\*"

::arc a -lc1024 -ld1024 -ep1 -ed -r -hp -pYourpassword -w.\ Data\data1.bin -mlegacy  "D:\Games\Steam\*"

echo *******************************************************************************

echo.

echo                Compressing is Completed

echo.

echo *******************************************************************************

pause

And here's my arc.ini:

Code:

[Compression methods]

; For Tiny SFX usage, dont use anything else other than that !!! 

sfx       =  msc+srep+delta+exe2+4x4:lzma:100m:max/$text=dict:p:128m+lzp:64m:105:d1m:s32:h22+ppmd:12:192m

sfxmax    =  precomp+msc+srep+delta+exe2+4x4:lzma:100m:max/$text=dict:p:128m+lzp:64m:105:d1m:s32:h22+ppmd:12:192m

lzma2     =  4x4:lzma:a1:mfbt4:d256m:fb273:mc1000:lc8

oldtxt    =  dict:p:128m+lzp:64m:105:d1m:s32:h22+ppmd:12:192m

; For Masked Compression Usage   (precomp:zl98:d1)

legacy    = msc+srep+delta+exe2+4x4:lzma:a1:mfbt4:d256m:fb273:mc1000:lc8

sonido    = msc+srep64+delta+exe2+zstd/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=dict:p:128m+lzp:64m:105:d1m:s32:h22+ppmd:12:192m

fast      = msc+srep64+delta+exe2+xz/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz/$precomp=msc+srep64+delta+zstd

optimal   = msc+srep64+delta+exe2+xz/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz

lztmax    = msc+srep64+delta+exe2+lzt/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz/$precomp=precomp+msc+srep64+delta+lzt

strong    = msc+srep64+delta+exe2+xz/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz/$precomp=precomp+msc+srep64+delta+lz77     

ultra     = precomp+msc+srep64+delta+exe2+lz77/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz

; these 2 methods provide ultimate compression ratios but they are BETA at the moment 

refl8     = msc+srep64+delta+exe2+xz/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz/$precomp=reflate+msc+srep64+delta+lz77 

devastate = precomp+msc+srep64+delta+exe2+nzmax/$jpgsolid=jpg+zstd/$mp3solid=mp3+zstd/$void=srep64+zstd/$text=nz



[External compressor:srep]

header = 0

packcmd   = srep -m3f -a1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = srep -d -s -- <stdin> <stdout>



[External compressor:srep64]

header = 0

packcmd   = srep64 {options} -m3f -d1g -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = srep64 -d -s -- <stdin> <stdout>



[External compressor:xz]

header = 0

packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d27:fb=273:mf=bt4:mc=1000000:lc=4:lp=0 -mmt=on -mx9 -si -so <stdin> <stdout>

unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout>



[External compressor:precomp]

header = 0

packcmd   = precomp -slow -t-j -o$$arcpackedfile$$.tmp  $$arcdatafile$$.tmp

unpackcmd = precomp -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp



[External compressor:reflate]

header = 0

packcmd    = reflate -process $$arcdatafile$$.tmp $$arcpackedfile$$.tmp 

unpackcmd  = reflate -restore123 $$arcpackedfile$$.tmp $$arcdatafile$$.tmp



[External compressor:zstd]

header = 0

packcmd   = zstd -f -9 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = zstd -d -f <stdin> <stdout>



[External compressor:rep,lzma,4x4,delta,exe]



header = 0



packcmd   = fazip -i0   compress:{compressor}{:option} <stdin> <stdout>



unpackcmd = fazip -i0 decompress:{compressor}{:option} <stdin> <stdout>





[External compressor:jpg,mp3]

header = 0

packcmd   = msr -fast -cn -t+j3 -o$$arcpackedfile$$.tmp  $$arcdatafile$$.tmp

unpackcmd = msr -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

solid = 0



[External compressor:lz77]

header = 0

packcmd = 7z a -txz -an -mcrc=0 -m1=lzma2:d200m:fb=273:mf=bt4:mc=10000:lc=4:lp=0 -mmt=2 -mx9 -si -so <stdin> <stdout>

unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout>



[External compressor:msc]

;Alt: c -v -f Forced //-frog=9 use this instead of -tak=9

header = 0

packcmd = MSC c -v -wav=1 -raw=1 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=1 -bmf=9s -tak=9 -dxt=1 -lzma=hc4,lc8,lp2,pb2,fb273,mc1000 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp



[External compressor:nz]

;options -cc=best -m3g=memory -nm 

packcmd   = nz a -r -v -cc -m3g -br128m -bw128m -p2 -t0 -nm $$arcpackedfile$$.tmp.nz $$arcdatafile$$.tmp   

unpackcmd = nz x -m1g -t0 -p0 $$arcpackedfile$$.tmp.nz $$arcdatafile$$.tmp     

datafile   = $$arcdatafile$$.tmp

packedfile = $$arcpackedfile$$.tmp.nz



[External compressor:nzmax]

;options -cc=best CM algo -m3g=memory -nm ; -cO fast option BWT algo

packcmd   = nz a -r -v -cc -m8g -br128m -bw128m -p4 -t0 -nm $$arcpackedfile$$.tmp.nz $$arcdatafile$$.tmp   

unpackcmd = nz x -m1g -t0 -p0 $$arcpackedfile$$.tmp.nz $$arcdatafile$$.tmp     

datafile   = $$arcdatafile$$.tmp

packedfile = $$arcpackedfile$$.tmp.nz



[External compressor:lzt]

;x64 version ;fast: -32 ;max:-p1 -49 -b200

packcmd   = lzt -p1 -49 -b200 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp   

unpackcmd = lzt -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp   

datafile   = $$arcdatafile$$.tmp

packedfile = $$arcpackedfile$$.tmp



[External compressor:precomp045]

; This is a method in test, dont use this unless you are a developer or Beta tester, I DO NOT PROVIDE PUBLIC SUPPORT YET (Felice dont update DLG with this :P )

header = 0

packcmd   = msr -intense0 -cn -zl98 -d1 -o$$arcpackedfile$$.tmp  $$arcdatafile$$.tmp

unpackcmd = msr -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

Here's my Unpacker.bat:

Code:

@echo off

setlocal enableextensions 





unarc.exe x -o+ -pYourpassword -dptest data.bin 



pause

And here's the arc.ini from the Decomp Tools folder:

Code:

[External compressor:srep]

header = 0

packcmd   = srep -m3f -a1 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = srep -d -s -- <stdin> <stdout>



[External compressor:srep64]

header = 0

packcmd   = srep64 {options} -m3f -d1g -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = srep64 -d -s -- <stdin> <stdout>



[External compressor:xz,lz77]

header = 0

packcmd   = 7z a -txz -an -mcrc=0 -m1=lzma2:d450m:fb=273:mf=bt4:mc=1000000:lc=4:lp=0 -mmt=4 -mx9 -si -so <stdin> <stdout>

unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout>



[External compressor:precomp]

header = 0

packcmd   = precomp -slow -t-j -o$$arcpackedfile$$.tmp  $$arcdatafile$$.tmp

unpackcmd = precomp -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp



[External compressor:reflate]

header = 0

packcmd    = reflate -process $$arcdatafile$$.tmp $$arcpackedfile$$.tmp 

unpackcmd  = reflate -restore123 $$arcpackedfile$$.tmp $$arcdatafile$$.tmp



[External compressor:zstd]

header = 0

packcmd   = zstd -f -9 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

unpackcmd = zstd -d -f <stdin> <stdout>



[External compressor:jpg,mp3,precomp045]

; to use precomp045 in this case you need to delete packjpg_dll1.dll packjpg_dll.dll and cls-precomp.dll from this menu

header = 0

packcmd   = msr -fast -cn -t+j3 -o$$arcpackedfile$$.tmp  $$arcdatafile$$.tmp

unpackcmd = msr -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

solid = 0



[External compressor:msc]

;Alt: c -v -f Forced //-frog=9 use this instead of -tak=9

header = 0

packcmd = MSC c -v -wav=1 -raw=1 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=1 -bmf=9s -tak=9 -dxt=1 -lzma=hc4,lc8,lp2,pb2,fb273,mc1000 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp



[External compressor:nz,nzmax]

;options -cc=best -m3g=memory -nm 

packcmd   = nz64 a -r -v -cc -m3g -br128m -bw128m -p2 -t0 -nm $$arcpackedfile$$.tmp.nz $$arcdatafile$$.tmp   

unpackcmd = nz64 x -m1400m -t0 -p0 $$arcpackedfile$$.tmp.nz $$arcdatafile$$.tmp     

datafile   = $$arcdatafile$$.tmp

packedfile = $$arcpackedfile$$.tmp.nz



[External compressor:lzt]

;x64 version ;fast: -32 ;max:-p1 -49 -b200

packcmd   = lzt -p1 -49 -b200 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp   

unpackcmd = lzt -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp   

datafile   = $$arcdatafile$$.tmp

packedfile = $$arcpackedfile$$.tmp

Quote:

Originally Posted by romulus_ut3 (Post 446358)

You've used mask with recompression: "/$mp3solid=mp3+zstd":rolleyes:

The "DLG Extractor v1.1" does not have support for MSR(precomp0.45 beta test)..;)

Quote:

Originally Posted by felice2011 (Post 446364)

You've used mask with recompression: "/$mp3solid=mp3+zstd":rolleyes:

The "DLG Extractor v1.1" does not have support for MSR(precomp0.45 beta test)..;)

I've tried it with the previous 0.38 precomp implementation, it still gave me the some error regarding disk being full.

http://i.imgur.com/dr6liKv.jpg

Bro, help me with this?

Quote:

Originally Posted by romulus_ut3 (Post 446373)

I've tried it with the previous 0.38 precomp implementation, it still gave me the some error regarding disk being full.

Compress with version (2.1) posted by Panker, version (2.2) does not use it, I think it works for new updates.
Give it a try and let me know if the extraction works with the DLG Extractor or Decomp Tools.

Edit: disable your antivirus during the operation.

is it possible to somehow make the progress bar during the installation smoothly? When using this method of compression is not a smooth progress bar and then hangs up and goes to 100%

Decomp Tools error

http://i.imgur.com/I83rWs1.png

This error occurs on Windows 7, I tested it on Windows 8.1 works, how to fix it?

Hello Dear Community, as of yesterday i am fixing a VERY VERY nasty BUG

i haven't finished fixing it , took me longer than i expected

but i hope when its done, Masks will surely look different

regarding this bug, it fixes masked compression file misdetection

i have found that if many masks are used in parallel detection will not work and files will be compressed using different method

so i decided to make new masks from scratch and write a new arc.groups
to make packing detect correctly and use all tools in the process

i am so taken in by this new work, that i didn't even log in the forums :/

allow me some time, and please use absolutely masks 2.1 legacy to work with

thanks for understanding :)

@andrey
i am working on a solution for smooth out the progress (progress.inf)

this will show the correct progress in the future and new precomp0.45 will be used extensively

Panker1992, see this
http://i.imgur.com/gtw3V4m.png

I used Masked Compression v2.2 and with Blackbox v2 updated one, which is modified for Masked Compression v2.2 by you! So can you tell me any solution for this problem!
Thanks in advance

I am in the middle of fixing a very serious bug, i have almost finished and my solution works

please use masked compression v 2.1, or wait for v2.3

i am testing my masks and my new v2.3 might be up by tomorrow

i cannot promise anything :) but i hope tomorrow it will be there

edit: Masked Compression v2.3 is up and running

i think the most solid work i did so far :)

This is a silent Release, none will notice it :P

http://i.imgur.com/8BbhvhM.png

Please help me with this!

And bro, one more question, I found two folders of Masked Compression v2.3 in the archive, so what is the difference or they are same

I forgot to include msr program in the folder :( i am updating the 7z file with msr inside :)

Bro, I tried the new updated Masked Compression v2.3 but Still I am getting this error(I used lztmax) Please help me bro!
http://i.imgur.com/MsoXfs0.png

Thanks in advance

i will help you gladly but .... provide info please ....

you said method lztmax what did you compress, did you turn off antivirus ??

was there any mp3 or jpg file to compress?? if you find a file names thumb.db delete it
it causes errors on some occasions