Information about Precomp

[Razor12911]

Hi guys. I am here to explain and give extra information about precomp.

First of all, a whole lot of newbies think that precomp compresses files, "If you use precomp you get better results", well I am here to clarify that.

Pecomp is not a compressor, it's more like a decompressor with the aim of decompressesing zlib and deflate streams that exist in a certain file meaning the output is suppose to be bigger than input.

Q: Why must the output be bigger than input?
A: So that there must be a lot of repetitions and similarities within the file that when you apply step or stronger compression such as LZMA which targets those repetition and similarities to make the output much smaller.

Q: Why does precomp take so long to process a file?
A: Data is represented as binary zeroes and ones, there are so many of them within a file, precomp has to scan for headers/traces of zlib or deflate streams then decompress them then looks for another.

Q: Why does precomp stop working sometimes.
A: it may be cause by a false detected stream within the file and when precomp tries to decompress it, an exception is raised because the stream was falsely detected via file headers.

Q: What if there were no streams detected?
A: Then do not use precomp for that particular file.

Q: How to know the count of streams found?
A: After precomp has processed a file, it shows a summary.

Q: How to know before precomp finishes that the results will be excellent?
A1: use verbose command -v. Turn it on.
A2: look at the temp file, original file and current progress, if temp is bigger than original file while precomp is still working then expect great results but if temp is more or less equal in comparison to original file and progress percentage agrees with the division of both then it is wise to stop process and remove precomp from method.

[Sarvagya]

Thanks for the explanation but i have one question is precomp good for gta v or reflate

[Razor12911]

I don't know, I don't have that game but many claim that reflate is better.

[Razor12911]

I was compressing Need For Speed The Run the other day.
Input was 15GB and final size was 4.34GB, precomp did all this.
Precomp made the input 29GB which is good.

Precomp v0.4.3 - ALPHA version - USE FOR TESTING ONLY
Free for non-commercial use - Copyright 2006-2012 by Christian Schneider

Input file: $$arcdatafile$$.tmp
Output file: $$arcpackedfile$$.tmp

Using packjpg25.dll for JPG recompression.
--> packJPG library v2.5a (12/12/2011) by Matthias Stirner / Se <--
More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg

100.00% - New size: 30842644270 instead of 16086403253

Done.
Time: 11 hour(s), 16 minute(s), 3 second(s)

Recompressed streams: 1122066/1369521
PNG streams (multi): 0/3
JPG streams: 0/8
JPG streams (progressive): 0/78
zLib streams (intense mode): 1122066/1369432

You can speed up Precomp for THIS FILE with these parameters:
-zl98 -d0

Fast mode does exactly the same for this file, only faster.

Errorlevel=0

There were more than 1 million zlib streams to precompress which is good for srep to find matches and similarities in data which brought 29GB down to 8GB.

Compressing 30,842,644,270 bytes with srep  -m5f $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
SREP 3.91 beta (June 27, 2013): input size 29413 mb, memory used 2503 mb, -m5f -l512 -c256 -a4/4 -hash=vmac -b8mb
100%: 30,842,644,270 -> 9,054,126,014: 29.36%.  Cpu 21 mb/s (1383.016  sec), real 2 mb/s (12086.196 sec)

from that, a strong compressor LZMA was applied, NanoZIP or PAQ can be applied to bring the size better than LZMA but doesn't show progress (Working on that)

Compressed 172 files, 16,086,403,253 => 4,670,475,225 bytes. Ratio 29.0%
Compression time: cpu 8678.03 secs, real 64392.04 secs. Speed 250 kB/s

Compression took 18 hours thanks to precomp which took 11 hours meaning you are disadvantaged with time but not with space.

[mikey26]

@Sarvagya guys do post sometimes on how long it sometimes takes to do conversion and the post the required files u need so i dont see a problem but do u know how many types of combinations of machine specs are out there.its near impossible to calculate how long its going to take to do a conversion everything can only be speculated.then u get guys that want the best compression possible and they running a pentium 2 with 256mb ram and wonder why conversion crash and take so long.look for thread for GTA V the game has been out for just over 2 months now and guys are still trying to figure out what is the best compression .when they find the best compression the complain it takes to long to install.then 2 weeks later a 5gb patch comes out and they want a update to the conversion the very next day its a no win situation for converters and coders.they do the best they can and guys should be grateful with what they get.or the should do the conversion themselves and read up on compression and how it works.i dont mean to be rude but thats how it is mate.

[mikey26]

@Razor hi bud hope u well.i have been reading a bit on the forums it looks like that the guy that was developing nanozip (Sami Runsas) might have actually abandoned the project.there are bugs in nanozip but is a great archiver i gonna play around with it and see how compression works.i sure you will work out the progress level issues with it.but then again do we actually need to see progress i mean we will all know when its done working and a conversion is done :-)

[Razor12911]

I am working on it, I can only do things in ASC. I managed with reflate, I can try working on nanozip too, at least in ASC to implement this like this is easy unlike in Inno Setup.