Information about Precomp

[Razor12911]

I was compressing Need For Speed The Run the other day.
Input was 15GB and final size was 4.34GB, precomp did all this.
Precomp made the input 29GB which is good.

Precomp v0.4.3 - ALPHA version - USE FOR TESTING ONLY
Free for non-commercial use - Copyright 2006-2012 by Christian Schneider

Input file: $$arcdatafile$$.tmp
Output file: $$arcpackedfile$$.tmp

Using packjpg25.dll for JPG recompression.
--> packJPG library v2.5a (12/12/2011) by Matthias Stirner / Se <--
More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg

100.00% - New size: 30842644270 instead of 16086403253

Done.
Time: 11 hour(s), 16 minute(s), 3 second(s)

Recompressed streams: 1122066/1369521
PNG streams (multi): 0/3
JPG streams: 0/8
JPG streams (progressive): 0/78
zLib streams (intense mode): 1122066/1369432

You can speed up Precomp for THIS FILE with these parameters:
-zl98 -d0

Fast mode does exactly the same for this file, only faster.

Errorlevel=0

There were more than 1 million zlib streams to precompress which is good for srep to find matches and similarities in data which brought 29GB down to 8GB.

Compressing 30,842,644,270 bytes with srep  -m5f $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
SREP 3.91 beta (June 27, 2013): input size 29413 mb, memory used 2503 mb, -m5f -l512 -c256 -a4/4 -hash=vmac -b8mb
100%: 30,842,644,270 -> 9,054,126,014: 29.36%.  Cpu 21 mb/s (1383.016  sec), real 2 mb/s (12086.196 sec)

from that, a strong compressor LZMA was applied, NanoZIP or PAQ can be applied to bring the size better than LZMA but doesn't show progress (Working on that)

Compressed 172 files, 16,086,403,253 => 4,670,475,225 bytes. Ratio 29.0%
Compression time: cpu 8678.03 secs, real 64392.04 secs. Speed 250 kB/s

Compression took 18 hours thanks to precomp which took 11 hours meaning you are disadvantaged with time but not with space.