From my "modest" compression experiments, I have concluded that with a given dictionary size "ex: d128m" and running on 1(!) thread, the occurrence of a given event, in this case RAM exhaustion, may also depend on how compressible and large the given dataset is.
For example, with TS12:
*.JA files, we are talking about nearly 9GB, with 16GB RAM, it can be compressed even with d1024. (~uses 12GB or more)
MSTS:
In the case of *.ACE files, if you get a total size of over 25GB even after the "xZLib+srep" phase, because we have so much starting data, then... (a lot of extra tracks, textures, compared to the 2CD release)
d128-d160, the safe limit, up to 25GB. Around 28-30GB, it's already a lot, here with d192, already at about the 80% stage, you'll run out of memory. (What's nice? After a 32-40 hour process.) If we have less data, the dictionary can be larger. This is probably due to too many matching results. After SREP, LDMF can find up to 1 million matches or even more in this example.
Example (test after xZLib+srep, With 8GB RAM from older times):
Code:
LOLZ v22c4b / LDMF1 / lolz:dt:dtb1:d160m:fba4096:mc1023:tt16:mtt0:mt1:ldmf1:ldc0:ldl5
100.00% ldmf dec_mem=0m 37:45:22
thread 0 : w/o manager mem_usage = 239941 kb, manager mem_usage = 49726 kb
alloc mem = 73728 kb, stat compr_size = 2934533
matches num = 1322053, additional mem overhead = 28329 kb
o1 model : 1'182'372 kb
raw graphic model 8 bit : 7'706'602 kb
raw graphic model 16 bit : 52'900 kb
raw graphic model 24 bit : 16'728 kb
raw graphic model 32 bit : 68'208 kb
dxt1 model : 1'275'819 kb
dxt3 model : 292 kb
dxt5 model : 302 kb
o1 model pos mod 2 : 99'564 kb
o1 model pos mod 4 : 61'374 kb
o1 model pos mod 8 : 562'345 kb
o1 model pos mod 16 : 76'373 kb
total size : 11'102'884 kb
total decode mem usage = 183mb
100.0%
Errorlevel=0
Compressed 1 file, 11,369,353,897 => 3,074,609,947 bytes. Ratio 27.04%
Compression time: cpu 38.00 sec/real 136505.89 sec = 0%. Speed 0.08 mB/s
All OK
Tech infos: ~9.5GB *.ACE file -> xZlib out: ~28GB -> srep out: ~10.6GB.
In the above case, instead of breaking up the files separately, a much better solution is to use multiple solid blocks, as in FreeArc. For such 100+ GB monsters, let's use solid blocks of say 15-25GB during packaging, if we force the use of the given dictionary size/compression configuration, we do not make concessions. In the above MSTS, 25GB example, the solid block was fragmented precisely because of the 100k file limit, so I was able to avoid the possibility of running out of memory.
Has anyone seen "bc5" DDS texture info after packaging? And what could these be: "float0, float1....4" ??