oggre optmization and dilemmas

[wareck]

I start this thread because I would like to solve a dilema about using oggre.

Actually I often use oggre_enc & oggre_dec (pre and post processing outside compression)
but I'm faced with a performance dilemma.

several small ogg files :
encode each file individually with oggre_enc
decode with oggre_dec
compare and make a diff file with hdiffz
in this case, fast installation (each file will be decoded, then patched)
but the final files are bigger
(several files, several hdiffz files, ogg_encoded files bigger)

one big file to process:
I merge all small ogg files in one archive or one chunk.
encode this big file with oggre_enc
decode with oggre_dec
compare and make a diff file with hdiffz
in this case, low speed installation (one big file to decode can take time)
but the final file is lower (one bigfile compressed, one hdiff file)

in fact after some analysis, with one big file, oggre_enc have better compression results due to ogg and frame deduplication.
With a big file we take the benefit of this step.
With several small file we gain in speed at the expense of this deduplication.

I try several things
-parallel processing (not working, because oggre do not like this)
-cut the big file in several part (working, but lost gain, more process = less deduplication)


example with numbers , game with 2.5Go of ogg files:
-single big ogg file processed with oggre => 1.65go, 7 minutes to uncompress and replace file in game
-big file cut in 4 chunk, then parallel uncompress => 1.92Go, 2 minutes to uncompress & replace files

if anyone has a better method, I'm interested.

[Masquerade]

The single big file is the only way to do it in order to get the benefit of deduplication.

Also mparallel does work with oggre, this is how wemtool works.