General Compression Discussion Thread

[romulus_ut3]

I used the search function and I was unable to find a thread for the sole purpose of troubleshooting/discussing just compression issues, hence I decided to start this thread.

The purpose of this thread is simple, troubleshooting issues related to compression and each and everyone is more than welcome to post their issues here.

I will go first.

So I have Left 4 Dead 2, which takes up 12 GB of space when installed, so I decided to compress it to a more convenient size for backup purposes.

I used the following method to compress the whole directory:

msc+pzlib64+srep64+xz

My arc.ini is as follows:

[External compressor:msc]
header = 0
packcmd = MSC c -v -wav=1 -raw=1 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=1 -bmf=9s -tak=9 -dxt=1 -lzma=hc4,lc8,lp2,pb2,fb273,mc1000 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

[External compressor:pzlib64]
header = 0
packcmd   = pzlib64 e -m2 -x -r9 -t100p $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp
unpackcmd = pzlib64 d -t100p - -o - <stdin> <stdout>

[External compressor:srep64]
header = 0
packcmd   = srep64 -m5f -d1g -a2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep64 -d -s -- <stdin> <stdout>

[External compressor:xz]
header = 0
packcmd   = xz a -txz -an -mcrc=0 -m1=lzma2:d1280m:fb=273:mf=bt4:mc=1000000:lc=4:lp=0 -mmt=3 -mx9 -si -so <stdin> <stdout>
unpackcmd = dec x -txz -an -y -si -so <stdin> <stdout>

When I compress the whole directory to a single archive, the output archive weighs 4.73 GB. 4.73 is just above the capacity of DVD5, a medium which I am planning to backup my game to. So if I move some files elsewhere and compress them separately using the exact same method, the total output becomes 5.27(!) GB.

What am I doing wrong that's causing the increase in output size by 600 MB opposed to a single archive?

[Gupta]

if i recall correctly, Left 4 dead 2 contains a lot of wav files so you can use oggenc2
this should get your Game into 3 Gbs
frankly speaking loss in quality can't be sensed at all
You can Find it here
http://fileforums.com/showthread.php?t=98819
in lossy Section

[romulus_ut3]

As I will be using the backup to restore the game in the future to be used on Steam, lossy compression is not an option. The game compresses just fine if I compress the whole folder, but if I move a folder worth 2GB and compress it in two separate archives, I see a 600 MB increase in size. MSC does a wonderful job with *.wav files, but for some reason I am unable to maintain the ratio.

Added an image to the OP for a better description of the issue.

[Bulat]

srep finds repetitions across entire 12 gb (even more after pzlib preprocessing), so shortly speaking these 600 mb are the same in two parts you are try to compress independently

you can try to use refalte to improve compression ratio

[romulus_ut3]

Originally Posted by Bulat

srep finds repetitions across entire 12 gb (even more after pzlib preprocessing), so shortly speaking these 600 mb are the same in two parts you are try to compress independently

you can try to use refalte to improve compression ratio

This actually makes sense, thank you.

[Razor12911]

I don't think this game uses deflate anywhere, I actually have it. So I don't think pzlib will give any diff.