new to this

[Crimzan]

Hello! Thank you for these explanations!

I'm also really new to this and I have read a lot but it is a lot of arbitrary seeming detailed information in these forums that make it incredibly hard to categorize and make sense of, so I have a lot of questions and I hope it's ok to just dump them all at once

I downloaded DiskSpan GUI, it seems like it's one of the more customizeable, yet very easy to use ones. Do you think this is a good tool to use or would I be better off using these exe's separately, or even a completely different software?

Originally Posted by KaktoR

In your example for "Sniper Elite 4" I guess you can just use srep+lolz if you want it strong, with cost of speed and pc resources, or srep+lzma with cost of compression ratio, but requires way less pc resources.

You said "srep+lolz" and "srep+lzma", does this mean, in DiskSpan GUI (if you are familiar with the software), I can select both srep first and then lolz / lzma and then it is basically what you had described?

To understand the logic behind this, when I select a folder to compress with DiskSpanGUI, "srep+lolz" would basically mean using srep first to deduplicate, and then compress the srep output with lolz?

I don't have Sniper Elite 4, but this is interesting -- how do you know srep+lolz is strong? Is it one of the strongest general combinations? Or how do you know, or in other words, how can I find out what compressors and thingies work the best?

Originally Posted by KaktoR

Some games requires that you first unpack the files and then compress them. In most cases you use xtool to unpack the game files, but you should know what compression method the game is compressed with. For example "Sniper Elite 5" is compressed with zlib (if I remember well), so you have to use xtool first to unpack the files, then apply srep to get rid of duplicates, then use lolz or lzma to use final compression.

Super interesting!
So, to make sure I understand what you said at the end, in DiskSpanGUI I would basically have xTool_zlib+srep+lolz (or lzma), is that correct?

Also how do you know it's compressed with zlib? Are you an experienced file magician and / or do you know of a scanner software that can test if a file is compressed, and what it is compressed with?

One final question, which is hopefully not too broad: What is your workflow finding 'the best' compression combinations (that is, not as slow as paq compression for instance but still really good compression ratio)? Is it Trial and Error? Like, do you just select a couple of files to compress, and try each and every combination? Because that's what I'm doing currently and this kinda sounds super inefficient so I suspect there's a way to find out what works with scanners or anything

Anyways, I'm eager to learning more about compression but I don't think I'm getting far without asking some dedicated beginner questions. I feel like I know some stuff but they are absolutely random bits of information in different fields and it's incredibly hard and really confusing trying to make sense of them

Thanks!
- Crimzan

[KaktoR]

Originally Posted by Crimzan

I downloaded DiskSpan GUI, it seems like it's one of the more customizeable, yet very easy to use ones. Do you think this is a good tool to use or would I be better off using these exe's separately, or even a completely different software?

I would say this tool is handy, but have it's limitations aswell. Most repackers use their own tool set anyway, but in the end all of them are using the same compressors, with some additions and techniques.

Originally Posted by Crimzan

You said "srep+lolz" and "srep+lzma", does this mean, in DiskSpan GUI (if you are familiar with the software), I can select both srep first and then lolz / lzma and then it is basically what you had described?

To understand the logic behind this, when I select a folder to compress with DiskSpanGUI, "srep+lolz" would basically mean using srep first to deduplicate, and then compress the srep output with lolz?

Yes.

Originally Posted by Crimzan

I don't have Sniper Elite 4, but this is interesting -- how do you know srep+lolz is strong? Is it one of the strongest general combinations? Or how do you know, or in other words, how can I find out what compressors and thingies work the best?

lolz is mainly good for texture compression. You know, games use textures. Save your time and don't try to use lolz for audio/video files and such.

For instance, took some DDS image files, and compare ratios between srep+lolz and srep+lzma.

Spoiler: lolz will be superior on these files in terms of ratio. It's just a matter on how long you want to run the process and how small the output you want.

Originally Posted by Crimzan

Super interesting!
So, to make sure I understand what you said at the end, in DiskSpanGUI I would basically have xTool_zlib+srep+lolz (or lzma), is that correct?

Yes.

Originally Posted by Crimzan

Also how do you know it's compressed with zlib? Are you an experienced file magician and / or do you know of a scanner software that can test if a file is compressed, and what it is compressed with?

It's like common sense sometimes. If you are in the process for years, you will get a "sense" of it. For example take any Unreal Engine 4 game. Most of these games are compressed with zlib/reflate, many others with oodle, and a few even with lz4 or zstd.

To check a file, you can use xtool's verbose mode, or manually check the file with HxD or similar for magic numbers and such (this requires that you know about these things).

I'm not a magican or something, I still have questions sometimes too

Originally Posted by Crimzan

One final question, which is hopefully not too broad: What is your workflow finding 'the best' compression combinations (that is, not as slow as paq compression for instance but still really good compression ratio)? Is it Trial and Error? Like, do you just select a couple of files to compress, and try each and every combination? Because that's what I'm doing currently and this kinda sounds super inefficient so I suspect there's a way to find out what works with scanners or anything

On larger archives (like in UE4 games the *.pak files) use xtool's verbose mode to find out. After you find out, pick the file and compress it with different combinations until you are happy with the outcome (ratio/time). There is no "magic tool" to give you all answers about a file in general.

Personally I don't use lolz because it is time consuming af and it tooks literally all PC resources while running, and I want it to be as fast as possible with a decent outcome in ratio, therefore for me my choice is in most cases 4x4:lzma.

[Masquerade]

Originally Posted by Crimzan

Also how do you know it's compressed with zlib? Are you an experienced file magician and / or do you know of a scanner software that can test if a file is compressed, and what it is compressed with?

Razor made drop + scan 4 zlib which is sometimes useful.

There's also this Zenhax page where Aluigi explains how to recognise different algorithms by their headers or magic numbers: https://zenhax.com/viewtopic.php?t=27

This requires a hex editor to view the data. I like HxD.