new to this
 

hello, im trying to get into repacking games for the first time ever, for my own private use. i have a very slow wifi where i live and its always a pain to download anything larger than a few gigs. so what i wanna do is compress the games im currently not playing for later use. ive been directed to this forum as a great resource to help me get started, and ive been looking through the sticky for a few days now and id be lying if i said it wasnt very overwhelming, especially since i know absolutely nothing about compression and repacking. i am willing to put the time and effort to learn but im really not sure where to even start? id very grateful if i could get some guidance as to where i need to start this process. currently all i have done besides reading the stickies is downloaded this <http://freearc.org/download/0.666/FreeArc-0.666-win32.exe> and this <https://jrsoftware.org/isdl.php#stable>. i have tried compressing a game, sniper elite 4 to be exact, since i dont have any smaller games to test on but after seeing how much time it was gonna take i aborted it for the sole reason of not being able to use my pc for that duration and not knowing whether or not it was gonna work

Depends on how much you want to size down your games. Generally spoken: The stronger the compression, the more time will it take and also CPU/RAM resources, to be simple.

To get the "quick and easy way" you can use DiskSpan_GUI to compress your files.
You can also unpack your compressed archives with it too (Tools -> Unpacker).

In your example for "Sniper Elite 4" I guess you can just use srep+lolz if you want it strong, with cost of speed and pc resources, or srep+lzma with cost of compression ratio, but requires way less pc resources. But I didn't checked the game so I cannot tell you for sure.

Some games requires that you first unpack the files and then compress them. In most cases you use xtool to unpack the game files, but you should know what compression method the game is compressed with. For example "Sniper Elite 5" is compressed with zlib (if I remember well), so you have to use xtool first to unpack the files, then apply srep to get rid of duplicates, then use lolz or lzma to use final compression.

Maybe you have noticed the problem for "newbies" here. That's nothing to explain in a few sentences.

thanks kaktor for the reply, helped understand a few things i was held up on, i got "srep+7zip multi volume" and "lzma" from the stickies as well as diskspan GUI, but i followed the link to lolz to krinkles.org and it seems i need to post stuff there before i can get download righs which sucks because i dont know anything so what would i post about. also diskpan gui is being flagged as malware, is that a false flag?

If DiskSpan GUI was flagged as a virus it is a false positive.
You can download without worries.
If you need to, disable your antivirus and after extracting the zip file from DiskSpan GUI, enable your antivirus again and check the extracted folder from DiskSpan GUI. Let me know exactly which file was captured by your antivirus so that if there is an alternative version maybe I can replace the captured file.

NOTES: DiskSpan GUI already includes all compressors reported by KaktoR, such as LOLZ.

EDIT: DiskSpan GUI is only False positive for viruses.
No virus detection using Kasperky Internet Security (See attached video below).

Hey Cesar82 thanks for the reply, i did what you said; Affected files:

containerfile: C:\Users\Neuromancer\Downloads\DiskSpan_GUI v2.0.1.3 (x64).7z

file: C:\Users\Neuromancer\Downloads\DiskSpan_GUI v2.0.1.3 (x64).7z->COMPRESSOR/Resources/TOOLS/WemTool/WemTool.exe

webfile: C:\Users\Neuromancer\Downloads\DiskSpan_GUI v2.0.1.3 (x64).7z|https://fileforums.com/attachment.ph...30245021635373

EDIT: after using the program for a bit i just wanna say great work i love the UI and the program is vey easy to use

Thanks for the test.
If possible download the WemTool file attached in the post HERE and let me know if your anti-virus also captures this updated version without UPX compression.

Sorry about the WemTool. It's probably going to trigger AV anyway because it's lazily built with bat2exe.

yes this one also gets flagged

Try compress using ASPack or MPress. Maybe it works.

ASPack will be even worse for detections.

You mean ASProtect ?!

Thanks for info!
Is there no other way or program you can use to convert your BAT that doesn't create false positives?

Probably could re-write the entire thing in C# but I'm currently so busy trying to balance a lot of IRL stuff I will not get time. It's something I will consider but it won;t be soon.

Hello! Thank you for these explanations!

I'm also really new to this and I have read a lot but it is a lot of arbitrary seeming detailed information in these forums that make it incredibly hard to categorize and make sense of, so I have a lot of questions and I hope it's ok to just dump them all at once :o

I downloaded DiskSpan GUI, it seems like it's one of the more customizeable, yet very easy to use ones. Do you think this is a good tool to use or would I be better off using these exe's separately, or even a completely different software?

You said "srep+lolz" and "srep+lzma", does this mean, in DiskSpan GUI (if you are familiar with the software), I can select both srep first and then lolz / lzma and then it is basically what you had described?

To understand the logic behind this, when I select a folder to compress with DiskSpanGUI, "srep+lolz" would basically mean using srep first to deduplicate, and then compress the srep output with lolz?

I don't have Sniper Elite 4, but this is interesting -- how do you know srep+lolz is strong? Is it one of the strongest general combinations? Or how do you know, or in other words, how can I find out what compressors and thingies work the best?

Super interesting!
So, to make sure I understand what you said at the end, in DiskSpanGUI I would basically have xTool_zlib+srep+lolz (or lzma), is that correct?

Also how do you know it's compressed with zlib? Are you an experienced file magician and / or do you know of a scanner software that can test if a file is compressed, and what it is compressed with?

One final question, which is hopefully not too broad: What is your workflow finding 'the best' compression combinations (that is, not as slow as paq compression for instance but still really good compression ratio)? Is it Trial and Error? Like, do you just select a couple of files to compress, and try each and every combination? Because that's what I'm doing currently and this kinda sounds super inefficient so I suspect there's a way to find out what works with scanners or anything :D

Anyways, I'm eager to learning more about compression but I don't think I'm getting far without asking some dedicated beginner questions. I feel like I know some stuff but they are absolutely random bits of information in different fields and it's incredibly hard and really confusing trying to make sense of them :D

Thanks!
- Crimzan

I would say this tool is handy, but have it's limitations aswell. Most repackers use their own tool set anyway, but in the end all of them are using the same compressors, with some additions and techniques.


Yes.


lolz is mainly good for texture compression. You know, games use textures. Save your time and don't try to use lolz for audio/video files and such.

For instance, took some DDS image files, and compare ratios between srep+lolz and srep+lzma.

Spoiler: lolz will be superior on these files in terms of ratio. It's just a matter on how long you want to run the process and how small the output you want.


Yes.


It's like common sense sometimes. If you are in the process for years, you will get a "sense" of it. For example take any Unreal Engine 4 game. Most of these games are compressed with zlib/reflate, many others with oodle, and a few even with lz4 or zstd.

To check a file, you can use xtool's verbose mode, or manually check the file with HxD or similar for magic numbers and such (this requires that you know about these things).

I'm not a magican or something, I still have questions sometimes too :)


On larger archives (like in UE4 games the *.pak files) use xtool's verbose mode to find out. After you find out, pick the file and compress it with different combinations until you are happy with the outcome (ratio/time). There is no "magic tool" to give you all answers about a file in general.

Personally I don't use lolz because it is time consuming af and it tooks literally all PC resources while running, and I want it to be as fast as possible with a decent outcome in ratio, therefore for me my choice is in most cases 4x4:lzma.

Razor made drop + scan 4 zlib which is sometimes useful.

There's also this Zenhax page where Aluigi explains how to recognise different algorithms by their headers or magic numbers: https://zenhax.com/viewtopic.php?t=27

This requires a hex editor to view the data. I like HxD.

I am happy you guys putting in the time to answer my questions, thanks a lot!

I don't know much about Hexadicmal, I do have HxD and tried a 7z (LZMA2) and a standard windows zip file (I suspect LZMA because it says so in the link?); however, none of the Headers shown in the link are actually seen in the Hexadecimal view of these files for some reason. Why is that? Could it be that the header is just offset or are some diferences in the Bytes to be expected? I expected to find pretty much the exact same Header in my file as in the link. Do you know why it differs on my end? :o


Would you mind explaining in a bit more detail how to use the XTool Verbose mode? I created a standard zip and also a 7zip file (LZMA2) and created a batch file based on what you showed in the link you provided (also changed the algorithm from mzlib to lzma2 (hoping that it works that way), but no matter what file I yeet onto xtool, all I get is this:

I didn't expect it to not find any lzma2 streams in an lzma2 file, or other files I tried out with the adjusted algorithm parameter. I expected, similar to the "Drop and Scan 4 ZLib" Application from Razor to have different values here, that recognize how many parts of a file actually correspond to the compression algorithm. Am I misunderstanding the function and expecting a result that this tool is not supposed to give? If it isn't too much to ask, could you maybe show me a working result on a file?

So, let's keep things simple for now and let's say I found a file is actually compressed with LZMA2 / 7zip -- I would use xtool to decompress it (precomp?), receive a larger file, and then use a more efficient combination to get a better compression ratio, right?

What does the 4x4: stand for? are these parameters in xtool or where does it come from? :confused:

Once again, I really appreciate you answering all these noob-ish questions, I really hope they aren't too bothersome! :D
Have a nice day guys!

- Crim

You cannot use -mlzma2 directly. As far as I know, zip/7z files contain always reflate streams (correct me if I'm wrong).

So for zip/7z files you have to use -mreflate if they are compressed.

Example for what it should look like:
Here are some possible options:


Yes.



4x4 is part of freearc. It utilizes all cores (max 16) for compression. This gives a good speed boost, but cpu usage is 100% most of the time.

7z archives most commonly use LZMA2 since it's the default option when you select 7z container in 7Zip creation window.

Zip files most commonly use deflate for their compression, although you can use any algo with a zip container (zip header is PK). For example, Crysis 2 Remastered uses LZ4 on its zip game archives. We Are Football uses ZSTD on its zip archive.

Afair there's no current easy precompressor (e.g. xtool) for lzma[2]. I know pLZMA exists on encode forum but it's not a precompressor like XTool.