Best Compression Methods for 'Specific' Games. Q&A

[Masquerade]

rishikrap
Using xzlib gives you no benefit, so there's no point to use it (UE3 does not use zlib). And then xlzo will not help either, since XTool cannot process these files IIRC (despite a .tfc file is given in example).

I'm downloading rocket league now from my Steam library, will give it a whirl then come back to you with some explanations on what to do.

[Masquerade]

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

[rishikrap]

Originally Posted by Masquerade

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

So, that means I was right after all, regardless of just using hit and trial and seeing fitgirl page to do it lol.

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

So, I have current compression running on CUI V3 b7.u4 with from around more than an hour and it hasn't yielded any errors either with these settings.

 xzlib+srep_new+lolz_mtx

https://prnt.sc/w8eac5
So, do I keep it running or do I cancel it and do with what you specified or, will what I'm doing give me proper results that I can extract too by making setup using the ciu script. I haven't used the $web or masking since there was no option of doing anything as such.

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

It was a little confusing not because of your explaination but because I don't know anything, but on reading multiple times I understood almost all of it
The only thing I'm confusd about is do I have to make/edit the arc.groups file manually or was the example just an example for explaination.
Thanks for all this, this explaination did make me understand the basic stuff that's happening which I didn't have any clue about.

[Masquerade]

The only thing I'm confusd about is do I have to make/edit the arc.groups file manually or was the example just an example for explaination.

I added arc.groups for you myself, but if you want to make future groupings, you must make them. If you don't have any issues, stick with what you have.

[rishikrap]

Originally Posted by Masquerade

I added arc.groups for you myself, but if you want to make future groupings, you must make them. If you don't have any issues, stick with what you have.

Thanks for doing that.
Now, just tell me the last thing, the settings I used have yielded me to 22% without errors, so should I use what you sent me or let CIU run its course and then create the setup using asis?
Screenshot 2020-12-23 113804.png
I'm thinking of making another virtual machine and running what you sent me there, my current machine runs Intel Xeon Platinum 8168 4cores @2.70 Ghz with 8gb ram.
Can I cheap out on cores and use a dual core AMD Epyc 7452 dual core @2.40Ghz for that with 8gb as well. I know the task is cpu intensive hence virtual cpu's are already slowing me down significantly.
I don't expect 2 cores do the task at same time but just an hour or so more would not hurt as much and would be half the cost of 4 cores on Azure and if the time difference would be more than that then 4cores it is.

[Masquerade]

I'd just let it run the course now.

[Snapppr6]

original size 7.35gb v1.7 and all previous patches
ripped some mp file (zone mp .ff )
method:
1) main .iwd files srep:m3f:a2+7z
2) zone .ff files precompx+srep+lzma
3) videos .bik files binkpack.exe
4) rest of files and .exe srep+nanozip

comprassed size 3.93gb