Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Originally Posted by rishikrap

Update: I'm now using xlzo+uelr+srep_new+lzma2 to recompress, I hope I got it right this time.

No.

uelr+srep_new+lzma2

I think you have to work with file list to avoid errors with uelr (only include tfc, upk, ... files for uelr) if I'm not mistaken. Long time not worked with uelr to be fair. To achive that you have to open up config with _DiskSpan_GUI_CONFIG.bat file and go to the file lists. Make 2 file list files somewhere (1.txt, 2.txt). In 1.txt write all files which have lzo streams inside (mainly tfc, upk, I think). You can use wildcards (for example: *.tfc, each file type in a new line). In 2.txt write just a "*" (without ").




Save, close, start the GUI, select game folder, start.

You can also use a freearc archive with masking data compression, but I have no compatible files on hand atm. Maybe someone else can hep you otherwise.

------------------------------------------------

There are currently only 2 installer scripts comptaible, namely "Lite Installer" and "CIU beta 7". The diskspan GUI will create a "Setup.dll" file which contains all used settings to install the files.

Compression speed depends on some more things then drive speed. First of all CPU speed and input.

UltraArc make a SFX only if you check the box "Self-Extractor". Keep in mind that this way you have to done things a bit different and you have to use/edit a mask method inside Config0.ini and maybe change arc.groups file. PS: In next update UltraArc will be removed because you can do all of this things directly inside diskspan GUI (testing atm). I'm sure my friend Masquerade will tell you more on how to config mask groups and methods properly (otherwise this here will be a whole book soon and I'm not in the mood to write one).

Using xtool or any other precompressor depends on input you have. Unreal engine games which use lzo codec aren't compatible with xtool (or vise-versa) because they use some special codec (if I remember right).

If you want to avoid the split file in UltraArc, open UltraArc.ini and change

FirstVolumeSize=9999999999
CustomVolumeSize=9999999999

or something even bigger, don't matter.

If you want max compression use lolz instead of lzma2.

[rishikrap]

Originally Posted by KaktoR

No.

Yep, learned i the hard way. I slept after putting this and it got error after 2hr (Part of conversion log attached).
Only if I had stayed up for 6 more minutes.
Then I also tried

Originally Posted by KaktoR

 uelr+srep_new+lzma2

Replaced lzma2 with lolz_mtx. I hadn't seen the screenshot so again got the same LZO error.
Now, I have followed all that you have mentioned and started again.
Used

 uelr+srep_new+lolz_mtx for Data1

And

 srep_new+lolz_mtx

for Data2 with 2.txt as included and 1.txt as excluded.
Also, does it matter if I use lolz or lolz_mtx, I just used mtx so it utilized full cpu, don't know if it would make difference in conversion time since I want the best conversion. And there are other lolz conversions n, f etc do they matter? I'm using default lolz_mtx
Thanks for the detailed screenshots and all the other informatin else I'd have done some mistake for sure.
https://drive.google.com/drive/folde...TJA5Qhd34FZAlb
This was my inital output, is there any way, like any way in which I can just extract these without any error ;__;

 bpk+cp_2077(Xtools 2020)+xzlib+srep_new+lolz_mtx

I get the error unsupported config in every installer I tried, even Light as you mentioned and not CIU either.
Are those files now useles since I can't extract them, should I delete them and accept that I wasted 21hrs and RDP money ;___;

Update: I used the config as mentioned above and got the same error again
https://prnt.sc/w8brei
I'm now using xtools and not uel(lzo) since you said it could be vice versa too. Plus fitgirl page also mentiones xtools library Also, I have reset the config to default without and file lists or so.
Currently retying with

 xzlib+srep_new+lolz_mtx

If I get an error I'll replace xzlib with xlzo, since I don't know what'll work, and I had gotten success with xzlib earlier, hnce I'm trying xzlib first.

[Masquerade]

rishikrap
Using xzlib gives you no benefit, so there's no point to use it (UE3 does not use zlib). And then xlzo will not help either, since XTool cannot process these files IIRC (despite a .tfc file is given in example).

I'm downloading rocket league now from my Steam library, will give it a whirl then come back to you with some explanations on what to do.

[Masquerade]

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

[rishikrap]

Originally Posted by Masquerade

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

So, that means I was right after all, regardless of just using hit and trial and seeing fitgirl page to do it lol.

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

So, I have current compression running on CUI V3 b7.u4 with from around more than an hour and it hasn't yielded any errors either with these settings.

 xzlib+srep_new+lolz_mtx

https://prnt.sc/w8eac5
So, do I keep it running or do I cancel it and do with what you specified or, will what I'm doing give me proper results that I can extract too by making setup using the ciu script. I haven't used the $web or masking since there was no option of doing anything as such.

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

It was a little confusing not because of your explaination but because I don't know anything, but on reading multiple times I understood almost all of it
The only thing I'm confusd about is do I have to make/edit the arc.groups file manually or was the example just an example for explaination.
Thanks for all this, this explaination did make me understand the basic stuff that's happening which I didn't have any clue about.

[Masquerade]

The only thing I'm confusd about is do I have to make/edit the arc.groups file manually or was the example just an example for explaination.

I added arc.groups for you myself, but if you want to make future groupings, you must make them. If you don't have any issues, stick with what you have.

[rishikrap]

Originally Posted by Masquerade

I added arc.groups for you myself, but if you want to make future groupings, you must make them. If you don't have any issues, stick with what you have.

Thanks for doing that.
Now, just tell me the last thing, the settings I used have yielded me to 22% without errors, so should I use what you sent me or let CIU run its course and then create the setup using asis?
Screenshot 2020-12-23 113804.png
I'm thinking of making another virtual machine and running what you sent me there, my current machine runs Intel Xeon Platinum 8168 4cores @2.70 Ghz with 8gb ram.
Can I cheap out on cores and use a dual core AMD Epyc 7452 dual core @2.40Ghz for that with 8gb as well. I know the task is cpu intensive hence virtual cpu's are already slowing me down significantly.
I don't expect 2 cores do the task at same time but just an hour or so more would not hurt as much and would be half the cost of 4 cores on Azure and if the time difference would be more than that then 4cores it is.

[rishikrap]

Rocket League Epic Games Version V1.0.10897.0 (Nothing Ripped)
[Steam Version Might work too, untested]

 8:17:03 AM - You have chosen the unlimited type for the output size.
 8:17:03 AM - ISO images will not be created when conversion is complete.
 8:17:03 AM - The computer will remain on when the conversion is complete.
 8:17:03 AM - Selected ARC method for Data1.bin was: xzlib+srep_new+lolz_mtx
--------------------------------------------------------------------------------------------------------------------------------------
 8:17:05 AM - Compression initialized in 23/12/2020 at 8:17:05 AM.
 8:17:05 AM - Starting Data1.bin archive creation.
 5:22:58 AM - Moving Data1.bin.001 file to media folder.
 5:23:04 AM - Compression finished in 24/12/2020 at 5:23:04 AM.
--------------------------------------------------------------------------------------------------------------------------------------
 5:23:05 AM - Getting size of the game: Rocket League
 5:23:05 AM - >> Installation folder size: 12.92 GB
 5:23:05 AM - >> Compressed game size: 6.30 GB
--------------------------------------------------------------------------------------------------------------------------------------
 5:23:05 AM - Overall input size: 12.92 GB
 5:23:05 AM - Overall output size: 6.30 GB
 5:23:05 AM - Overall conversion time: 21:05:58

Credits to

Originally Posted by KaktoR

No.

Originally Posted by Masquerade

I'd just let it run the course now.

For making me understand the basics and recommend me the best config for the best compression 48.7%

I still don't know how to make another installer work with my config, I'm just using Diskpan GUI from CIU V3 B7.U4 and the default Shadow of Tomb Raider Installer as included in the package.
I tried Light Installer, ASIS Installer and CorePack installer but they gave the error of not recognizing the archive format, and ASIS not even an error, just installed the default Command & Conquer™ Remastered Collection and gave me a desktop shortcut.

Any mods @JustFun or whoever, if you find this worth it, please do add this the Index for future reference for anyone else.