Best Compression Methods for 'Specific' Games. Q&A

[rishikrap]

Originally Posted by Masquerade

uhhhh ok

1. There's no need for bpk since there's no bink videos in this game

2. Why are you using the cyberpunk 2077 precompressor on not cyberpunk 2077??????

3. xzlib - we're getting somewhere, but this game is made in unreal engine 3 and is compressed with lzo and not zlib (zlib is ue4) - which is why you need to use UELR [Unreal Engine LZO Recompressor]

4. srep, great!

5. lolz_mtx - this has been proven to be slower IIRC and not worth whatever result you get. If you want a fast compression, use lzma2.

I don't know.
I'm totally new to this like literally 100%.
Found a config, used CUI V3 GUI for diskspan to do this and it took 21 hrs to do this.
Now I don't know how to extract this things again so finding the forum.

Originally Posted by KaktoR

Your chain does not make any sense, just saying.

I know, just googled and found the best options.

[Masquerade]

Sorry if I sounded harsh, not the intention, please feel free to ask questions

[rishikrap]

Originally Posted by Masquerade

Sorry if I sounded harsh, not the intention, please feel free to ask questions

1. Could you help me out with what can I do to create an installer for this thing.
2. It'll install correct anyways regardless of all the unwanted methods right?
Update: I managed to find a Corepack UI installer which just needed me to change the name of the file in compier.ini so I did that and it proceeded, but it gives me a the error https://prnt.sc/w85cc2
Update 2: Managed to also make installer with the default lego template provided in CIU V3 and it did start and extracted some files and hung at 0.9% without utilizing any resources. I guess my compression technique sucked more than I expected.
So, I won't be able to install it in any way or is there some way I can save time and use what I compressed initally?
2 (A). If not, could you give me a single config that I can copy paste in Diskspan GUI by CIU V3.
3. Does SSD help with speeds at a good margin. I'm using HDD (not a good one either).
4. Also if this is too much hassle, can you give me a way to make an SFX using Ultraarc.
And by the way I used Cyberpunk 2077 compresser because it said in bracket Xtools 2020 and I thought this'd be the latest Xtools. And fitgirl says it uses Xtools library by Razor 1911 on the Rocket League Page, so I figured I should too.
I made one and it was easy install too, no issues at all, I didn't know what Xtools was that time, so I used default Ztools as precompressor, Lolz, Lzma2, Srep but it only compressed to 10.1gb, I don't know if X tools would improve the compression.
Only problem I have with Ultraarc sfx is it splits files too much even in a,b,c. Any way to use diskspan to make an sfx? And also achieve this conversion.
Hope you could solve all the doubts.

Update: I'm now using xlzo+uelr+srep_new+lzma2 to recompress, I hope I got it right this time.
I'll just need some proper tutorial on making an installer, that's the only thing that has eaten my brain. I don't care about the graphics or anything, I just want it to work ;__;.
My aim with this is simply to compress the Epic Games version maximum possible such that I have to download less. I'm doing this on an RDP (Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz 4 cores 8gb ram) which doesn't have bandwidth limitations.

[KaktoR]

Originally Posted by rishikrap

Update: I'm now using xlzo+uelr+srep_new+lzma2 to recompress, I hope I got it right this time.

No.

uelr+srep_new+lzma2

I think you have to work with file list to avoid errors with uelr (only include tfc, upk, ... files for uelr) if I'm not mistaken. Long time not worked with uelr to be fair. To achive that you have to open up config with _DiskSpan_GUI_CONFIG.bat file and go to the file lists. Make 2 file list files somewhere (1.txt, 2.txt). In 1.txt write all files which have lzo streams inside (mainly tfc, upk, I think). You can use wildcards (for example: *.tfc, each file type in a new line). In 2.txt write just a "*" (without ").




Save, close, start the GUI, select game folder, start.

You can also use a freearc archive with masking data compression, but I have no compatible files on hand atm. Maybe someone else can hep you otherwise.

------------------------------------------------

There are currently only 2 installer scripts comptaible, namely "Lite Installer" and "CIU beta 7". The diskspan GUI will create a "Setup.dll" file which contains all used settings to install the files.

Compression speed depends on some more things then drive speed. First of all CPU speed and input.

UltraArc make a SFX only if you check the box "Self-Extractor". Keep in mind that this way you have to done things a bit different and you have to use/edit a mask method inside Config0.ini and maybe change arc.groups file. PS: In next update UltraArc will be removed because you can do all of this things directly inside diskspan GUI (testing atm). I'm sure my friend Masquerade will tell you more on how to config mask groups and methods properly (otherwise this here will be a whole book soon and I'm not in the mood to write one).

Using xtool or any other precompressor depends on input you have. Unreal engine games which use lzo codec aren't compatible with xtool (or vise-versa) because they use some special codec (if I remember right).

If you want to avoid the split file in UltraArc, open UltraArc.ini and change

FirstVolumeSize=9999999999
CustomVolumeSize=9999999999

or something even bigger, don't matter.

If you want max compression use lolz instead of lzma2.

[rishikrap]

Originally Posted by KaktoR

No.

Yep, learned i the hard way. I slept after putting this and it got error after 2hr (Part of conversion log attached).
Only if I had stayed up for 6 more minutes.
Then I also tried

Originally Posted by KaktoR

 uelr+srep_new+lzma2

Replaced lzma2 with lolz_mtx. I hadn't seen the screenshot so again got the same LZO error.
Now, I have followed all that you have mentioned and started again.
Used

 uelr+srep_new+lolz_mtx for Data1

And

 srep_new+lolz_mtx

for Data2 with 2.txt as included and 1.txt as excluded.
Also, does it matter if I use lolz or lolz_mtx, I just used mtx so it utilized full cpu, don't know if it would make difference in conversion time since I want the best conversion. And there are other lolz conversions n, f etc do they matter? I'm using default lolz_mtx
Thanks for the detailed screenshots and all the other informatin else I'd have done some mistake for sure.
https://drive.google.com/drive/folde...TJA5Qhd34FZAlb
This was my inital output, is there any way, like any way in which I can just extract these without any error ;__;

 bpk+cp_2077(Xtools 2020)+xzlib+srep_new+lolz_mtx

I get the error unsupported config in every installer I tried, even Light as you mentioned and not CIU either.
Are those files now useles since I can't extract them, should I delete them and accept that I wasted 21hrs and RDP money ;___;

Update: I used the config as mentioned above and got the same error again
https://prnt.sc/w8brei
I'm now using xtools and not uel(lzo) since you said it could be vice versa too. Plus fitgirl page also mentiones xtools library Also, I have reset the config to default without and file lists or so.
Currently retying with

 xzlib+srep_new+lolz_mtx

If I get an error I'll replace xzlib with xlzo, since I don't know what'll work, and I had gotten success with xzlib earlier, hnce I'm trying xzlib first.

[Masquerade]

rishikrap
Using xzlib gives you no benefit, so there's no point to use it (UE3 does not use zlib). And then xlzo will not help either, since XTool cannot process these files IIRC (despite a .tfc file is given in example).

I'm downloading rocket league now from my Steam library, will give it a whirl then come back to you with some explanations on what to do.

[Masquerade]

OK so I'd like to start by apologising to you for denying the use of xtool zlib, this is the first UE3 game I've seen use zlib on tfc/upk archives:

SAMPLE - xzlib

Compressed 24 files, 1,926,995,396 => 7,144,830,307 bytes. Ratio 370.78%
Compression time: cpu 1.89 sec/real 94.48 sec = 2%. Speed 20.40 mB/s
All OK

With that established, I have tagged below a sample compressor that will allow you to make an archive compatible with the ASIS script (I recommend using compact mode)

How to use:

1. Download the below 7z, extract it
2. Put your rocket league game folder in to In folder
3. Run the bat
4. Inside the bat, type:
   

   xprecomp+srep+lolz/$wem=srep+pmt_razor

5. Wait until done!

Compression method explanation:

xprecomp = xtool, configured for zlib.

srep+lolz - srep to remove duplicates and lolz for main compressor. If you want to use lzma2, swap lolz for "7z".

What does $wem mean?

This is a mask, and masks are a way of using multiple compression methods in one archive.

Each mask (and the file types included by it) is included in arc.groups.

This allows the main data to be compressed with xtool,srep,lolz then the audio gets compressed with srep,razor(pmt).

PMT is a chunk based accelerator for slow compressors like mtx, but ASIS doesn't have MTX yet, so PMT is what to use.

Because PMT is chunk based, we run srep beforehand to catch duplicates.

You can define a mask using the arc.groups file, and the grouping must begin with $groupname:

Example:

$help
*.bin
*.blob

This means you can separate off bin and blob files using the /$help identifier in your method.

If that's a bit confusing, I'm sorry!

[rishikrap]

Rocket League Epic Games Version V1.0.10897.0 (Nothing Ripped)
[Steam Version Might work too, untested]

 8:17:03 AM - You have chosen the unlimited type for the output size.
 8:17:03 AM - ISO images will not be created when conversion is complete.
 8:17:03 AM - The computer will remain on when the conversion is complete.
 8:17:03 AM - Selected ARC method for Data1.bin was: xzlib+srep_new+lolz_mtx
--------------------------------------------------------------------------------------------------------------------------------------
 8:17:05 AM - Compression initialized in 23/12/2020 at 8:17:05 AM.
 8:17:05 AM - Starting Data1.bin archive creation.
 5:22:58 AM - Moving Data1.bin.001 file to media folder.
 5:23:04 AM - Compression finished in 24/12/2020 at 5:23:04 AM.
--------------------------------------------------------------------------------------------------------------------------------------
 5:23:05 AM - Getting size of the game: Rocket League
 5:23:05 AM - >> Installation folder size: 12.92 GB
 5:23:05 AM - >> Compressed game size: 6.30 GB
--------------------------------------------------------------------------------------------------------------------------------------
 5:23:05 AM - Overall input size: 12.92 GB
 5:23:05 AM - Overall output size: 6.30 GB
 5:23:05 AM - Overall conversion time: 21:05:58

Credits to

Originally Posted by KaktoR

No.

Originally Posted by Masquerade

I'd just let it run the course now.

For making me understand the basics and recommend me the best config for the best compression 48.7%

I still don't know how to make another installer work with my config, I'm just using Diskpan GUI from CIU V3 B7.U4 and the default Shadow of Tomb Raider Installer as included in the package.
I tried Light Installer, ASIS Installer and CorePack installer but they gave the error of not recognizing the archive format, and ASIS not even an error, just installed the default Command & Conquer™ Remastered Collection and gave me a desktop shortcut.

Any mods @JustFun or whoever, if you find this worth it, please do add this the Index for future reference for anyone else.