FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

Masquerade 28-04-2022 09:45

Quote:

Originally Posted by kuyhaa (Post 496618)
anyone, have method for .bundle file from StreamingAssets\aa\StandaloneWindows64 ?

You can decompress that bundle with UBT.

Not much point in trying to recompress+patch it because a patch would be almost 17mb.

Masquerade 28-04-2022 12:02

The Riftbreaker [v27746]

Audio .zips:

Code:

msc+oggre+srep+lzma2
Everything else:

Code:

xzlib+srep+lolz
6.4GB ---> 4.4GB

Carldric Clement 30-04-2022 07:12

Need for Speed Carbon

Original Size: 4.91 GB
Compress Size: 2.41 GB

Method:
Code:

*  any jpg files = packjpg
*  rest of files = xprecomp+srep:m3f:a2+exe2+7z:d128m:mt2

XTool Used Plugins = ZLib, Reflate, LZ4HC


Carldric Clement 30-04-2022 07:42

Broforce v1131

Original Size: 457 MB
Compress Size: 289 MB

Method:
Code:

* jpg = packjpg
* rest of files = xprecomp+srep:m3f:a2+exe2+lolz:d128:mt4:mtt1:lmdf1
* asset bundle unity = xtool:zlib:unity+srep:m3f:a2+exe2+lolz:d128m:mt4:mtt1:lmdf1

Thanks to L33THAK0R for suggesting reduced size.
Update on 1 May 2022.

L33THAK0R 30-04-2022 17:33

Quote:

Originally Posted by Carldric Clement (Post 496643)
Broforce v1131

Original Size: 457 MB
Compress Size: 337 MB

Method:
Code:

* jpg = packjpg
* rest of files = xprecomp+srep:m3f:a2+exe2+lolz:d128:mt4:mtt1:lmdf1


you can reduce this to ~290mb (Incl. installer) if you decompress the Unity assets.

KaktoR 01-05-2022 01:40

Age of Empires II Definitive Edition
Update 61591, 1 Language, all DLC

Code:

17:20:52 - Selected ARC/DS method for Data1a-01.bin was: precomp_mtx+srep_new+4x4:lzma
 17:20:52 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:lzma
-----------------------------------------------------------------------------------------
 21:51:31 - Overall input size: 44.71 GB
 21:51:31 - Overall output size: 24.40 GB (Ratio 54.57%)
 21:51:31 - Data files creation time: 04:09:05

Code:

Data1a = *.aoe2scenario|*.aoe2campaign|*.scx|*.png|*.aoe2spgame|*.pdf

Why precomp? Xtool is ~200mb bigger

WemTool on wem files gives some mb, not worth it


kj911 01-05-2022 14:29

Quick noob decompression questions: This varying methods betterly safer from extraction process??

A: xZLib+delta+srep+lolz
B: xZlib+srep+delta+lolz

My test, its compression percentage differnece less than 0.1%. (B its smaller.)

Masquerade 02-05-2022 03:08

Project CARS 3 [v1.0.0.0.0705]
  • *.bff
    Code:

    xPCARS3+srep+lolz
  • *.bk2
    Code:

    bpk (hash fail on decompression so use bpk exes and hdiff patch them)
  • *.bank
    Code:

    msc+srep+lzma2
  • Everything else
    Code:

    precomp:d0+srep+lolz

49.4GB ----> 19.1GB

L33THAK0R 07-05-2022 05:04

Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript ), however I can't seem to figure out which library version to use, the best I've gotten so far is ~850MB. I think if I exhaust all my options I'll just look into re-importing the decompressed assets to a modified archive, since some Russian modders were able to do so, in order to apply a total conversion translation mod to the titular title in question.

Masquerade 07-05-2022 11:10

Quote:

Originally Posted by L33THAK0R (Post 496710)
Are there any reliable methods to discern which LZ4 library version works best with a given sample? I recently got interested in titles using LZ4, since I haven't tackled many, and thought I'd start off small with the 2013 title "Castle of Illusion", which has a singular archive containing all title assets, (decompressed archive inflates to 786 MB -> 1.20 GB, using the appropriate QuickBMSscript )

I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work ;)

I checked this game out and it uses lz4hc, not lz4. By using lz4 precompressor, you will end up getting 21/5539 streams. By switching the codec to lz4hc alone you will instantly get 4215/5539 [liblz4 v1.7.4].

This alone would seem pretty cool, until you see that the only streams that were inflated were the small ones, so you get very little size increase. In addition to this, you still have over 1000 compressed LZ4 streams still in the file.

By digging a bit deeper into the info XTool is giving me (using verbose output), my next intention was to lock precompression at a certain level of lz4hc. I chose lz4hc level 9. This is because most of the streams previously were compressed at level 9. After making a new database and testing, we get these results:
  • 5524/5539 streams [liblz4 v1.7.3, v1.7.4, v1.8.0].
  • 5488/5539 streams [liblz4 v1.8.1].
  • 5424/5539 streams [liblz4 v1.8.2, v1.8.3, v1.9.3].

This clearly shows that using anything newer than liblz4 v1.8.0 is going to yield extremely poor results. With the oldest 3 libraries, you will only miss 15 streams. In my opinion, this is completely acceptable. At this point, I stop testing and publish my findings in the XTool plugins thread.

We all have a lot to thank Razor12911 for.

L33THAK0R 08-05-2022 02:02

Quote:

Originally Posted by Masquerade (Post 496718)
I checked this game out and it uses lz4hc, not lz4.

Ah that'd definitely explain things, I didn't even think to see if I was using the wrong codec

Quote:

Originally Posted by Masquerade (Post 496718)
I think you are overthinking this.

First of all, bms2xtl exists so you half of the job is already done for you. The other half of the job is about 20 minutes and a bit of guessing work ;)

Yeah I was using bms2xtl, just thought rebuilding an archive would be the next step if I couldn't figure out how to pick up more streams.

Cheers for the help and testing mate :cool:

L33THAK0R 08-05-2022 21:26

Update on Zen Studios ".pxp" archives
 
2 Attachment(s)
Over the last couple days I've been testing "BMS2XTOOL" on ".pxp" archives to try and see if through this method it's possible to both decrypt + expand the compressed streams present within a given ".pxp" archive, to make re-compressing said archives a viable prospect. However I've hit a roadblock unfortunately, as I can get xTool (v0.5.0) to detect some streams, but can't seem to figure out what codec to use, to actually process the majority of said streams.

Currently I'm associating the compressor type, "ZENPXP" with the "zlib" codec for use with "BMS2XTOOL", and combining this with the "reflate" codec in my arc.ini.

For anyone interested, attached below is a sample testing suite, as well as just a copy of BMS2XTOOL, with the appropriate resources to generate your own database for said sample, should anyone be interested in looking at this. I've also tried to see if it's been documented anywhere within QuickBMS the compression algorithm used for a given "comtype" in a QuickBMS script but alas this doesn't seem to be the case

Spotless 10-05-2022 15:59

VALORANT: Episode 4 Act 3

srep+lolz

23.5 GB> 7.2GB

iamsmeg 11-05-2022 06:27

help please
 
okay I am having problems getting this to work what am I doing wrong
I place the wd1.xtl and xcompress in my xtool folder and use

xtool precomp -mwd1 common.dat common.tmp

it runs for a a few seconds and nothing happens it doesn't even make a common.tmp file

iamsmeg 11-05-2022 06:31

sorry I forgot to say this is for Watch dogs 1 using the database method posted here

pslove 13-05-2022 22:29

Hey guys, what's the best method for The Walking Dead Definitive Series? I'm using DiskSpan GUI btw. Thank you.

Masquerade 13-05-2022 22:59

Quote:

Originally Posted by pslove (Post 496800)
Hey guys, what's the best method for The Walking Dead Definitive Series? I'm using DiskSpan GUI btw. Thank you.

Telltale game, so all of the ttarch2 files that have headers beginning with ECTT are encrypted using Blowfish encryption.

In this case, DiskspanGUI has nothing to handle those files. I would use cls-ttgd by Shegorat, but that isn't a public tool.

Use this to decrypt the files: https://www.fileforums.com/showthread.php?p=483980

Then, compress with xtool(zlib+reflate)+srep+lolz. You can re-encrypt your decrypted files from earlier using the same tool to get MD5 perfectiom upon installation.

kuyhaa 14-05-2022 11:23

hi guys, i want try to repack Harvest Moon: One World

from 850 Mb to 736 Mb , method pak folder srep+lolz

i see on fitgirl , she is got 196 Mb

have info ?

Hexagon123 14-05-2022 12:18

Quote:

Originally Posted by kuyhaa (Post 496817)
hi guys, i want try to repack Harvest Moon: One World

from 850 Mb to 736 Mb , method pak folder srep+lolz

i see on fitgirl , she is got 196 Mb

have info ?

Seems to use Unity, try XTool or UBT?

kuyhaa 15-05-2022 00:40

Quote:

Originally Posted by Hexagon123 (Post 496820)
Seems to use Unity, try XTool or UBT?

precomp xtool doesn't work

Dragonis40 16-05-2022 06:21

Hello everybody! A simple question: does a more powerful PC give better compression results rather then a less powerful one?

KaktoR 16-05-2022 09:34

Quote:

Originally Posted by Dragonis40 (Post 496873)
Hello everybody! A simple question: does a more powerful PC give better compression results rather then a less powerful one?

No, only the speed is different.

pslove 16-05-2022 10:45

Quote:

Originally Posted by Masquerade (Post 496801)
Telltale game, so all of the ttarch2 files that have headers beginning with ECTT are encrypted using Blowfish encryption.

In this case, DiskspanGUI has nothing to handle those files. I would use cls-ttgd by Shegorat, but that isn't a public tool.

Use this to decrypt the files: https://www.fileforums.com/showthread.php?p=483980

Then, compress with xtool(zlib+reflate)+srep+lolz. You can re-encrypt your decrypted files from earlier using the same tool to get MD5 perfectiom upon installation.

I did this and the compression took a while but the results are dissapointing:

Code:

-------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------- CONVERSION DATE 15/05/2022 -------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------
 11:52:56 - Deleting files from the output folder.
 11:52:56 - Files have been deleted.
-------------------------------------------------------------------------------------------------------------------------------------------
 11:52:56 - You have chosen the DVD5 type for the media size.
 11:52:56 - ISO images will not be created when conversion is complete.
 11:52:56 - The computer will remain on when the conversion is complete.
 11:52:56 - Selected ARC/DS method for Setup-1a.bin was: xtool:mzlib:mreflate+srep_new+lolz
-------------------------------------------------------------------------------------------------------------------------------------------
 11:52:58 - Compression initialized in 15/05/2022 at 11:52:58.
 11:52:58 - Starting Setup-1a.bin archive creation.
 21:19:03 - Moving Setup-1a.bin » Setup-10a.bin files to media folder.
 21:19:05 - Compression finished in 16/05/2022 at 21:19:05.
-------------------------------------------------------------------------------------------------------------------------------------------
 21:19:05 - Getting size of the game: The Walking Dead - Definitive Series
 21:19:05 - » Installation folder size: 45.49 GB
 21:19:05 - » Compressed game size: 41.54 GB (Ratio 91.32%)
-------------------------------------------------------------------------------------------------------------------------------------------
 21:19:05 - Overall input size: 45.49 GB
 21:19:05 - Overall output size: 41.54 GB (Ratio 91.32%)
 21:19:05 - Overall conversion time: 26:37:47
 21:19:05 - Overall conversion discs: 10 DVD5
-------------------------------------------------------------------------------------------------------------------------------------------
 21:31:17 - Merging Records.ini into the Setup.dll file.
 21:31:17 - The Records.ini file was merged into Setup.dll successfully!
-------------------------------------------------------------------------------------------------------------------------------------------

Did I do something wrong? I used the tool you told me and I checked MD5s before and after decryption to make sure the files are decrypted, and they were different so I assumed the decryption was successful.

Masquerade 16-05-2022 12:11

Quote:

Originally Posted by pslove (Post 496876)
so I assumed the decryption was successful.

To assume makes an ass out of u and me :p

Anyways...

Make sure you used the correct key for decrypting the data. Check in hex editor that the decrypted file starts with ZCTT. ECTT will be an encrypted archive, ZCTT means you did it correctly.

Test on a single TTARCH2, decrypt and see if you can inflate it with XTool. If you decrypted it right, you will get correct ZCTT header and XTool will be able to precompress the data.

kj911 16-05-2022 14:22

Harvest Moon: Its probably use SREP and RZ (v1.00) and two LOLZ version (last and older v20d3) compressors used from final packing. And ISDone x64 v0.6.0.2!

This piggy and fbz compressor tell someone something?

KaktoR 17-05-2022 09:54

I need some (technical) information here.

Code:

xtool+msc+oggre
Compressed 1 file, 132,903,091 => 72,589,354 bytes. Ratio 54.62%
Compression time: cpu 0.08 sec/real 31.37 sec = 0%. Speed 4.24 mB/s
All OK

Extracting 1 file, 132,903,091 bytes. Processed  88%
ERROR: general (de)compression error in msc
Stream:# is not supported by this version of the library

Code:

xtool+oggre+msc
Compressed 1 file, 132,903,091 => 104,121,513 bytes. Ratio 78.34%
Compression time: cpu 0.13 sec/real 33.31 sec = 0%. Speed 3.99 mB/s
All OK

Extracted 1 file, 104,121,513 => 132,903,091 bytes. Ratio 78.34%
Extraction time: cpu 7.17 sec/real 9.96 sec = 72%. Speed 13.34 mB/s
All OK


kj911 17-05-2022 10:07

Kaktor: Trying probable two step compressions methods, don't work?? First archive use xtool+msc only and pack again the new arc file use oggre. FreeArc anomalies, bug, not the msc/srep or any packers.

L33THAK0R 17-05-2022 21:32

Sleeping Dogs (Original / Definitive Edition)
 
2 Attachment(s)
Hi all,

Been looking into Sleeping Dogs, and after a bit of discussion with some more experienced/technical tool engineers, I've got to say I'm quite intrigued. It uses a very aggressive "LZ" algorithm, with a given ".BIG" (archive format for the title in question) decompressing to over twice its original size! It also seems the WWise ".pck" files used for audio within the title have the potential for additional gains using "OGGRE"/"MSC". Unfortunately for the former I don't possess the skill-set (yet) to adapt the provided compression/decompression algorithm to be able to parse inputs for use with a chain of compressors, but none the less I think its still interesting enough for others to see.

Attached is a tool suite for processing the aforementioned ".BIG" archives, as well as a sample archive.

kj911 20-05-2022 08:08

Splinter Cell: Pandora Tomorrow v1.31 with Online+Maps Patch (Hungarian version!)

Results sizes its compressed only, without archive header. (Compressed directory size.)

Data files (more extensions with *.uax files):

992 file, 1 065 351 100 byte
Code:

xZLib+msc+srep:m3f:a0+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:fba4096:oh14:ldc0:tt32:mc1023
Results: 235 257 168 byte

Sound1 (MAPS.LM0/SM0, 2 file only):

Code:

srep:m3f:l32:a0+msc+4x4+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d160m:fba4096:tt16:mc1023
Results: 64 141 713 byte

Sound1 notes: MSC with tak=9 and raw=2/bmp=0 options! Others its defaults. (wav/mp3=1 ...)

Sound2 (*.hxb/hxc, *.LS0, *.SP0)

12 files, 192 933 113 byte
Code:

tta+lzp:3:64m:h22:d64mb+4x4 (without 2 small files, smaller than 6kB: tta+lzma:ultra:bt4:273:mc1023)
Results: 108 008 108 byte

Sound3 (*.SS0)

27 files, 250 086 228 byte
Code:

lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d128m:fba4096:tt32:mc1023
Results: 165 027 167 byte

Binaries (*.exe/dll)

62 files, 41 353 387 byte
Code:

rep+exe2+delta+lzma:ultra:mc384
Results: 10 422 472 byte

This Lossy parts:

Untouched videos size: 454 103 532 byte
Recompressed videos size: 200 621 456 byte

Code:

bpk+rep:64mb+lzp:128mb:3:h22:d256mb
Results: 143 422 866 byte

2006.894MB --> 692.649MB (34.513% ratio)

Masquerade 24-05-2022 13:26

Dolmen
  • *.PAK
    Code:

    oo2recm+srep+lolz
  • *.MP4
    Code:

    srep
  • Everything else:
    Code:

    srep+razor

Use this oodle core.

7.3GB ---> 5.7GB [Game + 60FPS + 30FPS videos]

6.8GB ---> 5.1GB [Game + 60 FPS videos]

6.6GB ---> 5GB [Game + 30 FPS videos]

kj911 26-05-2022 14:58

Reversed compression switches
 
1 Attachment(s)
Please test the sample archive, from decompression. Packed with SREP+MSC method. MSC use WAV/RAW/MP3=1 options and SREP 3.92 with -m3f:l32 switches. And TAK 2.3.1 version. Any methods, its few percentage worse ratios. (LOLZ not tested.) Decompression resources, its minimized, don't required arc/cls.ini files. Tested in use Arc.exe/ini files and any combinations its ok from WinXP machine.

1. Download the sample, and unpack to any blank dirs.
2. Run bat file and see results.
3. Move *.sha file to "Unpacked" dirs and checking use TC or any apps.

Thanks.

abror 30-05-2022 01:27

1 Attachment(s)
Hai,

I'm trying to compress WWE 2K22 with Diskpan GUI by Cesar82 and it's been running for more than 5 hours. It's written in the top left corner 46:20:09

Is that normal?

Compress method : xtool:mkraken:9+srep_new:1+lolz

https://i.postimg.cc/nrgymYrK/Capture.png

Masquerade 30-05-2022 02:09

Quote:

Originally Posted by abror (Post 497049)
Hai,

I'm trying to compress WWE 2K22 with Diskpan GUI by Cesar82 and it's been running for more than 5 hours. It's written in the top left corner 46:20:09

Is that normal?

Compress method : xtool:mkraken:9+srep_new:1+lolz

This game is encrypted + uses Oodle Hydra compression so you will get nowhere using xtool kraken.

abror 30-05-2022 04:09

Quote:

Originally Posted by Masquerade (Post 497050)
This game is encrypted + uses Oodle Hydra compression so you will get nowhere using xtool kraken.

is the method below correct?

xtool:mhydra:9+srep_new:1+lolz

in Diskpan GUI, Hydra is only in Xtool or should I replace it with oodle selkie from razor?

If something goes wrong please tell me

Or can you tell me the compression method so I can copy-paste?

Masquerade 30-05-2022 04:12

The Sims 4 [Deluxe Edition] (v1.88.228.1020 + All DLCs + Soundtrack)

Code:

xzlib+srep+lolz/$mp3=mpz
50.1GB ---> 30.5GB

Masquerade 30-05-2022 04:22

Quote:

Originally Posted by abror (Post 497051)
is the method below correct?

No. This game is encrypted. You cannot precompress any data without decrypting it first. Of which, is an incredibly horrid task for not much compression gain. Oodle cocecs are extremely efficient in their compression.
Quote:

Originally Posted by abror (Post 497051)
oodle selkie from razor?

Selkie and Hydra are completely different compression algos.

http://www.radgametools.com/images/o...pical_vbar.png

This graphic is slightly outdated since the Oodle family now includes LZNA and Hydra which do not appear on the above, but the image still illustrates my point.

kj911 30-05-2022 04:38

1 Attachment(s)
Quote:

Originally Posted by 78372 (Post 468838)
GTA SanAndreas

Everything in audio\streams folder *ONLY*

Method : oggrenc+srep

Original Size = 1.09GB
Compressed size = 981MB

Its really?? Game version? v1.00 or v1.01 or v2.0?? EU or USA versions? Trying compress v1.00 files, use this srep only compression get ~2% size ratio from compression. Use last LOLZ, reduced 1.10GB to 1.07GB size. (1 125 783kB) Use oggre v0.11 only have ~12MB size reduction. Combined with LOLZ in best results: 1 117 004kB size. This audio files encypted? (Retail DVD version.) Used older OGGRE_Enc version than v0.11??

Input size: 1 187 289 114 byte (Files creation date: 2005.02.10/11.)

Any tips, idea? And best methods from SFX folder compression?

UPDATE: My "SFX" and "streams" folder CRC's hash files added.

L33THAK0R 30-05-2022 05:53

Quote:

Originally Posted by kj911 (Post 497054)
Its really?? Game version? v1.00 or v1.01 or v2.0?? EU or USA versions? Trying compress v1.00 files, use this srep only compression get ~2% size ratio from compression. Use last LOLZ, reduced 1.10GB to 1.07GB size. (1 125 783kB) Use oggre v0.11 only have ~12MB size reduction. Combined with LOLZ in best results: 1 117 004kB size. This audio files encypted? (Retail DVD version.) Used older OGGRE_Enc version than v0.11??

Input size: 1 187 289 114 byte (Files creation date: 2005.02.10/11.)

Any tips, idea? And best methods from SFX folder compression?

UPDATE: My "SFX" and "streams" folder CRC's hash files added.

Might just be the files you've got, when I repacked the GTA series way back when I got similar gains with oggre + srep + lolz

kj911 30-05-2022 06:08

L33THAKOR: Did you get a similar result for you too? Max. ~35-42MB size reductions?
Whats differences? 78732 packed 981MB vs. my ~1.07GB (1091MB) archive. :confused:

New results with original files (not unpacked the game streams files):

srep+oogre+lolz: 1 121 406kB
oggre+srep+lolz: 1 116 863kB

The streams files unpacked via SAAT tools to *.OGG files.

Compress srep:m3f:l64 only: 1 099 616 kB (with l32: 1 095 842kB)
LZMA2/d26/lp0/pb0 --> 1 080 775kB
CAB/LZX:21 --> 1 079 857kB (better than LZMA2, LOL. Use PowerArc 7.00 CL version.)
LOLZ: 1 077 415kB

The SAAT exported OGG files reimported to original fileformats, totally lossless!! New created files (AA, ADVERTS, more..) no difference than original ones.

Testing now lolz and oggre compress its *.OGG files.

oggre:s3: 1 008 556kB (crunched again use srep:l32 --> 1 006 685kB)
oggre:s3+lolz: 1 006 630kB (yes, very minimally better than srep-ed versions.)

Note: All srep compressions, use -m3f defaults switches.

Conclusions? Make packed archive with WinXP compatible decompressors, use LOLZ only. Get better ratio and x64 systems? Use oggre+srep+lolz methods. (or and use SAAT tools from recreating original streams files, after decompression.)

Alternative ways, testing now, use OGG rehuffing, with ~2-3% or higher, taken from input/compressed size savings from *.OGG files packing without OGGRE.

KaktoR 30-05-2022 12:57

Sniper Elite 5

Code:

20:57:01 - Selected ARC/DS method for Data1b-01.bin was: srep_new:1+lzma2:d96m:fb273:lc4:mc1000000
 20:57:01 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib+srep_new:1+lzma2:d96m:fb273:lc4:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 23:24:34 - Overall input size: 76.51 GB
 23:24:34 - Overall output size: 46.48 GB (Ratio 60.76%)
 23:24:34 - Overall conversion time: 02:27:26

Data1a = *.pc and *.asr files



All times are GMT -7. The time now is 12:42.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com