Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Originally Posted by L33THAK0R

Whats the best method for processing unencrypted UE4 ".pak" files? I've tried just applying xtool's zlib+reflate / oodle codecs but whilst the streams are picked up I always have issues decompressing, with the archives being reported as corrupted. I used to just process the pak's by unpacking and compressing the loose files but with larger titles this isn't very efficient for the user's disk I/O and with some titles I'm investigating currently (such as "Phantom Doctrine"), seemingly random files fail to decompress, pushing me to investigate again into packing the compressed ".pak" files as they're originally presented instead.

I took your example "Phantom Doctrine" and it works fine with xtool+srep+lzma, so... ?

[L33THAK0R]

Originally Posted by KaktoR

I took your example "Phantom Doctrine" and it works fine with xtool+srep+lzma, so... ?

Ah cheers, maybe it's lolz then thats acting up, I'll definitely have to do a bit of testing I reckon, thanks for testing it on your end!

Originally Posted by KaktoR

I don't know where your "problem" is exactly now. You can ignore the AFR message about corrputed LZO-Streams. This just means that some streams aren't compressed with lzo, use another codec or lzo version or whatever else. Just ignore it.

Ah alright, I'll definitely give recompressing with another codec a shot, since that might be whats messing things up on my end, honestly cant think of what else it could be considering I'm getting decompression issues when just packing with afr+srep (to test).

[L33THAK0R]

Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

[KaktoR]

Originally Posted by L33THAK0R

Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

I don't know where your "problem" is exactly now. You can ignore the AFR message about corrputed LZO-Streams. This just means that some streams aren't compressed with lzo, use another codec or lzo version or whatever else. Just ignore it.

[L33THAK0R]

Originally Posted by L33THAK0R

Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

For anyone interested or who's encountered similar issues. I found out the cause of my troubles with both this issue and the immediate post prior to this one. Recently I streamlined and revamped my whole pipeline for packing titles and in the process accidentally included an outdated version of Srep somehow, using an updated version has fixed all my issues thankfully.

[KaktoR]

There is anvil plugin for xtool, you should give it a try.

[hwang4]

Masquerade , thank you for your help!

Finally , I found the right oo2core.dll!

oo2reck (7) + srep+ lolz

23gb ------> 16.6gb

[KaktoR]

Pathfinder Wrath of the Righteous
Enhanced Edition

 10:58:21 - Selected ARC/DS method for Data1a-01.bin was: xtool:munity,lz4hc,l12:lz4_193+srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 10:58:21 - Selected ARC/DS method for Data1b-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 10:58:22 - Compression initialized in 01.10.2022 at 10:58:22.
 10:58:23 - Starting Data1a-01.bin archive creation.
 12:46:03 - Moving Data1a-01.bin file to media folder.
 12:46:04 - Starting Data1b-01.bin archive creation.
 12:59:09 - Moving Data1b-01.bin file to media folder.
 12:59:10 - Compression finished in 01.10.2022 at 12:59:10.
-------------------------------------------------------------------------------------------------------------------------------------------
 12:59:10 - Getting size of the game: Pathfinder - Wrath of the Righteous
 12:59:10 - » Installation folder size: 37.88 GB
 12:59:10 - » Data1a-01.bin file size: 14.17 GB
 12:59:10 - » Data1b-01.bin file size: 3.85 GB
 12:59:10 - » Compressed game size: 18.02 GB (Ratio 47.58%)
-------------------------------------------------------------------------------------------------------------------------------------------
 12:59:10 - Overall input size: 37.88 GB
 12:59:10 - Overall output size: 18.02 GB (Ratio 47.58%)
 12:59:10 - Overall conversion time: 02:00:38

[KaktoR]

Command & Conquer Remastered Collection

I made this as a test and showcase with use of quickbms + bink patching. In this case use of quickbms is possible because game files are not compressed (or at least not seem).

Steps:

1. Make Backup of original archives (Data\movies_*.meg)
2. Unpack original archives (Data\movies_*.meg) with quickbms
   

   for %%f in ("MOVIES_*.MEG") do quickbms_4gb_files -d -o -Y petroglyph.bms "%%f"

3. Compress binks and make patches (bink_pack.exe, bink_unpack.exe, hdiffz.exe, hpatchz.exe)
   There are some bk2 files which even bink_unpack.exe has problems with, so patch size is same as original size (416mb diff size). Take care of them.
4. Null unpacked bk2 files (fill contents with zeros)
5. Reimport nulled bk2 files with quickbms into original archives (petroglyph.bms)
   

   quickbms_4gb_files -w -r petroglyph.bms "input .meg file" "input folder containing nulled bk2 files"

   Before reimporting you have to separate all the files in the unpacked folders, you only need bk2 files for reimporting
   Tbh I don't know if quickbms has a option to just import defined file types lol
   Remove Data\movies_*.meg.bak files if there are any (quickbms creates these files I guess)
6. Delete all (nulled) bk2 files, so you have just compressed binks and patch files + nulled .meg archives left
   These nulled .meg archives have a compression ratio by nearly 0%, so don't mind the size of these (14.8 GB -> some KB)
7. Compress the game
8. Include required files in Installer or game folder (before compress the game)
9. Let your Installer run a batch file to do it's job after extracting all files (example batch)

Installation/batch stuff will take some time, .meg archives are crc perfect at the end.

Results:

 15:11:02 - Selected ARC/DS method for Data1a-01.bin was: srep_new:1+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
 15:35:12 - Overall input size: 36.90 GB
 15:35:12 - Overall output size: 15.67 GB (Ratio 42.47%)
 15:35:12 - Overall conversion time: 00:24:08

Results without quickbms + bink patching stuff:
https://fileforums.com/showpost.php?...postcount=1772

[kuyhaa]

hi, how to TEXTURE Rebuild and compression .dds file use "crunch" ?

[KaktoR]

Dakar Desert Rally

bk2 (kb2n) = no bpk
pak = kraken (oo2core 7)

Streams: 17099/17099
Time: 00:12:38 (00:12:36)
Memory: 245 MB (245 MB)

Compressed 1 file, 2,125,021,534 => 4,260,018,174 bytes. Ratio 200.47%
Compression time: cpu 1.53 sec/real 125.97 sec = 1%. Speed 16.87 mB/s
All OK

[Edison007]

Came back to this one after a bit of a break and after learning about xtool's "verbose" mode, turns out LZO is used on the archives, however it seems that no matter what level depth setting or library I use no streams are able to be processed, does anyone know the reason behind this? Attached is an image supporting my findings.

No, in the game does not use LZO.
Тhis is a private(?) LZ algorithm. You can read this thread: https://forum.xentax.com/viewtopic.php?f=10&t=9481

hi, how to TEXTURE Rebuild and compression .dds file use "crunch" ?

crunch = lossy, not suitable for repacking

[L33THAK0R]

Originally Posted by Edison007

No, in the game does not use LZO.
Тhis is a private(?) LZ algorithm. You can read this thread: https://forum.xentax.com/viewtopic.php?f=10&t=9481

I remember reading through that entire thread when I was first researching this bugger of a title, had a hope perhaps the lads over at the Xentax forums had been wrong, and that the PMCQ algorithim was just a fork of LZO. I actually managed to talk to Ekey (a very talented developer who's quite experienced with developing various decompression suites for a variety of titles), but he unfortunately lost his source code for his Sleeping Dogs decompression tool. I'm not sure if you'd be able to answer this but do you know why xtool might be reporting lzo1x streams? Is this just as simple as being an error on xtool's part? Honestly might try again to adapt his original decompression function (which we still thankfully have the source code for) as an xtool plugin, but given my lack of success in previous attempts I doubt I'll make much progress.

[Edison007]

ad a hope perhaps the lads over at the Xentax forums had been wrong, and that the PMCQ algorithim was just a fork of LZO.

No This is a different algorithm.

Is this just as simple as being an error on xtool's part?

Yep. False positive detect.

Honestly might try again to adapt his original decompression function

You also need a compress function.

[L33THAK0R]

Originally Posted by Edison007

No This is a different algorithm.

Drat! It's always bizarre how complex some titles can be to properly process!

Originally Posted by Edison007

You also need a compress function.

Thats going to be a fun time trying to figure out, heres hoping its lying around somewhere on the web!