FileForums - Best Compression Methods for 'Specific' Games. Q&A

Also quick second question, has anyone else had issues with using AFR on AnvilEngine 1.0 titles (or whatever the engine was named prior to AnvilNext)? I'm getting some ".forge" titles being processed with a warning at the end of processing them with AFR: "WARNING! In file found corrupted LZO-streams. Check your data!" and I have to manually test each files or a batch of files to determine which produce this error. packing the files that produce this error separately to the ones that don't results in a consistent successful decompression however the size is inflated due to them not being de-duplicated together. Currently affected titles that I've noticed are "Assassin's Creed" "III", "Black Flag", "Freedom Cry" & "Rogue". I'll note that I've been using AFR v0.0.1.9, using the "-vr" parameter as well as Srep v0.392, since other versions have produced issues with the "*.forge" files from my experience.

For anyone interested or who's encountered similar issues. I found out the cause of my troubles with both this issue and the immediate post prior to this one. Recently I streamlined and revamped my whole pipeline for packing titles and in the process accidentally included an outdated version of Srep somehow, using an updated version has fixed all my issues thankfully.

L33THAK0R

04-10-2022 21:00

Sleeping Dogs Update

Quote:

Originally Posted by L33THAK0R (Post 496898)

Hi all,

Been looking into Sleeping Dogs, and after a bit of discussion with some more experienced/technical tool engineers, I've got to say I'm quite intrigued. It uses a very aggressive "LZ" algorithm, with a given ".BIG" (archive format for the title in question) decompressing to over twice its original size! It also seems the WWise ".pck" files used for audio within the title have the potential for additional gains using "OGGRE"/"MSC". Unfortunately for the former I don't possess the skill-set (yet) to adapt the provided compression/decompression algorithm to be able to parse inputs for use with a chain of compressors, but none the less I think its still interesting enough for others to see.

Attached is a tool suite for processing the aforementioned ".BIG" archives, as well as a sample archive.

Came back to this one after a bit of a break and after learning about xtool's "verbose" mode, turns out LZO is used on the archives, however it seems that no matter what level depth setting or library I use no streams are able to be processed, does anyone know the reason behind this? Attached is an image supporting my findings.https://i.imgur.com/8XrV9Jy.png

kuyhaa

05-10-2022 00:19

hi, how to TEXTURE Rebuild and compression .dds file use "crunch" ?

KaktoR

05-10-2022 04:21

Dakar Desert Rally

bk2 (kb2n) = no bpk
pak = kraken (oo2core 7)

Code:

Streams: 17099/17099

Time: 00:12:38 (00:12:36)

Memory: 245 MB (245 MB)



Compressed 1 file, 2,125,021,534 => 4,260,018,174 bytes. Ratio 200.47%

Compression time: cpu 1.53 sec/real 125.97 sec = 1%. Speed 16.87 mB/s

All OK

Edison007

05-10-2022 14:18

Quote:

No, in the game does not use LZO.
Тhis is a private(?) LZ algorithm. You can read this thread: https://forum.xentax.com/viewtopic.php?f=10&t=9481

Quote:

hi, how to TEXTURE Rebuild and compression .dds file use "crunch" ?

crunch = lossy, not suitable for repacking;)

L33THAK0R

05-10-2022 16:39

Quote:

Originally Posted by Edison007 (Post 498629)

No, in the game does not use LZO.
Тhis is a private(?) LZ algorithm. You can read this thread: https://forum.xentax.com/viewtopic.php?f=10&t=9481

I remember reading through that entire thread when I was first researching this bugger of a title, had a hope perhaps the lads over at the Xentax forums had been wrong, and that the PMCQ algorithim was just a fork of LZO. I actually managed to talk to Ekey (a very talented developer who's quite experienced with developing various decompression suites for a variety of titles), but he unfortunately lost his source code for his Sleeping Dogs decompression tool. I'm not sure if you'd be able to answer this but do you know why xtool might be reporting lzo1x streams? Is this just as simple as being an error on xtool's part? Honestly might try again to adapt his original decompression function (which we still thankfully have the source code for) as an xtool plugin, but given my lack of success in previous attempts I doubt I'll make much progress.

Edison007

06-10-2022 01:11

Quote:

ad a hope perhaps the lads over at the Xentax forums had been wrong, and that the PMCQ algorithim was just a fork of LZO.

No :) This is a different algorithm.

Quote:

Is this just as simple as being an error on xtool's part?

Yep. False positive detect.

Quote:

Honestly might try again to adapt his original decompression function

You also need a compress function. ;)

L33THAK0R

06-10-2022 05:51

Quote:

Originally Posted by Edison007 (Post 498633)

No :) This is a different algorithm.

Drat! It's always bizarre how complex some titles can be to properly process!

Quote:

Originally Posted by Edison007 (Post 498633)

You also need a compress function. ;)

Thats going to be a fun time trying to figure out, heres hoping its lying around somewhere on the web!

kj911

07-10-2022 09:07

Best compression methods from Stalker: Shadow of Chernobyl?? The game compressibble in smaller than 2-2.5GB's size? (with or without MP files. Multilang version installed size in has ~9.5GB's. Installer smaller than ~3.2GB with INNO only.)

Tested one gamedata archive file (*.db5), in few methods. This db5 archive contents extracted in specialized tools and extracted files compressed again into separated 3 archives and results its bigger than compress the one *.db5 file. Not understood!

File: gamedata.db5
Size: 677 500kB

CAB/LZX:21: 520 571kB
7z: 498 055kB
7z/PPMD: 529 581kB
SQX2: 492 547kB
DLZ only: 456 021kB
OGGRE+DLZ: 462 367kB

MSC only (BMP/DDSRAW/DXT=1): 550 356kB* (archive broken! Any data portions its missing??) --> Crunch in DLZ: 387 155kB (probably false...)
MSC only (DDSRAW=0, DXT=1): 677 377kB --> Crunch in DLZ: 478 327kB

Extracted the db5 file:

*.OGM: 68 290kB (6 file) --> DLZ: 59 064kB (Note: OGGRE+DLZ: 59 061kB)
*.OGG: 174 445kB (2052 file) --> OGGRE: 146 989kB
*.DDS: 446 886kB (1249 file) --> MSC+DLZ: 317 108kB / DLZ: 294 630kB
other: 908kB (17 file) --> MSC+DLZ: 103kB
__________________
Total: 690 528kB --> 523 264kB / 500 786kB (WTF!)

Masquerade

11-10-2022 11:47

Overcooked! 2

Code:

xtool_unity+srep+lolz

7.9GB ----> 3.7GB

liblz4 v1.8.0 used for best ratio.

Masquerade

11-10-2022 11:49

Prison Architect

Paradox Launcher files ripped.
All DAT files unpacked with WinRAR.

Code:

precomp:d0+srep+lolz/$ogg=oggre

670MB ---> 306MB

Thanks Nicolas Comerci for precomp STDIO fork.

L33THAK0R

12-10-2022 10:50

Grim Dawn LZ4 dropping streams

Hey all,

I've been giving "Grim Dawn" a looksie and was wondering if anyone might be able to clear something up

Code:

FreeArc 0.67 (March 15 2014) creating archive: Data.arc

Compressing 1 file, 145,766,328 bytes. Processed   0%

Compressing 145,766,328 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mgrimTEST -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 1396/14028

Time: 00:00:10 (00:00:36)

Memory: 160 MB (160 MB)



100%

Errorlevel=0

Compressed 1 file, 145,766,328 => 160,143,671 bytes. Ratio 109.86%

Compression time: cpu 0.09 sec/real 10.60 sec = 1%. Speed 13.75 mB/s

All OK

I've tried levels 1 - 9 so far and I only get maybe ~1 additional stream for each level increment, and I've tested essentially every version of LZ4 under the sun, (other versions only resulting in ~<100 streams being picked up), would anyone happen to know if its possible to pick up more streams or is it more likely that LZ4 was used very lightly and the other streams are inconsequential.

KaktoR

12-10-2022 14:23

Sample?

L33THAK0R

12-10-2022 18:21

1 Attachment(s)

Quote:

Originally Posted by KaktoR (Post 498703)

Sample?

Ah whoops I forgot to attach it, within the following attachment is a sample, a BMS script to decompress "Grim Dawn"'s assets (since the native LZ4 stream detection fails to pick up any at all), as well as the LZ4 library I've found success with.

All times are GMT -7. The time now is 08:00.

Page 214 of 312

214

Show 40 post(s) from this thread on one page