FileForums - Best Compression Methods for 'Specific' Games. Q&A

Has anyone had any luck with any LZ4 library for "State of Decay 2"? Threads around the net indicate it uses some sort of "custom" LZ4 solution, however QuickBMS is able to parse it with it's built in LZ4 comtype. Currently exhausted a good collection of ~20 libraries through trials and I'm yet to pick up or detect a single stream.

Attached is a sample ".pak" archive should this compression quandary tickle anyone else's fancy.

Masquerade

31-05-2023 09:58

Shame Legacy

*.pak:

Code:

ue4d:key=07E35C91100A9B400B1A87B15337BEEAC3545EEE30786F21CB2EE7283AF51743+xtool_zlib_dd5+lolz

Everything else:

Code:

razor-stdio

9.9GB ---> 6.9GB

Used CLS-UE4D library by Shegorat.
Thanks to Razor12911 and ProFrager for XTool and Lolz.
Thanks to Bulat Ziganshin for FreeArc and srep.

KaktoR

01-06-2023 09:09

Anyone can help me with srep options to use? I'm getting stuck here a bit.

Code:

ERROR! Can't allocate memory: 9705 mb required

Edit: -l# option seems to be the trick.

KaktoR

03-06-2023 16:39

Hitman 3 test using rpkg tool (because no plugin is available atm)

Code:

chunk0.rpkg (13.5GB)



 srep+lolz_fast

Overall input size: 14.03 GB

Overall output size: 12.10 GB (Ratio 86.29%)

Overall conversion time: 00:53:55

 

 

 srep+lolz_fast with rpkg tool

Overall input size: 15.50 GB

Overall output size: 11.41 GB (Ratio 73.61%)

Overall conversion time: 02:01:43

Input is bigger because chunk0.rpkg was extracted (lz4 variant is used). However output is smaller. But that time to process is insane and I recommend not to do this at all.

The time above was achived on a m2 ssd for a single test. Here again it is not recommended to do this kind of stuff on a ssd, even it is much faster.

Extracting all rpkg's on a HDD took barely 4 hours.
Next step is to make TAR files out of extracted files (this will take another bunch of hours), because somehow freearc seems to have problems with nearly 2 million extracted files.

Code:

*.WWEM

*.WWES



Rename them to *.WEM so you can use WemTool on them. After decoding rename them back to *.WWEM and *.WWES.

You can save approximately 10% of original size.





*.WWEV

They have "RIFF8-��WAVEfmt" but cannot do anything with them.

More info: https://fileforums.com/showpost.php?p=501212&postcount=3495

This is just a info test out of curiosity and I wanted to share some infos here.

dixen

05-06-2023 00:12

Quote:

Originally Posted by KaktoR (Post 499415)

Witcher 3 v4.0 Next Gen Update

Still the same algo from before
https://fileforums.com/showthread.ph...her#post498295

Code:

-mxtool:witcher3+xtool:zlib+xtool:lz4hc,l9



Compressed 1 file, 286,247,872 => 1,249,481,045 bytes. Ratio 436.50%

Compression time: cpu 0.41 sec/real 51.41 sec = 1%. Speed 5.57 mB/s

All OK



Extracted 1 file, 1,249,481,045 => 286,247,872 bytes. Ratio 436.50%

Extraction time: cpu 0.59 sec/real 37.01 sec = 2%. Speed 7.73 mB/s

All OK

Which version xtool and liblz4.dll needed?
With W3.xtl - decompression error for me:confused:

KaktoR

05-06-2023 00:20

XTool 0.6.9 + xdelta3_dll.dll + lz4_193 + doboz.dll

bms2xtl config

Code:

lz4=lz4hc:l9

dixen

05-06-2023 09:29

Quote:

Originally Posted by KaktoR (Post 501269)

XTool 0.6.9 + xdelta3_dll.dll + lz4_193 + doboz.dll

bms2xtl config

Code:

lz4=lz4hc:l9

It works but..VERY slow on lz4hc streams.

Masquerade

05-06-2023 09:39

Repella Fella

You can use ffmpeg to extract the vorbis audio tracks from the webm videos then make hdiffz patches to convert the oggs back into videos (note that the hdiff patches will be huge since they make up the entire video track).

Only issue is that oggre is buggy af with these oggs, so it may just be easier to use:

srep+lzma2 on the webm

precomps+srep+lolz on everything else.

Precomps = precomp stdin/stdio fork by Nicolas Comerci.

8GB > 7.2GB

I spent all day trying to get oggre working, but it will crash without much of a pattern to go off.

KaktoR

05-06-2023 09:53

Quote:

Originally Posted by dixen (Post 501273)

It works but..VERY slow on lz4hc streams.

Hm, no, not for me.

Did it today with latest version.

Code:

 11:16:00 - Overall input size: 61.98 GB

 11:16:00 - Overall output size: 44.43 GB (Ratio 71.68%)

 11:16:00 - Overall conversion time: 01:58:29

dixen

05-06-2023 13:00

Quote:

Originally Posted by KaktoR (Post 501275)

Hm, no, not for me.

Did it today with latest version.

Code:

 11:16:00 - Overall input size: 61.98 GB

 11:16:00 - Overall output size: 44.43 GB (Ratio 71.68%)

 11:16:00 - Overall conversion time: 01:58:29

Can you share your witcher3.xtl?

Masquerade

06-06-2023 11:06

Orcs Must Die! 3 [v1.2.1.0]

*.pak:

Code:

xtool_OMD3_dd5+lolz

Everything else:

Code:

xtool_zlib+razor-stdio

12.6GB ---> 6.8GB

Requires OMD3 database and lz4 library linked here.

Dragonis40

07-06-2023 10:08

Good evening, is there a way to create a .tmp file from a decompressed (or unpacked) file or batch of files in order to use hdiffz.exe and so to create a .diff file?
A lot of videogames have several files that can be unpacked using specific tools, but no way to repack them. Maybe hdiffz.exe is a solution, but it needs a .tmp file. How to create it? Thanks in advance for your help!

Masquerade

07-06-2023 13:57

HDiffz does not need a .tmp file, read the usage instructions.

Old file and new file makes differential patch.

Masquerade

09-06-2023 14:42

Hi,
Does anyone have any samples of LZMA compressed Unity Asset Bundles that they could share with me?
Thanks!

KaktoR

10-06-2023 00:27

Diablo IV
v1.0.2, incl. HD files

Code:

 10:50:35 - Selected ARC/DS method for Data1a-01.bin was: xtool:dd3+lzma2:d96m:fb273:lc4:mc1000000

 10:50:35 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzlib+xtool:mpreflate:dd3+lzma2:d96m:fb273:lc4:mc1000000

-----------------------------------------------------------------------------------------

 13:17:45 - Overall input size: 77.06 GB

 13:17:45 - Overall output size: 62.72 GB (Ratio 81.38%)

 13:17:45 - Overall conversion time: 02:27:08

All times are GMT -7. The time now is 08:02.

Page 234 of 312

234

Show 40 post(s) from this thread on one page