FileForums - Best Compression Methods for 'Specific' Games. Q&A

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

*.SPK from Steelrising are just zip files with what appears to be no compression. I cannot speak for the larger packs but the locales zips have data just stored.

Quote:

Originally Posted by Masquerade (Post 498249)

*.SPK from Steelrising are just zip files with what appears to be no compression. I cannot speak for the larger packs but the locales zips have data just stored.

Compressing datapc_initial.spk

Quote:

Compressed 1 file, 10,852,967,682 => 18,870,379,505 bytes. Ratio 173.87%
Compression time: cpu 9.08 sec/real 215.30 sec = 4%. Speed 50.41 mB/s
All OK

Files compressed zlib inside SPK

All SPK
66.5 gb > 116 gb.

Question.
datapchighres.spk - is optional data or not?

Quote:

Originally Posted by dixen (Post 498254)

Question.
datapchighres.spk - is optional data or not?

Nope, game is made in same engine as Greedfall. The file datapchighres.spk is basically the whole game data.

Quote:

Originally Posted by Masquerade (Post 498255)

Nope, game is made in same engine as Greedfall. The file datapchighres.spk is basically the whole game data.

Really? After all, the game works fine without these files. Especially on weak systems like mine

Quote:

Originally Posted by dixen (Post 498257)

Really? After all, the game works fine without these files. Especially on weak systems like mine

Really? I did not expect that. If you rip the highres file in Greedfall the game becomes unplayable

In this cases I always look on steamdb for optional depots.
If no info is provided then just rename the files/folders and play a bit and see if it works.

Have more tips, hints in recompress the old gamez, named in Space Empires V Hungarian Edition contains ~1GB's BMP files and 110+ MB's JPG files with WinXP compatibility setups. Use MSC this compression? Or any methods in better? Total game size: ~1.44GB

Quote:

Originally Posted by kj911 (Post 498264)

for bmp files use MSC or Internal Compressor (GRzip) and for jpg files use packjpg

Quote:

Originally Posted by Lord.Freddy (Post 498265)

for bmp files use MSC or Internal Compressor (GRzip) and for jpg files use packjpg

Where do I need to get this GRZip? It would be helpful if you posted this tool here for compression testing to use. :)

Quote:

Originally Posted by Lord.Freddy (Post 498265)

for bmp files use MSC or Internal Compressor (GRzip)

Isn't LOLZ better to use?

Caldric: This GRZIP compression algo its internally available from FreeArc! Use -mbmp switch from have compressing.

LOLZ its probably better. Quickly tested ~85MB's BMP files portion its 1GB's filepackage, from compare the compression. (x86 only compressors.)

Best: MSC+DLZ (better than MSC+LZMA combo.)
Default LZMA only its better than MSC packed one.*

*with 10-20% ratio from full input data, not packed in BMF. This MSC integrated BMF, not 100%-ish processing these bmp files within unsupported resolution issues.

LOLZ its testing in next days.

JPG compression:

My pretest in ~113MB's JPG files packing, get best results: "precomp043+srep:m3f:l32+lzp:3:16mb:h22" or replace precomp old to newest 0.4.8 version. This Precomp 0.4.8 (from KPS) ~4* faster in compression than released by profrager this v0.4.3 CLS-version.

Quote:

Originally Posted by Carldric Clement (Post 498269)

Where do I need to get this GRZip? It would be helpful if you posted this tool here for compression testing to use. :)

As kj911 said (GRzip) is a built-in compressor in FreeArc

Quote:

Originally Posted by Masquerade (Post 498273)

Isn't LOLZ better to use?

This is my test on my own (BMP) files:

Code:

This is For [BMP] files:



-----------------------------------------------------------------------------------------------------

Compression Method - 'mm+grzip:200mb:m1:l2048:h15:a'



FreeArc 0.67 (September 13 2014) creating archive: _OutPut\mm+grzip.arc

Compressed 22 files, 21,716,780 => 4,377,776 bytes. Ratio 20.16%

Compression time: cpu 6.69 sec/real 7.69 sec = 87%. Speed 2.82 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+grzip:200mb:m1:l2048:h15:a'



FreeArc 0.67 (September 13 2014) creating archive: _OutPut\msc+grzip.arc

Compressed 22 files, 21,716,780 => 3,525,547 bytes. Ratio 16.23%

Compression time: cpu 4.20 sec/real 106.58 sec = 4%. Speed 0.20 mB/s

All OK



[21,716,780 => 10,333,829 => 3,525,547]

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+lz-ma'



FreeArc 0.67 (September 13 2014) creating archive: _OutPut\msc+lzma.arc

Compressed 22 files, 21,716,780 => 3,689,764 bytes. Ratio 16.99%

Compression time: cpu 0.19 sec/real 105.72 sec = 0%. Speed 0.21 mB/s

All OK



[21,716,780 => 10,333,829 => 3,689,764]

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt2:mc1023'



FreeArc 0.67 (September 13 2014) creating archive: _OutPut\msc+lolz.arc

Compressed 22 files, 21,716,780 => 3,578,327 bytes. Ratio 16.48%

Compression time: cpu 0.16 sec/real 165.26 sec = 0%. Speed 0.13 mB/s

All OK



[21,716,780 => 10,333,829 => 3,578,327]

-----------------------------------------------------------------------------------------------------

Compression Method - 'lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt2:mc1023'



FreeArc 0.67 (September 13 2014) creating archive: _OutPut\lolz.arc

Compressed 22 files, 21,716,780 => 4,097,660 bytes. Ratio 18.87%

Compression time: cpu 0.20 sec/real 134.28 sec = 0%. Speed 0.16 mB/s

All OK

-----------------------------------------------------------------------------------------------------



MSC OutPut Console Window:

<--------------------------- B M P ---------------------------->

Total founded BMP: 22

BMP with not supported header value "bit_per_pixel": 9

BMP with not supported line size (should be line_size_bytes mod 4 = 0): 1

BMP filtered by size: 0



processed BMP: 12 (13'920'648 bytes)

How unpack / decrypt assets.pie from TMNT Cowabunga Collection?
It encrypted zip or not?

Quote:

Originally Posted by dixen (Post 498310)

How unpack / decrypt assets.pie from TMNT Cowabunga Collection?
It encrypted zip or not?

It's a zip file with no compression but encrypted. Currently looking into it.

https://i.ibb.co/B24fXfW/ex-tmnt.png

What is NBA2k23 using? Can't find any samples yet and don't want to download 122gb just for testing.

Quote:

Originally Posted by KaktoR (Post 498312)

What is NBA2k23 using? Can't find any samples yet and don't want to download 122gb just for testing.

Switch version uses multiple types of Oodle family, but can't comment on PC version. Sorry.

Quote:

Originally Posted by KaktoR (Post 498312)

What is NBA2k23 using? Can't find any samples yet and don't want to download 122gb just for testing.

Zlib

My test from BMP files portion the Space Empires V gamez:

This x86 ONLY compressors, its wins from ratio: MSC+DLZ
2nd place: MSC+BMP or MSC+GRZIP with any alterations.

This x64 ONLY or x86/x64 compressors, its wins from ratio: LOLZ
2nd place: MSC+LOLZ. Results ~200kB's smaller than MSC+DLZ combo.

Results:

Code:

This is For [BMP] files:



-----------------------------------------------------------------------------------------------------

Compression Method - 'msc'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test0.arc

Tested 164 files, 17,164,036 => 83,606,836 bytes. Ratio 20.53%

Testing time: cpu 109.84 sec/real 111.33 sec = 99%. Speed 0.75 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'bmp'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test1.arc

Tested 164 files, 14,300,345 => 83,606,836 bytes. Ratio 17.10%

Testing time: cpu 17.81 sec/real 19.20 sec = 93%. Speed 4.35 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+grzip:8388096b:m1:l2048:h15:a'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test2.arc

Tested 164 files, 12,463,714 => 83,606,836 bytes. Ratio 14.91%

Testing time: cpu 129.11 sec/real 130.56 sec = 99%. Speed 0.64 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+lzma'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test3.arc

Tested 164 files, 12,661,353 => 83,606,836 bytes. Ratio 15.14%

Testing time: cpu 126.25 sec/real 128.22 sec = 98%. Speed 0.65 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+lzma:a1:mfbt4:d72mb:fb273:lc8:mc1024'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test3-2.arc

Tested 164 files, 12,607,076 => 83,606,836 bytes. Ratio 15.08%

Testing time: cpu 126.55 sec/real 127.92 sec = 99%. Speed 0.65 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+lzp:64mb:3:h22:d128mb'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test4.arc

Tested 164 files, 15,073,522 => 83,606,836 bytes. Ratio 18.03%

Testing time: cpu 124.94 sec/real 126.44 sec = 99%. Speed 0.66 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+grzip:8388096b:m1:l2048:h15:a+lzp:64mb:3:h22:d128mb'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test5.arc

Tested 164 files, 12,462,437 => 83,606,836 bytes. Ratio 14.91%

Testing time: cpu 131.30 sec/real 135.17 sec = 97%. Speed 0.62 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+dlz:slp:d64m:lc8:lp0:fb337:mc8192'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test6.arc

Tested 164 files, 12,410,208 => 83,606,836 bytes. Ratio 14.84%

Testing time: cpu 114.17 sec/real 115.78 sec = 99%. Speed 0.72 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+grzip:8388096b:m1:l2048:h15:a+dlz:slp:d64m:lc8:lp0:fb337:mc8192'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test7.arc

Tested 164 files, 12,464,540 => 83,606,836 bytes. Ratio 14.91%

Testing time: cpu 122.03 sec/real 123.52 sec = 99%. Speed 0.68 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'msc+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mt4:d160m:fba4096:tt16:mc1023'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test8.arc

Tested 181 files, 12,309,746 => 83,606,836 bytes. Ratio 14.72%

Testing time: cpu 115.23 sec/real 117.13 sec = 98%. Speed 0.71 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mt4:d160m:fba4096:tt16:mc1023'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test9.arc

Tested 181 files, 11,111,413 => 83,606,836 bytes. Ratio 13.29%

Testing time: cpu 0.16 sec/real 3.45 sec = 5%. Speed 24.21 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'lzma'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test10.arc

Tested 164 files, 14,359,017 => 83,606,836 bytes. Ratio 17.17%

Testing time: cpu 2.39 sec/real 2.47 sec = 97%. Speed 33.87 mB/s

All OK

-----------------------------------------------------------------------------------------------------

Compression Method - 'lzma:a:72mb:normal:bt4:273:mc1024:lc8'



FreeArc 0.67 (March 15 2014) testing archive: bmp_test11.arc

Tested 164 files, 13,281,704 => 83,606,836 bytes. Ratio 15.89%

Testing time: cpu 2.41 sec/real 2.67 sec = 90%. Speed 31.29 mB/s

All OK

-----------------------------------------------------------------------------------------------------



MSC OutPut Console Window:

<--------------------------- B M P ---------------------------->

Total founded BMP: 142

BMP with not supported header value "bit_per_pixel": 0

BMP with not supported line size (should be line_size_bytes mod 4 = 0): 6

BMP filtered by size: 0



processed BMP: 136 (77'310'176 bytes)

Quote:

Originally Posted by kj911 (Post 498322)

Can you use this compressor?

Code:

[External compressor:bliz]

cmem      = 600

dmem      = 600

default   = 100

packcmd   = "Blizzard\bliz.exe" c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp {option}000000

unpackcmd = "Blizzard\bliz.exe" d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

Lord.Freddy: Nothing. I haven't met him yet, the Blizzard compressor. Testing now in next hours/days.

DLZ = (D)ds modeled (LZ)ma v0.2.2 by profrager

Quick tests: 2* Faster than LZMA and comparable ratio.

Code:

Tested 164 files, 13,760,715 => 83,606,836 bytes. Ratio 16.46%

Testing time: cpu 0.16 sec/real 29.28 sec = 1%. Speed 2.86 mB/s

Tested again, packed all files in one ARC file, in store mode and packing in Blizzard. Small, default blocksize its better!?

Code:

bliz.exe c bmp_store.arc bmp_.blz 300000000  ---> 13 762 777 byte

bliz.exe f bmp_store.arc bmp_.blz 300000000  ---> 13 963 348 byte

bliz.exe f bmp_store.arc bmp_.blz  ---> 13 604 705 byte

bliz.exe c bmp_store.arc bmp_.blz ---> 13 396 429byte

Quick note: Best results get in use 8MB blocksize (smaller/bigger, worse results): 13 369 598 byte

While we on this discusion right now, what's a good method for text files btw? Such as txt, xml, ini, etc.

Quote:

Originally Posted by KaktoR (Post 498327)

While we on this discusion right now, what's a good method for text files btw? Such as txt, xml, ini, etc.

Code:

----‐--------‐-------------‐--------‐-------------‐--------‐---------

1) ' nanozip:cc ' => Ultra

2) ' bsc ' => maximum

3) ' dict:p+lzp+ppmd:20:500mb:r1 ' => High

----‐--------‐-------------‐--------‐-------------‐--------‐---------

Good evening, I've extracted a .archive file and this is composed of several .bin files. Which is the best method to compress these .bin files? I've tried with precomp v0.4.7, then with xzlib, with no compression results at all. Is there a specific compressor or a mix of them? Thank you!

You should know what these bin files include.

The Death

srep+lolz

7.4GB ----> 2.5GB

Quote:

Originally Posted by KaktoR (Post 498331)

You should know what these bin files include.

Thank you for your reply! The .archive file i've extracted is a file from Cyberpunk 2077 and it's a language file, so I think .bin files inside are audio file or sound file. Do you think I have to use MSC or similar tools to compress them?

Quote:

Originally Posted by Dragonis40 (Post 498344)

It's WWise Vorbis

Quote:

Originally Posted by Masquerade (Post 498346)

It's WWise Vorbis

You mean I should compress .bin files with "How to correctly process WWise Vorbis Audio | Guide + WemTool" guide?

1GB's BMP compression test:

Code:

Original archive: bmp.arc (244,937,914 byte)

Methods: lzma:192mb:normal:bt4:128:lc8

Packed from grzip (with -mgrzip:256mb:h15:l2048:a):

Code:

5,483 files, 1,069,598,818 bytes, 320,784,780 compressed (not solid!)

tempfile+grzip..: 1,069,598,818 bytes, 321,101,123 compressed (solid!)

Packed from DLZ: 239,147,583 byte, Methods: -slp -d64m -lc8 -lp0 -fb337 -mc8192

Packed from bliz:

Code:

100m bytes dict. size: 258,506,104 byte

with 4096kB Dict size: 270,019,193 byte

tempfile+mm+bliz: 279,418,278 byte

mm+tempfile+bliz: 289,750,196 byte (not solid archive!)

Packed from MSC (BMF 2.0.1 with slow mode, bmf=9s) VERYSLOW:

Code:

Compressed 5,483 files, 1,069,598,818 => 340,840,020 bytes. Ratio 31.87%

Compression time: cpu 108.97 sec/real 17445.55 sec = 1%. Speed 0.06 mB/s

BMF Results:

Code:

<--------------------------- B M P ---------------------------->

Total founded BMP: 4549

BMP with not supported header value "bit_per_pixel": 0

BMP with not supported line size (should be line_size_bytes mod 4 = 0): 132

BMP filtered by size: 0



processed BMP: 4417 (919'082'710 bytes)

Packed again the MSC/BMF-packed previous archive:

Code:

7z/LZMA: 232 253 503 byte

DLZ only: 228 604 204 byte

Note: DLZ its finished in ~41 mins from compression. BLIZ its faster.

and any compressors:

RAZOR 1.03.7 x64 with 512m Dict. size: 195 443 238 byte

LOLZ v22c4b (Brute detect disabled!):

Code:

default lolz settings (ldmf1, 64MB dict ...): 187,784,213 byte

lolz:dt:mtt1:mt4:d160m:fba4096:tt16:mc1023 -> 187,109,257 byte

All compression tests, except RZ/LOLZ use only from very old x86 PC with 1GB RAM. RZ/LOLZ used x64 laptop with 8GB RAM.

I belive the *.ggpack* archives in Return To Monkey Island are XOR encrypted. None of the current keys from this list appear to be correct.

I'm very new to lzo1x pre-compression, could anyone explain why XTool in this example is able to detect some streams but not able to process any? I've tried levels 1 - 10 and the same amount of streams are detected but none are able to be processed

Code:

Creating Archive Please Wait...



FreeArc 0.67 (March 15 2014) updating archive: Data.arc

Compressing 1 file, 38,255,712 bytes. Processed   0%

Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 0/2854

Time: 00:00:01 (00:00:03)

Memory: 128 MB (128 MB)



100%

Errorlevel=0

Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%

Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s

All OK



All done.

The algorithm used is definitely lzo1x, and creating a database with bms2xtl detects an additional ~5000 streams but alas none are detected. I've also not been able to find many other lzo libraries to test with so if its a library issue, if anyone could point me in the direction to look for others it'd be greatly appreciated.

Quote:

Originally Posted by L33THAK0R (Post 498393)

Code:

Creating Archive Please Wait...



FreeArc 0.67 (March 15 2014) updating archive: Data.arc

Compressing 1 file, 38,255,712 bytes. Processed   0%

Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 0/2854

Time: 00:00:01 (00:00:03)

Memory: 128 MB (128 MB)



100%

Errorlevel=0

Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%

Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s

All OK



All done.

maybe you need to remove the level option, sometimes it works.

Quote:

Originally Posted by Lord.Freddy (Post 498395)

maybe you need to remove the level option, sometimes it works.

Unfortunately seems like some data actually got stripped away doing that, resulted in a 99.51% ratio :(

Quote:

Originally Posted by L33THAK0R (Post 498393)

Code:

Creating Archive Please Wait...



FreeArc 0.67 (March 15 2014) updating archive: Data.arc

Compressing 1 file, 38,255,712 bytes. Processed   0%

Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 0/2854

Time: 00:00:01 (00:00:03)

Memory: 128 MB (128 MB)



100%

Errorlevel=0

Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%

Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s

All OK



All done.

Maybe used pro-functions. Upload example file for testing

Quote:

Originally Posted by Edison007 (Post 498398)

Maybe used pro-functions. Upload example file for testing

Thanks for offering to have a looksie, really appreciate any insight you can offer, also in case its of any interest I've also got an LZ4 sample I've been having similar issues with, tried about a dozen libraries but no dice so far, xtool (0.6.5) is also detecting streams but unable to pick any up unfortunately.

For your lz4 sample you have to use lz4f with level 2 and pick the right lz4 library

Code:

Compressed 1 file, 32,882,867 => 158,560,939 bytes. Ratio 482.20%

Compression time: cpu 0.08 sec/real 1.18 sec = 7%. Speed 27.85 mB/s

All OK

You have to look which level produces the closest recompressed size to the original size

https://i.imgur.com/xXoiZtm.png

However in a quick test there are still some streams left behind.

Quote:

Originally Posted by KaktoR (Post 498403)

For your lz4 sample you have to use lz4f with level 2 and pick the right lz4 library

Code:

Compressed 1 file, 32,882,867 => 158,560,939 bytes. Ratio 482.20%

Compression time: cpu 0.08 sec/real 1.18 sec = 7%. Speed 27.85 mB/s

All OK

You have to look which level produces the closest recompressed size to the original size

https://i.imgur.com/xXoiZtm.png

However in a quick test there are still some streams left behind.

Ah didnt realise I had to specify lz4f, many thanks, also what tool are you using to get that console output if you dont mind me asking?

--verbose mode from xtool

Quote:

lzo_sample.rar

The old function LZO1X-1 was used. You can use these libraries. But not all streams are successfully recompressed, because lzo-detect in xtool is't ideal.

Quote:

Originally Posted by Edison007 (Post 498410)

The old function LZO1X-1 was used. You can use these libraries. But not all streams are successfully recompressed, because lzo-detect in xtool is't ideal.

Just ran a quick test, this works wonderfully, thanks so much! I've had this sitting on the back-burner for months, it's very satisfying to have it finally sorted!