FileForums - Best Compression Methods for 'Specific' Games. Q&A

Page 67 of 119

« First

117

Last »

Show 40 post(s) from this thread on one page

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

KaktoR

12-11-2021 10:06

Quote:

Originally Posted by Masquerade (Post 494818)

ENGLISH ONLY! All unneeded language files + deluxe movies ripped.

Start size: 51.3GB

Compression method:

Code:

oo2recm+xtool_zlib+srep+lolz

Do not use XTool chunk size of greater than 128mb otherwise it will get stuck during compression. (tested on my 64GB, 16 thread PC). XTool v0.12 used. Thanks to Razor12911 for the tools.

Final Size: 37.9GB

Code:

 13:03:18 - Selected ARC/DS method for Data1a-01.bin was: oo2recm8+xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 13:03:18 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

----------------------------------------------------------------------------------------------------------------------------------------

 19:01:31 - Overall input size: 51.51 GB

 19:01:31 - Overall output size: 44.30 GB (Ratio 86.01%)

 19:01:31 - Data files creation time: 05:31:39

Just 200mb for lolz, bad.

Even worse that with xtool applied the size is bigger then my last repack with just oo2recm https://fileforums.com/showpost.php?...postcount=2136

KaktoR

12-11-2021 11:32

They are Billions
v1.1.4

Code:

 19:48:52 - Selected ARC/SPLIT method for Data1a-01.bin was: srep_new+lolz:d64:mtt1:mt12:mc1023/$storepack=FA-MIN/$libpack=srep_new+razor_mtx/$oggretheyarebillions=oggre/$deflatetheyarebillions=xt_zlib+srep_new+lolz:d64:mtt1:mt12:mc1023

----------------------------------------------------------------------------------------------------------------------------------------

 20:30:49 - Overall input size: 5.16 GB

 20:30:49 - Overall output size: 1.84 GB (Ratio 35.70%)

 20:30:49 - Overall conversion time: 00:41:53

KaktoR

12-11-2021 11:59

Battlefield 2042 test

Code:

srep+4x4:lzma

Overall input size: 246.54 MB

Overall output size: 55.90 MB (Ratio 22.67%)



oo2reck+srep+4x4:lzma (oo2reck takes very long)

Overall input size: 246.54 MB

Overall output size: 54.91 MB (Ratio 22.27%)



xtool:kraken+srep+4x4:lzma

Overall input size: 246.54 MB

Overall output size: 54.94 MB (Ratio 22.28%)

KaktoR

12-11-2021 15:39

Skyrim Anniversary Edition test

Code:

Compressed 15 files, 9,955,981,643 => 16,269,399,549 bytes. Ratio 163.41%

Compression time: cpu 11.56 sec/real 320.70 sec = 4%. Speed 31.05 mB/s

All OK



Extracted 15 files, 16,269,399,549 => 9,955,981,643 bytes. Ratio 163.41%

Extraction time: cpu 17.92 sec/real 271.19 sec = 7%. Speed 36.71 mB/s

All OK



Data\Skyrim - Animations.bsa

Data\Skyrim - Interface.bsa

Data\Skyrim - Meshes0.bsa

Data\Skyrim - Meshes1.bsa

Data\Skyrim - Misc.bsa

Data\Skyrim - Shaders.bsa

Data\Skyrim - Textures0.bsa

Data\Skyrim - Textures1.bsa

Data\Skyrim - Textures2.bsa

Data\Skyrim - Textures3.bsa

Data\Skyrim - Textures4.bsa

Data\Skyrim - Textures5.bsa

Data\Skyrim - Textures6.bsa

Data\Skyrim - Textures7.bsa

Data\Skyrim - Textures8.bsa

Working bsa plugin https://fileforums.com/showpost.php?...&postcount=223
Use lz4 v1.8.3 library

PS: t100p stuck for me, so use a lower thread usage for xtool

kj911

14-11-2021 01:13

Microsoft Train Simulator *.ACE files:

older:

Quote:

xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt32:fba4096:mc1023

newest:

Quote:

xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt1:oh14:gm01:lm4:fba4096:mc1023

Green its optional switches, probably testing any game. Defaults, its 00, 11 or 21 worse results than 00.
Red the magic switch! Nice compression gain! And overall usage all physical RAM! (8GB and 87% usage vs. 97-99%)
oh14 its minimally better compression. Defaults: oh12
tt1 its very fast mode, max usage: tt16 or tt32!! Higher values not better and extremely slow compression.

Difference? Old version 2.93GB! New, with tt1 (compress again with tt16 and d160m switches in last time)* 2.80GB!! ~5% compression rate gain.

UPDATE: Aaarrgh! The test package don't decompresseable! CLS-ini error or any tips?

*tt32 its probably very best results. Calculated compression time avg. ~2days! Overkill.

Cesar82

16-11-2021 06:19

Question about LOLZ

Could someone familiar with LOLZ parameters help me with some questions?

How is actual memory usage calculated using for multi thread used An X value for dictionary (-d)?

Considering that the supported values for dictionary are from 0 to 2047 "-d[0..2047]): 64m". If you enter a value above 2047 (such as 2050) will the default value be set to the dictionary (64m) or will the maximum value (2047m) be used?

I need this information because I created presets for LOLZ in DiskSpan GUI.
The memory and thread information will be obtained when starting the DiskSpan GUI and then starting the compression will be calculated based on the threads and RAM of the system.
It will also be informed the number of system theads replaced <thread> by the number of virtual cores (threads) of the current CPU during the conversion.
So one more question arises...
In the lolz documentation it states that the supported values are 1 to 16 thread: "mt threads(-mt[1..16]): 1"
If using a Ryzen processor 9, 5900x is 24 threads, so if you use -mt24, lolz will set it to 1 (default) or to 16 (Maximum supported).

Below is the example where it will be in LOLZ presets in DiskSpan GUI.

Code:

LOLZ_HIGH = lolz:dtb1:d<expr:0.65*memory/(thread+1)/1048576>m:mtt1:mt<thread>:mc1023



Expression example: <expr:0.65*memory/(thread+1)/1048576>



0.65    = percentage of calcule (65%)

memory  = full system memory (RAM)

thread  = number of logical cores

core    = number of Physical cores

1048576 = convert result in bytes to mb

If someone can help me with these questions, I would appreciate it!

kj911

16-11-2021 08:47

Its calculated result correct?? See the linked picture from my friend laptop specifications. --> https://i.kek.sh/zUjUwL6MVZE.jpg

Example: Intel Core I3-1005G1 CPU, Win10 x64 with 8GB memory. (Total cores: 2, Total Threads: 4)

Available physical RAM 8GB --> 8589934592 byte

PreCalculate: 0.65*8589934592 = 5583457484,8
Cores --> 4+1 = 5
Final calculate: 5583457484,8/5/1048576 = 1064.96 MB?? Try it added extra /8 values? 132MB values its really.

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

Cesar82

16-11-2021 15:35

1 Attachment(s)

Quote:

Originally Posted by kj911 (Post 494938)

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

For your system it would result as if using from the command line:

Code:

LOLZ_HIGH = lolz:dtb1:d1065m:mtt1:mt4:mc1023



So lolz would use: 4 * 1065 = 4260 MB of 8192 MB.

I believe the memory usage of lolz is calculated as: theads*dictionary

Example: If using a Ryzen 5900x (24 theads) and 32 GB of RAM we would have:

Code:

LOLZ_HIGH = lolz:dtb1:d852m:mtt1:mt24:mc1023



So lolz would use: 24 * 852 = 20448 MB of 32768 MB.

It may be recommended to do the calculation using available memory rather than total system memory.

Maybe it's not just simply multiplying the dictionary by the number of threads, that's my question.
Can not understand. See the attached video.

its_gnar_again

16-11-2021 18:22

Has anyone here had any luck getting Final Fantasy X/X-2 HD to decompress without any issues? The method on here is basically xtool(xzlib)+srep+lolz, but that ends up with a corrupted archive error during decompression not even half-way. I have tried the following:

Tried different configs for LOLZ
Tried different configs for SREP
Tried outright replacing LOLZ with LZMA
Tried various versions of XTool and ZTool, and different configs for them as well.

I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

KaktoR

17-11-2021 09:39

Hellblade: Senua's Sacrifice Enhanced
Video documentary ripped

Code:

 17:51:27 - Selected ARC/DS method for Data1a-01.bin was: srep_new+LZMA-MT

 17:51:27 - Selected ARC/DS method for Data1b-01.bin was: bpk

----------------------------------------------------------------------------------------------------------------------------------------

 18:38:43 - Overall input size: 17.25 GB

 18:38:43 - Overall output size: 8.25 GB (Ratio 47.84%)

 18:38:43 - Data files creation time: 00:38:17

KaktoR

17-11-2021 11:48

Quote:

Originally Posted by its_gnar_again (Post 494944)

Tried different configs for LOLZ
Tried different configs for SREP
Tried outright replacing LOLZ with LZMA
Tried various versions of XTool and ZTool, and different configs for them as well.

I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

xtool:reflate works just fine for me.

Code:

[External compressor:xtool]

header    = 0

default   = -c64mb -t75p

packcmd   = "XTool.exe" precomp { -option} - - <stdin> <stdout>

unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

its_gnar_again

17-11-2021 17:06

Quote:

Originally Posted by KaktoR (Post 494957)

xtool:reflate works just fine for me.

Code:

[External compressor:xtool]

header    = 0

default   = -c64mb -t75p

packcmd   = "XTool.exe" precomp { -option} - - <stdin> <stdout>

unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

My xtool setup is pretty much the same from when I last tested it.

Quote:

[External compressor:xtool]
header = 0
packcmd = XT\xtool.exe precomp -mzlib -c64mb -t75p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

I haven't tested JUST xtool only to see if that works, but I will get back to you here from when I do. I don't understand what the problem is on my part really.

Grumpy

17-11-2021 22:22

@its_gnar_again

Quote:

forgot the pass & email I used on my old account lmao

Surely you remember one of them, you've had 4 or 5 accounts since 2020 :eek:

its_gnar_again

18-11-2021 19:05

Quote:

Originally Posted by its_gnar_again (Post 494959)

I haven't tested JUST xtool only to see if that works, but I will get back to you.

UPDATE: I tried specifically using reflate on xtool+srep+lolz, and still get a decompression error. However, I did try using xtool only and it decompressed just fine...what could honestly be the issue? I would like to get this figured out. :confused:

KaktoR

19-11-2021 00:32

Maybe check your CLS.ini settings for lolz and srep.

And make sure you're using the correct cls files for lolz and srep

Edison007

19-11-2021 08:10

Quote:

How is actual memory usage calculated using for multi thread used An X value for dictionary (-d)?

[dictionary*~11 + (mtb*2) + (memory_for_models)] * threads

frankiedylan

19-11-2021 15:07

ZLib and LolZ are probably the best. I used them on my old projects back in the day and they performed well.

Cesar82

19-11-2021 17:42

Quote:

Originally Posted by Edison007 (Post 494977)

[dictionary*~11 + (mtb*2) + (memory_for_models)] * threads

Thanks for the info.
Don't understand about "memory_for_models"?
You used '"[' and "]", have the same meaning as parentheses for the calculation?
It depends on what parameters?

In these presets below what would the calculation be, give me an example so I can understand.

Code:

LOLZ_FAST   = lolz:d32m:mtt1:mt<thread>:fba4096:mc2;

LOLZ_NORMAL = lolz:d64m:mtt1:mt<thread>:mc1023;

LOLZ_HIGH   = lolz:dtb1:d128m:mtt1:mt<thread>:mc1023;

LOLZ_ULTRA  = lolz:dtb1:d256m:mtt1:mt<thread>:mc1023

Thread is detected by DSG.
Consider as an example on a 24 thead cpu (I think lolz trunc to 16).

-----------------
This calculation is correct (This without adding the use "memory_for_models" which I didn't understand)?
To CPU 5900x (24 threads) lolz trunc to 16 threads
lolz default mtb is 32m

Code:

lolz:dtb1:d256m:mtt1:mt16:mc1023;

calc: (256m*11 + (32m*2) + (?)) * 16
(2816m + 64m + ?) * 16
2880m * 16 = 46080m

Edison007

20-11-2021 05:08

Quote:

You used '"[' and "]", have the same meaning as parentheses for the calculation?

Yes, priority of math operations)

Quote:

calc: (256m*11 + (32m*2) + (?)) * 16
(2816m + 64m + ?) * 16
2880m * 16 = 46080m

Yep, but the block size will be - 256 mb.

Quotes translated with Google Translate.

Quote:

-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, so the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;

Quote:

lolz test19g
- in the -mtt1 mode, when compressing, the block and dictionary sizes are aligned with each other according to the larger of them;

I don’t remember the required memory for the models. But this is a trifle

Cesar82

20-11-2021 07:13

Thank you very much Edison007.

Quote:

Originally Posted by Edison007 (Post 494985)

I think the problem was this. I was using default value for block size and was using value greater than 512 for dictionary using mtt1.

PS: I noticed that if you compress as small as 300MB the value set for dictionary in console output during compression is smaller than if you compress as much as 30GB. Dictionary appears to be input size dependent.

I will run some tests and see the results.
Thanks!

Edison007

20-11-2021 07:50

Quote:

PS: I noticed that if you compress as small as 300MB the value set for dictionary in console output during compression is smaller than if you compress as much as 30GB. Dictionary appears to be input size dependent.

Another quote from changelog:

Quote:

lolz test19c
- added reduction of the dictionary and the block to an acceptable size if the input file is smaller than the sizes of these parameters;

kj911

21-11-2021 10:09

MSC Questions: Use frog=9 switches equal to --bestnew or any OFR switches??

Example: chimes.wav file in 216 364 byte

MSC with frog=9 switches: 36 455 byte (internally from ARC archive: 36 233 byte)
Directly packing OFR with use --maximumcompression switches: 35 608 byte

Edison007

21-11-2021 11:09

kj911
MSC version 0064 (ofr 4.910b)
ArrOfFROGParams:array [1..9]of PAnsiChar=('
--mode fast --experimental --seek min --optimize best',
'--mode normal --experimental --seek min --optimize best',
'--mode high --experimental --seek min --optimize best',
'--mode extra --experimental --seek min --optimize best',
'--mode best --experimental --seek min --optimize best',
'--mode highnew --experimental --seek min --optimize best',
'--mode extranew --experimental --seek min --optimize best',
'--mode bestnew --seek min --optimize best',
'--maximumcompression --experimental --seek min --optimize best');

+

https://krinkels.org/threads/msc-med...e-7#post-33503

dixen

22-11-2021 05:49

What about *.shapes from Farming Sim 2022? Encrypt files?

Masquerade

22-11-2021 10:47

Quote:

Originally Posted by dixen (Post 495002)

What about *.shapes from Farming Sim 2022? Encrypt files?

Likely: https://zenhax.com/viewtopic.php?t=4030

dixen

27-11-2021 11:45

Hi, my friends)
What about decrypt/decompress *.FF from CoD BO 2?)

KaktoR

04-12-2021 02:09

Chorus
Do not use binkpack on video files. Again bpk fails because of 4k resolution, used codec or something else. RIP

Masquerade

04-12-2021 08:48

Grand Theft Auto III Definitive Edition

Grab oo2core from game files and rename from oo2core_8_win64.dll to oo2core_7_win64.dll.

Code:

oo2reck+srep+lolz

4.2GB ----> 2.9GB

Masquerade

05-12-2021 01:03

Grand Theft Auto Vice City Definitive Edition

Grab oo2core from game files and rename from oo2core_8_win64.dll to oo2core_7_win64.dll.

Code:

oo2reck+srep+lolz/$mp4=srep

8.7GB ----> 6.2GB (no credits video) / 6.3GB (with credits video)

Masquerade

05-12-2021 12:46

Grand Theft Auto San Andreas Definitive Edition

Grab oo2core from game files and rename from oo2core_8_win64.dll to oo2core_7_win64.dll.

Code:

oo2reck+srep+lolz/$mp4=srep

17.8GB ----> 13GB (no credits video) / 13.1GB (with credits video)

KaktoR

06-12-2021 04:38

Chorus

Code:

 12:00:47 - Selected ARC/DS method for Data1a-01.bin was: xt_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 12:00:47 - Selected ARC/DS method for Data1b-01.bin was: 1

 12:00:47 - Selected ARC/DS method for Data1c-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

----------------------------------------------------------------------------------------------------------------------------------------

 13:37:04 - Overall input size: 31.52 GB

 13:37:04 - Overall output size: 24.28 GB (Ratio 77.03%)

 13:37:04 - Overall conversion time: 01:36:12

For video files do not use bpk (crc error). Just use m1 or something similar.

dixen

08-12-2021 14:13

Halo Infinite

*.module - oodle algorithm (kraken)

Decompressed fine with Xtool v0.39 + oo2core_8_win64.dll
28 gb > 60 gb

jihack

09-12-2021 04:22

Hi! I'm new at game conversion. Is anyone know the best method to compress Sea of Thieves? And any tips for me as I am a beginner?

KaktoR

09-12-2021 04:34

@jihack
Maybe precompress with zlib because of UE4 engine (most of cases).
Could be that the files are encrypted.

Send me a sample .pak file

Decryption key should be 0x37A0BC3DC2E01D9EB4923CA266A5701F56A4802347F07927 FC3FC25C93B31B50

jihack

09-12-2021 04:41

Quote:

Originally Posted by KaktoR (Post 495118)

Yep I tried with some *.pak files with that eas key from rin but the ratio still 90%+. The tool I use is DiskPan GUI from Cesar82. I'll send the .pak file in pm.

jihack

09-12-2021 05:02

Quote:

Originally Posted by KaktoR (Post 495118)

Well, here's the paks file https://drive.google.com/file/d/1A2y...ew?usp=sharing
Bcs I can't pm anyone

KaktoR

09-12-2021 05:41

Maybe there is just "bad" data inside the files, hence the worse ratio.

KaktoR

09-12-2021 07:47

Halo Infinite

Code:

 12:29:36 - Selected ARC/DS method for Data1a-01.bin was: oo2reck:8+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 12:29:36 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

----------------------------------------------------------------------------------------------------------------------------------------

 16:44:16 - Overall input size: 48.00 GB

 16:44:16 - Overall output size: 36.37 GB (Ratio 75.77%)

 16:44:16 - Overall conversion time: 04:14:37

Data1a -> *.module, *.module_hd1
Data1b -> rest of the files

PS: You can rip languages from movie files
Academy_Tutorial_Intro.mp4 407mb -> 305mb with one language

KaktoR

09-12-2021 14:27

Horizon Zero Dawn Complete Edition
v1.11, one language

Code:

 19:32:12 - Selected ARC/DS method for Data1a-01.bin was: bpk

 19:32:12 - Selected ARC/DS method for Data1b-01.bin was: oo2reck:3+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

----------------------------------------------------------------------------------------------------------------------------------------

 23:23:00 - Overall input size: 57.77 GB

 23:23:00 - Overall output size: 41.74 GB (Ratio 72.25%)

 23:23:00 - Data files creation time: 03:21:54

Exclude the following files from bpk

Code:

MQ1_Intro_at_the_Hovel.bk2

mq02_prepare.bk2

MQ06_07_DeathAndRebirth.bk2

MQ16_Epilogue1.bk2

MQ16_Victory.bk2

MQ030_YearsOfTraining.bk2

MQ050_ProvingCllimb.bk2

Dark0226

10-12-2021 02:16

What algorithms should be used to process SpellForce 3: Reforced?

All times are GMT -7. The time now is 22:56.

Page 67 of 119

« First

117

Last »

Show 40 post(s) from this thread on one page