FileForums - Best Compression Methods for 'Specific' Games. Q&A

Page 83 of 119

« First

Last »

Show 40 post(s) from this thread on one page

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

KaktoR

17-11-2022 11:34

Is anyone here working with xtool's --dedup option?

I made many tests since yesterday and I think this will replace srep command entirely, at least for me.

Code:

xtool+srep+4x4:lzma

Compressed 27 files, 4,594,892,800 => 2,346,970,567 bytes. Ratio 51.08%

Compression time: cpu 3.69 sec/real 921.49 sec = 0%. Speed 4.99 mB/s

All OK



15min 23sec

Code:

xtool+4x4:lzma

Compressed 27 files, 4,594,892,800 => 2,346,861,976 bytes. Ratio 51.08%

Compression time: cpu 2.69 sec/real 660.86 sec = 0%. Speed 6.95 mB/s

All OK



11min 02sec

--dedup=(1..5)
srep.exe must be present next to xtool.exe for compression and decompression (no cls-srep required)

KaktoR

18-11-2022 04:40

Here is another test. You can use xtool without any precompression too.

Code:

xtool+4x4:lzma

Compressed 7,383 files, 31,661,584,708 => 8,601,282,118 bytes. Ratio 27.17%

Compression time: cpu 16.39 sec/real 1506.53 sec = 1%. Speed 21.02 mB/s

All OK



28min

Code:

srep+4x4:lzma

Compressed 7,383 files, 31,661,584,708 => 8,601,527,510 bytes. Ratio 27.17%

Compression time: cpu 24.27 sec/real 2312.41 sec = 1%. Speed 13.69 mB/s

All OK



38min

Code:

[External compressor:xtool]

header    = 0

default   = -c64mb -t100p

packcmd   = "{compressor}.exe" precomp { -moption} --dbase --dedup=3 - - <stdin> <stdout>

unpackcmd = "{compressor}.exe" decode -t100p - - <stdin> <stdout>

KaktoR

18-11-2022 07:54

Crysis 2 Remastered

Code:

 14:52:34 - Selected ARC/DS method for Data1a-01.bin was: oggre

 14:52:34 - Selected ARC/DS method for Data1b-01.bin was: xtool+4x4:lzma

 14:52:34 - Selected ARC/DS method for Data1c-01.bin was: xtool:mcrysis2remastered+4x4:lzma

 14:52:34 - Selected ARC/DS method for Data1d-01.bin was: xtool+4x4:lzma

-------------------------------------------------------------------------------------------------------------------------------------------

 16:53:20 - Overall input size: 53.76 GB

 16:53:20 - Overall output size: 34.79 GB (Ratio 64.72%)

 16:53:20 - Overall conversion time: 02:00:44



Data1a

GameCrysis2\music.pak

GameCrysis2\music_chk0.pak



Data1b

GameCrysis2\Video*.pak



Data1c

*.pak (excl 1a+1b)



Data1d

rest of the files

Thanks to Masquerade for database file

Masquerade

18-11-2022 13:57

For Spiderman Miles Morales, use XTool Spidey plugin with liblz4 v1.9.4 to get all streams.

KaktoR

18-11-2022 14:43

Edit: Have mixed up with another title :)

fabrieunko

18-11-2022 23:57

Quote:

Originally Posted by Masquerade (Post 499149)

For Spiderman Miles Morales, use XTool Spidey plugin with liblz4 v1.9.4 to get all streams.

thank you very much but where to find?

Masquerade

19-11-2022 00:20

Quote:

Originally Posted by fabrieunko (Post 499152)

thank you very much but where to find?

https://github.com/lz4/lz4/releases/tag/v1.9.4

C'mon man, just use Google

fabrieunko

19-11-2022 09:39

^^ I had used google and found the same link but I was not sure and I did not understand how to replace the dll...

KaktoR

19-11-2022 10:39

"msys-lz4-1.dll" is the file you need. Rename it to "liblz4.dll".

KaktoR

20-11-2022 04:54

Anyone knows how to deal with OGV video files?

L33THAK0R

20-11-2022 05:37

Quote:

Originally Posted by KaktoR (Post 499161)

Anyone knows how to deal with OGV video files?

I've had very hit or miss results with Theora (ogv) using Oggre, some, even from the same title and encoded with the same library will fail to decompress but others are able to decompress just fine, if I were a betting man I'd wager it might have something to do with the size of the Vorbis streams within a given OGV container but thats just a guess.

Masquerade

20-11-2022 05:39

Oggre will always fail with ogv in every time I've tested it. I usually just use razor or lzma on them.

shazzla

20-11-2022 07:42

@Kaktor : maybe ffmpeg can help you. Or not. Just an idea...

kj911

21-11-2022 06:54

@Kaktor: Try it use Masqueraqde posted compressors or combine with Theora rehuffing+any compressors if the fact that the rehuffed OGVs are not a problem, we get them in a non-lossless way at the output of the program. Nothing else is out of the question.

KaktoR

21-11-2022 07:30

Tbh I like it lossless without any recoding or something else like this. So I will stick with lzma2.

Snake288

21-11-2022 07:36

Hello

i'm getting an error in the institution, why is it caused?

https://i.hizliresim.com/6sm46ue.jpg

-----------------------------------------------------------------------------------------
-------------------------------- CONVERSION DATE 20.11.2022 --------------------------------
-----------------------------------------------------------------------------------------
21:08:34 - You have chosen the DVD5 type for the media size.
21:08:34 - ISO images will not be created when conversion is complete.
21:08:34 - The computer will remain on when the conversion is complete.
21:08:34 - Selected ARC/DS method for Data1a-01.bin was: xtool:c32mb:mspidey+srep_new:1+lolz
-----------------------------------------------------------------------------------------
21:08:35 - Compression initialized in 20.11.2022 at 21:08:35.
21:08:35 - Starting Data1a-01.bin archive creation.
17:28:42 - Moving Data1a-01.bin » Data1a-07.bin files to media folder.
17:28:44 - Compression finished in 21.11.2022 at 17:28:44.
-----------------------------------------------------------------------------------------
17:28:44 - Getting size of the game: Marvel’s Spider-Man Miles Morales
17:28:44 - » Installation folder size: 47,41 GB
17:28:44 - » Compressed game size: 27,49 GB (Ratio 57.99%)
-----------------------------------------------------------------------------------------
17:28:44 - Overall input size: 47,41 GB
17:28:44 - Overall output size: 27,49 GB (Ratio 57.99%)
17:28:44 - Overall conversion time: 20:20:08
17:28:44 - Overall conversion discs: 7 DVD5
-----------------------------------------------------------------------------------------

KaktoR

24-11-2022 08:24

Evil West

For bink2 files use bpk patch method.

15.9 GB -> 14.3 GB
(diff size = 228 MB is included)

Edit:

Code:

 17:28:19 - Selected ARC/DS method for Data1a-01.bin was: xtool:mreflate+xtool:mpreflate+xtool:db:-dedup=3+4x4:lzma

 17:28:19 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:-dedup=3+4x4:lzma

-------------------------------------------------------------------------------------------------------------------------------------------

 18:27:20 - Overall input size: 31.64 GB

 18:27:20 - Overall output size: 27.34 GB (Ratio 86.38%)

 18:27:20 - Overall conversion time: 00:58:59

Masquerade

24-11-2022 23:43

Automobilista 2 [v1.4.3.3]

*.bff, *.dds

Code:

xtool_automobilista2_dedup5_d1+lolz
Everything else:

Code:

razor

89.1GB -----> 17.7GB

shazzla

25-11-2022 03:23

XTool "dedup level 5" ?
Where ,how ? I cant find any information about that !
Is it another not published setting ,like -dd ?!

KaktoR

25-11-2022 03:44

Quote:

Originally Posted by shazzla (Post 499221)

XTool "dedup level 5" ?
Where ,how ? I cant find any information about that !
Is it another not published setting ,like -dd ?!

Code:

--dedup=[1..5]

srep.exe must be present next to xtool.exe for compression and decompression (no cls-srep required)

--dedup=5 is like you use srep -m5f setting

Example: https://fileforums.com/showpost.php?...postcount=3282

Especially on HDD the speed improvements are very good.

Maybe this is interesting aswell for some users

Code:

--compress=[l#,][hi0/1,][t#[p]]

default is l6

requires fast-lzma2.dll next to xtool.exe

KaktoR

29-11-2022 12:21

Stellaris
v3.6.0, incl all DLC, Soundtrack

Code:

 15:22:36 - Selected ARC/DS method for Data1a-01.bin was: xtool:mreflate+msc_tak+oggre

 15:22:36 - Selected ARC/DS method for Data1b-01.bin was: msc_tak

 15:22:36 - Selected ARC/DS method for Data1c-01.bin was: oggre

 15:22:36 - Selected ARC/DS method for Data1d-01.bin was: xtool:db:-dedup=3+lolz:d128m:mtt1:mt12:mc1023

 15:22:36 - Selected ARC/SPLIT method for Data1e-01.bin was: xtool:db:-dedup=3+lolz:d128m:mtt1:mt12:mc1023/$mp3pack=mpzmt/$deflatepack=xtool:mpreflate:mpng:d1:db:-dedup=3+lolz:d128m:mtt1:mt12:mc1023/$textpack=xtool:db:-dedup=3+lzma:a1:mfbt4:256m:fb273:mc1000000000:lc8

-----------------------------------------------------------------------------------------

 16:48:10 - Overall input size: 16.69 GB

 16:48:10 - Overall output size: 6.46 GB (Ratio 38.71%)

 16:48:10 - Overall conversion time: 01:25:32



Data1a = *.zip

Data1b = *.wav

Data1c = *.ogg

Data1d = *.dds

Data1e = rest of the files

KaktoR

29-11-2022 14:46

Mount & Blade II Bannerlord

Seems like tpac files use lz4hc level 12. On this level you will get low streams but output is much bigger then with other levels.

Code:

Compressed 1 file, 7,120,255 => 14,068,315 bytes. Ratio 197.58%

Compression time: cpu 0.00 sec/real 30.88 sec = 0%. Speed 0.23 mB/s

All OK

However it is slow af

KaktoR

30-11-2022 07:33

Assassin's Creed Valhalla
v1.6.2, 1 language, all DLC

Code:

 10:59:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:manvil:8+xtool:db:-dedup=3+4x4:lzma

 10:59:07 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:-dedup=3+4x4:lzma

-------------------------------------------------------------------------------------------------------------------------------------------

 16:29:11 - Overall input size: 143.49 GB

 16:29:11 - Overall output size: 50.14 GB (Ratio 34.94%)

 16:29:11 - Overall conversion time: 05:29:24

kj911

02-12-2022 08:53

Sorry from bumping the old post.

Quote:

Originally Posted by JRD! (Post 472175)

Here is: dlz_v0.2.2b.7z

The "dlz_unpack.exe" file its works?? Required special commands or any from decompression natively packed any data in used dlz_x86/64 executables without FreeArc? (aka ex: Bink_(un)pack or oggre_enc/dec...) Tested from Win XP x32 and Win7 SP1 x64 and crashing now. The CLS filter its works with FreeArc/Unarc usage.

KaktoR

03-12-2022 03:07

The Callisto Protocol

Code:

Compressed 2 files, 6,558,206,036 => 13,302,561,746 bytes. Ratio 202.84%

Compression time: cpu 2.78 sec/real 243.09 sec = 1%. Speed 26.98 mB/s

All OK

reflate
0x1958456F8B54A02D25DF32327C57C9D20DFCF7E948967D08 CFB48E8EBC65BEEB

Edit: Seems like you don't need the key. Without the key the output is the same.

L33THAK0R

06-12-2022 16:23

Quote:

Originally Posted by Gehrman (Post 499322)

Is there a way to control memory usage during decompression in SREP.exe?

LAST MINUTE EDIT: Just realized you're talking about using xtool's srep functionality, so not sure if this is useful at all :o

I use the following settings in my "CLS.ini" file, I do use the CLS version of Srep for decompression so not sure if that would have any impact if you're using a non-CLS package.

Code:

[Srep]

Bufsize=24m

transfer_ReadBufSize=512k

transfer_WriteBufSize=512k

Memory=60%-2048m

TempPath={app}



[Srep_new]

Bufsize=24m

transfer_ReadBufSize=512k

transfer_WriteBufSize=512k

Memory=60%-2048m

TempPath={app}

abror

06-12-2022 19:04

please tell me the best compress method for gta 5 game. I am using Cesar82 Diskpan Gui

Cesar82

06-12-2022 19:12

Quote:

Originally Posted by abror (Post 499326)

please tell me the best compress method for gta 5 game. I am using Cesar82 Diskpan Gui

DiskSpan GUI has compression preset in DATABASE for this game, just load the preset.

KaktoR

10-12-2022 11:18

Anno 1800
v16.0, incl all DLC, all languages, artbook, ost

Code:

 21:12:27 - Selected ARC/DS method for Data1a-01.bin was: xtool:mflac+xtool:mpreflate+xtool:db:-dedup=3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 21:12:27 - Selected ARC/SPLIT method for Data1b-01.bin was: xtool:db:dedup=3+razor_mtx/$mp3pack=mpzmt

-----------------------------------------------------------------------------------------

 23:12:09 - Overall input size: 86.73 GB

 23:12:09 - Overall output size: 31.42 GB (Ratio 36.23%)

 23:12:09 - Overall conversion time: 01:59:41



>> Unpacking finished successfully!

Edit: Method changed from "reflate" to "preflate" because of crc errors on file data13.rda

Masquerade

10-12-2022 11:21

@KaktoR
You can patch out the bink videos with XTool erase and squeeze the size down with bpk too

KaktoR

10-12-2022 11:34

@Masquerade
I know. bk2 files are spread all over inside many files and I for myself are just to lazy to do it :D

abror

11-12-2022 07:33

Quote:

Originally Posted by Cesar82 (Post 499327)

DiskSpan GUI has compression preset in DATABASE for this game, just load the preset.

thank you, do you know how to make the game install process not take long.
I have compressed the gta 5 game using the method that is in the database but during install (unpack) it takes very long around 5-7 hours

KaktoR

11-12-2022 11:00

Halo The Master Chief Collection
v1.3065.0.0 (7th December 2022), ripped multiplayer

Code:

 17:02:27 - Selected ARC/DS method for Data1a-01.bin was: bpk+xtool:db:-dedup=3

 17:02:27 - Selected ARC/DS method for Data1b-01.bin was: xtool:c128mb:mpreflate:db:-dedup=3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

 17:02:27 - Selected ARC/DS method for Data1c-01.bin was: xtool:db:-dedup=3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

-----------------------------------------------------------------------------------------

 19:06:36 - Overall input size: 103.82 GB

 19:06:36 - Overall output size: 64.34 GB (Ratio 61.97%)

 19:06:36 - Overall conversion time: 02:04:07



>> Unpacking finished successfully!



---



Data1b:

halo1\prebuild\shaders_2_cache\prebuild\*.sdc

halo2\prebuild\shaders_2_cache\prebuild\*.sdc

halo2\preload\paks\*.pck

halo2\h2_maps_win64_dx11\*.map

halo2\h2_maps_win64_dx11\textures.dat

halo3\maps\*.map

halo3odst\maps\*.map

halo4\maps\*.map

haloreach\maps\*.map

mcc\Content\Paks\*.pak

data\ui\Screens\*.gfx

With reflate you will get a crc error on "MCC-WindowsNoEditor.pak" file. Seems there are lz4 streams too but couldn't process any of them properly (also look in "Data\ui\texturepacks" folder). With old xtool the output is same as reflate option (~1gb smaller) but I prefer speed :p

128mb chunk size is the sweet spot here. However you will get some additional mb on 256mb.

bpk output is 22.5 GB, with srep 21.5 GB (many video stream duplicates).

KaktoR

13-12-2022 12:00

Crisis Core - Final Fantasy VII - Reunion

pak/ucas -> kraken
Use oo2core v2.9.6

Edit: Ok seems like zlib/reflate is in there too. What a weird mess I made, sorry :D

Code:

pakchunk0-WindowsNoEditor.pak



156mb -> 175mb (kraken) -> 199mb (reflate)

Edit 2:
pakchunk1-WindowsNoEditor.pak has no kraken streams, but only reflate.

Edit 3:
pakchunk5-WindowsNoEditor.pak, pakchunk6-WindowsNoEditor.pak -> seems like only usm video
pakchunk4-WindowsNoEditor.pak -> sounds

Masquerade

13-12-2022 12:33

Crisis Core - Final Fantasy VII - Reunion UCAS contains encrypted Kraken streams but the UnrealDB creator does not work for this game and requires an update.

Razor12911

13-12-2022 18:19

Crisis Core - Final Fantasy VII - Reunion

Used unreal_db
key 0x277041CBB2888087611DB5CBFEA119462920E1F67092C87E 87825725E38B320F
Oodle v2.9.5 dll

-mccff7r+kraken:l7 -d1 -c1gb -lm

Code:

pakchunk7-WindowsNoEditor.ucas

1.53 GB > 2.79 GB

idk, unreal db works for me, but there was a catch. The AES blocks can be larger than 700MB so just set a high chunk size value but add -lm parameter so that xtool doesn't tank your PC.

Edit: I recommend that you use two separate xtool processes (one for decryption and the other for processing kraken streams), since these aes blocks are big, xtool will use 1 thread in d1 mode for both encoding and decoding in an attempt to use the lowest possible memory, which will slow down installation tremendously if you plan to make a repack. This also eliminates the need for -d1, you can opt for oo2reck if you want, up to you.

KaktoR

14-12-2022 00:54

Thanks, this seem to work :)

Code:

-mff7rcc+xtool:kraken,l7



Compressed 2 files, 1,644,918,858 => 3,004,384,018 bytes. Ratio 182.65%

Compression time: cpu 1.22 sec/real 207.25 sec = 1%. Speed 7.94 mB/s

All OK



Extracted 2 files, 3,004,384,018 => 1,644,918,858 bytes. Ratio 182.65%

Extraction time: cpu 1.28 sec/real 191.11 sec = 1%. Speed 8.61 mB/s

All OK

Code:

[External compressor:xtool]

header = 0

packcmd   = xtool.exe precomp { -moption} -c64mb -t100p - - <stdin> <stdout>

unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>



[External compressor:ff7rcc]

header = 0

packcmd   = xtool.exe precomp -mccff7r -c1gb -d1 -lm -t100p - - <stdin> <stdout>

unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

KaktoR

14-12-2022 07:59

Witcher 3 v4.0 Next Gen Update

Still the same algo from before
https://fileforums.com/showthread.ph...her#post498295

Code:

-mxtool:witcher3+xtool:zlib+xtool:lz4hc,l9



Compressed 1 file, 286,247,872 => 1,249,481,045 bytes. Ratio 436.50%

Compression time: cpu 0.41 sec/real 51.41 sec = 1%. Speed 5.57 mB/s

All OK



Extracted 1 file, 1,249,481,045 => 286,247,872 bytes. Ratio 436.50%

Extraction time: cpu 0.59 sec/real 37.01 sec = 2%. Speed 7.73 mB/s

All OK

KaktoR

15-12-2022 02:01

Grand Theft Auto V
Build 10107212 (13 December 2022)

Code:

 17:17:06 - Selected ARC/DS method for Data1a-01.bin was: bpk+xtool:mreflate:db:-dedup=3+4x4:lzma

 17:17:06 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:-dedup=3+4x4:lzma

-----------------------------------------------------------------------------------------

 20:20:08 - Overall input size: 108.55 GB

 20:20:08 - Overall output size: 65.61 GB (Ratio 60.45%)

 20:20:08 - Overall conversion time: 03:02:59



Data1a

common.rpf

x64*.rpf

update\*.rpf

Edit: Use srep instead of xtool dedup. At least for me xtool just crashes in the middle, even with low settings. With srep cls it is working fine.

Gehrman

15-12-2022 12:01

Quote:

Originally Posted by KaktoR (Post 499426)

Grand Theft Auto V
Build 10107212 (13 December 2022)

Code:

 17:17:06 - Selected ARC/DS method for Data1a-01.bin was: bpk+xtool:mreflate:db:-dedup=3+4x4:lzma

 17:17:06 - Selected ARC/DS method for Data1b-01.bin was: xtool:db:-dedup=3+4x4:lzma

-----------------------------------------------------------------------------------------

 20:20:08 - Overall input size: 108.55 GB

 20:20:08 - Overall output size: 65.61 GB (Ratio 60.45%)

 20:20:08 - Overall conversion time: 03:02:59



Data1a

common.rpf

x64*.rpf

update\*.rpf

Edit: Use srep instead of xtool dedup. At least for me xtool just crashes in the middle, even with low settings. With srep cls it is working fine.

It is more optimal with --mem=10p

Code:

[External compressor: xtool]

header = 0

unpackcmd = "{compressor}.exe" decode -t100p --mem=10p - - <stdin> <stdout>

Also, use the 32-bit version of SREP.exe.

All times are GMT -7. The time now is 08:56.

Page 83 of 119

« First

Last »

Show 40 post(s) from this thread on one page