FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

KaktoR 27-08-2023 09:48

Not entirely sure. You can start the game normally, and the first videos + intro will show (if you cut bk2 videos from Data0.bdt). But I cannot speak for the whole game.

If you move the movie folder or rename it, the game will not even start.

shazzla 27-08-2023 10:09

Sorry for offtopic,but kaktor ,can u please check your msgs? A few days ago i sent u a pm.... thanks!:)

L33THAK0R 27-08-2023 23:30

Wolfenstein Cyberpilot
 
1 Attachment(s)
Uses Kraken, after testing with KaktoR's Oodle library collection, v2.9.0 seems to yield the best results, however even with this library a considerable number of streams are dropped. Against the attached sample 88/231 steams are processed (full recompression currently, as reported by XTool, indicates 6248/7387 streams are processed), verbose output indicates streams are at a depth of L4.

xtool 0.7.8 (-mkraken:l4:n128:t256+zlib+reflate) + srep + lolz

It should be noted that it seems at least with the current library re-compression isn't very effective, with the reinflated output only being ~10MB larger than the initial input. The original Oodle DLL matches the CRC-32 checksum for library v2.6.0, when compared against KaktoR's kindly supplied checksums for all current library.

Code:

16,606,744,418 => 14,348,641,273 bytes. Ratio 86.40%

Masquerade 28-08-2023 00:16

What does the Oodle srream scanner say?

KaktoR 28-08-2023 07:20

Quote:

Originally Posted by Danziel123 (Post 502129)
Hi, sorry if i join to discuss. How to do it in DSG? thank you;)

Database preset is now up for download (check in DSG for update).

However in CIU something has to be added in Setup.ini

Code:

[CMDCommands]
if exist "{app}\ac6for.xtl" (
"{tmp}\XTool_2020\XTool.exe" decode "{app}\ac6for.xtl" "{app}\Game\movie" "{app}\Game\Data0.bdt"
del /F /S /Q "{app}\ac6for.xtl"
TIMEOUT 3
)


L33THAK0R 28-08-2023 08:05

Quote:

Originally Posted by Masquerade (Post 502152)
What does the Oodle srream scanner say?

Here is the output for the stream scanner, as well as the output for the library checker.

Masquerade 28-08-2023 08:29

No, I meant the oo2scan_7_win64.exe by Razor12911, as it lists these algorithms:

Code:

Universal Oodle stream scanner

Created by Razor12911

[0] = Unknown/Invalid
[1] = Kraken/Hydra
[2] = Mermaid/Selkie/Hydra
[3] = Leviathan/Hydra

If you scan and you see [1] appearing, it could be that you are seeing Hydra streams, not kraken.

Try extracting a single stream from the sample you are testing on using XTool verbose output and a Hex editor and attempt to precompress it with oo2reck. If it *is* a Kraken stream, it will be inflated. Otherwise, try Hydra.

Jarmen_Kell 29-08-2023 03:31

Hey guys. Is there any suggested method for ARMA III?

L33THAK0R 29-08-2023 05:38

Quote:

Originally Posted by Masquerade (Post 502163)
No, I meant the oo2scan_7_win64.exe by Razor12911, as it lists these algorithms:

Code:

Universal Oodle stream scanner

Created by Razor12911

[0] = Unknown/Invalid
[1] = Kraken/Hydra
[2] = Mermaid/Selkie/Hydra
[3] = Leviathan/Hydra

If you scan and you see [1] appearing, it could be that you are seeing Hydra streams, not kraken.

Try extracting a single stream from the sample you are testing on using XTool verbose output and a Hex editor and attempt to pre-compress it with oo2reck. If it *is* a Kraken stream, it will be inflated. Otherwise, try Hydra.

I'm seeing results for both [1] and [3], however using XTool's GUI and testing each library against the sample, now with both the Kraken and Hydra codecs, still results in the same number of streams being successfully pre-compressed, when compared to only using Kraken. I've got a feeling it may in this case be Leviathan & Kraken, and I'm aware of how annoying Leviathan is to process since theres a notable manual element to pre-compressing this Oodle variant. I might investigate this title further, at some point in the future, once I'm more confident with working with undocumented Oodle titles.

KaktoR 29-08-2023 05:57

If I may quote Razor12911 on this:
Code:

He should give up
2.6.0 leviathan
2.9.0 kraken


L33THAK0R 29-08-2023 08:03

Quote:

Originally Posted by KaktoR (Post 502184)
If I may quote Razor12911 on this:
Code:

He should give up
2.6.0 leviathan
2.9.0 kraken


I'm nothing if not stubborn, well at least with that information I've got something to go off of, I'll see what I can do.

shazzla 29-08-2023 09:33

Quote:

Originally Posted by Jarmen_Kell (Post 502181)
Hey guys. Is there any suggested method for ARMA III?

AFAIK half of this game is RSA encrypted ,so it cannot be recompressed (EBO)

PBOs maybe some kind of LZO?!

Jarmen_Kell 30-08-2023 09:23

Quote:

Originally Posted by shazzla (Post 502189)
AFAIK half of this game is RSA encrypted ,so it cannot be recompressed (EBO)

PBOs maybe some kind of LZO?!

Then how Fitgirl compressed it from 146GB to 73GB?

Masquerade 30-08-2023 09:37

Quote:

Originally Posted by Jarmen_Kell (Post 502204)
Then how Fitgirl compressed it from 146GB to 73GB?

Because she decrypted it... duh.

shazzla 30-08-2023 09:57

IMHO its impossible ! RSA is asymmetric. So u can obtain the decryption key from the game ,but reencrypt...?! Only the devs knows the encryption key (Maybe its buried accidentally somewhere in a gamefile )...
Or am i wrong ?

btw Masquerade's answer seems good....

Edit:

Maybe start looking here :

https://www.unknowncheats.me/forum/arma-3-a/

KaktoR 31-08-2023 11:12

Trine 5: A Clockwork Conspiracy

Code:

19:18:46 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzstd:zstd_137+xtool:dd3+4x4:lzma
 19:18:46 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+4x4:lzma
-------------------------------------------------------------------------------------------------------------------------------------------
 20:12:09 - Overall input size: 16.92 GB
 20:12:09 - Overall output size: 12.42 GB (Ratio 73.38%)
 20:12:09 - Overall conversion time: 00:53:21


Masquerade 31-08-2023 12:09

You can rip voiceovers from audio1.fbq in Trine 5 using this QuickBMS script:

Code:

get VERSION long
if VERSION <= 3
    comtype zstd
else
    comtype lz4
endif
get FILES long
get BASE_OFF long
math BASE_OFF += 0xc
for i = 0 < FILES
    get NAME string
    get OFFSET long
    get NOZIP byte
    get SIZE long
    get ZSIZE long
    get CRC long
    math OFFSET += BASE_OFF
    if NOZIP == 4
        clog NAME OFFSET ZSIZE SIZE
    else
        log NAME OFFSET SIZE
    endif
next i

Made by eaZy41 from XeNTaX community.

Although comtype LZ4 is mentioned, Trine 5 uses only ZSTD and you can use libzstd v1.3.7 (v1.3.6 also seems to work). LZ4 is used on older versions of the FBQ archive structure.

KaktoR 31-08-2023 12:23

Better use zstd version "1.3.7", as it is stored in trine5.exe.
https://i.imgur.com/q4OTfAM.png

Although it doesn't make any sense, I looked through some of the streams in the texture archives.

Code:

#1 174760 > 174752 = 8 bytes
#2 349520 > 349504 = 16 bytes

#3 11184808 > 11184800 = 8 bytes
#4  5592400 >  5592384 = 16 bytes

#5  699048 >  699040 = 8 bytes
#6 1398096 > 1398080 = 16 bytes

#7    43688 >    43680 = 8 bytes
#8 22369616 > 22369600 = 16 bytes

Actually this is the repeating size pattern.

Code:

Signature(1),DSize(4),Unknown(8),ZstdHdr(4),Stream
This Unknown 8 bytes are actually the byte size difference for DSize. 8 and 16.

Masquerade 31-08-2023 15:17

Did your archive decompress alright? I'm getting this error from XTool when trying to decompress my archive:
I was using libzstd v1.3.7.
Code:

Unpacking 25,439,279,314 bytes with XTOOL decode -t100p $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
  0.0%XTool is created by Razor12911

Time: 00:00:15 (CPU 00:02:30)

  0.0%EThreadException: Error in the method 'zstd'

Errorlevel=1
  0.0%
ERROR: invalid compression method or parameters in XTZSTD


Wanterlude 31-08-2023 15:22

Quote:

Originally Posted by Masquerade (Post 502236)
Did your archive decompress alright? I'm getting this error from XTool when trying to decompress my archive:
I was using libzstd v1.3.7.
Code:

Unpacking 25,439,279,314 bytes with XTOOL decode -t100p $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
  0.0%XTool is created by Razor12911

Time: 00:00:15 (CPU 00:02:30)

  0.0%EThreadException: Error in the method 'zstd'

Errorlevel=1
  0.0%
ERROR: invalid compression method or parameters in XTZSTD


same, xtool just crash on unpack with 1.3.7 zstd lib
"ERROR: write error (disk full?) in compression algorithm zstd" in arc

edit.
1.3.5,1.3.6,1.3.7 - error (crash)
test file: gui1.fbq
Trine 5 [3 fbq files] - https://pixeldrain.com/u/cTSsXMes

Razor12911 31-08-2023 18:39

1 Attachment(s)
Quote:

Originally Posted by Wanterlude (Post 502237)
same, xtool just crash on unpack with 1.3.7 zstd lib
"ERROR: write error (disk full?) in compression algorithm zstd" in arc

edit.
1.3.5,1.3.6,1.3.7 - error (crash)
test file: gui1.fbq
Trine 5 [3 fbq files] - https://pixeldrain.com/u/cTSsXMes

I won't test as I'm currently busy with something else but maybe try this development version of xtool as a ton of issues were fixed however no update as of yet because it's currently in development so if it works, use it for only this case and revert back to 0.7.8 for everything else. (don't use -dd#, it is bugged in dev build)

Quote:

Originally Posted by L33THAK0R (Post 502186)
I'm nothing if not stubborn, well at least with that information I've got something to go off of, I'll see what I can do.

Code:

[Stream1]
// 2.6.0
Name=leviathan
Codec=leviathan:l4
BigEndian=0
Signature=0x0
Structure=Signature(16),DSize(8),CSize(8),OodleHdr(2),Stream
StreamOffset=-2
CompressedSize=CSize
DecompressedSize=DSize
Condition1=OodleHdr = 0x0C8C
Condition2=CSize>=256
Condition3=DSize>CSize

I made quickly made this the other day, not sure if it works very well.

KaktoR 01-09-2023 01:17

Quote:

Originally Posted by Masquerade (Post 502236)
Did your archive decompress alright? I'm getting this error from XTool when trying to decompress my archive:
I was using libzstd v1.3.7.
Code:

Unpacking 25,439,279,314 bytes with XTOOL decode -t100p $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
  0.0%XTool is created by Razor12911

Time: 00:00:15 (CPU 00:02:30)

  0.0%EThreadException: Error in the method 'zstd'

Errorlevel=1
  0.0%
ERROR: invalid compression method or parameters in XTZSTD


Yes it works fine here using libzstd v1.3.7, but tbh I am using the development version of xtool since days, there were some fixes.

Extracting could take a while because of zstd l22 btw.

KaktoR 01-09-2023 06:44

Sid Meier's Civilization VI
AUGUST 2023 UPDATE

Code:

14:01:31 - Selected ARC/DS method for Data1a-01.bin was: xtool:c256mb:mbk2:df20p
 14:01:31 - Selected ARC/DS method for Data1b-01.bin was: xtool:dd3+lolz:d64:mtt1:mt10:fba4096:mc2
 14:01:31 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+lolz:d64:mtt1:mt10:fba4096:mc2/$deflatepack=xtool:c128mb:mpreflate:mpng:d1+xtool:dd3+lolz:d64:mtt1:mt10:fba4096:mc2/$oggrepack=oggre
-------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------
 15:42:58 - Overall input size: 25.96 GB
 15:42:58 - Overall output size: 8.69 GB (Ratio 33.49%)
 15:42:58 - Overall conversion time: 01:41:25

Data1a = *.bk2
Data1b = *.blp
Data1c = rest of the files

PS: I used wemtool by Masquerade for the wem's


KaktoR 01-09-2023 09:11

Quote:

Originally Posted by Danziel123 (Post 502245)
Maybe someone know what best compress method for starfield? Thank you:)

Bruh
https://fileforums.com/showpost.php?...&postcount=379

KaktoR 01-09-2023 12:46

Baldurs Gate 3
v4.1.1.3686210

Code:

17:57:24 - Selected ARC/DS method for Data1a-01.bin was: xtool:mbg3+xtool:mzlib+xtool:dd3+4x4:lzma
 17:57:24 - Selected ARC/DS method for Data1b-01.bin was: xtool:c1024mb:mbk2:df20p
 17:57:24 - Selected ARC/DS method for Data1c-01.bin was: xtool:mbg3+xtool:dd3+zstd
 17:57:24 - Selected ARC/SPLIT method for Data1d-01.bin was: xtool:dd3+4x4:lzma/$deflatepack=xtool:c128mb:mpreflate:mpng:d1+xtool:dd3+4x4:lzma/$mp3pack=7zdll-pm:m0=mp3det:m1=packmp3c
-------------------------------------------------------------------------------------------------------------------------------------------
 21:42:21 - Overall input size: 137.01 GB
 21:42:21 - Overall output size: 88.23 GB (Ratio 64.39%)
 21:42:21 - Overall conversion time: 03:40:21


IRAQIGHOST 01-09-2023 21:19

Starfield

DiskSpan
xt_zlib+xtool:s:mzlib+srep_new:3+LOLZ_ULTRA

Test File (Starfield - Animations.ba2)
Size: 319 MB >> 711 MB
Final Size: 170.56 MB

Danziel123 02-09-2023 01:47

Quote:

Originally Posted by KaktoR (Post 502246)

Here is my DSG_Settings for starfiled, is this right or not?
Quote:

Data1.Name=Data1a-01.bin
Data1.InclText=*.ba2
Data1.ExclText=
Data1.Method1=xtool:mstarfield+xtool:mreflate:dd3+ LZMA-MT
Data2.Name=Data1b-01.bin
Data2.Method1=xtool:c256mb:mbk2:df20p+LZMA-MT
Data2.InclText=*.bk2
Data3.Name=Data1c-01.bin
Data3.Method1=xtool:dd3+LZM A-MT
Data3.ExclText=*.ba2|*.bk2
And what about for .bk2 files? is enough using chunk 256mb to compress it? Or i can increase the chunk size? Thank you:)

KaktoR 02-09-2023 04:37

Should be enough

KaktoR 02-09-2023 04:42

Quote:

Originally Posted by IRAQIGHOST (Post 502255)
Starfield

DiskSpan
xt_zlib+xtool:s:mzlib+srep_new:3+LOLZ_ULTRA

Test File (Starfield - Animations.ba2)
Size: 319 MB >> 711 MB
Final Size: 170.56 MB

Why use old xtool with new xtool? And why skip parameter?

IRAQIGHOST 02-09-2023 05:24

Quote:

Originally Posted by KaktoR (Post 502262)
Why use old xtool with new xtool? And why skip parameter?

for try to get more size

Fak Eid 03-09-2023 04:45

You tried compressing on just a single file. When you compress all the files it will be stuck forever. Do not try this as this is wrong.

Quote:

Originally Posted by IRAQIGHOST (Post 502255)
Starfield

DiskSpan
xt_zlib+xtool:s:mzlib+srep_new:3+LOLZ_ULTRA

Test File (Starfield - Animations.ba2)
Size: 319 MB >> 711 MB
Final Size: 170.56 MB


IRAQIGHOST 03-09-2023 05:30

Quote:

Originally Posted by Fak Eid (Post 502271)
You tried compressing on just a single file. When you compress all the files it will be stuck forever. Do not try this as this is wrong.

Yes, I notice that in some files.

KaktoR 03-09-2023 06:43

Quote:

Originally Posted by IRAQIGHOST (Post 502264)
for try to get more size

Wtf. Why not use the correct codecs with appropriate chunk size instead?

And do not use the skip option if you don't know what you do.

Danziel123 03-09-2023 21:40

Quote:

Originally Posted by KaktoR (Post 502261)
Should be enough

Why i get stuck at this?
http://i.ibb.co/sVbn5hm/Starfiled.png
Alredy tried with xtool:mreflate, xtool:mpreflate & xtool:mzlib still same. Maybe you know what cause it? Thank you:)

Fak Eid 04-09-2023 01:35

Because the codecs can't read the stream of the file. The files contains encoded lz4 that can be compressed only using a certain plugin 'mstarfield' (starfield.dll). Leave out the wwise sound files and compress using that algo.. wwise sound can only be compressed using srep+lzma (as of my knowledge right now)

Quote:

Originally Posted by Danziel123 (Post 502284)
Why i get stuck at this?
http://i.ibb.co/sVbn5hm/Starfiled.png
Alredy tried with xtool:mreflate, xtool:mpreflate & xtool:mzlib still same. Maybe you know what cause it? Thank you:)


Danziel123 04-09-2023 01:41

Quote:

Originally Posted by Fak Eid (Post 502286)
Because the codecs can't read the stream of the file. The files contains encoded lz4 that can be compressed only using a certain plugin 'mstarfield' (starfield.dll). Leave out the wwise sound files and compress using that algo.. wwise sound can only be compressed using srep+lzma (as of my knowledge right now)

Here is my DSG_Settings for starfiled, is this right or not?
Quote:

Data1.Name=Data1a-01.bin
Data1.InclText=*.ba2
Data1.ExclText=
Data1.Method1=xtool:mstarfield+xtool:mreflate:dd3+ LZMA-MT
Data2.Name=Data1b-01.bin
Data2.Method1=xtool:c256mb:mbk2:df20p+LZMA-MT
Data2.InclText=*.bk2
Data3.Name=Data1c-01.bin
Data3.Method1=xtool:dd3
Data3.ExclText=*.ba2|*.bk2

Masquerade 04-09-2023 02:13

Quite a lot of audio in Starfield is WWISE PCM - meaning you can use MSC on it.

Other type of WWISE audio is Vorbis, which you can read more about in WemTool topic.

KaktoR 04-09-2023 02:32

^Yes, all of the *WwiseSounds* are PCM. The Language files are Vorbis it seems.

Code:

Streams: 18613 / 18613
Time: 00:03:05 (CPU 00:16:04)
Duplicates: 109 (100 MB) [113 MB >> 35.3 MB]

Size: 19.3 GB >> 6.94 GB >> 6.90 GB

Additional decoding memory is 79.7 MB
Decoding speed improved by up to 1.01x

Done!!!

I used flac plugin with l8 option here. I guess msc is better in terms of ratio but alot slower.

PS: And compression for the other files, files which have "02 00 00 00" in the beginning are compressed with zlib, "03 00 00 00" are compressed with lz4.

KaktoR 04-09-2023 06:59

As for the WEM files in Starfield, seems like it does not make sense to process them.

xtool:dd3:l4
Code:

Compressed 125,234 files, 1,993,236,804 => 1,878,585,133 bytes. Ratio 94.25%
Compression time: cpu 6.13 sec/real 44.43 sec = 14%. Speed 44.87 mB/s
All OK

Compressed 1 file, 1,870,743,269 => 1,839,107,129 bytes. Ratio 98.31%
Compression time: cpu 0.52 sec/real 63.26 sec = 1%. Speed 29.57 mB/s
All OK


Fak Eid 04-09-2023 07:48

xtool:mstarfield+xtool:mreflate+srep_new+lolz/lzma seems correct for all .ba2 files except WWise Sound files. Also as of now, DSG won't be able to extract the files. You need to do it with your installer only.

Quote:

Originally Posted by Danziel123 (Post 502288)
Here is my DSG_Settings for starfiled, is this right or not?



All times are GMT -7. The time now is 18:40.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com