FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

Danziel123 04-09-2023 23:57

Quote:

Originally Posted by Fak Eid (Post 502295)
xtool:mstarfield+xtool:mreflate+srep_new+lolz/lzma seems correct for all .ba2 files except WWise Sound files. Also as of now, DSG won't be able to extract the files. You need to do it with your installer only.

Thank you for suggestion, change it to xtool:dd now can compress till finish:)

KaktoR 05-09-2023 10:31

BioShock Infinite
v31.08.2022

Code:

16:54:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:c320mb:mbik
 16:54:07 - Selected ARC/DS method for Data1b-01.bin was: uelr+xtool:c128mb:mzlib+xtool:dd3+4x4:lzma
 16:54:07 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:lzma
------------------------------------------------------------------------------------------
 19:08:31 - Overall input size: 43.20 GB
 19:08:31 - Overall output size: 15.32 GB (Ratio 35.47%)
 19:08:31 - Overall conversion time: 02:14:23


Fak Eid 05-09-2023 10:55

Why did everyone started using xtool:dd3 instead of srep_new:3. It it better?

Quote:

Originally Posted by KaktoR (Post 502320)
BioShock Infinite
v31.08.2022

Code:

16:54:07 - Selected ARC/DS method for Data1a-01.bin was: xtool:c320mb:mbik
 16:54:07 - Selected ARC/DS method for Data1b-01.bin was: uelr+xtool:c128mb:mzlib+xtool:dd3+4x4:lzma
 16:54:07 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:lzma
------------------------------------------------------------------------------------------
 19:08:31 - Overall input size: 43.20 GB
 19:08:31 - Overall output size: 15.32 GB (Ratio 35.47%)
 19:08:31 - Overall conversion time: 02:14:23



KaktoR 05-09-2023 11:11

Because it's faster.

Edit: "3" in dd# just represents the srep m#f setting m[1..5]. You can even use dd5 to get a bit better results in costs of speed.

Edit2: A bit old, but my test in the past when this feature was introduced: https://fileforums.com/showthread.ph...129#post499129

Masquerade 05-09-2023 15:35

@KaktoR did you try using xtool unreal 3 for the lzo? I'm interesred in the performance vs uelr.

KaktoR 05-09-2023 23:08

ANNO 1800
Update 18.0

Code:

20:37:16 - Selected ARC/DS method for Data1a-01.bin was: xtool:c1500mb:lm:mbk2:df20p+xtool:dd3
 20:37:16 - Selected ARC/DS method for Data1b-01.bin was: xtool:mflac,l8+xtool:mzlib:mreflate+xtool:dd3+4x4:lzma
 20:37:16 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+4x4:lzma/$mp3pack=7zdll-pm:m0=mp3det:m1=packmp3c/$bpkpack=bpk
-----------------------------------------------------------------------------------------
 00:12:23 - Overall input size: 101.44 GB
 00:12:23 - Overall output size: 30.63 GB (Ratio 30.20%)
 00:12:23 - Overall conversion time: 03:34:59

Notes:
- bpk was used on bink1 files which are located in data3.rda
- You have to use -lm for bk2 plugin if you don't have 64GB RAM
- The following files contain bink videos: data3.rda, data11.rda, data12.rda, data13.rda, data16.rda and data18.rda. Use quickbms with "-f "{}.bk2,{}.bik"" file mask to extracrt just bink1/2 files.
- Use XTool erase to create xtl file and zero the files out of the game archives.
- Use XTool decode + xtl file to patch the files back afterwards.
- Some rda files also contain WEM files. You can use wemtool there.
- The language files also contain WEM files, but they are PCM, so you msc/xtool:flac on them.


Fak Eid 06-09-2023 01:36

Thanks for the update. If dd3 can help me compressing the same size as srep and with less extraction time, I'll go ahead with it. My focus is to improve extraction speed at a cost of minimum difference in compression size. I'll try that for sure.

Quote:

Originally Posted by KaktoR (Post 502323)
Because it's faster.

Edit: "3" in dd# just represents the srep m#f setting m[1..5]. You can even use dd5 to get a bit better results in costs of speed.

Edit2: A bit old, but my test in the past when this feature was introduced: https://fileforums.com/showthread.ph...129#post499129


L33THAK0R 06-09-2023 10:38

Watch Dogs Database Generation
 
I know Masquerade made a database recently for the first "Watch Dogs" title, however I've found myself needing to re-generate a new database in my own conversion, as I've opted to include a variety of mods that restore features left on the cutting room floor as well as providing the end-user access to content previously only accessible through online-only interactions. This restoration overwrites a considerable amount of existing files, as such a new database is required.

My current database seems to be considerably smaller than Masquerade's however, with a final result of 3.84 MiB (versus Masquerade's 14.0 MiB). The only issue I can currently see is when the fat2_fat3.bms script attempts to process the "windy_city.fat/.dat" which gives the following error:

Code:

Error: the compressed zlib/deflate input is wrong or incomplete (-3)
Info:  algorithm  2
      offset      494c62b4
      input size  0x0000632b 25387
      output size 0x00010000 65536
      result      0xffffffff -1

Error: uncompressed data (-1) bigger than allocated buffer (224656)
      It usually means that data is not compressed or uses another algorithm

I'm assuming this is referring to

Code:

            elif ZTYPE == 2
                comtype deflate

however my bms2xtl.ini configuration file seems to be fine:

Code:

[Config]
SafeMode=0
UserInput=
Exclude=lzma_dynamic,snappy
gzip=zlib
deflate=reflate
zlib_noerror=zlib
deflate_noerror=reflate
zlib=zlib
oodle=leviathan
xmemdecompress=xmemcompress
lz4=lz4hc:l9
zenpxp=zenpxp
lzo1x=lzo1x
lzma_dynamic=reflate

I've ensured I've got the latest version of the xmemcompress plugin present, and its appropriate entry within "xtool.ini" is pretty standard

Code:

[xmemcompress]
Encode=xcompress64.exe d -window=131072 -partition=524288 <library>
Decode=xcompress64.exe e -window=131072 -partition=524288 <library>

I'm honestly scratching my head trying to figure out what I've messed up this time, even more so by the fact that the file "windy_city.fat/.dat" is one such file that hasn't been modified. Any tips would be greatly appreciated, its definitely been gnawing away at me.

Razor12911 06-09-2023 10:47

the window and partition for xmemcompress should be 32768 I believe, those configurations are for project cars

Code:

[xmemcompress]
Encode=xcompress.exe d -window=32768 -partition=32768 <library>
Decode=xcompress.exe e -window=32768 -partition=32768 <library>

as for the other errors, I'm not so sure

Masquerade 08-09-2023 01:04

DOOM Eternal v6.66 Rev 2.2
  • *.streamdb, *.resources, *.lightdb
    Code:

    xtool_kraken:l6_doom7*_dd+xtool_kraken:l6_doom8*_dd+srep+lolz
  • *.bk2**
    Code:

    xtool_bink2_dd5_fl2
  • *.bik
    Code:

    xtool_bink_dd5_fl2
  • Everything else:
    Code:

    razor
* Using these two oodle libraries.

** All bk2 except vs_first.bk2, vs_second.bk2 and doommarinesymbol_wide_fade.bk2.

88.7GB ---> 56GB

Masquerade 08-09-2023 05:52

The Sims 4 [v1.100.147.1030 + All DLCs]

Code:

xtool_zlib_dd5+LZMA2
58.7GB ---> 36.4GB

KaktoR 08-09-2023 11:42

Quote:

Originally Posted by Masquerade (Post 502353)
** All bk2 except vs_first.bk2, vs_second.bk2 and doommarinesymbol_wide_fade.bk2.

Why excluding? Does the modified bink plugin go crazy again? :D

Masquerade 08-09-2023 12:43

Quote:

Originally Posted by KaktoR (Post 502360)
Why excluding? Does the modified bink plugin go crazy again? :D

The modified import does indeed fix motion data warn but it does not fix err. Trying to compress the folder with those 3 videos present will just leave XTool hanging.

KaktoR 08-09-2023 23:31

Could you send me one of these files? Thanks.

Edit: I looked, and I don't have any problems with the patched bink plugin.

Code:

Compressed 3 files, 69,070,112 => 69,070,262 bytes. Ratio 100.00%
Compression time: cpu 0.06 sec/real 1.12 sec = 6%. Speed 61.71 mB/s
All OK

Extracted 3 files, 69,070,262 => 69,070,112 bytes. Ratio 100.00%
Extraction time: cpu 0.05 sec/real 0.60 sec = 8%. Speed 114.63 mB/s
All OK


KaktoR 09-09-2023 15:44

XCOM 2
version 28 March 2022

Code:

20:51:40 - Selected ARC/DS method for Data1a-01.bin was: uelr+xtool:mkraken:core_2.4.1+xtool:mzlib:dd3+4x4:lzma
 20:51:40 - Selected ARC/DS method for Data1b-01.bin was: xtool:lm:c1024mb:mbk2:df20p+xtool:dd3
 20:51:40 - Selected ARC/SPLIT method for Data1c-01.bin was: xtool:dd3+4x4:lzma/$oggre=oggre/$wemtoolpack=xtool:dd3+zstd
-------------------------------------------------------------------------------------------------------------------------------------------
 00:40:08 - Overall input size: 70.96 GB
 00:40:08 - Overall output size: 25.26 GB (Ratio 35.59%)
 00:40:08 - Overall conversion time: 03:48:26

WEMTool was used for *.wem files.

L33THAK0R 10-09-2023 18:52

Oggre Anomaly, a word of warning for those packing "Nioh 2".
 
Has anyone else experienced some issues with Oggre when processing a large amount of streams? I'm currently working on "Nioh 2", whose audio can be extracted into a readable Vorbis Ogg format. I individually packed every file (~9000) and compared the original file checksums to determine which files were incompatible and removed them accordingly. In the past this process has never failed me and has always without fail resulted in a successful decompression of those particular assets, however this time around it seems I'm left with a bit of a confusing result. Attempting to pack all the streams together results in the installer, presumably due to the CLS Oggre plugin, crashing.

Investigating the issue by creating a large stored archive of all files in their packable format shows a much clearer image:

Code:

Original Input, Stored Archive (7.79 GB) -> Processed with Oggre (5.29 GB) -> Unpacked with Oggre (4.67 GB)
My current solution I'll be employing for my conversion is simply post-install, unpacking the stored archive with the "temp" packable files, then using a patch made between the temp files and a stored archive of the original files.

I'll be then putting these 2 files alongside each other into one large archive, whilst praying I can squeeze some small gains but I'm under no illusion it'll almost certainly be a satisfactory result. Definitely not happy about the amount of additional R/W operations that'll be performed on the end-users drive, but its the best solution I can come up with.

I should mention I was able to process the audio from the original "Nioh" without any trouble, which follows the same structure as "Nioh 2", with a pretty decent final result of 20.3 GB for v1.21.04 of the Complete Edition release for "Nioh".

UPDATE: I ran an additional test with the original unmodified audio archives packed with a generic srep + lolz configuration, when alls said and done, srep actually works well with the duplicated data, resulting in an inflated final result of only 390MB. Given the negligible gains compared to the amount of data that would would need to be written (~16gb) for the original patching approach to work, I think for the time being, and in the interests of the end-user, I'll sacrifice this 390MB.

Masquerade 11-09-2023 00:17

Yeah, oggre is a mess.

L33THAK0R 12-09-2023 23:21

Mirror's Edge Catalyst Woes
 
Can anyone recommend which library and/or version of XTool to use for processing "Mirror's Edge: Catalyst"'s LZ4 streams? I recently tried running my collection of LZ4 libraries against a 100MB sample, using XTool 0.7.8 however seemingly none of them were of any use (see the following pastebin output). I've had success in the past with another LZ4 Frostbite 3 title, "Battlefield: Hardline", however attempting to use the same library for Catalyst results in 0 streams being detected/processed.

I can only find one user actually succeeding to pre-compress Catalyst, who indicates they used "xtlz4(v12)" but I've no idea if this refers to the library or tool.

Any advice would be greatly appreciated, I'm properly stumped with this one.

kj911 13-09-2023 02:52

Try its probably used old Xtool v0.12 with old lz4 libraries. (or used mixed/replaced in game related lz4 library dll.) Check it...

KaktoR 13-09-2023 07:04

If I remeber correct, 1.8.0+ should do it.

Edit:
Code:

Streams: 1790 / 1815
Time: 00:00:01 (CPU 00:00:00)

Size: 22.8 MB >> 60.2 MB

lz4 library 1.8.3. Try with others aswell

Masquerade 14-09-2023 00:15

Mary Skelter Finale

Code:

xtool_zlib_dd5+lolz/$mp4=srep/$bra=oggre
12.8GB ---> 8.7GB

Masquerade 14-09-2023 00:16

Golf With Your Friends [Build 12062185]

Code:

xtool_unity_dd5+lolz
5.1GB ---> 2.5GB

Masquerade 14-09-2023 09:08

Wild question, but would anyone happen to have a backup of the tool linked here?
https://forum.xentax.com/viewtopic.php?t=11106

The blog link is offline. The page is on the Wayback Machine, but the actual file download is blocked behind a xentax login so the file isn't on the wayback machine annoyingly.

Razor12911 14-09-2023 14:09

Quote:

Originally Posted by L33THAK0R (Post 502463)
Can anyone recommend which library and/or version of XTool to use for processing "Mirror's Edge: Catalyst"'s LZ4 streams? I recently tried running my collection of LZ4 libraries against a 100MB sample, using XTool 0.7.8 however seemingly none of them were of any use (see the following pastebin output). I've had success in the past with another LZ4 Frostbite 3 title, "Battlefield: Hardline", however attempting to use the same library for Catalyst results in 0 streams being detected/processed.

I can only find one user actually succeeding to pre-compress Catalyst, who indicates they used "xtlz4(v12)" but I've no idea if this refers to the library or tool.

Any advice would be greatly appreciated, I'm properly stumped with this one.

https://fileforums.com/showpost.php?...82&postcount=2
Use the frostbite 3 plugin for this game -mfrostbite3:lz4

L33THAK0R 14-09-2023 17:27

Quote:

Originally Posted by Razor12911 (Post 502497)
https://fileforums.com/showpost.php?...82&postcount=2
Use the frostbite 3 plugin for this game -mfrostbite3:lz4

I should have specified I was having difficulty with finding streams using the frostbite 3 plugin, turns out though I was just testing against a sample that didnt have any streams :o

Once I tried against a larger sample the plugin worked perfectly, many thanks for your hard-work on it!

kj911 15-09-2023 04:49

Quote:

Originally Posted by Masquerade (Post 502494)
Wild question, but would anyone happen to have a backup of the tool linked here?
https://forum.xentax.com/viewtopic.php?t=11106

The blog link is offline. The page is on the Wayback Machine, but the actual file download is blocked behind a xentax login so the file isn't on the wayback machine annoyingly.

Would that be it? -> https://www.moddb.com/downloads/dpk4util

Masquerade 15-09-2023 08:18

Thanks for the link but that is for DPK4, I was looking for DPK5 version. It's alright now though, the game in question only used zlib so I've done it and moved on.

IRAQIGHOST 18-09-2023 07:38

Forza Horizon 5 V 1.604.481

Quote:

xtool:c256mb:mpreflate:mzlib+xtool:dd3+LOLZ_ULTRA
Repack Size: 104 Gb
Final Size: 164 Gb

IRAQIGHOST 19-09-2023 14:06

PAYDAY 3

Quote:

xtool:dd3+LOLZ_ULTRA
Repack Size: 18.7 Gb
Final Size: 46 Gb

L33THAK0R 19-09-2023 22:11

New Oodle Library (v0.1.0, as reported by oodle_ver.exe) Assistance Needed
 
1 Attachment(s)
I've been investigating a 2017 title "Lone Echo", which seems to use a previously undocumented Oodle Library, v0.1.0, and whose CRCs don't match any other libraries, as posted by KaktoR.

Code:

   
File:  oodle_11_win64.dll
CRC-32: 15a0f169

Attempting to discern which member of the oodle family is used has proven to be a tad tricky, with each sample spitting back a result of [0] (Invalid/Unknown) as reported by Razor's Oodle Stream Scanner (oo2scan_7_win64.exe) utility. Attached is both a sample and the library in question. Since these streams are coming up as Invalid/Unknown is it not possible to re-compress them?

KaktoR 20-09-2023 07:57

I wouldn't say it is undocumented. The problem is that the game in question is exclusively for the oculus rift vr thing. Libraries can differ on different platforms.

And if I look at all the code in shared library, it differs much from the pc equivalent.

IRAQIGHOST 21-09-2023 14:44

Witchfire

Quote:

xtool:mzlib:dd3+LOLZ_ULTRA
Repack Size: 16.3 GB
Final Size: 23.5 Gb

KaktoR 28-09-2023 07:59

Cyberpunk 2077

I did a bit of digging in last few days. Here are some observations. Maybe you might find it usefull.

Code:

audio*.archive
lang_*.archive

  • Contains WEM's, BNK's and BIN's.
  • For the majority of WEM's and BIN's you can use WEMTool from Masquerade (rename *.bin to *.wem first).
  • For some of the files you have to use MSC, look out for this. Honestly I did not have checked how much files are affected by this, and I don't even know if there are tools out there which can check for specific blocks or decoded text to separate the files to be used by WEMTool or MSC.

Code:

basegame_5_video.archive
ep1_2_gamedata.archive

These two files contains bink2 files, for which you can use the xtool bink plugin. However the file "ep1_2_gamedata.archive" also contains other files alongside with bink2's, so I guess it should be wise to process the file together with all the other archive files if you don't want to extract the file entirely. However the binks are stored as plain text, so you can patch them out of the file and compress the extracted binks together with the "basegame_5_video.archive" file. Patch them back afterwards (not tested). Bink2 files have *.bin extension.

---

You can unpack the archives with WolvenKit.
I used the console version and used command like this
Code:

WolvenKit.CLI.exe <command> <InputFile> <option><OutputFolder>

WolvenKit.CLI.exe extract "G:\SteamLibrary\steamapps\common\Cyberpunk 2077\archive\pc\content\audio_1_general.archive" -o"E:\Test\audio_1_general"


Hexagon123 28-09-2023 13:10

Quote:

Originally Posted by KaktoR (Post 502650)
Cyberpunk 2077

I did a bit of digging in last few days. Here are some observations. Maybe you might find it usefull.

Code:

audio*.archive
lang_*.archive

  • Contains WEM's, BNK's and BIN's.
  • For the majority of WEM's and BIN's you can use WEMTool from Masquerade (rename *.bin to *.wem first).
  • For some of the files you have to use MSC, look out for this. Honestly I did not have checked how much files are affected by this, and I don't even know if there are tools out there which can check for specific blocks or decoded text to separate the files to be used by WEMTool or MSC.

Code:

basegame_5_video.archive
ep1_2_gamedata.archive

These two files contains bink2 files, for which you can use the xtool bink plugin. However the file "ep1_2_gamedata.archive" also contains other files alongside with bink2's, so I guess it should be wise to process the file together with all the other archive files if you don't want to extract the file entirely. However the binks are stored as plain text, so you can patch them out of the file and compress the extracted binks together with the "basegame_5_video.archive" file. Patch them back afterwards (not tested). Bink2 files have *.bin extension.

---

You can unpack the archives with WolvenKit.
I used the console version and used command like this
Code:

WolvenKit.CLI.exe <command> <InputFile> <option><OutputFolder>

WolvenKit.CLI.exe extract "G:\SteamLibrary\steamapps\common\Cyberpunk 2077\archive\pc\content\audio_1_general.archive" -o"E:\Test\audio_1_general"


Use VGMStream to check audio info.

KaktoR 29-09-2023 10:24

Cyberpunk 2077
v2.0, incl DLC, 2 languages, incl bonus content

Code:

08:53:27 - Selected ARC/DS method for Data1a-01.bin was: xtool:mcp2077+xtool:lm:c1024mb:mbk2:df20p+xtool:dd3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 08:53:27 - Selected ARC/DS method for Data1b-01.bin was: xtool:mcp2077+xtool:dd3+msc_tak+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
 08:53:27 - Selected ARC/DS method for Data1c-01.bin was: xtool:dd3+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
-----------------------------------------------------------------------------------------
 12:55:21 - Overall input size: 83.05 GB
 12:55:21 - Overall output size: 62.31 GB (Ratio 75.02%)
 12:55:21 - Overall conversion time: 04:01:50

Same as above but packed all archive files together

Code:

Overall output size: 62.60 GB (Ratio 75.38%)

kuyhaa 04-10-2023 12:22

hi, any info for Tony Hawk's Pro Skater 1 + 2 ?

Code:

unreal + 0x94DEC78FBD9762E1674A3418 BFE8637587659532C1C053D1F4AA650E2176286C

Code:

Streams: 49312 / 49737 49749
Time: 00:01:24 (CPU 00:33:41)

Size: 21.3 GB >> 21.3 GB

100.0%
Errorlevel=0

Compressed 45 files, 22,819,124,966 => 22,820,235,796 bytes. Ratio 100.00%
Compression time: cpu 94.69 sec/real 192.80 sec = 49%. Speed 118.35 mB/s
All OK

it seem not work, may be anyone have other aes KEY, i try Fmodel, pak file still encrypt

Hexagon123 04-10-2023 13:06

Quote:

Originally Posted by kuyhaa (Post 502708)
hi, any info for Tony Hawk's Pro Skater 1 + 2 ?

Code:

unreal + 0x94DEC78FBD9762E1674A3418 BFE8637587659532C1C053D1F4AA650E2176286C

Code:

Streams: 49312 / 49737 49749
Time: 00:01:24 (CPU 00:33:41)

Size: 21.3 GB >> 21.3 GB

100.0%
Errorlevel=0

Compressed 45 files, 22,819,124,966 => 22,820,235,796 bytes. Ratio 100.00%
Compression time: cpu 94.69 sec/real 192.80 sec = 49%. Speed 118.35 mB/s
All OK

it seem not work, may be anyone have other aes KEY, i try Fmodel, pak file still encrypt

Try QuickBMS if possible?

Masquerade 04-10-2023 14:42

Tony Hawk uses a non-standard encryption scheme such that decryption is possible using the attached script, but encryption is not.

https://files.catbox.moe/08vsy7.bms

prince55 04-10-2023 23:03

Forza Motorsport best method !
 
I tried xtool but can't precompress it, and lolz only does nothing
What i miss here :confused:

Masquerade 04-10-2023 23:28

^^
Please may you share a small sample?
Edit: https://www.fileforums.com/showpost....&postcount=383


All times are GMT -7. The time now is 18:53.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com