XTool 2020 (Database Maker)

[Razor12911]

Database Maker/Generator is part of xtool and this post is dedicated to explaining how it works.


If xtool does not have a plugin designed to work for a certain input or game but if for example you managed to extract (not unpack) streams a certain file has then you order xtool to create a plugin in a form of a database file which can then be used within the program.

The syntax is as follows:
xtool generate [parameters] extracted_streams original_data database_output
Parameters:
-m# - codec to use for precompression
-c# - scanning range of precompressor [16mb]
-t# - number of working threads [50p]

the inputs of both extracted_streams and original_data can either be a file or a folder (wildcards/masks) are accepted.

Example:
xtool.exe generate -mxcompress "VehicleHRLiveries.bff_extract\*" "VehicleHRLiveries.bff" pcars.xtl

[Razor12911]

bms2xtool is a separate project that helps generate database files from quickbms scripts.

Since this forum is either incredibly lazy or they can't code as there are no plugins for xtool submitted on the forum I thought of creating such a tool, QuickBMS has a lot of scripts made for a lot of games and has a community dedicated to making more of them which xtool doesn't. :o

The syntax is as follows:
bms2xtl script game_files database

the inputs of game_files can either be a file or a folder (wildcards/masks) are accepted.

Required files:
quickbms.exe or quickbms_4gb_files.exe
Get them from here (http://aluigi.altervista.org/quickbms.htm)

Source:
https://github.com/Razor12911/bms2xtool

[Razor12911]

I'm spamming forum

[Python-Boy]

I'm spamming forum
Then don't do that coz this is month of february not April.:D

[KaktoR]

I'm spamming forum

Please, spam a bit more of your usefull stuff :D

[Razor12911]

Dunia/Disrupt Engine Database maker

Instructions

Drag and drop game folder on make_db.bat, a database file (*.xtl) will be created

Supported games

Far Cry 3
Far Cry 4
Far Cry 5
Far Cry 6
Far Cry New Dawn
The Crew 2
Watch Dogs 1 (use bms2xtl)
Watch Dogs 2
Watch Dogs Legion

[L33THAK0R]

Dunia/Disrupt Engine Database maker

Instructions

Drag and drop game folder on make_db.bat, a database file (*.xtl) will be created

Supported games

Far Cry 3
Far Cry 4
Far Cry 5 (needs testing)
Far Cry 6 (I'm waiting for you)
Far Cry New Dawn (needs testing)
The Crew 2
Watch Dogs 1 (wait for xtool v0.4+ update)
Watch Dogs 2 (wait for R2)
Watch Dogs Legion (wait for R2)

Is "Far Cry 2" able to work with this? As far as I know it is one of two titles to use Dunia 1 (the other being the Ubisoft licensed "Avatar" title).

[Razor12911]

upload common.dat, common.fat and all other *.fat files as samples.

[:( Sad8669]

Razor, do you need WDL samples?

I asked because i have the game files right now.

[L33THAK0R]

upload common.dat, common.fat and all other *.fat files as samples.

Sample upload is structured as it appears in the title's directory (https://mega.nz/folder/WYIQyBJY#r1aXJzOA4jPb9488Z-qSBA), as some files have identical names. The second title mentioned "James Cameron's Avatar: The Game" does not have a sample included, as it does not seem to have any ".fat", or ".dat" files present, despite it using the same iteration of the Dunia engine as "Far Cry 2". It instead seems to house all of its required assets within ".pak" files (https://imgur.com/a/w07SxCT).

[KaktoR]

Here are some for testing

PS: db maker named them all as fc3.xtl (maybe it's a bug), so I have renamed them.

[ffmla]

Here are some for testing

PS: db maker named them all as fc3.xtl (maybe it's a bug), so I have renamed them.
some doubt arises.

did you change the fc3.xtl name in bat file?
for /R "%~1\" %%a IN (*.fat) do dunia2_db "%%a" fc3.xtl

[KaktoR]

some doubt arises.

did you change the fc3.xtl name in bat file?
for /R "%~1\" %%a IN (*.fat) do dunia2_db "%%a" fc3.xtl

Good point. Answer is no :D

[Razor12911]

Unreal Engine ucas Database maker

Instructions

Open batch file
Insert the name of game, enter the decryption key and then save
Drag and drop game folder on make_db.bat, a database file (*.xtl) along with key file (*.key) will be created

Use these with xtool.

[Razor12911]

Unreal Engine ucas Database maker updated

Changes

- Fixed decryption issues

Notes

That's all I can do for now, fixed some issues during my spare time. I think there's still a few left.

[Razor12911]

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for Far Cry 6

Notes

No idea if this works on the whole game, I tested on 1 sample.

Far Cry 6\data_final\pc\common.dat
Tested 2 files, 1,287,326,620 => 684,093,252 bytes. Ratio 188.18%
Testing time: cpu 0.80 sec/real 11.15 sec = 7%. Speed 61.36 mB/s

[KaktoR]

For FC6 use lz4 v1.8.2

fctber_disk.dat
Tested 2 files, 3,646,603,232 => 2,313,847,968 bytes. Ratio 157.60%
Testing time: cpu 0.86 sec/real 17.36 sec = 5%. Speed 133.32 mB/s

[Razor12911]

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for all* fat files in Watch Dogs Legion

Notes

I have no idea if all fat files are supported (I don't have game, only source is torrent files which my ISP does not appreciate so...)
This is perhaps why I stopped pirating games and started buying them

You'll need to wait for next xtool update for this to work, something is up with the main program causing it to crash.

[KaktoR]

Watch Dogs Legion

common.dat: 32781 stream(s)
commonengine.dat: 22 stream(s)
patch.dat: 113756 stream(s)
patch_brazilian.dat: 7255 stream(s)
patch_english.dat: 82028 stream(s)
patch_french.dat: 9519 stream(s)
patch_german.dat: 8264 stream(s)
patch_italian.dat: 7126 stream(s)
patch_japanese.dat: 3600 stream(s)
patch_russian.dat: 6981 stream(s)
patch_spanish.dat: 4819 stream(s)
shadersobj.dat: 168500 stream(s)
london.dat: 103303 stream(s)
london_brazilian.dat: 84705 stream(s)
london_cache.dat: 27030 stream(s)
london_english.dat: 84690 stream(s)
london_french.dat: 83419 stream(s)
london_german.dat: 84721 stream(s)
london_hires.dat: 41591 stream(s)
london_italian.dat: 84720 stream(s)
london_japanese.dat: 84718 stream(s)
london_preload.dat: 43551 stream(s)
london_russian.dat: 85156 stream(s)
london_spanish.dat: 84667 stream(s)
london_ultra.dat: 12124 stream(s)
Database created successfully

[Razor12911]

Information on main post added

Changes

- Added information showcasing database generation feature

[Razor12911]

Information on QuickBMS to Xtool added

Changes

- Added information and attachment showcasing bms2xtool (bms script to xtl database file)

[dixen]

I try it on LEGO I.J but


d:\RePack_tools\bms2xtool_0.1>bms2xtl ttgames.bms "GAME.DAT" lego.xtl
Exception: File operation not supported
UPD.
LEGO Batman The Videogame - same error((

[Masquerade]

I try it on LEGO I.J but



UPD.
LEGO Batman The Videogame - same error((

Recompression of LZ2K is not supported. QBMS can only extract LZ2K, not compress.

[dixen]

Recompression of LZ2K is not supported. QBMS can only extract LZ2K, not compress.
Damn..I forgot about that

[Razor12911]

QuickBMS to XTool updated

Changes

Fixed issues with some scripts not being parsed correctly (thanks Masquerade)
Added support for xtool v0.4+
Made overall progress to be visible from taskbar

Notes
You can find source code from here (https://github.com/Razor12911/bms2xtool)

[Razor12911]

QuickBMS to XTool updated

Changes

- Added user input option (auto write to cmd window any common prompts)
- Fixed issue with string to binary conversion of keys
- Fixed issue with handling streams that are both encrypted and compressed
- Fixed issue with storing incorrect decryption keys

Notes

It can be sometimes difficult to properly detect streams when you use xtool directly, especially oodle streams if decompressed streams is unknown however if a quick bms script was made, you can take advantage of this and quickly generate a database to use in xtool. Same goes for unreal engine games where ue4d plugin seems to not work as intended, you can loaded up the unreal_tournament_4 script, enter the key and everything should be taken care of.

Some results on Project Cars 3

bdt+oo2reck:
Compressed 1 file, 538,214,104 => 1,028,286,891 bytes. Ratio 191.06%
Compression time: cpu 0.77 sec/real 160.61 sec = 0%. Speed 3.35 mB/s

bdt+xtool:kraken:
Compressed 1 file, 538,214,104 => 1,035,969,556 bytes. Ratio 192.48%
Compression time: cpu 0.77 sec/real 141.78 sec = 1%. Speed 3.80 mB/s

xtool:pcars3 using (nfsshift.bms script):
Compressed 1 file, 538,214,104 => 1,058,023,182 bytes. Ratio 196.58%
Compression time: cpu 0.69 sec/real 25.66 sec = 3%. Speed 20.98 mB/s

Not only you get better results but the process is also faster. :)

[dixen]

Test database for Mortal Kombat 11

XTool+mk.xtl

Pack
Compressed 2 files, 1,694,239,720 => 3,013,304,346 bytes. Ratio 177.86%
Compression time: cpu 1.53 sec/real 137.23 sec = 1%. Speed 12.35 mB/s
All OK

Unpack
Tested 2 files, 3,013,304,346 => 1,694,239,720 bytes. Ratio 177.86%
Testing time: cpu 1.78 sec/real 120.46 sec = 1%. Speed 14.06 mB/s
All OK

*Requied oo2core_5_win64.dll from game folder

Xtoo(0.42)+mk11.xtl+srep+lzma2

Compressed 3,786 files, 81,540,296,959 => 59,769,328,360 bytes. Ratio 73.30%
Compression time: cpu 838.47 sec/real 23004.64 sec = 4%. Speed 3.54 mB/s
All OK
**Requied ~300 gb for decompressiion and srep proccess

[Razor12911]

QuickBMS to XTool updated

Changes

- Added safe mode feature

Notes

This issue is not related to bms2xtl but it's related to QuickBMS itself and was detected by KaktoR where quickbms will run out of memory when trying to process some streams especially ones where oodle is used, since quickbms internally uses old oodle library taken from Warframe, newer games seem to cause the old library to possibly have a memory leak.

Safe mode mitigates this issue (only use it if you have an issue when QuickBMS produces an error related to memory or something similar and it should be kept disabled under normal use)

[Luminior]

Sword and Fairy 7
Method: bms2xtool v0.3 + xtool v0.4.2 + srep + lolz

Compressed 16 files, 69,496,034,241 => 34,281,984,937 bytes. Ratio 49.33%
Compression time: cpu 1290.97 sec/real 41904.95 sec = 3%. Speed 1.66 mB/s
All OK

Notes: other libraries were not used.

[Razor12911]

QuickBMS to XTool updated

Changes

- Fixed issue with file inputs smaller than 4MB

[Razor12911]

Unreal Engine ucas Database maker updated

Changes

- Added support for new structure seen used in Ghostwire Tokyo

Notes

This still uses the old xtool 0.3.21 database structure, wait for another update if you plan on using it with 0.4+

[kuyhaa]

Unreal Engine ucas Database maker updated

Changes

- Added support for new structure seen used in Ghostwire Tokyo

Notes

This still uses the old xtool 0.3.21 database structure, wait for another update if you plan on using it with 0.4+

Compressing 2 files, 850,906,556 bytes
Compressing Snowfall-WindowsNoEditor_0_P.pak
Compressing Snowfall-WindowsNoEditor_0_P.ucas
Compressed 2 files, 850,906,556 => 850,907,110 bytes. Ratio 100.00%
Compression time: cpu 6.33 sec/real 8.72 sec = 73%. Speed 97.60 mB/s
All OK

-mgwtokyo+zlib -d1

unreal gwtokyo.xtl and gwtokyo.key near xtool

have wrong ?

[Razor12911]

kraken was used, not zlib

[Masquerade]

kraken was used, not zlib

Compressing 3 files, 851,793,200 bytes
Compressing Snowfall-WindowsNoEditor_0_P.pak
Compressing Snowfall-WindowsNoEditor_0_P.ucas
Compressing Snowfall-WindowsNoEditor_0_P.utoc
Compressed 3 files, 851,793,200 => 851,805,895 bytes. Ratio 100.00%
Compression time: cpu 0.84 sec/real 145.87 sec = 1%. Speed 5.84 mB/s
All OK

[External compressor: gwt]
header = 0
packcmd = xtool precomp -mgwtokyo+kraken -c256mb -t16 - - <stdin> <stdout>

xtool v3.2.1, oo2core_9_win64.dll

:confused::confused:

[Razor12911]

Compressing 3 files, 851,793,200 bytes
Compressing Snowfall-WindowsNoEditor_0_P.pak
Compressing Snowfall-WindowsNoEditor_0_P.ucas
Compressing Snowfall-WindowsNoEditor_0_P.utoc
Compressed 3 files, 851,793,200 => 851,805,895 bytes. Ratio 100.00%
Compression time: cpu 0.84 sec/real 145.87 sec = 1%. Speed 5.84 mB/s
All OK

[External compressor: gwt]
header = 0
packcmd = xtool precomp -mgwtokyo+kraken -c256mb -t16 - - <stdin> <stdout>

xtool v3.2.1, oo2core_9_win64.dll

:confused::confused:

-d1 :(

[dixen]

oo2core_7_win64.dll

*.pak
Streams: 401/402/402
Time: 00:00:23 (00:03:24)
Memory: 392 MB (392 MB)

100%
Errorlevel=0
Compressed 1 file, 622,183,516 => 848,346,910 bytes. Ratio 136.35%
Compression time: cpu 0.55 sec/real 25.16 sec = 2%. Speed 24.73 mB/s
All OK

*.ucas
Streams: 5550/5564
Time: 00:02:03 (00:08:04)
Memory: 776 MB (776 MB)

100%
Errorlevel=0
Compressed 1 file, 228,723,040 => 910,585,777 bytes. Ratio 398.12% 100%
Compression time: cpu 0.20 sec/real 124.58 sec = 0%. Speed 1.84 mB/s
All OK

[kuyhaa]

oo2core_7_win64.dll

*.pak


*.ucas

oo2core_7

Compressing 228,723,040 bytes

xtool321.exe precomp -mgwtokyo+kraken -c256mb -d1

XTool is created by Razor12911

Streams: 38/5640
Time: 00:02:53 (00:27:21)
Memory: 128 MB (128 MB)

100%

confiused :(

[dixen]

confiused :(oo2core_7_win64.dll 569.0 Кб

[kuyhaa]

oo2core_7_win64.dll 569.0 Кб

solved

[Razor12911]

Unreal Engine ucas Database maker updated

Changes

- Added support for newer xtool 0.4+

[Cesar82]

Unreal Engine ucas Database maker updated

Changes

- Added support for new structure seen used in Ghostwire Tokyo

Notes

This still uses the old xtool 0.3.21 database structure, wait for another update if you plan on using it with 0.4+

Unreal_db_R3 tool stopped working for Scarlet Nexus (test using samples).
The R2 version works correctly for the samples I have.
See attaced video file for more details.

[Razor12911]

Try R4

[Cesar82]

Try R4

The R4 version works perfectly (with XTool 0.4.6).
-------------------------------------------------------------------------------------------------------------------------------------------
Creating Data1a-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
Creating xtool plugin database...
global.ucas: 2 stream(s)
pakchunk0-WindowsNoEditor.ucas: 325 stream(s)
Database created successfully!
FreeArc 0.67 (March 15 2014) creating archive: Conversion_Output\Data1a-01.bin
Compressed 6 files, 274,587,614 => 660,942,430 bytes. Ratio 240.70%
Compression time: cpu 0.22 sec/real 6.21 sec = 4%. Speed 44.23 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------------
Extracting Data1a-01.bin file, Please Wait...
-------------------------------------------------------------------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) extracting archive: Conversion_Output\DISK\Data1a-01.bin
Extracted 6 files, 660,942,430 => 274,587,614 bytes. Ratio 240.70%
Extraction time: cpu 0.23 sec/real 5.78 sec = 4%. Speed 47.53 mB/s
All OK

[hwang4]

Masquerade , can you please upload "gwtokyo.xtl" & the better "oo2core" dll to use with ghostwire tokyo?thx :)

[hwang4]

hi ! need "oo2core_7_win64.dll 569.0 Кб " ! any help?THX:)

[Razor12911]

Dunia/Disrupt Engine Database maker updated

Changes

- Added support for newer xtool 0.4+
- Included updated libdunia.dll library for Watch Dogs series

[Razor12911]

try adding zipcrypto in xtool.ini

[KaktoR]

Dunia/Disrupt Engine Database maker updated

Changes

- Fixed/removed wrong parameter in source code which had the effect that after every processed file would reset the created xtl database so the xtl database only contained streams for the last processed file

PS: Razor12911 is on vacation

[dixen]

I was hoping so much...
Far Cry 5 - Common. fat & .dat

XTool v065 + fc5.xtl (maked by new version Dunia Database Maker)
178 mb > 263 mb

DeLZOREC
178 mb > 531 mb

[KaktoR]

I made a new test

FC3
Compressed 2 files, 192,353,931 => 445,118,342 bytes. Ratio 231.41%
Compression time: cpu 0.09 sec/real 8.21 sec = 1%. Speed 23.44 mB/s
All OK

FC4
Compressed 2 files, 128,161,319 => 304,134,400 bytes. Ratio 237.31%
Compression time: cpu 0.08 sec/real 6.21 sec = 1%. Speed 20.64 mB/s
All OK

FCNewDawn
Compressed 2 files, 175,906,959 => 297,921,068 bytes. Ratio 169.36%
Compression time: cpu 0.06 sec/real 2.26 sec = 3%. Speed 77.85 mB/s
All OK

FC6
Compressed 2 files, 1,189,251,048 => 2,039,057,467 bytes. Ratio 171.46%
Compression time: cpu 0.48 sec/real 31.33 sec = 2%. Speed 37.96 mB/s
All OK

WDL
Compressed 2 files, 1,745,094,516 => 1,941,707,042 bytes. Ratio 111.27%
Compression time: cpu 0.69 sec/real 40.02 sec = 2%. Speed 43.61 mB/s
All OK

[dixen]

Project CARS 2 Database for XTool 0.53

VehicleHRLiveries.bff

XTool v0.53 + pCARS2.xtl

Compressed 1 file, 935,792,689 => 12,945,491,383 bytes. Ratio 1383.37%
Compression time: cpu 1.28 sec/real 91.77 sec = 1%. Speed 10.20 mB/s
All OK

(All *.bff files - 45.6 gb > 102 gb)

BDT+OO2RECK

Compressed 1 file, 935,792,689 => 11,888,359,364 bytes. Ratio 1270.41%
Compression time: cpu 1.38 sec/real 459.26 sec = 0%. Speed 2.04 mB/s
All OK

https://drive.google.com/file/d/1-HgRPBcMhw_7SQ-dTV5dtJKD6n3Tmkck/view?usp=sharing

Note.
Requied -d1 parameter for XT

[deepshit]

Can someone help me with fast and furious crossroads *.bff files?
I'm trying to create database for this game using the bms script. database is created but using xtool and created database only decrypts the files. Im using -d1 but no decmpression happens. game uses leviathan.
Here is a sample file : https://www.mediafire.com/file/lg9tqgl793fj2em/VehicleHRLiveries.bff/file

and bms script is attached.
game uses oo2core_6_win64.dll

[Masquerade]

Can someone help me with fast and furious crossroads *.bff files?
I'm trying to create database for this game using the bms script. database is created but using xtool and created database only decrypts the files. Im using -d1 but no decmpression happens. game uses leviathan.

Working fine for me on the sample you provided using this config (using oo2core_8_win64 from Project CARS 3):

[Razor12911]

Several programs updated

Changes

- bms2xtl, dunia2_db and unreal_db updated to support xtool 0.7+