XTool 2020 (Main Project) [Archive] - Page 3

elit

03-07-2022, 14:55

During repacking of Final Fantasy XIII(PC version), I found that using xtool:d1:c64m:mzlib:mreflate cause crc error during unpacking of the game. Cause is almost certainly due to combining zlib with reflate as alone either one work. Maybe they are not together compatible with depth option, only d0?

[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t50p {options} - - <stdin> <stdout>
unpackcmd = _EC\xtool\xtool decode -t50p - - <stdin> <stdout>

EDIT: Its probably something else, I still got error today after using only reflate. Its strange I don't tend to get these kind of errors for no reason:

32015
32016

-----------------------------------------------------------------------------------
EDIT2:
Found the problem! It's with reflate. If I use zlib instead:
xtool:d1:c32m:mzlib+srep64:m3f:mem8g:a0+4x4:lzma:6 4mb:normal:32:lc8
, it works. I don't know why, in other cases I never saw issue with reflate.

ADDENUM: It's due to depth. Anything higher than -d0 cause failure during decompression. Zlib and preflate do not have this problem. Data tested is 'sys' directory(2.8gb) in Final Fantasy XIII(PC, fitgirl version).

Cesar82

03-07-2022, 15:14

elit

03-07-2022, 15:44

I could be wrong, but using this method line, I think something like this should be used in arc.ini:

Nope, still throws the error. Also it compress fine and show correct chain on archive info:
xtool:d1:c64mb:mzlib:mreflate+srep64:m3f:mem8g:a0+ 4x4:lzma:64mb:normal:32:lc8
, which is same regardless if I use {options} or { -option}

EDIT: Found the problem, see 2 posts above.

elit

03-07-2022, 15:58

Also, aside from what I stated above, there was another interesting thing I found. When inflating(with reflate option), if I use c32m-c128m range, xtool process about 25150 streams. Then when I use -c256m, it only process ~22000 streams. And yet, if I use -c512m option, it again process ~25150 streams. All other options same. Go figure.

elit

03-07-2022, 16:40

And even more interesting, reflate is actually faster than zlib at same options(with depth 1 and chunk size of 32m at least)???:

32013
32014

elit

04-07-2022, 05:20

And preflate is worse than zlib hmmm..:

32017

Razor12911

07-07-2022, 01:53

Update available

Changes

- added wav stream detector
- added flac codec
- added jpg stream detector
- added packjpg, brunsli, jojpeg codec
- added feature that allows input to be a directory
- added feature to extract detected streams
- updated database feature
- updated deduplication feature
- IO function decode updated

Notes

I have added wav audio compression as an alternative of msc for people who has issues with it or don't want to use Freearc's tta codec as it processes wav files individually. One thing to note, it's not as good as tta, tak or even frog as it is based on flac codec (It was the only open source codec I could work with...).

Also added jpg image compression codecs, you can pick between packjpg, brunsli or jojpeg. packjpg seems to have a memory leak, brunsli is fast but cannot deal with large jpg images and jojpeg can be used if you're after ratio as it is paq based (seems to be buggy so stick to packjpg or brunsli for now)

A new parameter is added --extract=[path], this will extract all detected streams to a directory... if you're interested in the streams themselves.

database and deduplication features have been updated and can now be used at all times without worrying about crc collisions.

Special thanks to KaktoR for running several tests and Shelwien for providing compiled libraries for brunsli and jojpeg.

@elit
These issues will be investigated in time.

dixen

07-07-2022, 02:52

Find little uncritical bug

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 1/1
Time: 00:00:02 (00:00:02)
Memory: 276 MB (276 MB)

100%
Errorlevel=0
Compressed 1 file, 77,790,338 => 53,378,764 bytes. Ratio 68.62%
Compression time: cpu 0.08 sec/real 2.78 sec = 3%. Speed 28.01 mB/s
All OK

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC --verbose -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 0000000001FFFFFF (33554432)
EAccessViolation: Access violation at address 0000000000409A77 in module 'xtool.exe'. Read of address 00007FF4FE6BFBA4

Errorlevel=1

ERROR: general (de)compression error in xtool

Anyway, thank Razor for update and new functions)

One question. Why TTA but not TAK?

Original.wav 74.1 mb
ENCTTA.tta 50.3 mb
ENCTAK.tak 48.7 mb

Masquerade

07-07-2022, 03:40

One question. Why TTA but not TAK?
TAK isn't Open Source.

elit

07-07-2022, 03:55

Interesting, I should note that compression difference between different wav codecs is minimal. Also for me in the past, I found wavpack the best, It handle more formats(WAV/BWF/RF64, Sony Wave64, Apple CAF, Philips DSDIFF, and Sony DSF (all with no size limitations)), more modes(8, 16, 24, and 32-bit integer PCM; 32-bit float PCM; DSD audio; mono, stereo, and multichannel; sampling rates from 6 to 192 kHz (and non-standard rates)), RIFF chunks, is open source, does support STDIO which for me work unlike flac, which for some reason during decompression kept running past original size inflating to infinity(when used in FA with stdio mode).

However, flac is great too because is non-symetric so decompression is fast(although wavpack also seem to support asym. cmp.).

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

Prince4

07-07-2022, 04:01

Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

Razor12911

07-07-2022, 05:27

Find little uncritical bug

your chunk size is too small for that input
Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

0.3.21 is an old version that supported old plugins and it is kept there until all plugins are updated to support 0.4+ versions, use 0.6.0 if you are not using old plugins.

Interesting, I should note that compression difference between different wav codecs is minimal.

https://wiki.hydrogenaud.io/index.php?title=Lossless_comparison

I picked codecs based on the information provided here, it's not that I haven't tried wavpack. I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

That's pretty much how it is intended to work, wav files are scanned within files and then fed to codecs.

elit

07-07-2022, 09:43

I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

That is optional feature with extra switches, by default it work same as flac - no lossy wav. As for thread safety, I believe its single threaded app so could be used chunk per thread but I don't know for sure.
I am also perfectly ok with flac as well so don't worry about it, I just though this one may have been better due to more formats supported.
https://www.wavpack.com

Razor12911

12-07-2022, 11:43

Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example
xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

CR2032

13-07-2022, 06:30

L33THAK0R

13-07-2022, 07:01

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

:( Sad8669

13-07-2022, 08:06

As far as i know, it was added to increase the portability of xtool.

Notes:
I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Cesar82

13-07-2022, 11:29

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -
XTool is going independent, abandoning its helpers, lol.
Maybe in the future it will get its own final compression with high compression gain and higher performance than lolz... It would be great!

Masquerade

13-07-2022, 12:14

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor :)

@Razor thanks for the continued updates!

CR2032

14-07-2022, 14:45

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.
Hello.
No. It's just a creative and really a gifted step, not more.:)

CR2032

14-07-2022, 14:55

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor :)

@Razor thanks for the continued updates!
Hello.
Exactly like that! Like a Precomp compression with the key -c.:)

Razor12911

15-07-2022, 10:50

Update available

Changes

- added feature to inject libraries to main executable

Notes

You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage
xtool.exe inject dll_file

More notes

Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.

Cesar82

15-07-2022, 11:57

CR2032

15-07-2022, 15:55

CR2032

15-07-2022, 16:19

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?
Hello.
1. Perhaps, just replace old to new.
For example:
If I have xtool.exe with integrated libraries:
xtool.exe inject _libraries\fast-lzma2.dll
xtool.exe inject _libraries\preflate_dll.dll
xtool.exe inject _libraries\zlibwapi.dll

And I want to replace only one a new version fast-lzma2.dll, then:
xtool.exe inject _libraries\fast-lzma2.dll

Or probably, every time there is a new version of xtool, you need to create a new integration.

3. Yes.

elit

17-07-2022, 18:15

I have found something that seem like a bug, and I am not sure if its FA or xtool, but likely an xtool issue.

Say you precomp a file with reflate, but you have an xtool.ini config that contain all kind of codecs, like xmemcompress, quickbms based and so on. But you only used reflate.

Now to decode successfully, you have to have all those exe files + untouched xtool.ini otherwise xtool will throw general error! Even if you don't need any of those for decompression.
That just happened to me. I had to copy all those pointless exe's and exact same xtool.ini for it to work. If for example I edited out those unneeded codecs from xtool.ini and/or deleted crilayla.exe or xmemcompress.exe, then xtool won't decode my reflate pack!

EDIT: No wonder it fails:
32214
Why it have to save all those things if it's not needed?

Razor12911

17-07-2022, 22:24

Razor12911

17-07-2022, 22:29

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

1) yes
2) possibly, in the main post there is actually link to the older releases I just removed 0.3.21 to make people not ask what is the different between this version and the recent update.
3) yes
4) imperfect streams use xdelta for patching, this function initially came from a dll but I separated all dlls from the main executable and gave the options to the user to include such a feature.

elit

21-07-2022, 02:50

Greetings Razor.
Back on page 34 where I wrote about reflate problems, causing crc errors... This time I was able to catch in on console:

32246

But the thing is, exact same run may pass successfully on second or more tries(and then maybe fail again, so random dice per run). I am starting to be cautious about HW issue possibility on my side, but then I had no such issues with zlib yet. Using -t8.

Looks like either thread race issue, or my HW memory. This is on xtool v0.5.3. I will investigate this further.

EDIT:
Latest v0.6.2 == same issue.
If xtool do not fail during compression(precomp), then it will always recompress successfully during unpacking - i.e. probably not HW issue after all.
Last stable version without this bug that I have is 0.3.21. Cannot replicate on it. I believe this one is stable. Unfortunately it miss a lot of streams that newer xtool-reflate can see.
EDIT2:
Forgot to add. Changing chunk size(or not using depth) can help sometimes, but on big enough data stream one chunk size may pass at one point and fail at different one(where again some other size could work and so on). Also, I started having these issues only recently which correspond with me upgrading to later xtool version than I was stuck with.
Also, I think this have way more likelihood of happening on files that do not contain(or very little) deflate chunks. When FA reported CRC fails it was pretty much files that happen to share same extension with those that do.
EDIT3:
Yup, not my HW. I just threw whole Halo Master Collection directory to xtool 0.3.21(without filter) and it completed successfully. Then I tried both v0.5.3 and v0.6.2 and it failed on address violation.
EDIT4:
I will further confirm it later here, but using [I]-lm *may* work as a workaround for reflate. Not due to a memory shortage, but the way chunks/data and threads are handled. No, it didn't help. [ADD: neither did -t1]
EDIT5:
Preflate works without issues, so I had to settle with it. Final repack size is ~1gb bigger though. I do hope reflate get fixed in the future as I prefer it.

Razor12911

25-07-2022, 08:42

samples?

elit

26-07-2022, 03:18

samples?

Unfortunately this wouldn't be possible, it need to be tested on whole Halo Master Collection directory. Because you are unlikely to trigger on small data. Even different arrangement of files(sorting) or chunk size can affect where or if the issue happen at all. Closest I was able to get for smaller data sample were 3 ipak files which is still ~1gb and less reliable to replicate. This could be a rare byte sequence that cause this. And if so I don't know if it's within reflate dll or xtool.
Best way to test is to run it on whole Halo MCC directory without any filter(all files), even better with FA gui instead cmd. That is the closest I know.

L33THAK0R

06-08-2022, 03:15

Is there a list of every valid codec/method that XTool supports? I'm talking about the ones outside of the codecs listed in the included documentation. It seems like a silly question but I havent kept that close an eye on XTool's development and with the dropping of GrittiBanzli support just thought I'd see if anyone has an up-to date list.

Snake288

07-08-2022, 10:04

KaktoR

07-08-2022, 10:24

KaktoR

07-08-2022, 10:42

Hello Razor12911
Dying Light 2 Stay Human compression XTool 2020 (Database you can provide support ?
Dying Light 2 Stay Human example Data
https://lifeboxtransfer.com/s/5d10431a-c36c-4ab8-a88d-ee4116099aea

Did they change compression? I'm sure in some older version of this game they used just zlib or sort of if I remember correctly.

dixen

07-08-2022, 14:14

Did they change compression? I'm sure in some older version of this game they used just zlib or sort of if I remember correctly.

No changes. ZLIB only in 2 *.pak files. RPACK is no compression

panker1992

26-08-2022, 06:21

Hello, it's been ages since i had an error !!

This Came directly from Xtool documentation
Xtool version used 6.2

this is my wav.ini
[Stream1]
Name=wav
Codec=wavpack
BigEndian=0
Signature=0x46464952
Structure=Signature(4),FileSize(4),FileType(4),Str eam
StreamOffset=-12
CompressedSize=FileSize + 8
DecompressedSize=0
Condition1=FileType = 0x45564157
Condition2=FileSize >= 4096

this is my xtool.ini
[wavpack]
Encode=wavpack.exe -hh -x4 <filein>.wav <fileout>.wv
Decode=wvunpack.exe <filein>.wv <fileout>.wav

i have been unable to make this even detect the sample FLAC internal codec works and so any other
NOTE: i use this as standalone and not in FA for testing purposes

Edison007

26-08-2022, 07:52

xtool.ini:

[wavpack]
Encode=wavpack.exe -hh -x4 <filein> <fileout>
Decode=wvunpack.exe <filein> <fileout>

or

[wavpack]
Encode=wavpack.exe -hh -x4 - - <stdin> <stdout>
Decode=wvunpack.exe - - <stdin> <stdout>

panker1992

26-08-2022, 08:26

Razor12911

05-09-2022, 02:16

Update available

Changes

- added universal lz4f scanner
- fixed issues with database feature
- fixed issues with executable plugin support
- updated lzo codecs

Razor12911

06-09-2022, 04:55

Update available

Changes

- fixed issues with lzo2a and lzo1c codecs

dixen

06-09-2022, 05:26

XTool v 0.6.4

[0] Processed lzo2a stream at 00000000199339FB (6669 >> 32768 >> 6669) using v999 successfully
[0] Processed lzo2a stream at 000000001993540C (6953 >> 32768 >> 6953) using v999 successfully
[0] Processed lzo2a stream at 0000000019936F39 (6965 >> 32768 >> 6965) using v999 successfully
[0] Processed lzo2a stream at 0000000019938A72 (7049 >> 32768 >> 7049) using v999 successfully
[0] Processed lzo2a stream at 000000001993A5FF (7357 >> 32768 >> 7357) using v999 successfully
[0] Processed lzo2a stream at 000000001993C2C0 (6483 >> 32768 >> 6483) using v999 successfully
[0] Processed lzo2a stream at 000000001993DC17 (6438 >> 32768 >> 6438) using v999 successfully
[0] Processed lzo2a stream at 000000001993F541 (7049 >> 32768 >> 7049) using v999 successfully
[0] Processed lzo2a stream at 00000000199410CE (10483 >> 32768 >> 10483) using v999 successfully
[0] Processed lzo2a stream at 00000000199439C5 (9063 >> 32768 >> 9063) using v999 successfully
[0] Processed lzo2a stream at 0000000019945D30 (11212 >> 32768 >> 11212) using v999 successfully
[0] Processed lzo2a stream at 0000000019948900 (13262 >> 32768 >> 13262) using v999 successfully
[0] Processed lzo2a stream at 000000001994BCD2 (19921 >> 32768 >> 19921) using v999 successfully
[0] Processed lzo2a stream at 0000000019950AA7 (14134 >> 32768 >> 14134) using v999 successfully
[0] Processed lzo2a stream at 00000000199541E1 (15579 >> 26778 >> 15579) using v999 successfully

Streams: 22550/22550
Time: 00:00:41 (00:00:38)
Memory: 264 MB (264 MB)

Errorlevel=0

Compressed 1 file, 429,260,800 => 829,963,641 bytes. Ratio 193.35%
Compression time: cpu 0.33 sec/real 42.34 sec = 1%. Speed 10.14 mB/s
All OK

Razor12911

09-09-2022, 00:19

Update available

Changes

- updated oodle scanner
- remove xdelta support from oodle and lzo codecs (crc mismatch often generates large diff files)

Notes

I've been getting reports of xtool taking a long time to process oodle streams (or getting stuck) so I reworked the scanner. I reverse engineered the code of oo2rec (I lost the source code) and rewrote parts of the code so xtool should produce better results while being faster.

There is a parameter rework as well, some people may know of the "n#" parameter which when set increases the chances of xtool capturing a stream. The default value now is 32, can be increased to 64, 128... Up to you, there is no limit just keep in mind that increasing this value also means longer precompression times.

Results on DOOM Eternal's "gameresources_4_1.streamdb"

oo2reck
Compressed 1 file, 1,353,405,353 => 2,584,084,262 bytes. Ratio 190.93%
Compression time: cpu 1.86 sec/real 590.20 sec = 0%. Speed 2.29 mB/s

xtool 0.4.7-0.6.4
Rumour has it that it's still processing as we speak (stuck on 5.7%) :rolleyes:

xtool 0.6.5 (-mkraken:l6:n128)
Compressed 1 file, 1,353,405,353 => 2,589,860,614 bytes. Ratio 191.36%
Compression time: cpu 1.80 sec/real 370.32 sec = 0%. Speed 3.65 mB/s

panker1992

10-09-2022, 03:29

kuyhaa

15-09-2022, 08:07

confiused . .

xt 065 , ( same result with other version )

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi09StmwygiMwNFDlHymDFcJ5knlACn8qEov9U9sJ24PB-iedNuxBDWqzQLg5BOEMPbLIpfoEo6-bYB9ly_P0Kx1TYpoWgV7sPZ8fsYjlQbFtG0MtzqzT1cizhD4Ik UBU7fEH5-AMRmBadyyAgPdA_ZDQ6A4eCtJ4DlEY5tdh2LsOjlsXEazZeuR8 rqCw/s1147/frusta.png

shazzla

28-09-2022, 10:20

Hi guys!

I need a little help....
Something is wrong here. Tried to precomp a file with deduplication ,but XTool 0.6.5 doesnt make it. It looks like it ignores the corresponding parameter,even .bin file not created! V0.3.1 works fine!
What am i do wrong?

[External compressor:x065-dedup]
header = 0
packcmd = xtool065\xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup=chor065.bin - - <stdin> <stdout>

[External compressor:x065-nodedup]
header = 0
packcmd = xtool065\xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>

[External compressor:x031-dedup]
header = 0
packcmd = xtool031\xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup=chor031.bin - - <stdin> <stdout>

[External compressor:x031-nodedup]
header = 0
packcmd = xtool031\xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>

Results :

Xtool 0.3.1 ,deduplicated : 40.2 GB
Xtool 0.3.1 ,not deduplicated : 42.9 GB

Xtool 0.6.5 ,deduplicated : 42.9 GB
Xtool 0.6.5 ,not deduplicated : 42.9 GB

All processes finished without error. V0.6.5 is the "pure" version ,directly from Zee's ZIP-file...

Btw where to insert --verbose command?
Only one place is accepted in my experiments ,but no verbose infos are displayed...

Masquerade

28-09-2022, 10:30

Btw where to insert --verbose command?
Only one place is accepted in my experiments ,but no verbose infos are displayed...

Only use --verbose for testing. It limits XTool to 1 chunk processed at once. Add it before input / output. You shouldn't combine stdin/stdout with --verbose, use $$arcdatafile$$.tmp/$$arcpackedfile$$.tmp instead.

KaktoR

28-09-2022, 10:55

You can make a batch file near xtool.exe and drag&drop a file onto it. This way you don't use freearc for it.

Example
@echo off
cd /D "%~dp0"
xtool.exe precomp -mzlib -c16mb -t100p --verbose - - < %1 > %1.out
pause

shazzla

03-10-2022, 09:29

Nobody ? :(

Found. Damn...
Parameters were changed...

L33THAK0R

14-10-2022, 08:18

Masquerade

14-10-2022, 11:17

Quick question, or I guess just asking for some clarification, in regards to "bms2xtl", can you only process a given file with your generated database, if the codec is one that xtool supports? So for example xtool wouldn't be able to process a COMTYPE if it isn't deflate, lz4(hc/x/f)... etc.? I guess what I'm asking in essence is am I correct in saying external, quickbms-native COMTYPEs aren't supported.

Use xtool.ini to add COMTYPEs that QuickBMS supports. Bear in mind that QuickBMS built in compression can vary wildly from that of what the original input file is compressed with so this is really only suitable for encryption algorithms where you're more likely to get a correct output.

Also, not every algorithm that QBMS can decompress it can also compress - such as LZ2K, of which no compression code exists publicly.

You also have totally different implementations of algos althogether such as LZSS where it's basically different every time you see it.

In short xtool.ini compression with QuickBMS is hit/miss.

shazzla

15-10-2022, 04:59

Hi!

How can i pass the --dedup parameter into the -m chain ?

Masquerade

15-10-2022, 05:31

Hi!

How can i pass the --dedup parameter into the -m chain ?

Just put --dedup into XTool packcmd in arc.ini.

shazzla

15-10-2022, 06:40

Currently using that method. Is it possible to pass --dedup ?!

dixen

15-10-2022, 07:24

Currently using that method. Is it possible to pass --dedup ?!
Just delete --dedup

shazzla

15-10-2022, 07:37

Delete ?!
Why ?
Not really understand....

Edit:
I mean pass --dedup parameter to chain. I dont want a new arc.ini section.

Razor12911

15-10-2022, 14:40

There were undocumented changes in xtool 0.6.5, --dedup can be used as -dd so you are able to pass this via freearc provided your arc.ini looks like this

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -option} -c32mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

then your method will be something like -mxtool:mzlib:dd

shazzla

15-10-2022, 23:46

Thanks ,Z !
If there are other undocumented changes ,it would be nice to publish! :)

bunti_o4u

02-11-2022, 23:21

I am trying to compress Madel of Honor Airborne with xtool but it's not inflating the input files.

Here is the sample:

https://www62.zippyshare.com/v/3b32KZZm/file.html

Result with uelr:
Compressing 2 files, 177,974,335 bytes using uelr:uv

Compressed 2 files, 177,974,335 => 181,783,854 bytes. Ratio 102.1%
Compression time: cpu 0.13 secs, real 2.30 secs. Speed 77,496 kB/s
All OK

[External compressor:uelr]
header = 0
packcmd = uelr {option} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

Result with xtool:
Compressing 2 files, 177,974,335 bytes using XTool2K20:mue3:c128m:t75p

Compressed 2 files, 177,974,335 => 177,974,456 bytes. Ratio 100.0%
Compression time: cpu 0.13 secs, real 3.12 secs. Speed 57,116 kB/s
All OK

[External compressor:XTool2K20]
header = 0
packcmd = XTool precomp {-option } - - <stdin> <stdout>

IDK what is wrong with xtool compression.

dixen

03-11-2022, 00:48

IDK what is wrong with xtool compression.

XTool niot supported decompress ue3 lzo files

Razor12911

08-11-2022, 07:54

Update available

Changes

- fixed issues with exporting precompression database
- fixed issues with deduplication feature consuming a lot of system memory
- fixed oodle codec from auto enabling selkie method
- fixed reflate related checksum issues due to false positives

dixen

08-11-2022, 08:39

XTool 0.6.6

Test Psychonauts 2

Tested 1 file, 69,667,416,567 => 31,496,410,866 bytes. Ratio 221.19%
Testing time: cpu 11.56 sec/real 443.22 sec = 3%. Speed 71.06 mB/s
All OK

Before that, absolutely all versions issued CRC ERROR in zlib+reflate mode.
@Razor12911, thank you for fix

Masquerade

08-11-2022, 08:58

@Razor
Thanks for the update and your continued work :D

shazzla

13-11-2022, 22:41

@Razor12911 :

Xtool 066 bug.

xtool.exe precomp -mreflate -c32mb -t100p --verbose - - < %1 > %1.out

Streams: 11678/11678
Time: 00:00:25 (00:00:22)
Memory: 291 MB (291 MB)

arc.ini :

xtool066\xtool.exe precomp { -option} {options} -t100p - - <stdin> <stdout>

cmd :

arc a -mxtool:mreflate:c32mb p:\output.arc "p:\input.pak"
FreeArc 0.67 (March 15 2014) creating archive: p:\dd.arc
Compressed 1 file, 185,868,230 => 516,491,144 bytes. Ratio 277.88%
Compression time: cpu 0.17 sec/real 6.18 sec = 3%. Speed 30.10 mB/s
All OK

But when dedup enabled: --dedup, -dd, :dd (via arc.ini ,cmd line pararmeter ,etc. Same effect):

xtool.exe precomp -mreflate -c32mb -dd -t100p --verbose - - < %1 > %1.out

EStreamError: TBufferedStream.Seek not implemented

arc.ini :

xtool066\xtool.exe precomp { -option} {options} -dd -t100p - - <stdin> <stdout>

cmd :

$ arc a -mxtool:mreflate:c32mb p:\output.arc "p:\input.pak"
FreeArc 0.67 (March 15 2014) updating archive: p:\dd.arc
Compressing 1 file, 185,868,230 bytes. Processed 100%
ERROR: general (de)compression error in xtool:mreflate:c32mb

P/Reflate,Zlib ,it doesnt matter ,same error.

Razor12911

14-11-2022, 22:07

Update available

Changes

- fixed oodle scanner exceptions when incorrect library is used
- fixed issues with deduplication feature

aleison

18-11-2022, 12:28

How can I add the new version of Xtool to ultraarc to compress using SREP+LZMA+XTOOL? and how to add the same decompression method to an inno setup script? please help I have not been able to follow, thank you very much.

Razor12911

25-11-2022, 13:56

Hi, I recommend that you use DiskSpan GUI (https://fileforums.com/showthread.php?t=104507) as the project currently has support and UltraARC has long been abandoned.

aleison

01-12-2022, 10:40

Hi, I recommend that you use DiskSpan GUI (https://fileforums.com/showthread.php?t=104507) as the project currently has support and UltraARC has long been abandoned.

Thank you very much Razor what happens is that when compressing a game using the method Precomp (Ztool) + LOLZ + SREP, I get the following error when decompressing:

An error occured while unpacking: Does not match checksum!
Unarc.dll returned an error code: -12
Error: file D:\Program Files Plague Tale Requiem REPACK JPW\DATAS\SHARED.DPC failed CRC check.

Other games I compress and decompress with this method without problems, but I don't know why I get this error with this game, I have tried to compress it several times and I get the same error when decompressing it. Can you please help me if you can do something? thank you very much for your help.

Masquerade

01-12-2022, 10:42

^^
This is the XTool topic and you are asking about ZTool.

ZTool has been long since discontinued and is missing countless improvements added to both old XTool and XTool 2020.

aleison

01-12-2022, 11:32

^^
This is the XTool topic and you are asking about ZTool.

ZTool has been long since discontinued and is missing countless improvements added to both old XTool and XTool 2020.

Thanks for answering, I have another doubt, could you tell me exactly what information I have to enter in ARC.ini to decompress a compressed file using XTOOL(xZlib)+SREP+LOLZ methods? I did everything, but I always get an error: unarc.dll -2 compression method not supported xZlib+SREP.

Masquerade

01-12-2022, 23:40

^^
... this is the XTool topic now you are asking about your issues with DiskSpanGUI.

DiskSpanGUI has a built in decompressor area for your archives so you can decompress them without issues.

leftnomemes

12-12-2022, 08:32

Sorry if dumb question, but where can I find the documentation for the tool. New to this scene, looking to learn the tool properly.

shazzla

12-12-2022, 08:42

Doc (.chm) available in the xtool package. :)

Razor12911

16-12-2022, 17:31

Update available

Changes

- added advanced configuration based plugin support
- added UI mode when xtool.exe is launched with xtoolui.dll present
- added skip verification mode
- xtool now enforces w15 deflate stream detection by default
- fixed oodle scanner exceptions when incomplete stream is detected
- fixed issue with zlib codec not accepting streams from database plugins
- updated command line syntax

Notes

advanced configuration plugin is a way of writing more complex ini files for stream detection without the need of coding skills or the need of creating a library based plugin.

spidey plugin is an example, regular configuration files could not add support for this game however, advanced configuration allows this and here's an example of how that looks like

[StreamList1]
Name=lz4
Codec=lz4
BigEndian=0
Signature=0x1000352415344
Structure1=Signature(8),Count(4),Unk1(4),Unk2(16)
StructureN=DPos(8),CPos(8),DSize(4),CSize(4),Unk3( 8)
StructureS=Stream
CounterStart1=1
CounterEnd1=Count
CounterStep1=1
StreamPosition=CPos
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize
Condition1=

The user interface mode has been added and it's not meant to compete with tools like DiskSpanGUI or other similar programs but it's meant to help newbies operate the program as everything is made as simple as possible. The program can still be useful for regular users too, you can use the user interface mode to get theoretical outputs without doing actual (pre)compression, if you set output to none (leave blank in cli mode) and enable skip verification then essentially, you are performing a scan on the input (similar to what Drop and Scan for Zlib does) but for all supported codecs.

An example is Cyberpunk 2077 as shown below.
33329
33330

From here, you can find out the output size if you were to do precompression (in about 5 mins or less, all depends on drive speed).

More uses? Well people have a problem deciding how much chunk size to use, you can change the chunk size and see how the stream found or how the size differs to decide what chunk size to use.

The UI is a bit borked at the moment but the next update will improve upon it.

xtool enforces w15 streams because I figured out that this is what causes reflate to produce crc errors because there are more false positives this way. So how does this affect results, it doesn't because 99% of games use w15 anyways except Dishonored 2 and Dishonored Death of the Outsiders (set w10 for them)

Syntax has been updated as per request from Cesar82:
--dedup can also be -dd
--dedup=#, -dd#
--dbase, -db
--diff=#,-df#
--verbose, -v
--skip, -s

Razor12911

18-12-2022, 19:17

Update available

Changes

- added library checker (trial and error)
- improved user interface
- fixed bugs related to oodle scanner
- skip verification no longer applies to encryption codecs

Notes

Library checker allows you to find out what library was used through trial and error, all you do is pick a directory with a list of either lz4, zstd or oodle libraries and set output to none then every single library found in that directory is loaded by xtool one by one while showing you how many streams were processed by each and their respective precompressed outputs. This should allow you to maximize compression in one click rather than doing it manually.
33335
33336

More syntax changes as per request from Cesar82:
--zlib= can also be -zb
--lz4=#, -l4#
--lzo=#, -lo#
--zstd=#, -zs#
--oodle=#, -od#
--srepmem=75p (when decoding) as requested by Gehrman

Announcement

This is the last update for xtool, I am at a point where I think I have done enough for this project. I have dragged development of the project longer than I should have but I guess it's a habit of mine of not leaving something unfinished and this is the creative vision I had for this tool from the start, it took longer than I expected but I'm glad it's done.

So what does that mean I am leaving the forum? No, I'll stick around for the time being, it's just that the main project is no longer getting updates... and I had to stop at the magic number 69 :D. So if there are bugs and issues, you'd have to refer to the older releases which are made available on the main post.

dixen

31-12-2022, 06:00

Xtool v0.6.9 Elden Ring Edition

Two xtool.exe with injected oo2core_5_win64.dll and oo2core_8_win64.dll

Compressing 1 file, 2,655,581,216 bytes
Compressing Data3.bdt
Compressed 1 file, 2,655,581,216 => 4,846,074,313 bytes. Ratio 182.49%
Compression time: cpu 4.28 sec/real 347.49 sec = 1%. Speed 7.64 mB/s
All OK

[External compressor:xt8,xt5]
header = 0
packcmd = {compressor} precomp -mkraken:l6 -c32mb -t100p - - <stdin> <stdout>
unpackcmd = {compressor} decode -t100p - - <stdin> <stdout>

All with Happy New Year!!! Be Happy!!

Hexagon123

23-02-2023, 11:56

Razor12911

23-02-2023, 17:08

Tried using LZ4, but no streams.
Anyone make a plugin for this XTool.

Seen Puyo Puyo Tetris 2, Sonic Frontiers, and a few titles that uses this compression.

Ref1:http://wiki.xentax.com/index.php/Sonic_Games_PAC
Ref2:https://github.com/nickworonekin/puyo-pac

The sample you've provided is too small and only contains one stream, can't make a plugin off of this.

XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 000000000000081F (2080)
[0] Actual lz4 stream found at 0000000000000040 (2005 >> 9880)

[0] Processing streams on block 0000000000000000 to 000000000000081F (2080)
[0] Processing lz4 stream at 0000000000000040 (2005 >> 9880 >> 2171) using a0 has failed
[0] - Patching stream at 0000000000000040 (2005 >> 2171) [1151] has failed

Streams: 0/1
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)

elit

04-03-2023, 08:10

"I had to stop at the magic number 69.."
^So you are a Scorpio :)

Anyway I am glad you made my lz_scan (almost)obsolete because this is way better. I will later check in more detail how to do a multi-lib scan. From the pictures it seems you create a version directory and put a dll there, like 2.8.1/codec.dll, 2.8.2/codec.dll and so on, while dlls are all of same name.
As opposite to my lz_scan where I had a single directory with like liblz4_112.dll, liblz4_120.dll etc.

Advanced config is what I love about this new version and will use, really kudos for that.

I recommend if you want to call it a day, don't add a new features anymore but hang around to collect and fix last remaining bugs that users report over time. What you achieved here is a project that gives any institutional budget projects run for their money. I say lets not leave it half backed, after all you do remember Freearc(and srep) fiasco. Both a buggy, half assed projects, but no proper alternative unfortunately(at least for my usage as a GUI archiver).

After some time once project is bug-hardened with no more changes, maybe I try to port it one day to C and/or multiplatform if I decide to switch to Linux in the future. Thank you for all you did for us so far.

Wanterlude

12-03-2023, 02:22

Good afternoon, everyone!
I decided to pack Transport Fever 2 using zlib+dedup+lolz, but srep takes up all RAM when unpacking. I looked on the forum to see if this was a problem before and saw the --mem=# parameter. I tried "20p" "800m" "800mb" but nothing ended up working. This is what I have prescribed in arc.

[External compressor:xtool]
header = 0
default = -c128mb -t100p
packcmd = xtool.exe precomp { -moption} -db -dd5 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p --mem=10p - - <stdin> <stdout>
xtool - last ver.
srep 32-bit v392
ASIS script (i try v732,v744)
https://i.ibb.co/L5B5S7Z/sk.png

Razor12911

12-03-2023, 02:25

it should be --srepmem=...

elit

19-03-2023, 05:24

I would like to bring to your attention an issue I am currently experiencing with xtool.

In the past, I compressed several games with it under Windows 7. Probably most used version by me back then was 047 and 053. I test every archive after creation and back then every one of them tested fine.

However about 2 months ago I switched to Windows 8.1 and I find myself facing decompression errors.
Specifically this almost exclusively relate to reflate(I had one zlib giving me error as well but not all). It is 100% reproducible and do not matter which xtool version I use(I keep ones I compressed anything with just in case). Reflate in archive = 100% fail. Any archive. Due to that I excluded disk issue.

Note that ztool with -cm0 work fine under Win8.1, as well as 32bit xtool-crilayla version. Problem is only xtool->reflate(maybe occasional zlib) from what I seen so far. Thankfully majority of my archives are ztool based due to me stubbornly sticking to it for so long but still quite few of new ones I made use xtool already. xtool doesn't have any fancy switches like -cm0, I tried -t1 to no avail.

I am currently trying all archives to make list of broken ones, I also already installed Win7 under Hyper-V late night and will try more on it later but don't know if environment will be 1:1, maybe xtool is having issues due to memory/exception handling of Win8.1.
Also, I did of course already tried Win7 compatibility(all relevant exe files in chain) as well as running as an admin(real one, enforced through group policy).

If anyone have an idea, I will appreciate it.

elit

19-03-2023, 06:37

Use compatibility mode
Setup.bat
ECHO OFF
Reg.exe add "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe" /t REG_SZ /d "~ DPIUNAWARE WIN7RTM" /f
START Setup.exe
Reg Delete "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe" /f
Exit

That's interesting, so it seems a known issue. Thank you, I will try later after I scan all my archives and report back. I hope this will really fix it.

elit

19-03-2023, 07:03

^Unfortunately, putting both freearc.exe and xtool.exe into registry like above did not help.34189

EDIT: Not all reflate are broken, in fact many are fine. I am starting to see potential pattern but will confirm it later whether its case. Ones with the issue so far are the ones with custom settings, like chunk size -c32m etc., whereas all the good ones are just default -mxtool:reflate so far.

EDIT2: I see the pattern. This error affect both reflate and zlib, when high chunk size(>32mb) is used. This was working fine on Win7. Changing thread number doesn't seem to cut it. And I have same rig with enough memory(16G). I will try few more things but it seems as if Win8.1 is handling memory or relevant functions differently.

EDIT3: Windows 7 under Hyper-V did not help. I still believe archives are good and this is because I don't run under native.

Gehrman

20-03-2023, 14:27

@elit
Increase the size of the Windows page file (pagefile.sys).

Razor12911

20-03-2023, 15:41

could you try isolating the bug and send over the sample that causes xtool to bug out

elit

20-03-2023, 17:25

@Gehrman
I have 16G RAM + 16G swap. I also tried -t1 option. This is certainly not the case of RAM but also not of disk space.

@Razor12911
I have looked into it today and it does look like it may have to do with chunk size. All archives that bugged out had -c more than 32mb(64m, 128m), but those without any option or up to 32m were all fine. Best at this point will be for you to see yourself. I will upload 2 of them + FA whole pack that I use. With that you will be able to understand how I use it and try for yourself. Again this all worked under W7 native install. Also I am sure among those hundreds of archives where ztool was used, there had to be at least several with chunk size >32m and those passed fine(I tested my whole collection).
I will upload things somewhere and post the links by tomorrow.

elit

20-03-2023, 18:09

@Razor12911,
Ok I already found a site that was capable to test my gigabit/hdd limits :)

Anyway, this link gives you 2 archives and FA bundle:
https://ufile.io/f/hjbf7

In FA bundle, I use freearc.exe(GUI). In bin\_EC\xtool\ directory are the relevant xtool files. Archives use xtool.exe, so you overwrite one of different versions to that. I believe I used v047 most of the time, but can't say for sure which one I used on those archives. However, none of them successfully unpacked under Win8.1.

You will be able to test 1F.arc to the almost end, which confirm that last chain with big chunk size set is the one problematic(see archive info from GUI). Inside 1R.arc is a single file and will bug out at the very beginning.

Please let me know what you think. Also if anyone will be able to unpack those without error, then it would confirm my suspicion about my Win8.1.

EDIT: Also, when creating new archive under Win8.1(same parameters), it will test and extract correctly. I tested it on "1R" content. Still don't believe this to be bitrot/disk corruption etc. More like different OS environment+HW is causing intolerance in between versions. Before I had Win7 and i5-4690K CPU. Now I have Win8.1 and Ryzen 5800X. This is not a good thing if the case, change of HW/OS should not affect decompression reliability.

MAY/O

21-03-2023, 08:05

Did you ever think about making a linux build?

Razor12911

24-03-2023, 13:15

elit

28-03-2023, 06:27

With -mpng.. method, xtool may hang, remaining unresponsive. This probably happen with non-png data. For example if doing whole "Data" directory in "Total War: Warhammer 2".

With -mpng though, more than 20k new streams can be found in those .pack(s) and inflation of another 500mb can be achieved.
Test passed without hanging when -c32m was used, but I fear this is more like a workaround/cheating and I fear of using bigger chunks due to risk of not unpacking data sometime in the future again(as it was the case already as mentioned in my posts above).

34259

EDIT: I have narrowed down which specific file is causing the bug: https://www.udrop.com/7YDG/local_en_2.7z
It is replicable on my comp. Xtool069 fails/hang with reflate+zlib+png:d1 in its own GUI scanner, while it doesn't hang with c32m added.

L33THAK0R

05-04-2023, 20:16

Are there any switches to disable the 256kb minimum file size for the "Erase" functionality? I've got some tens of thousands of small ~20kb files that I'm trying to rip/erase but alas the minimum limit is making it a bit tricky.

Masquerade

05-04-2023, 23:24

The filter is for bytes, not kilobytes.

L33THAK0R

06-04-2023, 08:37

The filter is for bytes, not kilobytes.

Huh, guess I must have messed something else up then!

kj911

20-04-2023, 03:07

Special questions: Hardly ways integrate in last xtool internal (not srep related!) deduplications functions adding to old xtool v0.12 versions?? Or make very tiny xtool compressors, named in xdedup or any names, in has ONE deduplications functions with stdio mode? (No zlib/reflate or many more functions..) Would I make sense?

Razor12911

22-04-2023, 07:51

Hi, the older versions of xtool and the current version have different code infrastructure and porting the deduplication code would be time consuming, also a standalone deduplication program wouldn't make sense simply because the xtool version is stream based meaning, it relies on whatever streams are detected, however if you just wanted deduplication in stdio mode, you can just use srep in -m0 mode rather than -m#f, not sure if that answers your request :)

Razor12911

26-04-2023, 14:34

Update available

Changes

- added ability to redirect base directory for plugins and libraries
- added restrictions to avoid errors with experimental codecs
- added optimize option to speed up the decoding process for zstd and oodle codecs
- added dictionary parameter for fast-lzma2
- added memory caching when decoding to alleviate speed bottleneck
- fixed bug with download feature for inputs in URL format
- fixed issues with exporting precompression database
- fixed issues with executable plugin support
- fixed issues advanced configuration based plugin support
- fixed potential decoding issue upon using plugin support functions
- fixed issues with deduplication feature
- fixed issues with jojpeg codec
- replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature
- replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios
- improved user interface
- improved oodle codec performance for 2.6.0+ libraries
- improved encoding speed when using internal codecs
- improved processing speed when depth is used
- removed fast lzma2 multi threaded decompression due to excessive memory requirements
- removed debugging information when using the patch function
- removed ability to toggle database feature and ability to export database files (now enabled by default)
- updated deduplication virtual memory allocation
- updated reflate codec to verify streams prone to data corruption

Notes
Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools.
Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added.

Speed when using -d# parameter has been improved as show below on the game Star Ocean The Divine Force's chunk_2.cpk file.

0.6.9
Compressed 1 file, 641,373,728 => 1,104,208,408 bytes. Ratio 172.16%
Compression time: cpu 0.53 sec/real 37.94 sec = 1%. Speed 16.90 mB/s

Tested 1 file, 1,104,208,408 => 641,373,728 bytes. Ratio 172.16%
Testing time: cpu 0.41 sec/real 73.39 sec = 1%. Speed 8.74 mB/s

0.7.0
Compressed 1 file, 641,373,728 => 1,104,208,460 bytes. Ratio 172.16%
Compression time: cpu 0.61 sec/real 30.71 sec = 2%. Speed 20.88 mB/s

Tested 1 file, 1,104,208,460 => 641,373,728 bytes. Ratio 172.16%
Testing time: cpu 0.45 sec/real 22.72 sec = 2%. Speed 28.23 mB/s

dixen

28-04-2023, 01:47

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

shazzla

28-04-2023, 04:34

Something is wrong with Xtool 07 hotfix.

It doesnt catch up any stream on Snowrunner's EDITOR.PAK ,while Xtool 069 does ! (069 fails at the end on this file with reflate ,zlib works fine !)
See below :

......

Edit #2 :
Ohh ,maaan! :D ALL DLL's must be in the libraries folder ! Not only the external ones ,like unreal ,unity,etc... :D

Razor12911

28-04-2023, 05:52

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

Streams: 58080 / 58080
Time: 00:01:27 (CPU 00:15:18)
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

I'm not sure what the question is, but the information shows up when you use the program w/o FA.

@shazzla

be mindful of base directory or -bd#, this allows xtool to relocate libraries and plugins (including xtl and ini files) from another directory

I added this because there were just too many files near xtool.exe, also if you want to compress games but rather than moving and deleting oo2core..., you can just change the base directory.

shazzla

28-04-2023, 06:08

This "base directory or -bd#..." is not so clear...
How to use/set -bd# ?

Razor12911

28-04-2023, 06:18

Let's say you want want to move all libraries and plugins to a folder called "libs" which is near xtool.exe, you specify base directory as -bdlibs or -bd.\libs. or you can just leave it empty to make xtool work the same as before.

dixen

28-04-2023, 10:30

Sorry for dumb question, but how get this info in xtool?

Streams: 58080 / 58080
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

KaktoR

28-04-2023, 10:32

Sorry for dumb question, but how get this info in xtool?

With these settings
https://i.imgur.com/KrVjKWN.png

Razor12911

28-04-2023, 11:37

The information is always visible and verbose option isn't necessary because that will force xtool to use 1 thread, whether you use GUI or CLI. It's always there.

Razor12911

28-04-2023, 13:13

There's a feature I'd like to point out which was briefly added in 0.6.8 but wasn't finished until this current release and that is on the fly compression, basically if you wanted to quickly find out the theoretical compressed size of a game without consuming disk space.

Reasons you may want to use this feature:
+ If you just wanted to find out the compressed size
+ If a game uses kraken and wanted to see if it was worth it to compress it at all
+ If the game is big and you don't have space for compression
+ If you don't want to reduce lifespan on SSD

https://i.ibb.co/1n8BL2G/otfcom1.png
XTool is created by Razor12911

Streams: 1415335 / 1415335
Time: 00:01:21 (CPU 00:04:03)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB

Done!!!

13GB is the size that srep is expecting, so you change duplication preset to srep, then set input size to at least 15GB then enable compression, set desired level and dictionary.
https://i.ibb.co/L0Kdzrs/otfcom2.png
Streams: 1415335 / 1415335
Time: 00:04:36 (CPU 00:34:05)
Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 751 MB [5.15 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.03 GB >> 4.65 GB

Done!!!

So then you know that 15.3 GB becomes 4.65 GB in less than 5 mins but without creating any temp files as the whole compression takes place in memory. You also know srep decompression memory too.

_________________________________

The other thing, as you know how much I tried to make xtool to perform as fast as possible, the thing that always bothered me was that lzma was slow and that's why I added fast-lzma2. I noticed something interesting about this compression library in the sense that it's able to increase the number of threads used for compression without increasing memory usage.

Here's a few results

4x4:lzma (via fazip) (b128mb:t6), ultra, 64mb dictionary, 12 threads used 2781 MB ram
Compressed 288 files, 16,391,993,193 => 4,869,896,897 bytes. Ratio 29.71%
Compression time: cpu 13.50 sec/real 662.85 sec = 2%. Speed 24.73 mB/s

fast lzma2, level 10, 64mb dictionary, 12 threads used 403 MB ram
Compressed 288 files, 16,391,993,193 => 4,929,451,310 bytes. Ratio 30.07%
Compression time: cpu 11.84 sec/real 425.39 sec = 3%. Speed 38.53 mB/s

Not only is memory used by fast lzma2 low, but it's much faster while producing similar results and what is impressive is, if you had like 32 thread cpu, you can increase its speed even further without increasing memory usage by too much but if you use the 4x4 method, the memory usage will increase even more.

infovs

30-04-2023, 00:13

Thank you Razor12911 for this fantastic tool...have one question, if it is not too much trouble, can you please consider naming x64 srep.exe (since xtool is hardcoded using this exact name) like Bulat original named it as srep64.exe?
I'm using original naming for many years, I need 32bit srep.exe for some compatibility reasons of my own...and your renaming 64bit srep64.exe as srep.exe is causing problems.
Maybe at least consider making xtool using srep64.exe name first, if found, then if not found use srep.exe or whatever srep name you consider acceptable.

Thanks in advance,
best regards

kj911

30-04-2023, 08:29

Streams: 58080 / 58080
Time: 00:01:27 (CPU 00:15:18)
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

Final compressed size without srep?? Better or worser?

FLZMA2: This really? Small memory footprint + utilize extra more cores? :eek: (This feature available from native 7-Zip releases?)
Likelys the feature from LOLZ. More 4-12C/4-24T CPU's from in visible this ONE virtually core utiiized during in compression. (Faster speed in LDMF1 mode.)

Wanterlude

13-05-2023, 01:37

Guys, does anyone know how to fix this?
Game: A Plague Tale Requiem
Files: *.DPC
Xtool: last ver 0.7.0 hotfix
Problem: after unpacking 1 file (28.9gb) xtool goes beyond its allocated memory. (x4)
The picture shows the information from windows resource monitoring. (I have 16 GB RAM)
https://i.ibb.co/tBFnGqR/111759.png
[External compressor:xtool]
header = 0
unpackcmd = "xtool.exe" decode -t100p -dm20p -sm20p - - <stdin> <stdout>

[External compressor:xtool]
header = 0
unpackcmd = "xtool.exe" decode -t100p -sm20p --mem=20p - - <stdin> <stdout>
In 1 case it happens as shown in the picture. In 2, the memory increase can be seen even in the windows task manager.

arc.ini:
[External compressor:xtool]
header = 0
default = -c128mb -t100p
packcmd = xtool.exe precomp -mzlib -d1 -db -dd5 - - <stdin> <stdout>
-----------------------
upd. (14 may): preflate has the same problem | xtool 0.6.9 - same
-----------------------
upd. (15 may): fixed by 2nd xtool and -dm20p -sm20p parameters.
I think the problem was that I used 1 xtool for both zlib and dedup
-----------------------

Razor12911

22-05-2023, 04:47

Update available

Changes

- fixed issues with fast-lzma2 being unable to set correct compression level
- updated deflate stream scanner

huluwizard

03-06-2023, 07:49

Forgive me if I have missed something obvious, but the first post in this thread mentions documentation, but I'm not able to locate any documentation for xtool. Specifically, I'm trying to figure out the command line. I can get the results I want from the GUI plus a plugin, but ultimately I'd like to pair it with a batch file that does the same thing.

KaktoR

03-06-2023, 08:07

xtool.chm is what you are looking for.

huluwizard

03-06-2023, 08:12

xtool.chm is what you are looking for.

Oh wow, it really was right there all along. Thanks!

dixen

08-06-2023, 02:17

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

Cesar82

08-06-2023, 03:10

And yet I don't understand how to use the memory limit in the -dd# function. The --srepmem=# parameter does not give anything at all. Maybe I'm doing something wrong?

I think the parameter for srep memory usage in XTool 0.70 is "-sm#" and not anymore "--srepmem=#".

shazzla

08-06-2023, 05:10

It would be nice if someone document this parameters.... :)

Masquerade

08-06-2023, 09:09

It would be nice if someone document this parameters.... :)

:confused::confused:

CMD> xtool precomp

XTool is created by Razor12911

precomp - data precompressor

Usage:
xtool precomp [parameters] input output

Parameters:
-m# - codecs to use for precompression (separate with "+" if more than one)
-c# - scanning range of precompressor [16mb]
-t# - number of working threads [50p]
-d# - scan depth [0]
-dd - use stream deduplication
-l# - compress data using fast lzma2 (separate params with ":")
d# - dictionary size
-lm - low memory mode
-s - skip stream verification
-v - enables verbose
-df# - set xdelta threshold to accept streams [5p]
-x# - extract streams to directory path
-dm# - deduplication memory usage limit (#=size) [75p]
-sm# - srep memory usage limit (#=size) [75p]

Razor12911

08-06-2023, 09:55

Executing xtool with the command as the only parameter will display information of how the command is used alongside the available parameters. The reason certain parameters remain undocumented is simply because they were added for personal usage or as a gimmick and not a lot of testing has been done and as such, they may not work as intended and may corrupt your data in certain instances.

Any of the listed parameter is fine to be used by you, the end user however anything not listed here is limited to advanced users who may have gone through the source code over on github to even know of these parameters existence.

shazzla

09-06-2023, 01:53

Thanks,but i miss exactly those undocumented parameters! :D

Somehow i missed github. lol
Going there now,btw thanks!

Razor12911

12-06-2023, 07:21

Update available

Changes

- memory usage optimizations

Notes

Resources utilised by zlib, lzo, zstd and some other codecs have been made to initialize only when used, this was down to reduce memory usage and allocation.

If you're users like shazzla however, who want to utilise features of xtool before they are tested and ready for, there's a new parameter introduced in this release which makes xtool utilise the GPU's VRAM to improve precompression speed and reduce memory usage even more when decoding by offloading deduplication memory requires onto the GPU.

-g# (# may be a percentage or specific value), default value is 0

So how does it work? When precompressing, xtool reads, processes then writes, it does this again and again until it finishes. The problem with this however is there's a bottleneck when it is writing especially if you're repacking on HDD because xtool may be processing faster than it is actually writing to the disk so that's where caching feature introduced in 0.7.0 and the new GPU feature comes in. Instead of writing straight to disk, the data is written to the GPU and as xtool reads and processes the next batch of data, the GPU will be writing to the disk making sure that the processing aspects of xtool are not slowed down.

When decoding, xtool can sometimes use the ram for the duplicated streams resulting in high memory usage when installing a repack, usually when you're installing a game, the GPU isn't doing anything so these duplicated streams are stored on the GPU, this is done to reduce memory usage. 75% of the allocated GPU is dedicated to this while 25% is dedicated to caching data from srep+lolz/lzma or what it is that you use where xtool would be reading data in advance to reduce bottlenecks even more.

TLDR; how to enable this feature? just add -g75p when encoding/decoding (can be both, up to you)

How to know if it's working? Check in task manager and you should see xtool utilising the GPU with the Engine "Copy".

Benchmarks

0.7.1
XTool is created by Razor12911

Streams: 1415315 / 1415335
Time: 00:07:20 (CPU 00:44:37)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

0.7.2
XTool is created by Razor12911

NVIDIA GeForce GTX 1060 6GB (4.50 GB loaded)
Streams: 1415315 / 1415335
Time: 00:05:31 (CPU 00:46:09)
Duplicates: 1134302 (1.99 GB) [7.11 GB >> 16.1 GB]
Srep decompression memory: 738 MB [5.13 GB*]

Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.06 GB >> 4.65 GB

Done!!!

There was a 30% speed improvement, your mileage may vary but only use this feature if you repack on HDD. There won't much speed gains on SSD.

OpenCL was used to achieve this so if your PC does not have the library in system32 folder, you should place the dll near xtool.

If feature does not work as intended then you must understand why I keep some features undocumented.

Masquerade

12-06-2023, 10:11

Nothing short of incredible, always love seeing XTool updates. Thanks again Razor!

infovs

12-06-2023, 10:27

Razor12911

12-06-2023, 16:00

Thank you Razor12911 for your incredible work! Two short questions if it is not too much trouble...how to use undocumented option introduced in 0.70?
- added optimize option to speed up the decoding process for zstd and oodle codecs
Looking at source it is used by adding parameter -o ...but not sure where, at decoding or encoding... I tried used it at both places, precomp some oodle data -> decoding speed was exactly the same as before without -o parameter :\
Also, is there any chance or parameter that can make xtool use and utilize srep64.exe instead of renamed srep.exe?
Thanks in advance, again, big thanks for everything!

Hi, optimise was added but not documented simply because the conditions needed for the benefits to even be noticeable are obscure, essentially the game Returnal was the reason this feature was added. The game used leviathan which takes a long time to process, to put it into perspective, if a stream was compressed using level 9 and let's say it takes 20 seconds to process, what optimise does is try lesser compression settings that give the same output, like level 8 which takes half the time on average compared to level 9 so that's about 10 seconds, 50% gain in speed, it does this again and again until the compression setting no longer gives the same output. As a result, some level 9 compressed streams of Returnal could be compressed with level 6, some level 7.

With this in mind, the installation should be faster right? Well yes it is but I wasn't impressed by the improvements so I left it undocumented.

Benchmark on pakchunk2_s8-WindowsNoEditor.pak of Returnal

without -o
XTool is created by Razor12911

Time: 00:22:17 (CPU 02:12:41)

with -o
XTool is created by Razor12911

Time: 00:18:50 (CPU 01:52:02)

We got about 15% speed improvement. You might see this as a win, but precompression took 32 mins with -o than 23 mins without -o so 40% more time for precompression.

As for srep/srep64, just let 32-bit programs die. :rolleyes:

janb14

19-06-2023, 07:16

Hello everyone,
first of all thank you Razor12911 and eveyone else who made this possible.
Im currently experimenting with xtool and noticed no dedicated handling of ogg files. Im trying to compress some of my minecraft backups where mods with ogg sound files contribute a significant share.
I have already extraced some and compressed them with standalone oggre. Now i want to include them the same way panker1992 (i recreated his wavpack config and it worked ;D) did but im not really a hex guru. Did anyone already have a working ogg config ?
So far ive found the Signature but im not sure what to fill in for the rest of the ini values:

[Stream1]
Name=ogg
Codec=oggre
BigEndian=0
Signature=0x5367674f
Structure=Signature(4),FileSize(4),FileType(4),Str eam
StreamOffset=-12
CompressedSize=FileSize + 8
DecompressedSize=0

Hello, it's been ages since i had an error !!

This Came directly from Xtool documentation
Xtool version used 6.2

this is my wav.ini
[Stream1]
Name=wav
Codec=wavpack
BigEndian=0
Signature=0x46464952
Structure=Signature(4),FileSize(4),FileType(4),Str eam
StreamOffset=-12
CompressedSize=FileSize + 8
DecompressedSize=0
Condition1=FileType = 0x45564157
Condition2=FileSize >= 4096

this is my xtool.ini
[wavpack]
Encode=wavpack.exe -hh -x4 <filein>.wav <fileout>.wv
Decode=wvunpack.exe <filein>.wv <fileout>.wav

i have been unable to make this even detect the sample FLAC internal codec works and so any other
NOTE: i use this as standalone and not in FA for testing purposes

Masquerade

19-06-2023, 08:17

^^
I'd advise against using XTool with oggre because oggre has built in deduplication that will not be working when each ogg is compressed individually with XTool.

janb14

19-06-2023, 09:46

Do you know any other ogg compressors ? Either way running it on single stream still resulted in a noticable reduction and i would see it as an example of how to correctly fill out these ini files. Can you help me with that ? From the xtool chm im not quite able to do that yet.

Razor12911

29-06-2023, 15:44

Update available

Changes

- Minor bug fixes

infovs

02-07-2023, 04:13

Update available
Changes
- Minor bug fixes
Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?
And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only.. :)
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards

Razor12911

03-07-2023, 16:18

And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only.. :)
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards
Initially only the x86 libraries existed and after the developer of reflate returned years after, he provided x64 versions however they function the same. (Also the libraries included in 0.3.21 are actually x64, that xtool release was also x64 :confused:)

Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?
I've actually added Denuvo to limit installation of xtool compressed games to five per day. :rolleyes:

infovs

04-07-2023, 03:16

I've actually added Denuvo to limit installation of xtool compressed games to five per day. :rolleyes:
Glad to see you added older Denuvo version otherwise xtool.exe size would be like 1GB .. :p :rolleyes:

Joking aside, regarding x86 hif2raw_dll.dll/raw2hif_dll.dll..you're right - 0.3.21 version did not contained x86, 0.3.19 did but only x86 xtool.exe.
x86 hif2raw_dll.dll/raw2hif_dll.dll libs versions I'm mentioning are contained with xtool_1908_R6, they seems newer than ones distributed with 0.7.3, hence the confusion :confused:

kj911

04-07-2023, 05:55

Hi Razor12911
Please add -a option to Srep. Sometimes, for big games like FH5, -a0 should be used to use less memory.
...

Used -a0 in many times ago! :D

Try it -m5f mode, its minimally smaller memory eat than -m3f. And combined in -a0. This m5f:a0 smaller memory footprint from decompression than m3f:a0??

In last newer ~20kB sized hif/raw dlls, in have different results than older ~80kB sizd versions. 50% case bigger inflated/compressed results.
The one problematic parts, from x86 build packages, the "preflate_dll.dll" file, not compatible from WinXP and not working from any precompressors usage. (ex: ZREC) Still exist working one in have preflate library available in many precomp 0.4.8 and older versions. (x86 releases.) ;)

Razor12911

05-07-2023, 08:58

Update available

Changes

- user can specify srep parameters to use via -sp#
- updated zlib codec

Notes

the -sm# parameter is removed, now if you want to control the memory srep is bound to use, you use refer to the list of srep's parameters and use it as such -spmem75p or whatever it is you prefer, -sp is also available when encoding in order to specify additional srep parameters like the user Gehrman has requested. -spa0:l512.

The executable got bigger because I've been busy trying to add some features to the program that have not come into fruition but if you're a nosy person, you can check the source code to see what caused the exe to get big.

KaktoR

06-07-2023, 06:19

What is wrong here? I don't get it.

Streams: 12050 / 12446
Time: 00:00:03 (CPU 00:00:18)

Size: 78.9 MB >> 211 MB

Compressed 2 files, 84,074,653 => 84,074,928 bytes. Ratio 100.00%
Compression time: cpu 0.06 sec/real 2.99 sec = 2%. Speed 28.16 mB/s
All OK

[External compressor:xtool_test]
header = 0
packcmd = xtool.exe precomp -mscarletnexus+zlib -c128mb -t100p -d1 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

Gehrman

07-07-2023, 04:47

Thank you Razor12911 for the new version

XTool 0.7.4 test
Forza Horizon 5

134 GB > 198 GB > 78.8 GB

Creating archive: FH5PE - 1.594.508 - 001.Bin using xtool:c32mb:mzlib:dd3:spa0:hash-+4x4:lzma:fb273:lc8:mfbt4:64m+diskspan:1395864372: 1557135360
Compressed 17,216 files, 143,891,581,840 => 1,395,864,412 bytes. Ratio 0.97%
Compression time: cpu 477.42 sec/real 28649.30 sec = 2%. Speed 5.02 mB/s
All OK
CT: 07:57:29

Tested 17,216 files, 1,395,864,412 => 143,891,581,840 bytes. Ratio 0.97%
Directory 171,831 => 873,633 bytes. Ratio 19.67%
Testing time: cpu 181.02 sec/real 6340.69 sec = 3%. Speed 22.69 mB/s
All OK

DT: 01:45:40 (In another compressed version that used XTool 2020, the extraction time was 1 hour. It seems that this version has decreased in decompression speed.)

a problem
Heavy CPU consumption in Normal CPU Priority mode, which causes Mouse Lag.
This problem does not exist in "Below Normal" CPU Priority.

Razor12911

07-07-2023, 15:27

The log information via xtool when repacking FH5:
Duplicates: 409792 (8.55 GB) [13.1 GB >> 22.2 GB]
Srep decompression memory: 5.30 GB [19.9 GB*]

Something you have to keep in mind when repacking Forza Horizon 5 is that this game has a lot of duplicates because its also a big game. If you use srep, it will request 20 GB for decompression memory or (5 GB if you used -dd while xtool itself will request 8.5 GB, about 13 GB overall, ~7 GB memory savings) and since you made a post that you wanted to control srep's a# parameter, I can only assume that when you ran decompression before, you had more system memory available than when you ran it this time.

The speed reductions seems to be related to you not having enough memory to install the game without srep or xtool making virtual temp files which incur speed penalty.

Games like these are the reason I added GPU memory feature in xtool because it would offload some memory onto the GPU which typically isn't used when installing games (if you have 8GB GPU, xtool would require 1GB instead of 8.5 GB, while 7.5GB will be on the GPU then installation would need 6GB instead of 20 GB which srep initially required), though not ready because it's not thoroughly tested on a lot of systems to see if it works without issues. :(

I ran a test between 0.7.4 against older releases, they all produce the same results and have the same speeds when decoding (at least within margin of error) and 0.7.0+ encodes faster. Decoding speed in 0.7.4 is slightly slower but reasonable because of memory usage optimisations that were made in 0.7.2, all versions used 220 MB+ for decoding while 0.7.4 used 73 MB.

ForzaHorizon5\DLC\MicrosoftExpansion2FH5\media\Tra cks\Falcon\bin.zip
0.7.4
Compressed 1 file, 1,893,356,830 => 2,847,735,336 bytes. Ratio 150.41%
Compression time: cpu 1.36 sec/real 20.95 sec = 6%. Speed 90.39 mB/s

Tested 1 file, 2,847,735,336 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.19 sec/real 22.40 sec = 1%. Speed 84.52 mB/s

0.7.0
Compressed 1 file, 1,893,356,830 => 2,847,735,336 bytes. Ratio 150.41%
Compression time: cpu 1.41 sec/real 20.57 sec = 7%. Speed 92.05 mB/s

Tested 1 file, 2,847,735,336 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.33 sec/real 22.06 sec = 1%. Speed 85.84 mB/s

0.6.9
Compressed 1 file, 1,893,356,830 => 2,847,735,092 bytes. Ratio 150.41%
Compression time: cpu 1.48 sec/real 25.08 sec = 6%. Speed 75.50 mB/s

Tested 1 file, 2,847,735,092 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.14 sec/real 21.56 sec = 1%. Speed 87.81 mB/s

0.6.5
Compressed 1 file, 1,893,356,830 => 2,847,735,092 bytes. Ratio 150.41%
Compression time: cpu 1.38 sec/real 24.58 sec = 6%. Speed 77.03 mB/s

Tested 1 file, 2,847,735,092 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.30 sec/real 22.06 sec = 1%. Speed 85.83 mB/s

0.5.3
Compressed 1 file, 1,893,356,830 => 2,847,735,104 bytes. Ratio 150.41%
Compression time: cpu 1.39 sec/real 25.08 sec = 6%. Speed 75.51 mB/s

Tested 1 file, 2,847,735,104 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.16 sec/real 21.56 sec = 1%. Speed 87.81 mB/s

0.4.0
Compressed 1 file, 1,893,356,830 => 2,847,735,104 bytes. Ratio 150.41%
Compression time: cpu 1.38 sec/real 24.39 sec = 6%. Speed 77.64 mB/s

Tested 1 file, 2,847,735,104 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.41 sec/real 21.56 sec = 2%. Speed 87.82 mB/s

Cesar82

07-07-2023, 16:27

@Razor12911, can the new -sm# parameter be used from the freearc command line using comma "," to separate srep parametters?
xtool:mzlib:dd3:spa0,s100gb
I would also like to know if running xtool with the -g90p parameter on a system without a dedicated video card will also work or could it crash the system?

Thanks!

Razor12911

07-07-2023, 21:50

Update available

Changes

- added footer feature in configuration plugins
- added thread priority controller via -T# parameter

Notes

Footer detection has been added to configuration which means if there's no way of knowing the size of a stream but if within the stream itself has a footer or end of file/stream marker, you can input that as a signature for xtool to look for which will help you in determining the size of the stream.

For example, jpeg image file has the header 0xE0FFD8FF but if you looked at the last 3 bytes, you'll notice that in all of them, they end with 99 FF D9, this will be the footer and your configuration should look something like this

[Stream1]
Name=jpg
Codec=packjpg
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

The offset is -4 because the signature is part of the stream, you have Footer=0xD9FF99 which xtool will search for after finding Signature=0xE0FFD8FF, the compressed size is then Footer + 7 because the signature + the footer are both part of the stream.

Another use case for this is if you have streams stored in sequence, if their signature is 59 61 7A 30 and the next stream has the same signature, you can write the next stream as the "footer" of the current stream.

[Stream1]
Name=Yaz0
Codec=Yaz0
BigEndian=0
Signature=0x307A6159
Footer=0x307A6159
Structure=Signature(4),Stream,Footer(4)
StreamOffset=-4
CompressedSize=Footer+4
DecompressedSize=0

This time, the compressed size doesn't added both signature and footer because the footer is actually the header of the next stream, there's one caveat with doing it this way however, the last stream will not be captured.

Thread priority option is added and can be used via -T and the values are 0..6 with zero being idle and 6 being real-time (time critical), if you have laggy mouse while installing a game knowing full well that you have set 100p for xtool and have lolz or lzma working and toasting your CPU and saying it's xtool's problem then you should use this but know that setting this too high means your system will be unresponsive or setting too low making installation longer, well... up to you. Too many notes...

@Cesar82

You can use commas to separate parameters.

For GPU, xtool performs a check to see if the system has a GPU with at least 1GB VRAM and even then it makes sure that there's at least VRAM 512MB left unused. If the condition is not met, it doesn't use GPU at all.

Razor12911

10-07-2023, 19:33

shazzla

14-07-2023, 01:23

Damn. I'm lame :D

How can i use this with XTool076 for JPG stream compression?

[Stream1]
Name=jpg
Codec=brunsli
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

Solved :
My sample JPG have different header.

infovs

14-07-2023, 07:54

Update available
Changes
- Minor bug fixes
Notes
This release contains recompiled zlibwapi.dll, preflate_dll.dll, lzo2.dll (to remove Visual C++ 2010 requirement) and xdelta3_dll.dll simply because there are people who are using Windows XP in this day and age, shocking I know...

Thank you Razor12911 for your updated xtool and libraries...anyway, tested unpacking speed of your last three x64 zlibwapi.dll library compiles, multiple times (did not test x86 version at all)...

Unpacking speed of zlibwapi.dll 1.2.11/2020-07-18/535552 bytes is _always_ faster than any of your other, newer, 1.2.13 compiles (2022-12-31/544768 bytes and 2023-07-11/512000 bytes)
Newest XP compatible zlibwapi.dll is by far slowest during unpacking, probably because it does not use some of advanced CPU instructions available after XP. Anyway, I truly welcome XP support because I consider games that are playable on XP should be able to unpack on XP...but if making .dlls XP compatible comes with obligatory speed loss than maybe it is better to provide separate XP compiled libs, just my thoughts of course. Ideally, one compile should work on all OSes with max speeds but don't really sure if this is possible...

Following are examples of decompression speed of _same_ arc packed with xtool:zlib+srep:m3f+lolz ,all that is changed is zlibwapi.dll before unpacking (encoding speed is not primary important to me, arc is packed using one of mentioned zlibwapi.dll but this does not impact decompression speed, tested)
Tested on Windows 11, 20 threads CPU, xtool 0.7.4 used, -t100p in arc.ini

Using zlibwapi.dll 1.2.11/2020-07-18/535552 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 0.97 sec/real 74.49 sec = 1%. Speed 62.87 mB/s
All OK

Using zlibwapi.dll 1.2.13/2022-12-31/544768 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 1.59 sec/real 83.01 sec = 2%. Speed 56.42 mB/s
All OK

Using zlibwapi.dll 1.2.13/2023-07-11/512000 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 0.63 sec/real 106.93 sec = 1%. Speed 43.80 mB/s
All OK

Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it..

Regards

kj911

15-07-2023, 05:33

infovs

15-07-2023, 06:05

infovs: Please testing more version older dlls...
Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.
This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.

I tested all zlibwapi.dll versions that I could find since ztool..Razor12911 version from 2020-07-18 (535552 bytes) was the fastest at decompression. I also did my own compile of zlibwapi.dll 1.2.11 with MS VC 2010 and also 1.2.13 with MS VC 2010/2015/2022 -> VC 2022 1.2.13 compile produced best result but Razor12911 version was still fastest at unpacking, although not by much, speeds were pretty close. Actually only this latest 2023-07-11/512000 bytes version speed drop was noticeable for me, that's why I posted my observation.
Also, if you use MS VC and want your compile to be XP compatible you are pretty much forced to use VC 2010..yes, you can use XP platform toolset on some later MS VC versions but I had some issues with it. Never compiled with MinGW though so I can't comment on that.

Razor12911

15-07-2023, 06:22

vc22_vista_1.2.13: 24688ms 24687ms 24750ms 24750ms 24782ms
vc17_winxp_1.2.13: 24375ms 24172ms 24156ms 24266ms 24359ms
rad_studio_1.2.13: 24359ms 24219ms 24203ms 24265ms 24313ms

vc19_vista_1.2.11: 21734ms 21672ms 21610ms 21843ms 21578ms
vc22_vista_1.2.11: 21391ms 21359ms 21391ms 21343ms 21282ms
vc17_winxp_1.2.11: 21453ms 21469ms 21359ms 21328ms 21360ms

vc22_vista_1.2.8: 21282ms 21281ms 21266ms 21281ms 21281ms

zlib-ng_1.2.13: 7843ms 7797ms 7781ms 7813ms 7828ms

Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it

I didn't do anything to 1.2.11 to make it fast, 1.2.13 itself is the one that is slow as you can see with the benchmarks, winxp compile has little to no impact on performance.

There's an issue opened on github regards to the speed which can be found here.
https://github.com/madler/zlib/issues/820

There was also a security vulnerability discovered in 1.2.11 and I don't know if patching this issue is what made zlib slower or not moving forward.
https://www.cybersecurity-help.cz/vdb/gnu/zlib/1.2.11/

The beauty of xtool is that the user can swap libraries if they prefer to use their own, there's also zlib-ng (utilizes modern CPU features) benchmarks added, as you can see it is faster however I doubt it produces matching crc as standard zlib but good to know nonetheless. ;)

Razor12911

29-07-2023, 14:56

Update available

Changes

- added universal scanner for DirectStorage gdeflate streams
- added the use of gpu for caching and virtual memory purposes
- updated depthing feature to improve stream detection when used by plugins

Notes
With the release of the game ratchet & clank: rift apart which used direct storage's gdeflate compression within its game files, I thought... hm, why not just add a scanner for these streams and so I did after investigating the stream structure (was easy to figure out actually). Use via -mgdeflate.

Will games use direct storage? I don't know but if they do then at least xtool has support for the streams used in the games, I just added it so that repackers don't flood me with DMs when they have troubles repacking games, maybe Starfield might use this...

After several tests, the gpu feature can now be used by the general public, use via -g#

L33THAK0R

31-07-2023, 01:29

Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?

daveyrob

01-08-2023, 07:22

Razor12911

08-08-2023, 16:28

Update available

Changes

- fixed an issue where xtool would crash on Windows XP and other systems upon closure
- memory optimisations
- improvements with database "based" plugins

infovs

09-08-2023, 12:07

Thank you Razor12911, also..is there a reason not mentioning there is also new CLS version of xtool in your 0.7.8 update? Not stable/recommended yet?

Masquerade

09-08-2023, 12:19

^^
Run some tests and see what you find. That's what I'm gonna do when I return to my PC.

Please note that FreeArc is an x86 program, so in order to use the x64 CLS variant, you will first need to use some kind of wrapper such as wrapcls by 78372 or FAZip (not FAZip32).

It's gonna be a lot of fun to play with. A cls is what I've been secretly wishing for, thanks Razor :D

shazzla

06-09-2023, 04:23

@Razor :

Maybe an XTool 0.7.8 bug ,using FLAC. Why is that increased memory consumption ?

Compression : -mflac:l8:c128mb , 1 thd ,approx. 800 MB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

Compression : -mflac:l8:c128mb , 4 thds ,approx. 1.8 GB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

All decompression settings used % to set the number of threads. Using T4 instead of t100p cannot decompress the archive :
ERROR: general (de)compression error in flac:l8:c128mb

Maybe other cases affected. Dont know.

i5-6600

Razor12911

06-09-2023, 10:56

There's two different code sets for decoding in xtool, one is for multi threading which caters for speed at a cost of increased memory usage because the streams have to be reorganised before they are written to disk and as such, high memory usage because they are stored in memory first. The other code is when you have set 1 thread of if a block has 1 stream in it, for this as soon as a stream is processed, it is written to disk hence why the reduced memory usage in -t1.

For flac, I'd suggest using -t1 for decoding simply because flac is decompressing when installing unlike other codecs where they require multi threading to be fast as they are doing the opposite (compressing) during installation.

A small test on some audio files

Tested 39 files, 745,011,186 => 1,379,878,320 bytes. Ratio 53.99%
Testing time: cpu 1.08 sec/real 4.68 sec = 23%. Speed 295.10 mB/s

Using 12 threads, used 1.23 GB ram

Tested 39 files, 745,011,186 => 1,379,878,320 bytes. Ratio 53.99%
Testing time: cpu 0.97 sec/real 10.18 sec = 10%. Speed 135.58 mB/s

Using 1 thread, used 185 MB ram.

The single threaded speed is reasonable while using 12x number of threads only gave 2.2x speed benefits while using nearly 7x memory. Obviously due to IO bottleneck show that it's better to just use 1 thread.

infovs

08-09-2023, 06:17

Hi Razor12911, I know similar question has been posted before but can you please re-consider renaming hardcoded srep.exe call in x64 xtool version to original, srep64.exe ? Since you made xtool available for both x86 and x64 versions I started using both when appropriate (making XP compatible installers for games that can be played on XP...) and hardcoded srep.exe is giving me headaches because both x86 and x64 are executing srep.exe. x64 xtool version should be executing srep64.exe IMHO :o
Anyway, thanks again for your xtool, best regards

Razor12911

10-09-2023, 06:32

Update available

Changes

- added recompress streams feature
- added reassign streams feature
- added dummy codec
- added data transfer buffer for srep when dd# is used
- added memory checks to ensure memory usage does not run wild
- exectuable plugins (via xtool.ini) are no longer required when decoding if they were never used
- internal stream deduplication now reports speed and memory usage benefits
- configuration based plugins can now have multiple names (if multiple games use the same configuration)
- fixed xtool crash if an incorect path for plugins was provided
- fixed lz4/lz4hc codec bug when used directly without any plugin
- fixed an issue where using fast-lzma2 compression would crash when decoding
- fixed an issue in DirectStorage gdeflate codec
- fixed an issue with execute command in stdio mode
- deduplication memory requirements improved
- memory optimisations
- reduced memory requirements for large streams that require patching
- removed ability to inject libraries to exe (buggy)
- removed patch command (uses too much memory)
- removed archive command (no one uses it)
- updated oodle scanner
- updated lz4/lz4hc universal scanner
- updated lz4/lz4hc/lz4f codecs
- updated zlib codec
- updated zstd codec
- replaced gpu caching feature with normal system memory cache (it doesn't work well on AMD gpus), use -p#
- replaced xdelta3 with zstd patching engine
- srep64 executable considered in x64 build of xtool

Notes

First and foremost, huge thanks to KaktoR (https://fileforums.com/member.php?find=lastposter&t=99554) for helping with this update, he was full of energy when we started but I think I broke him towards the end...
35375

This update was supposed to be released a long time ago, but he would find more bugs hence the long list of changes.

I won't say much other than notable changes, xdelta was replaced by zstd patching engine (yes... zstd can make patches too, shocking I know...), reason for replacement xdelta gives me less options coding wise to work with in terms of memory management.

Injecting libraries feature also removed as some libraries when injected completely stop working or worse, cause xtool to crash.

I've quietly added cls plugin in the previous update and did not tell anyone about it simply because it was never tested, for this update It was tested and it works so long as you never used fast-lzma2 in any of the archives when using it with installations created with inno setup as it would simply cause a crash.

I've left 0.7.8 on the main post in case there are issues with this update even after extensive testing.

L33THAK0R

12-09-2023, 22:27

Very embarrassed to ask such a simple question regarding database creation, but I can't seem to wrap my head around it. I've got some assets I'm attempting to rip some assets that are compressed inside an archive using deflate.

To do this I'm pretty confident in saying xtool just needs a database to reference, with the command looking like

xtool erase <folder> <archive> <database>

as a user pointed out to me a while ago.

Where I get stuck is creating the database, the title in question is "The Wanderer" a UE4 title (unencrypted) with a singular ~30GB ".pak" archive containing said assets, I've extracted the assets I'm interested in ripping to their own folder and have so far tried using xtool 0.7.9's GUI to generate the database since I'm not too confident the syntax I'm using is correct when running xtool from the command line.

Here is how I've setup the database creation though I'll admit I'm unsure of whether I've got the "Input" and "Source" correct.

https://i.imgur.com/2IEnbQ7.png

However all I get from this configuration is the following message, which hangs for a few minutes despite some disk activity as per task manager, and then indicates the process is completed however no database file is created.

https://i.imgur.com/az0bW9p.png

Very unsure and confused about what parts I'm mucking up or whether XTool just has a limitation on the size of inputs that can be processed.

Masquerade

13-09-2023, 00:13

Don't use the GUI. In that screenshot, you are using the wrong database generator and your inputs+outputs need to be reversed.

And you cannot rip compressed data. It must appear in plaintext in order to be erased by xtool.

L0v3craft

14-09-2023, 15:16

Someone knows how to use rz compressor with xtool like xbcm?

Edit: thanks to Razor12911, now xrz has been added to XTool!

MAY/O

20-09-2023, 13:57

I once did but it would cause me headaches as a Windows user and as such, the testing would most likely be relegated to Linux users and on this forum, there's really not that many. I mean even tests on Windows are too much for ask for at times :D

Now I got some more experience with compression, I am always open to help if you want.:)
But I totally get it if not! So don't worry

felice2011

25-09-2023, 08:35

Hi guys, I like to take this opportunity to say hello to old contacts who have known me here on Forum since 2011, and the new guys who are passionate about compressing and installing games. I'm glad that the forum is still active, even if not as I remembered...:)
A big hello to friend Zee, who still works flawlessly with his work to this day.
I've been away a lot for work, so I'm quite rusty on the subject.:)
I wanted to ask someone here in the thread if they had managed to implement the added function of srep in xtool, or at least if I understood correctly if could be done and how....
Added in last xtool "- srep64 executable considered in x64 build of xtool" - //- "-sp# - srep parameters (separate params with ":")". thankss

Masquerade

19-12-2023, 02:41

I have archived XTool 2020's release history, related tools and plugins to a github repository in case of another FileForums outage. This repository also contains collections of oodle, lz4 and zstd libraries for precompression and decoding.

https://github.com/Masquerade64/XTool_2020_Archive

I have also archived Razor12911's legacy tools into a separate repository as they may still be desired:

https://github.com/Masquerade64/RazorTools_Archive

-------------------------------------------------------------------

For those who don't know, XTool progress has been moved to Patreon
https://www.patreon.com/xtool

Well, after one of my drives suddenly stopped working I just had a thought. Why exactly do I keep developing xtool with no incentive whatsoever, I mean back when I started, I did it because I was on capped internet and had slow PC and never liked the prolonged installations that repacks come with, but now that this is a thing of a past for me, why do I keep doing it? This is probably the third drive that I have lost, people would tell me to make a backup but why should I? The drives in most cases die from the excessive testing that I perform on my PC as I develop the project, who pays for these replacement drives? Me of course, and that's why when I decided to monetize the development of this project.

Consider joining to help support Razor12911.

-------------------------------------------------------------------

Latest Release: XTool v0.8.1 (6/12/23)

Changes

added stream count terminator for library checker
fixed issues related to upgrade from RAD Studio 11.3 to 12
database structure changes have been reverted

Notes
RAD Studio updates from 11.3 to 12 brought a lot of changes which greatly affected xtool, for one the source needed to be corrected in certain places while some parts of the code needed to be rewritten which I guess is why 0.8.0 seemed to have a lot of hotfixes because I can confirm that not even those fixed everything however with this release, things should be greatly improved.

I've also added stream count terminator when using library checker in UI mode, this is added for large inputs that have really long precompression times and by default, first 10 streams are considered for checking and from those the user should decide which library to use for precompression.

Database structures have been reverted to the way they were before and this is due to the amount of work needed to support the new unreal engine utoc/ucas format because of recent updates to the engine that make precompression using xtool/oo2rec seem unfeasible especially for the newly released games.
https://www.patreon.com/posts/release-v0-8-1-94158307

Jahan1373

02-01-2024, 04:31

Hello
I have been told that you are very forgiving
Can you recommend a good compressor?
Easy and simple work, high compression and works with all site installations, thank you

Masquerade

03-01-2024, 00:33

^^
This question has been asked hundreds of times.

7-Zip.

Jahan1373

07-01-2024, 04:07

^^
This question has been asked hundreds of times.

7-Zip.
I don't know that Fitgirl put all these compressions with 7zip program here

Masquerade

07-01-2024, 05:37

^^
Obviously FitGirl doesn't use 7-Zip. But you asked for:

Easy and simple work, high compression and works with all site installations

So, 7-Zip is your answer. If that isn't easy enough for you, try built in Windows zip maker :p

Masquerade

11-01-2024, 10:23

New Release: XTool v0.8.2 (11/1/24)

Changes

additional mathematics functions added (ceil and floor)
fixed issue with zstd recompression
fixed issues with decode/extract functions

Notes

Not much to say but special thanks to Panker1992 and KaktoR for discovering bugs and for offering support especially over the holidays, and for everyone else, happy new year and I hope you keep compressing :)

https://www.patreon.com/posts/release-v0-8-2-96291433

Masquerade

16-01-2024, 11:43

New Release: XTool v0.8.3 (16/1/24)

Changes

improved encoding speed when deduplication is used under certain conditions
removed deduplication decode memory calculations (takes too long to compute)

https://www.patreon.com/posts/release-v0-8-3-96616797

Wanterlude

25-01-2024, 02:53

New Release: XTool v0.8.4 (25/1/24)
Changes

older zstd versions such as 1.0 can now work alongside newer versions of the library

https://www.patreon.com/posts/release-v0-8-4-97128362
You can always support Razor for his work on his patreon

Wanterlude

03-02-2024, 02:46

New Release: XTool v0.8.5 (03/2/24)
Changes
added -X parameter, to only extract streams that could not be processed
fixed issue with extract stream feature where it would make excessive writes to disk
fixed issues with executable plugin support
updated oodle scanner
updated zstd scanner
Notes
The oodle scanner has been updated, I suspect it to be slower than before but it should detect a bit more streams.
Mermaid test
0.8.4
Compressed 1 file, 104,857,600 => 221,155,278 bytes. Ratio 210.91%
Compression time: cpu 0.00 sec/real 8.73 sec = 0%. Speed 12.02 mB/s0.8.5
Compressed 1 file, 104,857,600 => 294,542,000 bytes. Ratio 280.90%
Compression time: cpu 0.00 sec/real 13.80 sec = 0%. Speed 7.60 mB/s
https://www.patreon.com/posts/release-v0-8-5-97747742
You can always support Razor for his work on his patreon

dixen

15-02-2024, 04:38

New Release: XTool v0.8.6 (15/2/24)

Changes

- flac library renamed
- fixed an issue with execute command
- fixed issues with executable plugin support
- updated reassign stream feature
- updated oodle scanner

Notes

Reassign stream was first introduced in 0.7.9 but it was not implemented correctly and with this update it got major improvements. What reassign essentially does is transfers streams to another codec for them to be processed for example, if there's zstd streams detected but for whatever reason, internal zstd cannot process them via libzstd.dll however, the original zstd.exe is capable of processing these streams, you can set a method that looks like this
-mzstd=zstd147, where the equal sign "=" transfers all streams detected by xtool to be processed by zstd147 where your custom zstd.exe will be called.

The same applies to plugins that have been created thus far so for example the game tony hawk pro skaters 1+2 with its custom encryption, if you were to create a plugin that contains this custom implementation, the method should be something like -mue4=thps2decrypt, so instead of streams being decrypted by aes via xtool, they will be passed to your custom decryptor.

The same could work for lzma compressed unity game files (though not tested), if you've set up your lzma decoder/encoder, the method becomes -munity=lzma_dynamic but then again, I think with this codec in particular, -munity:lzma_dynamic should work just the same. Once again, this is theoretically it should work this way I just do not have samples to test.

Don't like flac? set up your own external codec like wavpack, tak or whatever, the method will be -mflac=wavpack, where all wav streams detected instead of being processed by flac, they'll be passed onto wavpack to be processed.

There's just a lot of use cases for the stream reassign feature and it's all up to the end-user how they plan on improving recompression of streams that cannot be processed correctly by what xtool has to offer.

Masquerade

22-02-2024, 14:02

New Release: XTool v0.8.7 (22/2/24)

Changes

updated media codecs (flac, brunsli, packjpg...)

Support Razor12911 on Patreon (https://www.patreon.com/posts/release-v0-8-7-99004708)

Redmist2013

07-03-2024, 14:42

Is there an issue with LZ4 stream detection in recent versions of XTool?

The LZ4 streams in the following sample file are correctly detected in old XTool 012 but not in recent versions...or am I missing something?

File sample.pak

XTool 012

xtool.exe e:precomp:c16mb:lz4 sample.pak sample.pak.xt

9,327,964 => 13,522,962

XTool 0.8.7

xtool.exe precomp -mlz4 sample.pak sample.pak.xt

It just seems to get stuck in a loop forever.

Here is the link to sample.pak -:

https://drive.google.com/file/d/1Yrjuyb96DAuXSGUMDwu2zjv9W2o24Uuq/view?usp=sharing

KaktoR

08-03-2024, 06:13

You cannot use lz4 directly. Plugin is required always if you want to use lz4/lz4hc.

However your sample seems to be compressed with lz4hc, because recompressed stream is bigger then original stream.

xtool.exe precomp -mlz4:s64k -t100p -c16mb -v - - < %1 > %1.out

[0] Processing streams on block 0000000000000000 to 00000000008E555B (9327964)
[0] Processing lz4 stream at 000000000002C850 (58101 >> 65536 >> 61165) using a1:b0 has failed
[0] - Patching stream at 000000000002C850 (58101 >> 61165) [42205] has failed

Streams: 0 / 81
Time: 00:00:03 (CPU 00:00:03)

Size: 8.90 MB >> 8.90 MB

xtool.exe precomp -mlz4hc:s64k -t100p -c16mb -v - - < %1 > %1.out

[0] Processed lz4hc stream at 00000000008D8B64 (51701 >> 65536 >> 51701) using l9:b0 successfully

Streams: 46 / 81
Time: 00:00:04 (CPU 00:00:04)

Size: 8.90 MB >> 9.36 MB

I don't know from what game this sample is. Either make a plugin configuration (ini file) or search for bms script and use bms2xtl with correct settings.

Redmist2013

08-03-2024, 18:15

Yes, you're right - new XTool seems to detect some of the streams as LZ4HC. Not sure why the old XTool works perfectly when choosing generic LZ4 though.

The file is from CoD:MW Remast btw. All of the scene repacks are horribly out of date so I was compressing it myself.

KaktoR

09-03-2024, 06:20

In this case you have to make a configuration plugin in order to precompress this game with newer xtool versions, as I am pretty sure there is no bms script for this game.

dixen

26-03-2024, 02:10

Application cls-srep.dll instead of srep.exe is it possible to implement this when using -dd#?

Masquerade

26-03-2024, 03:01

^^
Nope. cls-srep_x64.exe works differently to a typical srep.exe. It's focus is decompression only and uses different parameters afaik.

dixen

08-04-2024, 08:24

Hmmm..
xtool execute doesn't work on v0.8.7?

daveyrob

28-04-2024, 05:19

Hmmm..
xtool execute doesn't work on v0.8.7?

i had multiple issues with the execution options in general including the ones using xtool.ini, some exes using xtool.ini seem to get confused with the in-out like tak or ofr, the only one i actually got working for wav was wavpak. in some versions the exe options kinda work wile in others not at all. i now just use mtx for mt, msc for wav and biks in containers i just extract myself to save jumping around xtool versions.

edit:
after changing xtool to version 086 all the stuff i could not get to work before seemed fine.

Masquerade

15-06-2024, 10:07

New Release: XTool v0.8.8 (15/6/24)

Changes

added zipcrypto encryption support
added library config based plugin support

https://www.patreon.com/posts/release-v0-8-8-106259359

wareck

15-06-2024, 12:09

do you know how to use the new feature zipcrypto ?

Masquerade

20-06-2024, 02:00

^^
It must be used via plugin, configuration or database. It cannot be called directly.

wareck

20-06-2024, 12:53

I understand that you have to use bmscrypt and zip.bms + password.
I've also started a draft ini file but it's a bit dry....

But anyway, with my ugly method and block processing, I've reduced the game to 9.7GB, which isn't so bad.
The only problem is that the CRCs are different, which could pose a problem for future updates.
but not for sure, updates are patch_00x.pak files that are added...
it's not really for the game itself but more to understand the process and improve...

Masquerade

03-07-2024, 10:24

New Release: XTool v0.8.9 (3/7/24)

Changes

fixed issues with base directory and file/directory pathing in general
updated verbose mode log

Thanks to Razor12911 for the update. Please support his work on patreon.

https://www.patreon.com/posts/release-v0-8-9-107298667

dixen

13-07-2024, 11:51

Can you explain how to link quickbms to XTool? I am interested in working with the giants.bms script and files.GAR from FS 22

Masquerade

13-07-2024, 13:39

^^
From this, what I retrieved from the wayback machine:

https://web.archive.org/web/20221212200247/https://0bin.net/paste/xYAx24us#2umJQj-g1+CQYJ29GGaDtWP0yYUz1VJ4U/CEKOWrwzW

- This file format appears to be encryption on top of LZMA.

- XTool cannot natively handle LZMA so that would need to be plugged in. You'd need the right version and parameters for lzma.exe from the lzma sdk.

- The decryption function itself is not sourced, it's embedded into the script as MEMORY_FILE3 (which you must then prompt for qbms to load at runtime or with -Y argument)

- Can this function do encryption too? I do not know.

- Razor said a while back that QuickBMS support was fixed in XTool.ini (because I wanted to use it for Unity games), but I still couldn't quite get it to work correctly.

- So likely you'd need to generate a database with bms2xtl, but this database doesn't even get generated correctly since xtool detects "calldll" as the encryption algortihm (calldll is just a function in qbms to load external code/functions) and then plug qbms and lzma into xtool

This is a lot of trouble and I don't think I can offer any further help.

wareck

19-07-2024, 22:42

hello guys
I would like to erase some files in a stream.

xtool.exe erase extracted_ original db.xtl

it works but I want's also to remove small file

Skipped Engine\Plugins\VirtualProduction\LiveLinkCamera\Co nfig\BaseLiveLinkCamera.ini (Smaller than 256)

there is an option for that ?

KaktoR

20-07-2024, 01:29

No. However the file is smaller than 256 bytes, so this shouldn't be a problem.

LILU

21-07-2024, 14:34

Good afternoon. The questions are as follows.
1. For which algorithms (for “uncompressing”) (Zlib,Kraken,Leviathan,Mermaid,Selkie,ZL4F,ZSTD,Cr ilayla,Reflate,Preflate) can GPU be used?
2.Which Gpu's are supported? AMD,Nvidia? How much video memory?
3.Do you need any additional Windows customization(software installation etc.)?
4. What version of Xtool has GPU support?
Thanks in advance.

Masquerade

22-07-2024, 00:37

^^
Hi, none. This was a cancelled feature.

LILU

22-07-2024, 02:41

^^
Hi, none. This was a cancelled feature.

Are you talking to me?

KaktoR

22-07-2024, 02:55

No, he was talking with himself, don't worry.

...

Yes, he was talking to you. GPU feature was canceled, at least for AMD gpus.

LILU

22-07-2024, 03:36

No, he was talking with himself, don't worry.

...

Yes, he was talking to you. GPU feature was canceled, at least for AMD gpus.

Okay. Which Nvidia models are supported?

ZiLATAN

10-10-2024, 04:03

Can anyone give me the download link of SREP, LZMA/Lolz Please?

dixen

10-10-2024, 04:30

Can anyone give me the download link of SREP, LZMA/Lolz Please?

https://fileforums.com/showthread.php?t=104866

mosath

04-08-2025, 03:34

What is the difference between xtool 2019 / 2020 / 2020 extended support?

Hexagon123

04-08-2025, 06:41

What is the difference between xtool 2019 / 2020 / 2020 extended support?

Compatibility improvements?

Unreal support and others.

aman71711

11-08-2025, 11:33

Hello everyone, I'm trying to create a repack for Cricket 24 using the method provided by masqurade in this forum and have a quick question. I have the full game folder, libzstd.dll, and all the necessary xtool files. This guide https://fileforums.com/showpost.php?p=502793&postcount=3768 specified using the parameters -mzstd:l3 -df5p -dd5, and since that didn't work directly, I used the precomp command to run it like this: xtool.exe precomp -mzstd:l3 -df5p -dd5 "CRICKET 24" "CRICKET_24_PRECOMP". The process started, but my 90 GB game folder is creating an output CRICKET_24_PRECOMP folder that has already grown to over 190 GB. As I'm new to this, I just wanted to ask if this massive increase in size is normal for the xtool precomp process and what the correct next step is after this finishes. Thanks in advance for any guidance!

Masquerade

11-08-2025, 11:53

^^
Seems to be working correctly, the precompression process is removing the existing zstd compression and writing the decompressed data to disk.

naxil

27-08-2025, 01:31

exist a way to use it under linux?

NERV

17-01-2026, 08:22

New Release: XTool v0.9.1 (12.01.26)
Changes

- minor code optimisation and fixes to prevent errors in certain scenarios

- fixed issues with reassign stream feature

- fixed issues library config based plugins

- removed recompress stream feature

Notes:

Reassign stream was added because of inconsistent game engines like Unreal Engine because you could write a program/library to work on a list of games then a few months later, they change something in how the files are stored which renders your tool outdated because it no longer works on the newer games. unreal.dll is a plugin to support ue3-5 games and for as long as I remember, ue3 games were compressed with lzo then came mass effect legendary edition and it used leviathan, so I updated the plugin and then Gears of War Reloaded came out and it used lz4hc so rather than updating the plugin (I've lost source code), you can reassign all those ue3 streams to lz4hc, this feature was bugged but now it is fixed and should work like so:

Usually, you'd use -mue3 to precompress the game but it doesn't work because now the game uses lz4hc so, you need to notify xtool that streams need to be reassigned to a different codec and the new method becomes -mue3+ue3=lz4hc:l9.

Library config based plugins were added in 0.8.8 because I'm a person who seems to always lose their source codes due to hard drives dying, so what I've done is allowed dlls to be imported through ini which xtool can make use of, so rather than write code internally in xtool or through a plugin which is bound to have its code lost, I can just share a ini just like other config based plugins and if there's something that needs changing or updating, either the user can do it themselves or I can just use notepad to provide a quick solution. I've provided a sample.

Recompress stream was removed because it was an overly ambitious feature I thought of added, the idea behind it was that since some compression algorithms, zlib, zstd, lz4, and oodle to a certain extent do not error out if the incorrect compressed size was provided. So what does this mean? It means if a game could not be precompressed at all, like some special edition of zstd or oodle was used and let's assume it was compressed using level 6, recompress feature was meant to allow you to recompress the streams with a stronger level, let's assume level 7 and by doing this, you ensure that your new stream will be smaller and could replace the level 6 stream that could not be precompressed, a new game file is produced and that will be your new game files and then you can use xtool normally to precompress the game and repack it.

So, what's the problem? Well by doing this, you're opening a can of worms because you now have to test if the game works without issues (maybe from start to finish) because the game files were "modified", it is lossless but it is no longer bit-perfect and you might as well use other tools to rebuild the game files after unpacking them because that's even safer due to headers being updated too and this is why it was removed. 0.9.0 is the last version to have this so if you want to use it, then look for that version here.
https://www.patreon.com/posts/148017425

Butza

09-03-2026, 02:10

Tell me how to scan oo2scan_7_win64.exe files when they are encrypted (unreal)

Redmist2013

13-03-2026, 20:39

So, to clarify, do we need an older version of XTool for Unity/LZ4 files? I've noticed it doesn't seem to work with recent versions.

KaktoR

13-03-2026, 20:57

Send sample

Redmist2013

15-03-2026, 05:03

Send sample

No problem - attached

Redmist2013

16-03-2026, 21:55

OK, it's fixed in 0.9.4. Thanks, Razor12911.

NERV

17-03-2026, 04:00

New Release: XTool v0.9.4 (17.03.26)
Changes

- Added internal version check when dragging liblz4, libzstd and oodle dlls onto xtool

- Added cls config

- Fixed an issue with plugins not being loaded correctly

- Updated library checker when using UI

- Updated leviathan codec

https://www.patreon.com/posts/xtool-v0-9-4-153238519

wrathma

18-03-2026, 12:13

im trying xtool 0.9.4 on latest version of no man's sky but it keeps failing. zlib also doesnt work.
test on NMSARC.MeshPlanetDESERT.pak (0.41kb) -
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 987) using l16:f0:w0:b0 has failed
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 936) using l17:f0:w0:b0 has failed
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 922) using l18:f0:w0:b0 has failed
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 921) using l19:f0:w0:b0 has failed
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 922) using l20:f0:w0:b0 has failed
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 922) using l21:f0:w0:b0 has failed
[0] Processing zstd stream at 0000000000009E50 (909 >> 65536 >> 922) using l22:f0:w0:b0 has failed

Streams: 0 / 3
Time: 00:00:00 (CPU 00:00:00)

Size: 0.04 MB >> 0.04 MB

Done!!!

edit:
fixed by replacing old libzstd.dll with a new one from https://github.com/facebook/zstd/

NERV

28-03-2026, 23:25

New Release: XTool v0.9.5 (21.03.26)
Changes

- updated DirectStorage gdeflate and nvCOMP libgdeflate codecs

Notes:

This is a complete implementation of both libraries meaning compression level can be specified and as such, the game Ninja Gaiden 4 now requires you to specify the correct level for precompression which is level 6 because stream patching takes place. Don't mind the streams that get left behind for that game because they are just too small to be patched without incurring negative ratio.

https://www.patreon.com/posts/xtool-v0-9-5-153584845

hwang4

20-04-2026, 07:09

i compressed using "dd3" parameter in xtool , but got error in decompression , need help or a tuto for how to use/configure correctly "dd3" instead of srep!thx

Dunnowho69

01-06-2026, 23:52

Changes

- added tracking for sizes of stream patches

- added stream coverage tracking

- added option to specify multiple oodle libraries to use (-oodl#)

- fixed issues with reassign stream feature

- fixed minor issues with DirectStorage codecs

Notes

Patreon now seems to have virus scanner in attachments that I upload and has false detected xtool, this has delayed the 0.9.6 update because I was stuck in a situation where I wondered, should I add a password to the archives to prevent scanners from blocking the post? But then that would raise suspicions to some people using this tool especially with some programs like cpu-z and such getting malware recently (it's good to have trust issues) and so, I want to make it clear. The software is provided as is and I shall not be held liable for any damages.

With that out of the way,

xtool now tells you how many streams were patched and how big the patches are that way you can play a bit more with -df parameter to maximise ratio for imperfect streams.

Streams: 5359 / 5547 (95,43%)

Patched: 5359 (0,15 MB)

It also tells you the stream coverage in percentages, 95,43% as an example for that input means 95% of the file had streams and 5% was something else, this is good feature that tells you if said file contained something else other than streams, maybe audio, video or encrypted data.

Streams: 17850 / 17850 (7,72%)

ForzaHorizon6\media\Tracks\Brio\GeoChunk0.minizip

This file is 37GB, only 7% contains zlib streams, what is the remaining 93%? That's for you to investigate...

I've also added the feature that allows you to specify multiple oodle libraries for precompression, if one fails the next one is attempt because there's some games that seem to use more than one dll which makes complete precompression in one go not possible, requiring multiple passes of xtool which can be slower and use more memory.

Keep in mind, that the same parameters for oodle dlls is required when decoding and the primary oodle dll -oodle### should be a newer version meaning, do not use oo2core_4_win64.dll for the first one and oo2core_6_win64.dll for -oodl2###

gameresources_20_patch2.streamdb

xtool.exe precomp -mkraken:l6 -c32mb -oodleoo2core_9_win64.dll -oodl2oo2core_9_win64_2.dll - -

with one library

Streams: 1045 / 1154 (96,90%)

Size: 356 MB >> 534 MB

with both libraries

Streams: 1150 / 1154 (96,90%)

Size: 356 MB >> 605 MB

https://www.patreon.com/posts/xtool-v0-9-6-159873167
Password: 123